Earlier this week, we noticed that emails sent to @newcircleconsulting.com and a few other domains were either delayed or not being delivered. All these domains are hosted on the same Media Temple DV server. After a few chat sessions with the helpful Media Temple reps, we were able to found and solve the issue.
Step 1: Check email queue
% ssh username@hostname
$ sudo su
$ /var/qmail/bin/qmail-qstat
We had over 10K emails in queue when we first checked. We were told that anything more than 100 could indicate problem.
Step 2: Check and clear email queue
We had good experience with a tool called amHandle. We followed instructions on this page to get it installed – a quick and easy process, no compile was needed. Here are the steps and we put this tool under a private folder outside httpdocs.
% ssh username@hostname
$ wget http://jaist.dl.sourceforge.net/sourceforge/qmhandle/qmhandle-1.3.2.tar.gz
$ tar -xzvf qmhandle-1.3.2.tar.gz
$ cd qmhandle-1.3.2
$ ./qmHandle -s (to see summary of the queue)
$ ./qmHandle -D (to delete everything, could take several minutes if you have more than 10K of messages like me)
$ ./qmHandle (to see all options available)
At the beginning, when we had more than 10K of messages queued, no way to sort through and deal with each individually. So we had to use the -D option to remove all things being queued. After that, we were able to use ‘-l’ check and see what messages were coming to the queue. And these message details helped us find the root cause.
Step 3: Fix the compromised site
Different people could have different reasons, our case the emails were caused by a compromised WordPress site. There are plenty of articles talking about how to run security scan and clean up, so we are not going to repeat here.
In addition, there are two helpful Linux command (need root privilege) that you might need when working with qmHandle:
$ /sbin/service qmail start
$ /sbin/service qmail stop
Step 4: Test email then continue monitor
After email queue was cleared, we tested a few emails and all were delivered properly. We continued to monitor the queue for a few days and the number of emails in queue has been around 10-15. We consider this issue being fixed.
Code sample conventions used on New Circle blog:- % – local shell prompt
- $ – server side shell prompt