The War on Spam

It was earlier this year that I last detailed some steps taken in an attempt to thwart the growing junk email (a.k.a. SPAM) epidemic. dspam has continued to work like a champ, with an accuracy rate currently standing at 99.13%.

However, the volume of spam continued to increase, almost as inevitable as the rising tide. I didn't let it bother me too much, as I could simply delete the messages sent to my spam folder, although I had to admit that looking through all of those messages for false positives was getting a bit tiresome.

And then it happened... The event that moved me past the tipping point, and thrust me forcefully back tot he front lines of the war on spam. The event seemed innocent enough: I went to send an email, and our smtp server (qmail) did not respond. wtf?

I shut down qmail, restarted it, and it seemed to work for a while. However, after an hour or so, the same problem arose. wtf?

Looking at the process list, I noticed that there were 40 qmail-smtpd processes running. What were they trying to send? Time to get my hands dirty and learn a bit about qmail's queue. I read a bit on the queue, and installed a tool called qmqtool. By using this new tool, I discovered that I had over 6,000 messages awaiting deliver in my mail queue!

Why so many? Because for every spammer that sends an email to a bogus address, a mail server will craft a bounce message and attempt to send it back to the (usually faked) sender. Gah!

As a first step, I decrease the queue lifetime to 24 hours. After 24 hours, any message still left undelivered in the queue would now be deleted. Then, I cleaned out the queue and restarted qmail. It worked!

...for a while. Ok, so let's do some more furious reading on the subject of spam prevention in the qmail environment. First of all, I needed to increase the maximum number of qmail-smtpd processes which were allowed to run simultaneously by tcpserver. By adding a "-c 400" to the tcpserver command used to execute qmail-smtpd, I increased the maximum number of simultaneous connections from 40 to 400. That ought to be a good start.

Now, how else can I decrease the volume of spam? I had heard of greylisting before, and thought that it might be a good next step. Luckily, there is a simple greylist implementation for qmail called qgreylist. It installs fairly quickly, and does a fairly good job. However, it is written in perl and with all of the spam emails flowing in, it can cause a bit of a performance hit.

So, maybe there is something another check I could do before sending the incoming email to qgreylist? How about an RBL check! When another computer attempts to deliver an email to our mail server, the RBL (Real-time Blackhole List) check sees if the sender's IP address in a list of know spam servers. If it is, then the connection is dropped and the email never even is seen by the greylist app.

With the above changes made to our mail server, I am now receiving about 10% of the spam volume that I was before the changes were made. Furthermore, our remote queue sits at a fairly steady level of 400 messages.

Only time will tell for how long this solutions will suffice! Wish me good luck and my email godspeed!

Upgrading to Ubuntu 7.10 - Gutsy Gibbon

As mentioned in previous postings, I use Ubuntu as a desktop OS both at home and at work. At home, I have had Dapper Drake installed on an older Dell Inspiron laptop with a D-Link DWL-650+ wireless PCMCI networking card, which functions perfectly. The kids will even hop on it to browse the web with Firefox or write a paper using OpenOffice.

At work, I have a dual Opteron Dapper Drake box on which I do any Linux and/or C based development. It has been rock solid for well over a year. I also recently installed the 64-bit version of Feisty Fawn (7.04) on a spare AMD64-3200+ system to test 64-bit development using Free Pascal.

When I heard Ubuntu 7.10 was released, I looked through the notes and thought that it might well be time to upgrade my Dapper boxes, as Firefox was getting rather long in the tooth (the Dapper version being stuck on the 1.5 branch), and I also wanted to investigate mono development (more on this in another post!). Since mono development is still rapidly evolving, I wanted to look at a recent distribution.

The first box I went to upgrade was the AMD64 system. I had little to lose on the system, and so elected to do a clean install, nuking the previous distribution. Everything installed smoothly, so I played around with the distribution for a while to see what was new.

The installer had detected the motherboard's integrated Nvidia video controller, and offered to install the restricted driver for me to increase performance. I accepted, then turned on some of the flashy desktop effects provided by compiz, and was pretty impressed with some of the flash and sizzle. To get finer tuned control over what compiz does and does not do, I installed the compizconfig-settings-manager using synaptic.

Next check - Firefox and the network. The integrated NIC was detected and automatically setup - no work necessary on my part (since I recently switched our work network to providing DHCP addressing). Firefox seemed to work well, and when I visited a page that used Flash, it offered to install the 32-bit flash plugin with all of the necessary wrappers and libraries so it could be used in my 64-bit Firefox. Until Adobe can provide a 64-bit flash plugin, this is the next best thing and it worked flawlessly.

Ok - the box looks and works well, so it was time to test the next system in my parade of installs. Hmm... Let's try the laptop at home. I burned a live cd of Gutsy and took it home. I booted up the laptop with the CD, and wireless networking was dead - frak! This has been a problem with this laptop for the last three ubuntu releases. I decided to come back to the laptop at a later time.

Next stop - the dual opteron development workstation. What to do, what to do? I didn't want to do a clean install this time, being a somewhat lazy sob. I decided to back up my important files to another computer and do an inplace upgrade from Dapper to Edgy, then from Edgy to Fesity, and finally from Feisty to Gutsy. This seemed to me to be somewhat risky, but I wanted to see how Ubuntu fared when doing inplace upgrades. If it came through THREE upgrades, well that's a pretty damn strong endorsement.

So I first upgraded from Dapper to Edgy by following the instructions here. It took a while, I had to confirm a couple of file overwrites, and about 3 hours later it said it was done. Reboot the machine, and all looked well. Did it again for Edgy to Feisty, and one more time for Feisty to Gutsy.

Once the upgrade to Gutsy was done, I took a close look at the system. Compiz was not working, and I found tha I needed to install xserver-xgl for compiz support. Fonts were a bit tweaked, but I managed to get them workig with a couple of corrections to the FontPath section of the file /etc/X11/xorg.conf.

Overall, I was *very* impressed with the three distribution upgrade. This gave me the confidence to try it with the laptop at home, and so I did. To make a long story short, I am now writing this post from the Dell Inspiron upgraded to Gutsy, with wireless networking and flashy desktop effects turned on. It's not super speedy, but once things are loaded into memory, everything runs swimmingly.

I just checked fonts in emacs, and they are fairly screwed at this point in time. I may have to check out my font paths on this system, and if I recall correctly, I had to change the default fonts emacs uses to get non-horrid looking fonts on the dual opteron system.