Rest In Peace, Ace

Ace, our dog of almost 13 years, was laid to rest on Friday March 21st, 2008. He leaves behind a family that misses him terribly.

I've mentioned Ace in several of my posts, and if you have ever been to my house over the last 13 years, you have certainly met him. He was a large and friendly dog that my wife and I picked out from the local animal shelter approximately one year before our first child was born. It turns out that his other siblings were not able to be adopted, as all of them were sick. We were lucky that we picked him out of the litter before the vet had a chance to look at him.

During the early years of his life, he provided my wife with both companionship and security as our children were born. She wasn't ever worried about being home alone while I was at work, so long as Ace was around. Later on, he was a friend and protector of our children.

During his last year of life, we saw his health steadily decline. His hearing deteriorated quite a bit, and his bouts of nervousness increased. Near the end, he had an eye infection in his left eye that he just couldn't seem to shake off, despite medication. His appetite was a mere fraction of what it used to be. Finally, he lost his footing with his rear legs and hips, and went to the floor. After attempting to get him up and mobile that day, and after several trips to the vet over the previous two weeks, my wife and I decided that it was time to let Ace go.

I have never owned a dog from puppyhood through death, so taking him to the vet on his last day was a new experience for me, and for my kids. The vet and techs were very understanding and helpful. It took four of them to get Ace out of the car and onto a stretcher, such was his lack of mobility. He never made a peep, and seemed content and ready.

After we said our goodbyes to him, I held him as they injected him with the sedatives that would release him. I can only hope he understood how much all of us loved him.

The old cliche goes, "you don't know what you've got until it's gone". I find myself "seeing" him out of the corner of my eye sometimes, only to realize upon further inspection that it was a blanket or shadow. I have to battle to not leave the back door open for him, and still expect to see him first when I come home from work in the afternoon (he was always the first to greet me). I miss his company in the mornings when only he and I were awake in the house as I prepared for work.

Rest in peace, Ace. We love you, and you will never be forgotten.

Now Using Greylite

In this old post, I spilled the beans on my new and improved spam filtering triple play: dspam, qgreylist and RBL checking. Overall, the system has worked well over the last four months, though I have noticed the increasing volume of spam showing up in my spam folder.

I suspected that the very simple form of greylisting implemented in qgreylist was the culprit, and after a bit of investigation, I found out that this indeed was the case. So, I set out to see if there was another greylisting implementation which could be used in my qmail installation.

Using my google-fu, I quickly zeroed in on Greylite. After reading up on it, I found that it held several advantages over qgreylist:

1. Written in C instead of perl, so it should perform a bit snappier and be less of a burden on the mail server.
2. Instead of only considering the IP address of the sender as qgreylist does, greylite considers the complete triplet of {IP address, from, to} before validating an IP address.
3. All data is stored in a single sqlite3 database, whereas qgreylist stored the verified IP addresses as files in a single directory, which clutters the filesystem and increases access time in that directory as more IPs are validated.
4. Greylite appears to have some enhanced functionality called `tuning suspicion' which allows you to customize how it behaves in certain circumstances.

To install it on my debian box, I first had to install sqlite3 (and the associated development libraries). Then, I downloaded the source code and followed the simple directions that came with it.

Less than 5 mintues later, and the server was up and running with its new greylisting implementation up and working perfectly. Flawless victory!

Baseball Season Update

So, here we are over two weeks into our little league baseball season - how is our team doing? Well, we have 0 wins and 4 losses, so one could make the argument that we are not doing too well. I would have to agree.

The kids seem to be having a good time, however. So long as they continue to enjoy the game and improve over the course of the season, I'll be content. Wish us luck!

By Request - My dspam Training Script

In a post I made about one year ago, I mentioned a script which I created which trains dspam to recognize missed spam email, and corrects it when it falsely identifies a good ( or "ham") email as spam. Someone has requested that I post that script, so here it is. Please note that my qmail installation uses the maildir format!

--- start file: train-spam.sh ---
#!/bin/sh

# train-spam.sh
#
# Description: Checks each user's /home/Maildir/.Spam.Missed
# directories to see if the user placed any "missed" spam
# messages which got through SpamAssassin to their INBOX.
# If there are messages in this directory, then the script
# invokes sa-learn to update the site-wide tokens to try
# and improve the defenses for next time...
#

# learn_spam - Function which takes a directory and a user as
# arguments, and then feeds that directory to our anti-spam
# applications for further SPAM training.
#
# Arguments:
# $1 - Directory name containing SPAM emails. Required
# $2 - User name. If it is not provided, $USER will be used.
#
# Example:
# learn_spam /home/alank/Maildir/.Spam.Missed/cur alank
#
function learn_spam {

# loop through all emails in given directory
for email in $(ls $1); do

# process SPAM email using DSPAM
/usr/local/bin/dspam --mode=teft --source=error --class=spam --feature=chained,noise --user $2 < $1/$email
echo -n "."

# delete SPAM email
rm $1/$email

done # end of email loop

} # end function learn_spam

# learn_ham - Function which takes a directory and a user as
# arguments, and then feeds that directory to our anti-spam
# applications for further HAM training.
#
# Arguments:
# $1 - Directory name containing HAM emails. Required
# $2 - User name. If it is not provided, $USER will be used.
#
# Example:
# learn_ham /home/alank/Maildir/.Spam.NotSpam/cur alank
#
function learn_ham {

# loop through all emails in given directory
for email in $(ls $1); do

# process HAM email using DSPAM
/usr/local/bin/dspam --mode=teft --source=error --class=innocent --feature=chained,noise --user $2 < $1/$email
echo -n "."

# delete HAM
rm $1/$email

done # end of email loop

} # end function learn_ham

#
# Script starts here!
#

# loop through all user home directories
for file in $(ls /home); do

# if there is a Spam/Missed maildir
if [ -d /home/$file/Maildir/.Spam.Missed/cur ]; then

# then process any missed SPAM
echo -n "missed spam for $file: "
learn_spam /home/$file/Maildir/.Spam.Missed/cur $file
learn_spam /home/$file/Maildir/.Spam.Missed/new $file
echo ""

fi # end if

# if there is a Spam/NotSpam dir
if [ -d /home/$file/Maildir/.Spam.NotSpam/cur ]; then

# then process any falsely identified spam, i.e. HAM
echo -n "false positives for $file: "
learn_ham /home/$file/Maildir/.Spam.NotSpam/cur $file
learn_ham /home/$file/Maildir/.Spam.NotSpam/new $file
echo ""

fi # end if

done # end for loop

echo "Done!"
--- end file: train-spam.sh ---

I place the above script in /root and create a cron job to run it every day in the early morning. You will need to edit some parts of the script if your missed spam and not spam directories are named differently. Good luck, and I hope it is helpful in your continuing battle against spam!

Yes, I Am Still Here

Wow - it's a little dusty around here, isn't it? I haven't updated the ole' blog in months, and every time I did think about updating it, I said to myself, "but it's been so long!" That is a nice self-perpetuating situation, so now it's time I break free from the shackles of stupidity and update the damn thing.

I am very well aware that I still need to post pictures up from last summer's vacation, but those will have to wait for another day. Instead, I'll try to catch you up with what has happened over the last four months or so.

Little League

As I may have mentioned previously, I am on the board of directors for our local little league. Finding volunteers is always a challenge, as many people are just too busy (or unable) to help out. I have managed my son's team for the last three years, and really enjoyed it and the kids and parents in the community that I have been lucky enough to meet and get to know. Since the league was in dire straits to find a new board, I volunteered as the information officer.

It has been a lot of work, but overall, I think it is worth it. The people on the board are all good people, doing their best to make sure that the kids of our community have the opportunity to baseball in a safe and fun environment. I have spent late nights at monthly board meetings, weekends doing sign-ups and field prep work, and time during the day responding to board emails and the like.

I am not managing this year, and am instead, merely coaching. It's sort of driving me nuts not being in control, but I'll get over it.

Christmas

I took my normal two weeks off after Christmas, and promptly got sicker than I was all year. During the coarse of 2008, I took two sick days off from work. While on vacation, I was bed ridden for nearly three days. That'll teach me...

Other than that, we had a very nice Christmas and New Years. No Christmas cards were sent this year, due to the ebola plague that swept through the house. I like to think of it as a minor inconvenience to our friends, and a minor bonus for the tress.

Tech-talk

The day after Thanksgiving, I braved Fry's and purchased parts for a new computer at prices almost too good to be true. Actually, they were too good to be true because of those damn rebates. Here it is, four months later, and I am still waiting on my final rebate to arrive from Abit.

The system is an Intel Core Duo 2.6 Ghz machine with 2 GB ram, and nice Nvidia video card, and a decent SATA hard drive. It plays Team Fortress 2 beautifully, and I haven't really bought any other recent games due to time limitations. It should last me for the next couple of years easily.

So, that is a very brief update of what I've been doing over these last few months. I'll try to update the blog a bit more regularly as time and obligations permit.

The War on Spam

It was earlier this year that I last detailed some steps taken in an attempt to thwart the growing junk email (a.k.a. SPAM) epidemic. dspam has continued to work like a champ, with an accuracy rate currently standing at 99.13%.

However, the volume of spam continued to increase, almost as inevitable as the rising tide. I didn't let it bother me too much, as I could simply delete the messages sent to my spam folder, although I had to admit that looking through all of those messages for false positives was getting a bit tiresome.

And then it happened... The event that moved me past the tipping point, and thrust me forcefully back tot he front lines of the war on spam. The event seemed innocent enough: I went to send an email, and our smtp server (qmail) did not respond. wtf?

I shut down qmail, restarted it, and it seemed to work for a while. However, after an hour or so, the same problem arose. wtf?

Looking at the process list, I noticed that there were 40 qmail-smtpd processes running. What were they trying to send? Time to get my hands dirty and learn a bit about qmail's queue. I read a bit on the queue, and installed a tool called qmqtool. By using this new tool, I discovered that I had over 6,000 messages awaiting deliver in my mail queue!

Why so many? Because for every spammer that sends an email to a bogus address, a mail server will craft a bounce message and attempt to send it back to the (usually faked) sender. Gah!

As a first step, I decrease the queue lifetime to 24 hours. After 24 hours, any message still left undelivered in the queue would now be deleted. Then, I cleaned out the queue and restarted qmail. It worked!

...for a while. Ok, so let's do some more furious reading on the subject of spam prevention in the qmail environment. First of all, I needed to increase the maximum number of qmail-smtpd processes which were allowed to run simultaneously by tcpserver. By adding a "-c 400" to the tcpserver command used to execute qmail-smtpd, I increased the maximum number of simultaneous connections from 40 to 400. That ought to be a good start.

Now, how else can I decrease the volume of spam? I had heard of greylisting before, and thought that it might be a good next step. Luckily, there is a simple greylist implementation for qmail called qgreylist. It installs fairly quickly, and does a fairly good job. However, it is written in perl and with all of the spam emails flowing in, it can cause a bit of a performance hit.

So, maybe there is something another check I could do before sending the incoming email to qgreylist? How about an RBL check! When another computer attempts to deliver an email to our mail server, the RBL (Real-time Blackhole List) check sees if the sender's IP address in a list of know spam servers. If it is, then the connection is dropped and the email never even is seen by the greylist app.

With the above changes made to our mail server, I am now receiving about 10% of the spam volume that I was before the changes were made. Furthermore, our remote queue sits at a fairly steady level of 400 messages.

Only time will tell for how long this solutions will suffice! Wish me good luck and my email godspeed!

Upgrading to Ubuntu 7.10 - Gutsy Gibbon

As mentioned in previous postings, I use Ubuntu as a desktop OS both at home and at work. At home, I have had Dapper Drake installed on an older Dell Inspiron laptop with a D-Link DWL-650+ wireless PCMCI networking card, which functions perfectly. The kids will even hop on it to browse the web with Firefox or write a paper using OpenOffice.

At work, I have a dual Opteron Dapper Drake box on which I do any Linux and/or C based development. It has been rock solid for well over a year. I also recently installed the 64-bit version of Feisty Fawn (7.04) on a spare AMD64-3200+ system to test 64-bit development using Free Pascal.

When I heard Ubuntu 7.10 was released, I looked through the notes and thought that it might well be time to upgrade my Dapper boxes, as Firefox was getting rather long in the tooth (the Dapper version being stuck on the 1.5 branch), and I also wanted to investigate mono development (more on this in another post!). Since mono development is still rapidly evolving, I wanted to look at a recent distribution.

The first box I went to upgrade was the AMD64 system. I had little to lose on the system, and so elected to do a clean install, nuking the previous distribution. Everything installed smoothly, so I played around with the distribution for a while to see what was new.

The installer had detected the motherboard's integrated Nvidia video controller, and offered to install the restricted driver for me to increase performance. I accepted, then turned on some of the flashy desktop effects provided by compiz, and was pretty impressed with some of the flash and sizzle. To get finer tuned control over what compiz does and does not do, I installed the compizconfig-settings-manager using synaptic.

Next check - Firefox and the network. The integrated NIC was detected and automatically setup - no work necessary on my part (since I recently switched our work network to providing DHCP addressing). Firefox seemed to work well, and when I visited a page that used Flash, it offered to install the 32-bit flash plugin with all of the necessary wrappers and libraries so it could be used in my 64-bit Firefox. Until Adobe can provide a 64-bit flash plugin, this is the next best thing and it worked flawlessly.

Ok - the box looks and works well, so it was time to test the next system in my parade of installs. Hmm... Let's try the laptop at home. I burned a live cd of Gutsy and took it home. I booted up the laptop with the CD, and wireless networking was dead - frak! This has been a problem with this laptop for the last three ubuntu releases. I decided to come back to the laptop at a later time.

Next stop - the dual opteron development workstation. What to do, what to do? I didn't want to do a clean install this time, being a somewhat lazy sob. I decided to back up my important files to another computer and do an inplace upgrade from Dapper to Edgy, then from Edgy to Fesity, and finally from Feisty to Gutsy. This seemed to me to be somewhat risky, but I wanted to see how Ubuntu fared when doing inplace upgrades. If it came through THREE upgrades, well that's a pretty damn strong endorsement.

So I first upgraded from Dapper to Edgy by following the instructions here. It took a while, I had to confirm a couple of file overwrites, and about 3 hours later it said it was done. Reboot the machine, and all looked well. Did it again for Edgy to Feisty, and one more time for Feisty to Gutsy.

Once the upgrade to Gutsy was done, I took a close look at the system. Compiz was not working, and I found tha I needed to install xserver-xgl for compiz support. Fonts were a bit tweaked, but I managed to get them workig with a couple of corrections to the FontPath section of the file /etc/X11/xorg.conf.

Overall, I was *very* impressed with the three distribution upgrade. This gave me the confidence to try it with the laptop at home, and so I did. To make a long story short, I am now writing this post from the Dell Inspiron upgraded to Gutsy, with wireless networking and flashy desktop effects turned on. It's not super speedy, but once things are loaded into memory, everything runs swimmingly.

I just checked fonts in emacs, and they are fairly screwed at this point in time. I may have to check out my font paths on this system, and if I recall correctly, I had to change the default fonts emacs uses to get non-horrid looking fonts on the dual opteron system.

Mammoth Lakes Vacation - Part 1

Wow, have I been slacking off on the blog lately, or what? I've been very busy at work and at home this last month, so please pardon the virtual dust gathering around here.

For our family summer vacation this year, we decided to rent a trailer and go camping up in the Mammoth Lakes area. There are many of campgrounds in the area, and quite a few activities to keep us busy.

Our trip up was fairly uneventful. We a made a quick stop in Lone Pine to stretch our legs, and take in the beautiful view of Mt. Whitney.



After that brief stop, we grabbed lunch in Bishop and then made our way to Mammoth Lakes and our remote campground. The wilderness up there is just pristine, with jaw-dropping views surprising you around each new bend in the road.



I took the kids fishing for the first time, but we were skunked. From what I understand, it is best to avoid the easy-access lakes, as they are fished heavily. Even though we didn't hook up anything, the kids had a good time trying their hand at it.


It was odd seeing Mammoth Mountain without snow, as I had never been up in this area during the summer. It is a truly beautiful area, and I bet we make our way back up here sometime in the near future.


I'll have parts 2 and 3 coming up in the next week or so. To see more photos from this picture set, click here.