Spam fallout
I’m going to make a stab at writing up all the things I’ve meant to write up in recent memory. Will probably end up burning out for another few months, but such is life.
The months of June/July were mostly devoted to learning new programming gadgets. Spent some time working with my friend’s MacMonkies site and his news feed. Also spent a substantial amount of time playing with and tinkering some CSS positioning and PHP/Database layout for my site. I never quite got things to a finished state, so it all just sat on my laptop. Let me just say that until you’ve programmed/developed on your own server, you have NO idea how wonderful it can be.
Around the end of July, our sysadmin at Trib decided that it was time to make a major attack against the spam problem. He built a really nifty array of servers to process incoming mail, run SpamAssassin, and then deliver the mail normally.
First, a little retrospect. Our old system consisted of a metric of a variety of different keywords and message headers used to identify a message as spam. As long as the sysadmin kept that list of keywords up to date, it did a pretty bangup job of reducing spam. Unfortunately, keeping up with the necessary keywords was a full-time job in itself, and the list was usually several months out of date. This kept spam at a ‘reasonable’ level. I got the occasional spam message, and the only ones who got really bombarded were the ones who had signed up for it some way or another. The real downside to this system was that we threw out the messages tagged as spam. This resulted in about 1-2 calls per month of people who just lost certain messages, and there wasn’t much we could do other than whitelist those addresses in the future.
The new system was designed around preventing those calls, and reducing the overall spam load on the system. To be fair, the new system has been dead-on for me. In the two months since activation, I’ve gotten maybe two false-negatives and zero false positives on my primary account, and only 6 false-negatives and zero false positives on my secondary account which used to get 20+ spam messages daily. We’ve also heard from a few customers who are absolutely thrilled that they no longer receive hundreds of spam messages a day.
Unfortunately, we’ve also heard from a large number of customers who are getting a fair number of false-positives. For those unfamiliar with the terminology, a false-negative is a spam message that the filter mistakenly lets through and delivers as normal mail. A false-positive is a normal message that the filter mistakes for spam and delivers to the junk mail folder. These false-positive problems typically come from one of a few sources. Companies or other business organizations who have a wide variety of contacts and news organizations that are vital to their operation. Random users who have friends on AOL or who use some wacky email program to “prettify” their email. People who never received much junk mail in the first place, and don’t really need the filter at all.
The people who don’t need the filter are easy. We turn off the filter, and let them know that if spam ever becomes a problem, we can turn it back on. They’re happy.
The random users aren’t too hard either. Once we explain that a message with dancing smiley faces, a recipient list a mile long, and a few banned words is nearly indistinguishable from spam, they’ll usually relent. We add the necessary addresses to their whitelist and move on.
The businesses are easily the worst. First, for whatever reason, our sysadmin chose not to send out a letter stating the changes to the system, so these business persons are absolutely incensed that we would change something without giving them advance notice in triplicate. Forget the fact that Joe Random user wouldn’t understand even one sentence of said notice. Forget the fact that we did post a notice about it on our website. Forget the fact that our contract (which everyone must sign) says that we can make changes to the system whenever necessary.
So after spending 15 minutes reaming me out for something over which I had zero control, I fix their problem, either by adding a large number of whitelisted addresses or turning off the spam filter so they can deal with it. I then get another 15-20 minutes getting reamed out, on how important email is to their business, and how much money we’re costing them. The funniest part is when one of these business people calls back a week later, after wading through the 300+ daily spam messages that our system was filtering out, begging to have the filter put back into action.
In the end, it comes down to simple economics. These people are using a cheap online service, intended for home use, to run and provide service for business level applications. It’s like a racing company renting a Yugo for the Indy 500 and then wondering why it doesn’t take first place.
Tomorrow: Religion