Tuesday, November 25, 2008

Thanksgiving Wishes


So here is my prayer for Thanksgiving:

I want to give thanks to all the people that I associate with via the blogosphere. Like the Pilgrims and the Indians, we often come from different backgrounds. And like them, we don't always agree, but we need to find a way to get along, even when we disagree (especially when we disagree). We spend a lot of on-line time working together in all forms of groups. These groups often become polarized, and much anger and vitriol is spewed against those who are "less" enlightened than us. I love passion, and passionate arguments, even when I disagree. It is energizing to have a strong passion for what we do, otherwise why get out of bed?
Thanksgiving is an opportune time to reflect about our relationships to others. As Charles Dickens said so well in "A Christmas Carol" that it is a time to "think of people (around us) as if they really were fellow-passengers to the grave", which should give us pause to reflect on the shortness of our time here.
We are all on this planet for only 78 round trips. Some get a few more, many get a lot less. So above all, the first thing that I am (and all of us should be) thankful for is good health. Without it, all the rest really does not count for very much.
So, I would like to summon this spirit to wish everyone a Happy Thanksgiving, and hope that we can re-learn what we were taught as young children: to be nice to others. This is especially needed in the blogosphere, where anonymity and distance cloaks civility and open discourse which oftentimes provokes needless harsh words and enmity. In that spirit of brotherhood (or should I say community), I give thanks to all the people in my life, associates at Maysoft, fellow workers, and fellow bloggers everywhere for the chance to spend a few moments, in person or virtually, together, enriching our shared experience here on earth. Isn't that, after all, what the Indians and Pilgrims were doing at the first Thanksgiving in the year 1621?


Picture: "The First Thanksgiving", painting by Jean Leon Gerome Ferris (1863–1930).


Tuesday, November 18, 2008

A World without Spam Filters


After looking at our spam blocking statistics for yesterday (we blocked 279,706 messages on our 2 Domino SMTP gateways), I wondered what the world would be like if we had no spam filters?After I pondered this question for some time, I finally decided that the only way to really know the answer is to stop the automated blocking and just see what happens. After all, SpamSentinel has been working so well
for so long that I've forgotten what it is like to live without a Spam filter. So I have decided to do a one hour test, bypassing the filter on our main SMTP gateway, from 11 AM to 12 noon today, for our Lotus Notes email users.


10:55 AM
I am writing this part of the blog before the test, not knowing if a small mutiny will ensue, or not. All is calm now, at 10:55 AM. The test begins in five minutes, at 11 AM, and runs for one hour. I am afraid to run it for more time than that, as this is our
live email and people have work to do.


10:59 AM
So, it is 10:59 now, and I have just sent one last warning email about the test to everyone ....

"At 11 AM EST today, we are running a test of a World Without Spam Filters.All automated Spam blocking will be turned off for sixty minutes. We expect all users to block their own spam manually"

11:00 AM and we're off...the "World Without Spam Filters" test begins

11:21 AM
So, 21 minutes into the test, my Lotus Notes Inbox is being flooded. Here is a snapshot:
A picture named M2
11:38 AM
Blackberry users are complaining about the spam. Allison Cote wants to know what is happening. She had to leave her Blackberry and delete a bunch of messages from her Inbox. My test is distracting her, and others, from their work.

11:42 AM
I am checking mail1.box and mail2.box. There are 3,341 messages in one, and about the same number in the other. The router is going crazy trying to send some of these back to the senders as delivery failure reports. Aaaargh... this is worse than I thought.

11:46 AM I have to stop this test early. The amount of pain everywhere is too much. I am turning on the SMTP mail filter now as the only prudent course of action.
Blackberry users are still complaining.
Good mail is buried in the spam.
The router task on the server is working way too hard, trying to deliver these spam messages or delivery failure reports.  Some messages are being sent back out as Backscatter.
I have stopped the router task while I delete the contents of mail1.box and mail2.box.
Good messages that were in mail1.box or mail2.box may have been lost when I deleted all the pending messages to clear out the spam
I just restarted the router.  It is quiet now. I will have all users resend all messages between sent between 11:30 and 11:45, as they were most likely deleted by me.


11:58 AM Looking around at other mail boxes to see the damage. Most people got about 10 messages, which tracks well with the 200 a day most people receive on average.
I found this gem in one Inbox, containing malware and a funny subject, "CNN: Aliens send us cartoon messages!"
A picture named M3

12:06 PM All the mailx.box files are clean now. Users are not making any more noise, but they have questioned my sanity.


So what did I learn from this Social Experiment? What would a World Without Spam Filters be like?

1. For starters, deleting real messages is a highly likely possibility, both by users in a hurry to clean their Inbox and by the Lotus Notes email administrator.
2. It took me about 15 minutes to clean up 45 minutes of spam. That makes the clean up and delete job, as an administrator,
to be about 15 minutes per hour, or 2 hours per day.

3. Spam is still heavily reliant on random dictionary attacks, hence the amount of delivery failure reports.
4. Backscatter causes collateral damage to others outside the organization.
5. Servers would be overwhelmed with the workload, trying diligently to deliver every message as if it were a valid email.
6. Blackberry Users: It is not just the inconvenience of hitting the *delete* button. It is the mind-numbing distraction of having the Blackberry go off every time an email message comes, especially when you are in a meeting but also waiting for a critical email that needs an immediate response. The tension over false alarms kills concentration.
7. Attention Deficit Disorder: Some studies have been done which detail how much productivity is lost for every interruption. The cost of a Spam email isn't just the time it takes for our minds (and hands) to figure out it is Spam and ignore/delete it, it is also the amount of time it takes us to get our mind focused again which can magnify the loss of productive thinking time. Without a Spam filter, I would NEVER enable any sort of pop-up 'you have email' alerts in my mail client, as I do a lot of programming and project work where I need time to work for a stretch without interruption in order to keep up the quality of my output.
8. The cost of spam without a filter is nearly 120 times what the filter costs. This is based on the fact that, between deleting spam and hunting for good messages in the mess takes about 30 minutes a day, or about 120 hours annually. At a pay rate of $30 (25 Euros) per hour, which approximately equals the annual cost of spam filtering, the payback is only a single day of usage. Myself, I couldn't stand even one hour with the filter off, let alone one entire day.
9. Spam Filters, although imperfect, are an absolute critical necessity.
10. In a World Without Spam Filters, email would be almost useless.

SpamSentinel for 64 Bit Domino

I now know why 64 bit Domino is gaining in popularity. It really gives Domino access to so many more memory resources. Performance is not limited by the old 2 gig RAM barrier for applications. So we are happily releasing the 64 bit version of SpamSentinel to beta customers this week. The conversion from 32 bit to 64 bit was no fun, but that is why they call it work!
SpamSentinel also has less constraints on processing, and it shows! We are seeing much faster throughput, past the 2 million messages per server per day mark!
We have been running this on our internal server for 5 days and it has remained stable. It is running on our internal mail server, a 64 bit server running Windows 2003, along with Domino version 8.0.2, where all of our mail files exist. This is our own internal "vote of confidence", meaning we will not release software until we are confident enough to put it on our internal production servers, as errors can cause dreaded work interruptions.
So, for those of you who like to have your anti-spam and anti-virus solution run natively on the Domino server, you can use this solution. As before, you can expect a 99.44% block rate.
Any customers who want this version can call us at our main line at (978) 635-1700 or email me, frank_paolino@maysoft.com to obtain an advance copy.

Friday, November 14, 2008

How to Build a Botnet

So, one Washington Times reporter did what a whole "Can Spam" act couldn't do, make a significant dent in spam. It is still down 2/3, which worldwide is huge!

As a designer of anti-spam software at www.maysoft.com/, it did make me start to think that if we could figure out how the botnets work technically, then we could begin to shut them down. To defeat an enemy, you must think like them. So, to understand how to build a botnet in detail, I decided to do a "thought experiment" of building a sustainable fault tolerant botnet system. So, the first thing we need are some requirements. The "requirements" if one were to build a botnet, would be the following:

  1. A master server (host) that gives instructions to all the bots
  2. Bots need to check into a host daily to get instructions
  3. Host needs to be able to give it pointers to a list of email recipients and the message(s) for today
  4. Host needs to give bot a number of messages to run (or instructions to run the whole list)
  5. Fault tolerance - Botnet needs to tolerate if a host is unavailable, having the ability to look for another host

Requirement 5 is interesting, as it applies to McColo going off-line, and the recent reduction of spam by 2/3 because the bots are not sending billions of messages. Why? Because they have no controlling host to get instructions from. Where are the botnets going to go next? The spammers have deployed an amazing grid computing platform. I have to believe anyone capable of building such a platform would have thought about what happens if the host becomes unavailable, making it a single point of failure? My design would be for it to try other servers if the default host is unavailable for 1 day.
How to do that? This was a puzzle at first. If you just hard code IP addresses, you give computer forensic researchers (anti-virus companies, government agencies, and hackers) a list of all of your servers. Not a good idea if you run an illegal operation. So, you would generate DNS names randomly. Then look for them. But if you do it randomly, how can the host know which DNS names to create?
The answer for my design would be to have the bots try to connect to a domain name and use DNS to find the server. For example, spammer1.com, spammer2.com, etc. This is still pretty easy to stop. So my design would be to have the domain names generated by the bots based on an algorithm based on date, so they are dynamic. The spammer creates the new domains as needed and points them to the new host computer. This could change daily to avoid the problem of cutting off one host. (Frank's note: This answer is too basic. I think their actual system is more clever than this simple approach).

So that is my idea of how to design a botnet. So when these zombies start reconnecting, and my guess is soon, unfortunately these millions of compromised computers will start to spew more spam.
So the answer here seems to be to find ways to locate and disable the host computers. I prefer technical solutions to legal ones. But in this case, if the techies can notify the government(s) where these hosts are located, maybe that one-two punch will really help to "Can Spam" !

(Read my other postings about Lotus Notes Spam at http://blog.maysoft.org )

Thursday, November 13, 2008

Worldwide Spam Volume Down 60% !!!

One spam controlling source, McColo in San Jose, was shut down. The result was that the botnet it controls basically had nothing to do. This botnet is responsible for 60% of spam, and its demise means everyone is noticing a 60% drop in spam worldwide in one single day! We have noticed it internally, dropping from 250K message daily to 90K.

Our phones at Maysoft are ringing with SpamSentinel customers who want to know where their spam is. Well, it never arrived in the first place!


Maybe if we can get 3 or 4 more of these firms shut down, I can take a vacation!


Seriously, though, I fear for the millions of rogue computers who no longer have a way to phone home to McColo. What if they have some malicious code built in if they cannot communicate to the controlling servers for x days? Sort of like a rogue army, with no leader?
I would prefer that they self-destruct, but, alas, that is just wishful thinking.