Why Is Email So Complicated? Part 101: There’s Just Too Much Of It

When I tell people that I’m writing about why email is so complicated, the most common reaction is disbelief:  “Email seems simple to me.”   The average email user has no more of a clue about the complexity of email than a hitchiker has about the size of the universe.  Email is complicated by a host of, technical, political, ethical, social, and historical factors, all of which I hope to discuss in due course.  But one of the biggest factors –and it’s hard to believe how vastly hugely mind-bogglingly big it is — is the sheer size and scale of modern email.  Listen…As I type this, Cisco is reporting that a single IP address has generated 15 million messages so far today, all of them likely spam.  Overall they estimate 40 billion spams per day, almost 15 trillion per year.  (Non-spam messages are only 1 or 2 percent of that volume.)  At Mimecast, we protect our customers from tens of millions of spam messages daily — and that doesn’t make us unique in the least.  Finding the few good messages in all those millions is just basic table stakes in the game of email security.A theoretician might tell you that there’s no major difference between processing 10 messages and 10 million; it’s just a matter of scaling up your infrastructure.  But the people who work for email service providers know differently.   The problem is that there are more things that need to scale than meet the eye, and any one of them can become a chokepoint for your service.

So, as volume scales, you need to get more machines and bandwidth.  That’s pretty obvious.  Almost as obvious is the need for more skilled operations personnel — by the time you have tens of thousands of machines, like Google, one person couldn’t even turn them all on or off in one day.

But even if you’re on top of these things, there are some that might creep up on you.  Can the line from the power company handle all your needs or do you need them to lay another?  Do you have enough HR people to hire your new admins?  Enough room for all your servers?  Enough rows in all your databases?  Enough failover plans?  Enough compliance monitoring?  Is your basic software architecture hurtling towards its capacity, with a major rewrite in the works?

It’s nearly always the things you don’t expect that burn you. For a while a major limiting factor in Mimecast’s growth was the speed with which we could absorb new customers’ often vast existing email archives.  A simple one-time auxilliary process can become a major problem as you scale up.

Email isn’t unique in this regard, but it suffers greatly from its own success.  Beyond the challenges for operators hinted at above, there’s the bozo factor.  If a billion people are sending email, you can guarantee that quite a few of them are doing so using broken or amateur software, out of compliance with the standards, causing untold mischief.   There are relatively few such problems, but even a few out of billions can create headaches for mail admins around the world.

The scale problem is shared with any other service that sees these kinds of volumes, but those are relatively few.  Email’s extraordinary scale only exacerbates its fundamental complexity, as I’ll be explaining in future installments in this series.

Of course, that’s still oversimplified …. Nathaniel Borenstein <[email protected]>