Live from Inbox: AOL Meltdown Leaves E-mail Executives Scrambling

AOL’s meltdown today sent e-mail service provider executives into a gut-wrenching scramble as they tried to figure out what was happening, and most importantly, what they should tell their clients.

The incident also shed light on an unusual relationship among a group of executives who work for competing e-mail service providers.

Millions of AOL users had trouble sending and receiving e-mail yesterday as AOL worked to identify and fix a software problem.

At the same time, a panicky e-mail from an AOL employee asked bulk e-mailers to stop sending messages into AOL’s system until the Web portal could figure out why its e-mail system had crashed.

“We are having a massive internal systems failure and are asking anyone and everyone that sends us a lot of mail to either stop (bulkmailers) or choke it (ISPs, hosting companies),” said the message from “Anna.” “If any of y’all can help, we would be most grateful. I do not ask this of anyone lightly. I know it’s a huge and costly pain in the ass. Believe me, it is for us also, and right now we are tempfailing [temporarily blocking] the world anyway, so if you can help take some of the load off, your mail is more likely to arrive sooner anyway.”

Another source inside AOL, however, said nothing was wrong, leaving executives at e-mail service providers in a serious bind: Should they tell their clients to stop mailing and possibly lose the revenue AOL addresses represents?

AOL addresses typically make up between 25% and 30% of a business-to-consumer marketer’s list.

“I don’t want to go into fire drill mode here and put my clients into a panic until I can figure out what’s going on,” said one e-mail executive just after the message from Anna arrived. The executive asked not to be named.

As the AOL crisis unfolded, an estimated 15 to 20 deliverability specialists for the industry’s top e-mail service providers—some of whom were attending a keynote session at the Inbox conference—were e-mailing, calling and instant messaging one another trying to verify that AOL was, indeed, having a problem, and trying to figure out what they should tell clients.

A couple of the group’s members managed to get Anna’s message verified by two other AOL employees and then the e-mail executives began figuring out what to do next.

These e-mail deliverability specialists—also known as ISP relations executives—work for companies that battle one another ruthlessly for market share, yet they are reportedly in daily contact with one another on an Internet discussion group to solve deliverability problems.

“It’s the type of discussion group that’s very open,” said Spencer Kollas, director, deliverability services for Premiere Global Services. “If someone is having an issue or noticing something going on they’ll e-mail the entire group saying ‘what do you know about this?’”

They even socialize at conferences.

“We’re all trying to get the same thing accomplished,” said Kollas. “We’re trying to get the e-mail into the inbox, so it doesn’t matter who your clients are versus who my clients are.”

Tricia Robinson, vice president of marketing and strategy for Premiere Global Services, added: “That is the one area of our industry that is collaborative. The rest of us don’t work together like that.”

And today, those relationships paid off handsomely.

For its part, once Premiere verified to problem, it immediately halted 11,000 of its clients’ messages that were ready to be sent, and then contacted clients to let them know what was going on.

As of this evening, AOL had sent a message to e-mail service provider executives saying the problem had been resolved and they could begin mailing to AOL addresses again at 3 a.m. Friday eastern time.