More Give and Take on the False Positive Problem

Editor’s Note: The June 22 issue of DirectTips included a story about a test run by e-mail delivery service Pivotal Veracity to judge the extent to which large companies’ e-mail program can be affected by “false positives”: having their permission-based e-mail blocked or delivered into spam folders. The Pivotal Veracity experiment used three dedicated e-mail accounts on MSN Hotmail, Google’s Gmail and Yahoo! Mail. The test found that out of 90 companies, 54 had some of their e-mail blocked by spam filters during a six-week period. Another six turned up blocked e-mails just after the test period ran out.

Pivotal Veracity found that some of those blocked e-mails were sent out under a “bonded sender” program. In fact, the experiment’s results found that the rate of false positives was slightly higher among companies that used that form of delivery assurance. According to the survey, 55% of companies used a bonded sender program experienced at least one false positive, compared to 53% of those that did not.

In fairness, and to stimulate a productive debate, we are publishing the following response from Matt Blumberg, president and CEO of Return Path, which operates the Bonded Sender e-mail accreditation program. We also felt it fair to permit a reply from Deirdre Baird, president and CEO of Pivotal Veracity, the company that conducted the original test.

Mr. Blumberg of Return Path writes:

I saw Brian Quinton's article today regarding Pivotal Veracity's deliverability "study," and I would like to set the record straight regarding the claims the study makes about the Bonded Sender Program, which we run. There are two major issues with the data you posted regarding accreditation programs.

1) This study tracked delivery at for 100 clients - only 13 of which use Bonded Sender. The other "third" they report is certified by TRUSTe - which in no way impacts email delivery rates. Thirteen is hardly a statistically significant sample.

2) The ISPs covered in the study are Yahoo!, Gmail and Hotmail. Of those, only Hotmail accepts the Bonded Sender program. Any study looking at Yahoo! or Gmail for clues of Bonded Sender helping delivery would not show any conclusive evidence of improvement, since they do not use the service at this time. That is common knowledge in the industry, and it certainly would not be a surprise to Pivotal Veracity.

As for Hotmail, being a Bonded Sender does indeed help delivery-- to the tune of 25% lift on average. By tracking delivery rates across companies using Return Path's delivery monitoring tools both pre- and post- Bonded Sender accreditation, we have seen significant increases in delivery across the board. Of the ISPs publicly stating they use Bonded Sender in their filtering evaluation process, the results were as follows: Hotmail 25%; Road Runner 22%; and MSN 15%. Also of note, companies accredited with Bonded Sender experience an average of 95% Inbox success across the board for their campaigns.

It is important to note that the general methodology used by Pivotal Veracity in this study is NOT scientific or controlled. The numbers - especially around Bonded Sender - are not statistically valid and should not be stated as such indisputable fact. For example, it's unclear from the study that they even tracked Bonded Sender clients' mailings by looking at the actual sending IP addresses (which can be bonded) as opposed to simply domains and company names (which can not be bonded).

We agree that false positives are a huge issue in the market today, and we are working hard to help companies and [e-mail service providers] overcome them.

Best Regards,
Matt Blumberg, President and CEO, Return Path

Deirdre Baird of Pivotal Veracity offers a response to this critique:

In the “Limitations and Study Methodology” section, we clearly state that our study was limited to 100 companies - nothing more, nothing less - and while the results speak for themselves as it pertains to the mailers in our study, there is no basis or claim that the data can be extrapolated across all mailers. Since Return Path offers deliverability tracking that relies upon sample sizes that are statistically insignificant even as it pertains to each mailer they are tracking, we imagine Return Path is not disputing the value of directional data but rather is concerned about the fact the data in this study does not reflect positively on one of Return Path's products. Unfortunately, the sum total results of the study illustrate that there is no silver-bullet and, as much as we would all like it to be, Bonded Sender was not one either.

As for the domains covered by Bonded Sender, we, unfortunately, do not "know better" than any other company reading the sales literature on Bonded Sender, which claims that Bonded Sender can ensure inbox delivery at thousands of domains. Since Return Path does not disclose this full list, we stated in our survey that "Bonded Sender does not publish a full list of all the ISPs against which this claim [‘ensures inbox delivery’] is applicable." It has always been our assumption the bulk of the domains "using Bonded Sender" are not ISPs but rather domains using Spam Assassin, a free open-source spam filter in which use of Bonded Sender slightly helps one’s spam score but by no means guarantees inbox delivery. We very much appreciate Return Path's clarity and confirmation that only two major ISPs, MSN Hotmail and Road Runner, use Bonded Sender.

We did not track Road Runner in the study but Bonded Senders (as identified by the sender's IP address) did not "ensure inbox delivery" at Yahoo, Gmail OR Hotmail. In addition to the mailers tracked in our study, none of which were Pivotal Veracity clients, we also service a number of companies who use Bonded Sender. Our experience has been that MSN Hotmail routinely filters mailers on a variety of content and mailing-practice-related issues that can easily be resolved by the mailer without the use of Bonded Sender. In fact, both our staff and clients were in the audience at the INBOX Conference in California recently where Matt Blumberg himself gave a speech in which he stated that if mailers have sound infrastructure and good mailing practices they don't need Bonded Sender; so it appears, in fact, we all agree.

Bonded Sender is a useful last resort for MSN Hotmail but is by no means a silver-bullet and can not guarantee inbox delivery anymore than whitelisting directly at the ISP can. We recommend mailers always test whether any solution delivers better results and to never be misled by any solution that states much less infers that it is a guarantee of inbox delivery. As the ISPs will tell you, a single such solution simply does not exist.

Deirdre Baird, President and CEO, Pivotal Veracity