Slogging Toward E-mail Metrics Standardization

Quick: How do you define “open rate”?

If you’re not sure of the exact definition, you’re not alone. 

“A lot of terms that we use in e-mail marketing don’t really reflect what people think they reflect,” notes Stephanie Miller, global markets catalyst for Return Path. “Plus the definitions that people use for the same terms are different, and how they measure them is different.”

These discrepancies are the reason that the DMA’s Email Experience Council (EEC), of which Miller is roundtable leadership chair, set out several years ago to standardize the definitions of common e-mail metrics and, just as important, convince marketers and service providers to use the standardized metrics.

For marketers, the primary benefit of standardization is obvious: It’s much easier to compare statistics and benchmarks from one service provider to the next and across industry segments when everyone is defining the metrics the same way. In a survey conducted by the EEC, 89.1% of the 250 respondents said they were in favor of standardization, with just 6.5% against it.

But agreeing to the concept of standardization is different from agreeing on the standardized definitions. “There was a lot of debate,” says Miller of the council’s efforts to create metrics and definitions that the industry could get behind. “People who have done things in certain ways felt very strongly about them,” she adds diplomatically.

Among the proposed definitions is “accepted” in lieu of “delivered,” defined as “Any email that is not rejected by a server, including emails delivered to the inbox or the spam or junk folders and including those that are missing but did not receive a bounce reply.” According to Miller, some but not all e-mail service providers (ESPs) take into account bounced e-mails when reporting their delivered rates, an inconsistency that this definition aims to eliminate.

As for “opens,” the EEC proposes changing this to “renders,” as in “The total number of times an email is displayed (whether fully opened or within the preview pane) by any user and recording using only a tracking pixel for a unique subscriber address. If the user opens the email multiple times, one e-mail render is counted for each occurrence. This metric is based off of HTML-formatted emails only.”

The debate regarding open/render rates and related metrics was particularly spirited, judging from responses to a post on the EEC’s blog last year, when the definition was proposed. “My issue with this whole Render Rate proposal is it's adding a layer of complexity that isn't necessary,” wrote Chris Bryan of EmailDirect.com. “Why do we need Unique Emails Rendered and Unique Actions? (Unique Actions would be how EmailDirect calculates Opens right now.)” Matt Vernhout of EmailKarma added, “I'm not sure why we need a new metric... You can still count clicks without images as a successful open, just not a successful render. In truth well-designed messages don't need images to be fully usable by subscribers. Personally speaking, there are plenty of messages I don't call images for but still fully interact with.”

“It took a while to get consensus,” Miller concedes regarding this and the other definitions. “It’s not something that we rammed through.” Now, though, “no one is disagreeing with our definitions, so far as I know.”

The council is currently working to get the industry to begin using the new definitions and metrics. According to Miller, 15 service providers have so far verbally agreed to do so. But adoption entails reprogramming software, changing dashboards, and the like. “We probably won’t see any changes to dashboards and implementations for at least six months,” she says.

Granted, there’s nothing the EEC can do to force ESPs to implement the new metrics. “There’s no stick,” Miller admits. “We may make it a requirement for membership in the EEC. We may use ‘fame and shame’ as an incentive—as an association that’s the most powerful tool you have.”

In the meantime, the EEC is encouraging marketers to sign a petition (http://www.ipetitions.com/petition/eec-email-definitions/) supporting the standardization. Again, this is a somewhat toothless means of encouraging adoption (and so far only 31 people have signed), but Miller says it’s a way of showing service providers that their customers and prospects would prefer to do business with companies that adhere to the standards and that ESPs could eventually use their adherence as a competitive advantage.