April 27, 2017
Seedlist-based inbox monitoring the right way.
*Editor’s Note: As of January 2019, Gmail no longer allows apps to scrape and send inbox data to third-party providers, including those who offer panel data to inform marketers. Whether other mailbox providers will follow suit is yet to be determined, but at this time, there is no Gmail data informing panel data solutions. Read more here.*
When sending commercial email, it’s important to understand what’s likely to happen the next time you send to a recipient at a particular mailbox provider. Are your messages hitting spam at Yahoo or going missing at Comcast? Are you sacrificing ROI because you lack the proper visibility into performance? Seedlist-based inbox monitoring is a key data point in solving this issue.
- It’s important to understand the differences between traditional, non-optimized seedlist testing (testing with a seedlist that isn’t weighted to match the composition of your list) versus optimized seedlist testing (weighting seedlists to match your list composition).
- The gold standard of inbox monitoring, with the launch of 250ok ESP Integrations, is now optimized seedlist-based inbox monitoring and user-based engagement data.
- Sales pitches like “seed testing is dead,” and other tropes from the Sender Mythology canon are marketing fluff; learn how inbound filtering works.
- The perceived weaknesses of seed testing are actually its strengths: An unbiased look at perimeter defenses such blacklists, global spam filters, and so on.
Seed testing (a controlled, static behavior environment) is not intended to measure user-based engagement, and neither is email panel data (a heavily biased source lacking statistical relevance, among other issues).
- The non-optimized seedlists offered by most mailtech vendors treat all seeds as equals, whereas 250ok’s Seedlist Optimizer (patent pending) analyzes your mailing list to determine which mailbox providers matter to you.
- Not all seedlists are created equally; investigate and compare. Lookout for duplicate seeds when comparing seedlists among vendors.
The New Gold Standard Approach
Our expert team, a group that cut their teeth in email over the past two decades at places like ExactTarget (now Salesforce), Microsoft/Hotmail, Adobe, SparkPost, Oracle, Aprimo, Return Path, and more, agree the gold standard for inbox testing is through analysis of optimized seedlist performance—seedlist testing the right way—while looking for meaningful changes in user-based engagement data.
The focus of this blog is on the differences between non-optimized seedlists—the historical status quo and common approach—and the optimized seedlist analysis made possible with 250ok’s Seedlist Optimizer. We will also explain why email panel data and non-optimized seeds fail as a viable alternative to the gold-standard solution.
Sender Mythology Smackdown
We’ve detected an uptick in the number of myths promoted by other seed testing and panel data vendors, so we want to address some of their greatest hits.
“Emails seeds do not work.”
Source: Google AdWords, April 27, 2017
Isn’t that a dandy? Please note: eDataSource sells seed testing. If seeds are worthless, why do they offer them?
And we certainly do not consider currently available email panel data from any vendor an “upgrade” from seeds. To be clear: Seeds and panel data are not comparable email tools. The attempt to force an equivalence between them is flawed logic.
Additionally, we cannot advocate for the use of current email panel data offerings from vendors like Return Path or eDataSource. Find out why, and why we refuse to offer a similar panel data product, here: The Truth About Email Panel Data.
“Seed-based inbox monitoring is drawing to close.”
We just spat out our LaCroix. Thanks.
Seeds are like spam traps: By the time the red flag goes up, the canary’s not in jeopardy, she’s already dead. While seeds themselves are only one of several valuable data points you must track, no serious sender operates without frequently testing seeds. That’s why ESPs like SparkPost, SendGrid, Marketo and many of the world’s biggest brands rely on our seeds every day.
“Seeds have no statistical significance.”
Bizarrely, we hear this comment from panel vendors that also sell seeds. Why would they sell a worthless product? It’s likely just a (strange) sales strategy, but this approach exemplifies how business goals can override transparency concerns for consumers.
The bottom line: If you send a campaign to your list and those messages are successfully delivered to seed addresses, what you learn is that your reputation is sufficient to get you past the perimeter defenses: IP blacklists, global spam filters, and so on.
From a statistical significance perspective, panel data fails to achieve the desired significance for the same reasons, among others, as seeds: Do they accurately represent the behavioral demographics of your list? Are all recipient archetypes represented? If the pushback is that panel members interact with the messages, and therefore personalized filtering comes into play, our response is: So what? That actually pollutes your data because it is now cluttered with individual bias. Unless you have a mechanism correlating a given panel member with segments of your list and proves that panel member’s actions are an accurate proxy for the behavior within their segment, you do not have data providing a more accurate measure of deliverability than seeds.
There’s no evidence suggesting there are large enough panels to guarantee sufficient coverage for most brands. Learn more about the statistical relevancy issues with panel in The Truth About Email Panel Data.
“Seeds cannot represent human behavior and therefore cannot represent how deliverability filters interpret personal preferences.”
Bravo. That’s exactly the point of seeds: They’re a control group modeling unbiased mailbox behavior. They provide a valuable data point because Gmail, Outlook, and every other major ISP have different global filtering behaviors.
Additionally, at smaller ISPs and private domains which use either heuristic filters (i.e., hand-crafted rule sets) or commercial-filtering products (e.g., Cloudmark, Barracuda, etc.) which are essentially “shared” global filters operating from the same cloud-based threat profiles, seed testing remains relevant because of the relative lack of sophistication of those filtering schemes.
We asked 250ok’s Vice President of Engineering, Paul Midgen, who previously served as the Microsoft/Hotmail senior program manager, to chime in:
“Seed accounts and panels are popular tools for forecasting the outcome of a future email campaign or to help understand why current campaigns are not performing as expected. But their validity varies widely based on how closely they match your audience. In other words, you can only achieve highly accurate results by using your existing audience as a panel or for seed testing.
This is, for obvious reasons, impractical; the lesson, however, is even more clear: Choose a solution that models your audience as closely as possible in order to get the most accurate information. Better yet, choose a solution that keeps pace with your audience as it grows and changes.
Midgen offered additional clarity, “If you’re looking for visibility into recipient behavior, then you don’t want to look at email panel data or seeds. Everyone sending commercial email today already has the best and most relevant panel in the world: the people they’re sending mail to.”
Based on this realization, we kicked off our 250ok integrations program in early 2017 and recently opened it up for beta testing. How will integrations drive a meaningful evolution for email analytics?
- For the first time, all of your delivery (e.g., delivered rates, bounces, deferrals, opens, clicks, unsubscribes) and reputation (e.g., blocks, complaints, spam traps) metrics will be housed under one roof with data feeding directly in from your ESPs in real time.
- Instead of noisy charts with no context, you’ll finally have a comprehensive picture of your mailstreams to identify trends and truly understand your performance.
At that point, you will finally have total statistical relevance, but even then, you’ll still need optimized seedlist testing. Let’s get into it.
Global Seedlist Coverage
When considering the use of seeds, it begins with coverage.
Top vendors offer seeds at more than 100 mailbox providers around the world, with an average of approximately 10 seeds per provider. When shopping for a seedlist provider, be on the lookout for low seed count per provider, a low provider count, and an overinflated provider count due to duplicate seeds.
What are duplicate seeds?
Several mailbox providers offer regionalized versions of their service based on country. However, if the mail for various regions all resolve back to recipient servers that apply identical global filters, the mail is treated equally. In these cases, adding multiple seeds in those regions is redundant and does not buy you any greater insight or value.
Yahoo, for example, maintains an international presence in more than 50 countries. But the mail for a majority of their regional mailboxes resolves back to recipient servers that apply identical global filters: South America and Australia are covered by yahoo.com, Yahoo UK covers Europe, and so on. Based on how Yahoo handles mail, we maintain seeds for every unique gateway—Yahoo US, Yahoo UK, Yahoo HK, Yahoo JP, and Yahoo TW—and ignore duplicates.
We have witnessed vendors advertise seedlists that include duplicates. In the spirit of transparency and clean data, we do not. The point here is that you should remove any duplicate seed count when assessing the actual coverage of a seedlist, and make sure you’re getting an apples to apples comparison.
Why Non-Optimized Seedlist Testing Isn’t Enough
Non-optimized seedlists (those offered by other mailtech companies) treat all seeds equally. While we agree you need a vendor with massive global coverage, what matters just as much is bringing greater context to your seeds.
So, what are optimized seedlists?
Do you send to the same number of addresses at Gmail as you do Roadrunner or AT&T? Do you know how many of the addresses you message are hosted by Office 365? If you answered “no,” you’re not alone. Giving context to seedlists, in other words making them specifically relevant to the composition of your recipient list, is why we built the Seedlist Optimizer.
As you analyze a list or segment of your addresses, the optimizer provides a breakdown of the networks receiving your mail. If your list segment did not include any addresses for a given provider, by creating an optimized seedlist, neither will your 250ok seedlist.
Test results provided by the optimizer will show the total count of recipients and the percentage of your list for each mailbox provider (organized by region), hosting, and filtering company.
Also, you can apply the weighting of your optimized seedlist to your reporting, so the results of each test match your list composition. By doing so, you see a truer inbox rate score of your mail. With a more accurate view into the health of your inbox placement, you can stop focusing on issues at the providers you don’t mail to.
For ESP or agency users
How can this functionality be used if you’re an ESP or agency sending messages on behalf of several customers and you don’t want to isolate a specific list? While viewing the optimizer results, you can drill down into each of the mailbox providers to view the domains that make up that receiving network.
And knowing your domain distribution is valuable for many reasons: Onboarding a new IP/domain and warming up traffic, mitigating an issue to know volumes, and overall hygiene and list/segment modification updates are all examples.
Additionally, this will improve your visibility. Using traditional domain -> provider mapping (i.e., yahoo.com = Yahoo), ESPs in the B2C space often see 20-30% of a customer’s total sending volume classified as “other.” With the additional visibility generated by the Seedlist Optimizer, the “other” classification typically drops down to less than 5% (e.g., rocketmail.com is no longer classified as “other,” it’s now classified as Yahoo).
There you have it. In this blog, we covered:
- The gold standard for inbox testing is optimized seedlist-based inbox monitoring and actual user-based engagement data.
- We shot down Sender Mythology promoting email panel data and non-optimized seeds as a viable alternative to our recommended gold standard—they are not.
- Not all seedlists are created equal, and look out for duplicate seeds.
- We gave you a tour of Seedlist Optimizer (patent pending) and demonstrated its unique value to your mailtech portfolio.
What are you waiting for? Request a demo today and start seedlist testing the right way.
You may also like...
It’s hard to believe we are nearing the end of yet another exciting year in email, and 2019 proved to be one of the most momentous and active years to date. Over the past year, the number of new technologies, mergers and acquisitions, mailbox provider (MBP) announcements, news, and highlights is evidence of the versatility […]
I recently gave a presentation on global privacy regulations to a post-graduate marketing class and one of the things I noticed while preparing was that even within a single country, privacy is complicated. On a global scale, it is really complicated. For example, Canada has one federal private sector privacy law, three similar provincial laws, […]
*Update: This article was featured on email influencer Jordie van Rijn’s emailmonday blog! To see it in action, plus a great round-up of other articles and thought leadership looking forward to the future of email, click here.* The Black Friday emails are deleted, marketers’ email lists are checked twice, we pretty much know which senders […]