Skip to content

Book a call

How to evaluate a cold email test: set the pass criterion before launch.

To evaluate a cold email test, write the pass criterion down before launch: how many interested leads per 100 contacts the company will treat as a result worth further work and how many contacts the result is assessed on. A criterion set after the fact is easy to fit to the result, and then the test settles nothing.

Published
Updated
Reading time
6 min
Category
Results
Author
DYNTRO team

Why write the criterion down before launch?

After a few weeks of sending, the team has figures in front of it: contacts, replies, interested leads. If nobody agreed beforehand what counts as success, the discussion starts from scratch. Some see a promising start, others see wasted time. Usually the measure that happens to look best wins, and that need not be the measure that says anything about sales.

A criterion written down before launch turns the test into a decision. It is known in advance what will happen if the result is above the threshold and what if it is below: the campaign continues, the segment or the argument changes, or the test ends. Nobody has to defend the result or explain it.

The second reason is less obvious. When writing down the criterion, the company has to answer the question of how many conversations it needs. That question concerns its own sales, not the campaign, and it is worth asking before the first email goes out.

What should cold email test criteria include?

The criterion is a few sentences that both sides understand in the same way. Ideally, it covers:

  • The measure. Interested leads per 100 contacts, not opens or the number of replies alone. We explain why opens are misleading in the article on cold email metrics.
  • Definitions. Who is a contact and whom we count as an interested lead. Without shared definitions, the same result can be counted in two ways. Our definitions are in the How we measure results section.
  • The level of interest. The threshold is best based on interested leads who are ready to talk, with “needs review” shown alongside, separately.
  • The threshold. A figure the company derives from its own sales: how many conversations with interested leads its salespeople turn into proposals, and how many proposals into customers. Averages found online will not help here, because they come from other markets and other offers.
  • The number of contacts on which the result is assessed, and when the assessment takes place.
  • A decision for each outcome: what we do if the threshold is met and what we do if it is not.
Example of a criterion written down before a test

An example training company sells customer service training to retail chains. This is example data. The company calculates the threshold from its own sales, which is why we give no figure in the example.

FieldEntry
Segmentclothing retail chains in Poland, people responsible for in-store sales
Channelemail, one sequence
Contacts for assessmentthe whole segment after exclusions, 800 people with the first message delivered
Main measureinterested leads ready to talk per 100 contacts
Thresholdthe number of interested leads ready to talk per 100 contacts that the management board calculated from the company's own sales: how many conversations with interested leads turn into proposals and how many proposals into customers
Shown alongsideinterested leads needing review, replies, bounces, unsubscribes
If the threshold is meta campaign to a second segment, footwear retail chains
If the threshold is not metreview of replies, then a new argument or the end of the test

How many contacts does a campaign test need for the result to mean something?

Enough that a single reply does not decide the result. With 100 contacts, one interested lead more or less changes the result by 1 per 100. With 500 contacts, the same single person changes it by 0.2 per 100. The fewer the contacts, the easier it is for one company that happens to be looking for a supplier to turn an average test into a good one, and for one week of holidays in the industry to turn a sound campaign into a weak one.

That is why the number of contacts for assessment is set before launch, together with the threshold. The assessment only takes place once the whole sequence has gone out to all contacts and replies have had time to arrive. Some of them come after the later messages, not after the first.

When the market is small, the test covers the whole segment after exclusions. If that is still too few, it is better to broaden the profile, e.g. to a second role or an adjacent industry, than to assess a result from a few dozen contacts. We explain how to choose a segment that is large enough and consistent in the article on contact list segmentation.

How do you assess several segments and two channels?

Each segment and each channel is counted separately: contacts, replies, interested leads. Then you know what worked, even when the campaign covers several segments or both email and LinkedIn from the start.

A segment means one reason for contact and one group of companies in a similar situation. That is why each segment needs its own number of contacts for assessment. A combined result mixes groups that may have reacted quite differently.

Adjustments during the test are normal, e.g. a new subject line after the first report. It is best to change one thing at a time and record the change with the date, so that after the test you know which results came before it and which after it.

What should you compare after a campaign test?

The answer “threshold met” or “not met” on its own is only the start. The most useful comparisons are:

  • Groups within the segment. If the segment covered two roles or two closely related industries, it is worth calculating interested leads per 100 contacts separately for each of them. With small groups, do so with caution, for the reasons described above.
  • Subject lines and arguments. After which message the replies came, which argument the interested leads refer to and which objections keep coming up. A recurring “someone else handles that here” says more about the segment than about the content.
  • Speed of response. How much time passed between an interested company's reply and the salesperson's response.
  • Bounces and unsubscribes. Many bounces say something about the quality of the contact list, and many unsubscribes say that the message was reaching people for whom the topic was not relevant.

What does a well-planned test give you?

A clear decision: whether to keep working in this segment and what to change in it. The test shows whether this message, in this segment, at this time produced conversations at the level written into the criterion.

  • A good result is worth confirming in the next segment, in another market or with a larger number of contacts.
  • A weaker result shows what to change in the segment or in the argument. It concerns this segment and this argument, not cold email in general.
  • A conversation with an interested company opens the sale. Buying cycles in companies can be longer than the test, so revenue from the campaign often shows only after the day of assessment.

Summary.

  • The pass criterion is written down before launch: measure, definitions, threshold, number of contacts and a decision for each outcome.
  • The main measure is interested leads per 100 contacts, and the threshold comes from the company's own sales.
  • The fewer the contacts, the bigger the role of chance. In a small market, the test covers the whole segment or a broadened profile.
  • Each segment and each channel is counted separately, so you know what worked.
  • A test gives a clear decision: where to keep working and what to change.

We run email and LinkedIn campaigns to decision-makers, from data and domains to replies, while your salespeople focus only on conversations. See the service: B2B email campaigns.

Book a call

Questions and answers.

When should you assess the result of a cold email test?

Only once the whole sequence has gone out to all contacts and replies have had time to arrive. Some replies come after the later messages, not after the first. The moment of assessment and the number of contacts for it are set before launch, together with the threshold.

Can the open rate be the criterion for a campaign test?

No. The main measure of a test is interested leads per 100 contacts, not opens or the number of replies alone. The company derives the threshold from its own sales: how many conversations with interested leads its salespeople turn into proposals, and how many proposals into customers.

Related articles.

All articles

Let's start talking to the right companies.

Leave your details.

Your answer helps us see which channels bring enquiries.

DYNTRO sp. z o.o. processes the data from this form to reply to your enquiry. More in our privacy policy.

What happens next.

  1. On the first call, we define your customer profile, market and exclusions.
  2. We choose the scope: email, LinkedIn or both channels, in one or several segments.
  3. If working together makes sense, you receive a proposal with terms for your company.

Prefer email? kontakt@dyntro.pl