How to Run a Headline Test Without Wasting Traffic
A headline test answers one question: does this headline convert better than that one, on the traffic you actually have. Coframe makes generating variants cheap, which creates the temptation to test six at once on a page that supports two. The result is a flat chart and a wasted month.
Here is the arithmetic, then the design that respects it.
How much traffic does a headline test need?
Rough numbers for a page converting at 3 to 5%: detecting a 10% relative change needs a few thousand visitors per variant; detecting a 25% change needs several hundred to about a thousand per variant.
| Baseline CVR | Lift to detect | ~Visitors per variant |
|---|---|---|
| 3% | 10% relative | 12,000+ |
| 3% | 25% relative | 2,500 |
| 5% | 25% relative | 1,600 |
| 5% | 50% relative | 600 |
If your page gets 800 visits a week, a two-variant, 25% test runs about six weeks. That is the honest timeline. Longer tests collect seasonality noise; shorter ones collect wishes.
What is the most common design error?
Changing the headline and the hero image in the same variant. Every practitioner does it once. The winner tells you the pair worked, not which element did the work, and you cannot reuse the lesson.
The second error is segment mismatch: the variants get traffic from different channels because a campaign launched mid-test. Tag the test window and check that traffic mix held steady.
How do you set up the run in Coframe
- Freeze the baseline and note the CVR with its date range.
- Generate two variants (three maximum) that differ only in headline text.
- Lock the image, CTA and layout.
- Set a sample size per variant before launch, using the table above.
- Do not open the dashboard daily. Check at 50% and at the end.
- Record the result and the decision in your experiment log.
The log line matters more than the number. “June: headline B beat A by 6%, sample 2,400 per arm, shipped” is what helps next quarter.
A test that ends without a written decision is a data collection hobby.
What happens if the result is flat?
It often is. Flat results on a headline usually mean the page’s promise and the audience’s search intent already agree; the words were never the constraint. Move the effort to the pricing section or the proof above the fold.
Flat results also happen when traffic is too small. Check the sample size before concluding anything.
If you are testing full pages rather than headlines, and the copy itself is the problem rather than the framing, a content-generation tool plus honest editing gets you further than more variants. Generators such as Coframe are strongest when the hypothesis is about wording, not about offer.
Who should not test headlines
Pages under a few hundred visits a week, pages with one dominant source of traffic that changes weekly, and any page mid-launch. Test after the campaign settles.
Quick Summary
- Headline tests need thousands of visitors per variant to detect a 10% lift.
- With under a thousand weekly visits, run a two-variant 25% test and expect weeks.
- Change one element per variant; pairing headline with image voids the lesson.
- Pre-commit sample size and check only at 50% and end.
- Flat results usually mean the offer, not the words, is the constraint.
