How to Run an Appcues A/B Test on In-App Copy That Means Something

October 31, 2026

Appcues includes flow-level A/B testing, so you can split traffic between two versions of a tour and compare completion or a downstream action. It works, but only if you decide the metric, the sample and the stopping rule before the test starts.

What should you actually test?

Rarely the thing people test first. Changing a modal headline from “Welcome” to “Let’s get you set up” almost never moves anything measurable. What does move: the number of steps, where the tour anchors, whether a step blocks the UI, and what the final step asks for.

Start with these, in order of likely effect:

  1. Number of steps (2 vs 3 vs 5).
  2. Final call to action: a direct link versus a passive “Got it”.
  3. Whether the tour is a modal or a non-blocking tooltip.

Copy variants are a fine second test once the structure is settled. Testing them first is how a team spends a month learning nothing.

How large does the sample need to be?

Bigger than most SaaS products can gather in a week. To detect a move from 40% to 45% completion, you need roughly 1,500 users per arm at conventional confidence. For a structural change from 40% to 55%, about 200 per arm is enough.

That gap is the whole argument for testing structure first. Small apps can detect structural wins and cannot detect copy wins.

If you cannot reach a few hundred per arm, do not A/B test. Ship the version you believe in, watch a PostHog funnel, and revisit. A test that cannot reach significance is a coin flip you paid for.

How do you set the test up in Appcues?

  1. Build the candidate as a complete flow, not as an edit to the live one.
  2. Open the flow and enable A/B test on the variant.
  3. Assign traffic 50/50. Uneven splits cost statistical power for no benefit.
  4. Attach exactly one success metric.
  5. Set identical audience rules on both arms — different audiences invalidate the comparison.
  6. Launch, note the start date, and do not open the results page for a week.

Step 6 is the one people ignore. Checking daily and stopping when the chart looks good inflates false positives, and it is the most common reason in-app tests produce results that never replicate.

What metric should count as success?

Not tour completion. Completion measures whether people clicked through the tour, which is a property of the tour, not the product.

The useful metric is the downstream action the tour is meant to cause: project created within 7 days, integration connected, invite sent. If the winning tour raises completion but not that action, you built a more entertaining tour and taught nobody anything.

This is also where Appcues alone is thin. It tracks the event you send but does no cohort analysis on what happened next, so pair it with your product analytics tool.

What does a real result look like?

The pattern in most public write-ups: a short tour ending in a direct CTA link beats a long tour ending in “Got it”, and the gap comes from the final step rather than the shortened tour. Segment your numbers before you conclude, because often only the last step carries the effect.

A flat result is a legitimate finding. Report it and leave that surface alone for a quarter.

When is A/B testing the wrong tool?

When sample sizes are tiny, when the flow is obviously broken (fix it, do not test it), and when the decision is already made for legal, brand or accessibility reasons. Testing to look data-driven burns users on variants nobody intended to ship.

Userpilot and Pendo both offer variants too, and the method above transfers unchanged.

Key Takeaways

  • Test structure before copy: step count, anchoring and final CTA move numbers; headline wording usually does not.
  • Detecting a 5-point copy lift needs about 1,500 users per arm — most apps cannot get there.
  • Pick one downstream success metric, not tour completion.
  • Fix the stopping rule before launch; peeking inflates false positives.
  • A flat result is a valid finding. Report it and move on.
Tags: