Skip to main content
sendurl, home
Start free

A/B test calculator for clicks and conversions

Enter sends and clicks for two versions and the A/B test calculator tells you whether the difference is statistically significant.

Runs in your browser.

Compare two versions

Version A

Click rate of version A2.30%

Version B

Click rate of version B3.02%

Sends: emails delivered, or people who saw it. Clicks: people who clicked (or converted), if you can count them.

These are the example numbers from our A/B testing guide.

Result

Significant at 95%

B's click rate is 3.02% against A's 2.30%, 31.5% higher. Significant at 95% (p = 0.044): a gap this big would show up by chance about 4 times in 100 if A and B were the same.

A's click rate
2.30%
B's click rate
3.02%
Lift (B vs A)
+31.5%
z-score
2.01
p-value (two-sided)
0.044
1 − p
95.6%
Copies the numbers as plain text.

Fix the sample size before you start and don't stop early. Count people, not clicks, where you can: previews and bots inflate clicks. More in the A/B testing guide.

Settings95% confidence, two-sided
Confidence level

95% is the usual choice. A higher level needs more data to call a winner.

Test

Two-sided asks whether the versions differ either way. One-sided only asks whether B beats A; choose it before the test, not after.

Counting clicks per version? Give each one its own SendURL short link. Create a free account to keep both and see their clicks.

Updated

How the calculator decides

It runs a two-proportion z-test, the same test as the A/B testing guide. Each version's click rate is its clicks divided by its sends. The test pools both versions into one rate, works out how far apart the two rates would usually land by chance at these sizes, and measures the real gap in those units (z).

pooled = (clicks A + clicks B) / (sends A + sends B)
se     = √(pooled × (1 − pooled) × (1/sends A + 1/sends B))
z      = (rate B − rate A) / se

The p-value is the chance of a z at least that far from 0 if the two versions were really the same. A p-value under 0.05 means a gap this large would turn up by chance less than 1 time in 20, which is what "significant at 95%" means. The confidence figure is 1 − p, a common shorthand; it is not the chance that B is better.

The guide's example: 4,000 emails per version, 92 clicks on A and 121 on B. That is 2.30% against 3.02%, z = 2.01 and p = 0.044. Just under 0.05, so B probably wins, and it is worth repeating before you rewrite every email.

How many sends you need

Work this out before you send, with Plan a test. It uses the standard formula for two proportions, per version, with 80% power:

n = (z_alpha + z_beta)² × (p1 × (1 − p1) + p2 × (1 − p2)) / (p1 − p2)²

p1 is your usual click rate and p2 the rate you hope to reach. z_alpha is 1.96 for 95% confidence two-sided (1.64 one-sided) and z_beta is 0.84 for 80% power. With a 2% click rate, spotting a lift to 2.5% takes about 13,807 recipients per version; a lift to 3% takes about 3,823. If your list has 5,000 people, test bold changes, not commas.

When a result isn't significant, the calculator uses the same formula on the rates you got to say how many sends per version it would take to detect that difference. Treat it as a size for the next test, not a reason to keep this one running until it crosses the line.

Before you trust the result

  • Decide when the test ends before it starts. A sample size, or a fixed period such as seven days after the send. Evan Miller's How Not To Run an A/B Test (2010) explains why: "Repeated significance testing always increases the rate of false positives."
  • Count people, not clicks. The test assumes each recipient either clicked or didn't. SendURL counts every click, repeats included, and also counts link previews and email scanners; it has no unique-click count and no bot filtering. Remove automated clicks and repeats first, or use your email tool's unique clickers.
  • Split one audience at random, at the same time. Two posts at different times or two posters in different places differ in more than the message.
  • Run an A/A test once. Send the same message through two links. The calculator should say "not significant" about 19 times in 20; if it finds a winner, look for an uneven split before trusting any A/B result.

SendURL has no split or rotating redirect: one short link always goes to one destination. For a test, make one short link per version and send each to its half of the audience.

Frequently asked questions

How do you calculate A/B test significance?

With a two-proportion z-test. Work out each version's click rate, then the pooled rate of both together, then the standard error: the square root of pooled × (1 − pooled) × (1/sends A + 1/sends B). z is the difference in rates divided by that, and the p-value is the chance of a z at least that far from 0 if the two versions were really the same. 92 clicks from 4,000 against 121 from 4,000 gives z = 2.01 and p = 0.044.

What does 95% confidence mean in an A/B test?

That the p-value is under 0.05: a gap as large as yours would turn up by chance less than 1 time in 20 if the two versions were really the same. The calculator shows 1 − p as the confidence figure, a common shorthand. It isn't the probability that B is better.

How many people do I need for an A/B test?

It depends on your usual click rate and the smallest lift you care about. At 95% confidence and 80% power, detecting a move from 2.0% to 2.5% takes about 13,807 recipients per version; from 2.0% to 3.0% takes about 3,823. Use Plan a test above before you send.

Should I use a one-sided or a two-sided test?

Two-sided, unless you decided before the test that you only care whether B beats A. A two-sided test asks whether the versions differ in either direction. A one-sided test reaches significance sooner, but it can never tell you that B is worse.

Can I stop an A/B test as soon as it is significant?

No. Checking again and again and stopping at the first significant result finds winners that aren't there. Fix the sample size or the end date before you start, and read the result once you reach it.

Does the calculator send my numbers anywhere?

Everything is worked out in your browser and the page's address doesn't change as you type. The site's page-view analytics record a visit to /tools/ab-test-calculator, with any query string removed.

Read next