A/B Test With Short Links: One Link per Variant

To A/B test links, give each variant its own short link, send each one to a fair share of the same audience, and compare clicks against the size of each share. The link tells you which version people clicked more; a two-proportion test tells you whether the gap is bigger than chance.

SendURL has no split or rotating redirect: one short link always goes to one destination. That doesn't get in the way, because most tests worth running with links compare two messages (a subject line, a button, a caption, a poster), and those need separate links anyway. This guide is part of the link tracking handbook.

What you can test with links

Clicks happen before the page loads, so they measure the message that carried the link. The page behind it needs its own test.

The first three are clean tests: a random split of one list gives both sides the same audience. The poster and social tests are weaker, because place and time differ between the variants. Run them anyway, but repeat them before you trust them.

For landing pages, clicks will be roughly equal by design (the message is the same). Tag each destination with utm_content=a or utm_content=b and compare conversions in GA4; see the UTM parameters guide.

Set up the variants

Two links, identical except for the slug and, if you use GA4, utm_content:

Slugs on snu.to are shared by all SendURL users, so a dated prefix keeps them yours. Creating both through the API:

for v in a b; do
  curl -s -X POST https://api.sendurl.to/api/v1/links \
    -H "Authorization: Bearer $SENDURL_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"destinationUrl\": \"https://example.com/autumn-sale?utm_source=newsletter&utm_medium=email&utm_campaign=2026-10-autumn-sale&utm_content=$v\", \"slug\": \"2026-10-autumn-$v\", \"title\": \"Autumn test: variant $v\"}"
  echo
done

Change the slugs before you run it: once a slug exists, a second create with it is a 409 slug_taken. Note each response's id; you'll need it to read the clicks.

How many clicks you need

Decide before sending how small a difference you care about, then work out the audience it takes to see it. The A/B test calculator does this without code: under Plan a test, enter your usual click rate and the lift you want to detect. The standard formula for two proportions, per variant:

n = (z_alpha/2 + z_beta)^2 × (p1 × (1 − p1) + p2 × (1 − p2)) / (p1 − p2)^2

p1 is your usual click rate, p2 the rate you hope to reach, and the z values come from your confidence level (1.96 for 95%) and power (0.84 for 80%). In Python:

import math
from statistics import NormalDist

def per_variant(p1, p2, alpha=0.05, power=0.8):
    """Recipients needed per variant to detect p1 -> p2 (two-sided test)."""
    z = NormalDist().inv_cdf
    spread = p1 * (1 - p1) + p2 * (1 - p2)
    return math.ceil((z(1 - alpha / 2) + z(power)) ** 2 * spread / (p1 - p2) ** 2)

print(per_variant(0.02, 0.025))  # 13807
print(per_variant(0.02, 0.03))   # 3823

With a 2% click rate, spotting a half-point lift takes about 13,800 recipients per side; a full point takes about 3,800. If your list has 5,000 people, test bold changes, not commas. (The 2% is an example rate for the arithmetic, not a benchmark; use your own past campaigns.)

Is the difference real?

Once the test has run its course, compare the two click rates with a two-proportion z-test. To check one result by hand, enter the four numbers into the A/B test significance calculator; the code below does the same for a script or a sheet. The example numbers are made up for illustration: 4,000 emails per variant, 92 clicks on A and 121 on B.

from math import sqrt
from statistics import NormalDist

def two_proportion_test(clicks_a, sent_a, clicks_b, sent_b):
    p_a, p_b = clicks_a / sent_a, clicks_b / sent_b
    pooled = (clicks_a + clicks_b) / (sent_a + sent_b)
    se = sqrt(pooled * (1 - pooled) * (1 / sent_a + 1 / sent_b))
    z = (p_b - p_a) / se
    p_value = 2 * (1 - NormalDist().cdf(abs(z)))
    return p_a, p_b, z, p_value

p_a, p_b, z, p = two_proportion_test(92, 4000, 121, 4000)
print(f"A {p_a:.2%}  B {p_b:.2%}  z={z:.2f}  p={p:.3f}")
# A 2.30%  B 3.02%  z=2.01  p=0.044

A p-value under 0.05 means a gap this large would turn up by chance less than 1 time in 20 if the variants were really the same. Here it is just under, so B probably wins, and it is worth repeating before you rewrite every email. The same test in JavaScript (Node or a browser console):

// Normal CDF via Abramowitz and Stegun 7.1.26 (error below 1.5e-7).
function normalCdf(x) {
  const t = 1 / (1 + (0.3275911 * Math.abs(x)) / Math.SQRT2);
  const poly = ((((1.061405429 * t - 1.453152027) * t + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t;
  const erf = 1 - poly * Math.exp(-(x * x) / 2);
  return x >= 0 ? (1 + erf) / 2 : (1 - erf) / 2;
}

function twoProportionTest(clicksA, sentA, clicksB, sentB) {
  const pA = clicksA / sentA;
  const pB = clicksB / sentB;
  const pooled = (clicksA + clicksB) / (sentA + sentB);
  const se = Math.sqrt(pooled * (1 - pooled) * (1 / sentA + 1 / sentB));
  const z = (pB - pA) / se;
  return { pA, pB, z, pValue: 2 * (1 - normalCdf(Math.abs(z))) };
}

console.log(twoProportionTest(92, 4000, 121, 4000));
// { pA: 0.023, pB: 0.03025, z: 2.01…, pValue: 0.044… }

Count people, not clicks

The test assumes each recipient either clicked or didn't. SendURL counts every click, including repeats and link previews and email scanners, and it has no unique-click count. Before testing, clean the numbers: drop automated clicks, then count distinct IP address and user agent pairs as an estimate of people. This script does both for each variant (install requests first with pip install requests):

import os, re, time

import requests

API = "https://api.sendurl.to/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['SENDURL_API_KEY']}"}
BOT_UA = re.compile(r"Slackbot|facebookexternalhit|Twitterbot|bot/|crawler|spider|curl/|python-requests", re.I)

def people(link_id):
    """Distinct (IP, user agent) pairs among clicks that don't look automated."""
    seen, params = set(), {"limit": 10}
    while True:
        r = requests.get(f"{API}/links/{link_id}/clicks", headers=HEADERS, params=params, timeout=10)
        r.raise_for_status()
        page = r.json()
        for c in page["clicks"]:
            ua = c["userAgent"] or ""
            if ua and c["browser"] and not BOT_UA.search(ua):
                seen.add((c["ip"], ua))
        if not page["pagination"]["hasMore"]:
            return len(seen)
        params["cursor"] = page["pagination"]["nextCursor"]
        time.sleep(0.25)  # stay under the API's rate limit

for name in ("LINK_A", "LINK_B"):
    print(name, people(os.environ[name]))

Feed those counts into two_proportion_test with the number of emails delivered to each half. If your email tool reports unique human clickers per variant, use its numbers as a cross-check.

Run an A/A test first

Before your first real test, send the same message to both halves through two links. The two rates should come out close, and the test should say "no difference" (a p-value above 0.05) about 19 times in 20. If it flags a winner, look for the cause before trusting any A/B result: an uneven split, one half going out hours later, or bots hitting one link more.

When to stop

Pick the end before you start: a sample size from the formula above, or a fixed period such as seven days after the send. Then don't call the result early because it looks good on day two. Evan Miller's How Not To Run an A/B Test (2010) explains why: "Repeated significance testing always increases the rate of false positives."

To watch the test without deciding on it, read daily counts per variant with GET /links/:id/stats (daily clicks reference):

curl "https://api.sendurl.to/api/v1/links/$LINK_A/stats" \
  -H "Authorization: Bearer $SENDURL_API_KEY"

The response is a list of { "date": "2026-10-14", "count": 9 } rows in UTC, days without clicks left out.

Results template

Test:            2026-10 autumn sale, button text
Hypothesis:      "See what's 30% off" gets more clicks than "Shop the sale"
Audience:        newsletter list, random 50/50 split in the email tool
Links:           snu.to/2026-10-autumn-a (A), snu.to/2026-10-autumn-b (B)
Planned size:    3,823 per variant (2.0% -> 3.0%, 95% confidence, 80% power)
Sent:            2026-10-14 09:00 UTC, both halves together
Ends:            2026-10-21 09:00 UTC
Delivered:       A 4,000 | B 4,000
People clicked:  A __ | B __   (automated removed, IP + user agent deduplicated)
Result:          z = __, p = __
Decision:        __
Repeat on:       __

Checklist

• One question per test, written down before sending
• One short link per variant, same destination unless you're testing the page
• A random split of one audience, sent at the same time
• Sample size or end date fixed in advance
• An A/A test run once on this channel
• Automated clicks removed and repeats deduplicated before testing
• Two-proportion test on people and audience size, not raw clicks alone
• Results logged, including the tests that found nothing

For turning clicks into a rate in the first place, see how to measure click-through rate. To compare posting times rather than messages, use your best time to post from click data.