A/B testing with low traffic: four ways to get a real answer
A 2% demo page at 4,000 visits a week needs 39 weeks to catch a 10% lift. Four moves bring a low-traffic A/B test down to a few weeks you can wait.

A/B testing with low traffic works when you shrink the sample the test needs. Test a change big enough to show, measure a goal that converts more often, pool pages that share a template, and send whatever still won't fit to five people in a usability session. Each move cuts the visitors a test needs from tens of thousands to a number a B2B site reaches in weeks.
Here's the page these moves are built for. Your demo page gets 4,000 visitors a week, and 2% of them ask for a demo. You want to know whether a new version does better. Below is what that test costs, and how to bring the bill down.
Start with the math, then change the inputs
Sample size has four inputs: your baseline conversion rate, the smallest lift worth catching, confidence and power. Hold the last two at 95% and 80% and the estimate fits on one line: 16 x p x (1 - p) divided by the absolute lift squared, per variant. We walked through where that rule comes from in how much traffic you need before an A/B test means anything.
Run it on the demo page. A 10% relative lift on 2% is 0.2 points, so the test needs about 78,400 visitors per variant. Split 4,000 a week in two and you get 2,000 per arm: 39 weeks. Nobody waits nine months for a headline test. So change the inputs, and let our A/B test sample size calculator price each version of the plan below.
One input hides inside the formula: how noisy your metric is. If your testing tool offers CUPED, which adjusts each visitor's result using what they did before the test, turn it on. Microsoft's team reported cutting variance by about 50% at Bing, the same power with half the users. It needs pre-test data on the same people, so it pays on pages your users come back to, like pricing or a logged-in app, and does little for a page of first-time visitors.
Test a change big enough to see
The lift you aim for is the input you control most, and it moves the number most. Halve the effect and the sample roughly quadruples. Run that the other way: aim for a 30% lift on the same 2% page and the test needs about 8,700 visitors per variant. At 2,000 per arm a week, that's five weeks.
A 30% lift sounds greedy until you look at what produces one. Button colours and word swaps rarely get there. Changes to what the page asks of the visitor can: a form cut from nine fields to four, a price shown where "contact us" used to be, a demo booked straight into a calendar, a different offer altogether. On a quiet page, the tests worth running change a decision the visitor makes.
The cost is diagnosis. When three things change at once and the variant wins, you know the package worked and you don't know which part did. That's fine. The winner becomes the new control, and you can take it apart later, when the traffic allows.
Move the goal up the funnel
The baseline is the other input you can move. A demo request sits at the end of a long page, so it converts rarely. The step before it converts more often: say 12% of visitors click through to the demo form. Test against that click and the same 10% lift needs about 11,700 visitors per variant, six weeks at your traffic. A 20% lift on the click needs about 2,900, so the test ends at the two-week floor.
Keep the real goal on the dashboard as a guardrail. A variant can win more clicks by promising something the form doesn't deliver, and then demo requests fall. Call the test on the click, and check that requests held before you ship it.
Pool pages that share a template
If one page is too quiet, test the template. Five feature pages at 800 visitors a week each become one test at 4,000 a week, as long as they share a layout and a call to action and the variant changes the same element on all five. The result tells you how the template performs, which is usually the decision you were about to make anyway: few teams redesign one feature page on its own.
Two rules keep a pooled test honest. Randomise by visitor, so someone who reads three feature pages sees the same version on all three. And read each page's result once the test ends. If one page moves against the rest, its traffic is different, and it deserves a look of its own.
Send the rest to five people
Some pages won't fit any of this. A pricing page with 300 visitors a week can't settle a test this year. There, a usability session is the better instrument. Nielsen and Landauer's model, the source of the five-user rule, says five people find about 85% of the problems in a task, and Jeff Sauro at MeasuringU spells out the condition: that holds for problems that affect 31% or more of your users. Those are the problems a low-traffic page needs fixed first.
Give five people from your audience a real task, watch where they stall, fix it, then run five more. What comes back is a list of the places visitors get stuck and why, the problems a test on that page would have taken a year to hint at.
If you want help sorting which of your pages can carry a real test and what to try on the rest, that's what we do.