UI A/B tests: a full example report

A walk through a real UI A/B test report from top to bottom: what you upload, the conclusion, the key numbers, what to do next, the detailed findings, who responds and the segments.

The report screenshots come from one example report: a checkout page test for a fictional store with 100 virtual consumers in the US. The treatment shows the total including shipping upfront, with free returns and express pay. UI A/B tests take up to 100 virtual consumers without a plan, 500 on Base and 2,000 on Pro.

AI simulation for pre-testing, not a sales forecast.

What you upload

Two screenshots of the same page: the control (what runs today) and the treatment (your change). You define the positive action, such as joining, and the negative one, and each virtual consumer decides on both versions.

The conclusion

The report opens with a conclusion: a recommendation for the next step based on the simulated change in buy rate, up to three short reasons and one action. A positive result means: take this change into a live A/B test. A negative one means: redesign before you spend live traffic on it.

Example report: Checkout page test, 100 virtual consumers. “Switch to the treatment”, with the reasons and one action.

Key numbers and what was tested

The key numbers as tiles: the buy rate for the control and the treatment, the difference with its interval, and how many shoppers bought more often, the same or less often. Then the two versions you tested, side by side.

Example report: Checkout page test. Buy rate 39.4% to 98.6%: 95 of 100 shoppers bought more often, 5 showed no change, none bought less often.
Example report: The control and treatment checkout pages for a fictional store. The treatment shows the total including shipping upfront, free returns and express pay.

Recommendations

Recommendations in three groups: what to do, what to improve next (with the behavioural-economics principle behind each idea) and what to check after the change. They build on this test’s own results.

Example report: Checkout page test. Switch to the upfront total and keep the package together; then check support questions, returns and drop-off at the payment step.

Detailed findings

The detailed findings follow, with the key findings, creative and UI ideas and the risks to consider. When a report asks shoppers whether they noticed the change, the answer is shown too. Here almost every shopper gave the same answer on each version, so noticing cannot show who responds, and the report leaves the mechanism analysis out and says why.

Example report: Checkout page test. “Did you notice that the total including shipping is shown before payment?” Yes in 0% of control sessions and 100% of treatment sessions.

Segmentation analysis

The buy-rate difference by age, gender, income, education, industry, home ownership, pets, area type, children and relationship status, each with a short insight.

Example report: Checkout page test, four of ten segment cards. Lower-income shoppers +75.4 points against +20.0 for higher-income; female shoppers +71.9 points.

Who benefits: feature importance

Which shopper traits predict a stronger response to the treatment, with the model’s cross-validated R², so you can see how much of the difference they explain.

Example report: Checkout page test. Income (0.184) and gender (0.135) lead, and age adds almost nothing. R² = 0.291, so much of the variation stays unexplained by shopper traits alone.

AI mockups from a recommendation

Turn a recommendation into a mockup of the next version. Each click creates 2 variations. Mockups use your own Gemini API key.

Who the virtual consumers are

Each virtual consumer is a persona with its own profile: where they live, age, household, home, job and income. Choose the region: the US, the EU or Asia.

Each persona comes with cards for demographics, psychology, and lifestyle and behaviour. Rotate through five example personas on the virtual consumers page.

Limits

  • Up to 100 virtual consumers per test without a plan, 500 on Base, 2,000 on Pro
  • Audiences in the US, the EU and Asia
  • 2 free tests in total, shared by both test types
  • Results usually in 10–30 minutes. We email you when the test is ready.
  • Reports in English
  • Reliable results need larger persona samples. Larger tests are paid from your balance: up to 100 personas without a plan, 500 on Base and 2,000 on Pro.

Find your stronger page before launch

2 free tests, no card needed. Add a card for a $25 credit.