A/B Testing explained
A/B testing shows two versions of an experience to different groups and compares a metric defined in advance. Results should be interpreted in the context of the tested change and metric. Comparable groups are assigned the two versions randomly and at the same time. The primary metric, test duration, and potential negative effects should be defined before the test starts.
Why it matters.
It turns preference debates into measurable questions. Qualitative research may still be needed to understand why one version worked better. A higher number is not always a better experience: clicks may rise while completed applications fall. Results need to be read alongside sample size, uncertainty, and other relevant measures.
Let's look at an example.
Think of a sign-up prompt.
Version A might say ‘Start free’ and version B ‘Create account.’ The team compares their effect on completed registrations with an appropriate sample. A label may attract more clicks while sign-up completion falls or people arrive with the wrong expectation. Following the whole flow shows whether a local gain causes a larger problem.
How to apply it.
- 01
Define one change and the metric it should affect.
- 02
Show each version to comparable groups.
- 03
Interpret results within the limits of the sample and context.
“A measurement gains meaning from the right question.