SRM: sample ratio mismatch

An SRM (sample ratio mismatch) is a statistically significant divergence between the traffic split you configured and the split you observed — say 55/45 exposures when you allocated 50/50. It almost always means something upstream is broken: assignment not called on every path, bot traffic hitting one variant, a redirect dropping users, or client-side caching. Results computed on SRM'd traffic are untrustworthy, however good they look.

The hard gate

LLMJury runs a chi-squared test comparing observed per-variant exposure counts against the configured allocation on every analysis run. When the SRM p-value falls below 0.001, the pipeline halts analysis for that experiment-day instead of publishing numbers — the dashboard shows a sample-ratio-mismatch alert with the p-value.

What to check when it trips

1. Every code path that shows a variant calls assign and tracks an exposure.
2. Bot/crawler traffic is excluded or spread evenly.
3. No variant-dependent redirects or errors drop users after assignment.
4. You didn't change the allocation mid-run — that requires a new experiment version (see bucketing & sticky assignment).

Related: the glossary entry on the marketing site.