LLM-as-judge

Conversion events can't tell you whether an answer was good. LLMJury scores a sample of your model's outputs with a judge model against a rubric, turning quality into a metric you can run statistics on.

How it works

1. Sampling — a deterministic sampler picks a fraction of model_call events per experiment (quality signal doesn't need every event; budgets do).
2. Judging — the judge model scores each sampled output against the metric's rubric into a structured output schema (a number, a 1–5 ordinal, a boolean…).
3. Caching — identical (metric version, output) pairs are served from cache, not re-judged.
4. Budgeting — a hard per-experiment judge budget with alerts at 50/90/100%; safety metrics are exempt from budget cuts.

Built-ins and custom metrics

Quality, safety, and relevance ship as built-in judge metrics. You can define your own as a natural-language rubric + output schema — see the custom metrics guide. Rubrics are versioned: editing one mints a new metric version, locked per experiment run, so results are always attributable to the exact rubric that produced them.