Custom metrics

A custom metric turns "did the answer get better, by our definition?" into a judged, testable number. It has two parts:

1. A rubric — natural language for the judge. Be concrete about the scale and the failure modes, e.g. "Score 0–1 how directly the answer resolves the user's question. Penalize hedging and unrequested caveats."
2. An output schema — the JSON structure the judge must fill, e.g. {"type": "object", "properties": {"score": {"type": "number"}}}. Structured output keeps judging machine-readable and cache-able.

Category and method

Pick the metric's statistical category (continuous, binary, ordinal, percentile, count) — it routes to the recommended test automatically, and advanced users can override per metric with an inline divergence warning (see method selection).

Versioning

Metrics are versioned: editing the rubric or schema mints a new metric version, and each experiment run is locked to the version it started with — results always trace to the exact rubric that produced them. Custom metrics get the same sampling, caching, and hard judge budget as built-ins (see LLM-as-judge).

Build one in the dashboard under Metric builder, or attach it in the create-experiment wizard.