FAQ

Answers to the questions teams ask most often. Can't find yours? The concept pages go deeper, and llmjury.com has the product overview.

Getting started

Do I need a credit card to try LLMJury?

No. Every plan starts free — sign up, grab your org's publishable key, and run the quickstart. Upgrade only when you outgrow the free allowance.

Which languages have an SDK?

Python (3.8+), TypeScript/JavaScript (Node 18+ and the browser), and Java (11+). All three implement the same frozen bucketing hash, so the same user resolves to the same variant in every language. Not on this list? Use the server-side POST /v1/assign endpoint.

Do I have to run experiments through your dashboard?

You create and configure experiments in the dashboard, but assignment and tracking happen in your app via the SDK. Reading results is available in the dashboard and via GET /v1/experiments/{id}/results.

How it works

Will assign slow down my app?

No. assign is pure local compute against a config the SDK polls in the background — there is no network call on the hot path, and it never blocks or throws. track only appends to an in-memory buffer that flushes asynchronously.

Why did assign return null?

The experiment config isn't cached yet (the first fetch is still in flight). Treat null as "use my control behaviour." To avoid it at startup, pass the experiment ids to prefetch — experiments: [...] in the SDK constructor — and await readiness before the first call. See any SDK quickstart.

When do results appear?

After the daily pipeline runs with enough judged events (roughly 200 evaluated events per arm). Until then the dashboard shows a first-class "collecting" state rather than a broken chart.

Why are my results marked "preliminary"?

Someone ran a manual analysis over partial data (the "Preliminary look" on the results tab), so the numbers may still move. Every manual run is stored permanently and appears in the day picker labeled manual with the exact time it ran; the default view switches to the nightly analysis once it lands. Preliminary numbers are safe to watch but not yet to decide on.

How is significance decided?

On the FDR-corrected p-value only (Benjamini–Hochberg across the experiment's metrics), at p < 0.05 — never the raw p-value. This controls false positives when you track many metrics. See FDR correction.

My results were withheld for "sample ratio mismatch" — what happened?

SRM is a hard gate: when the observed traffic split diverges from your configured allocation (p < 0.001), analysis halts because the comparison would be untrustworthy. It usually points to an assignment-integration, bot-filtering, or redirect bug. See SRM.

Data & keys

Where do I get my API key, and where do I put it?

Keys are issued per organization from the dashboard under Settings → API keys; the raw key is shown once when issued or rotated, so copy it immediately. Put it in the LLMJURY_API_KEY environment variable (shell, CI secret, deployment environment, or .env file) — every SDK reads it automatically, so Client() / new Client() / LlmjuryClient.builder() need no key in code. If your platform injects secrets another way, pass it explicitly via the constructor (api_key= / { apiKey } / builder(apiKey)). See the getting started walkthrough.

What's the difference between the publishable and secret key?

The publishable key (llmj_pk_…) is safe in client-side code: it's write-only and rate-limited (assign, track, and read config). The secret key (llmj_sk_…) is server-side only and unlocks privileged operations like bulk import. Never ship the secret key to a browser.

Can I bring my own historical data?

Yes — bulk-import past events as newline-delimited JSON with a secret key. See Bring your own data.

What happens to events if the SDK can't reach the API?

track retries with backoff and can spill undeliverable events to an offline store for bounded replay on the next start, so a transient outage doesn't lose data. assign keeps returning the last-known variant (or null), never an error.

How long is my data kept?

Raw events are retained per your plan (Free 7 days → Business 90 days); computed results are kept so your experiment history stays intact. Retention bounds storage only — it never shortens an experiment's analysis window. See analysis windows & retention.

Billing & plans

How am I billed, and what counts as an event?

By monthly event allowance. Every track (exposure, model_call, business_event) counts; assign is free. See Plans & pricing for allowances and overage.

Why isn't LLM-as-judge on the Free plan?

Every judge evaluation is a real model-inference call with real cost. Free (50k events/mo) includes the metrics the SDK captures for free — latency, token cost, and business events — and judge-scored quality metrics unlock at Pro with a monthly evaluation allowance (50k on Pro, 500k on Business). See Plans & pricing.

How do team seats work?

Seats are per organization: Free 1, Pro 10, Business 50. Use Invite team in the dashboard sidebar to add teammates by email; everyone shares the org's experiments and results. At the seat cap, the invite flow shows the upgrade path.

Can I change or cancel my plan?

Yes, from the dashboard. Downgrades apply at the next cycle and never delete experiments — only the plan limits change.