Getting started

Ship your first statistically defensible LLM experiment in about five minutes. You need an LLMJury account (every plan starts free, no credit card).

1. Get your API key

In the dashboard, open Settings → API keys and issue a key for your org. You can view or copy it again any time from the same page (admins only — the key sits behind a masked field).

Use the publishable key (llmj_pk_…) in your app. It's safe in client-side code: it can only assign, track, and read config. Secret keys (llmj_sk_…) are server-side only and unlock privileged operations like bulk import — never ship one to a browser.

2. Install an SDK

Pick your language — each has a dedicated quickstart with the full API surface: Python · TypeScript · Java.

All three SDKs are open source under Apache-2.0: llmjury-python · llmjury-typescript · llmjury-java.

pip install llmjury-sdk      # Python 3.8+ (imports as `llmjury`)
npm install llmjury-sdk      # Node 18+ / browser
# Java 11+: com.llmjury:llmjury-sdk (Gradle/Maven)

3. Give the SDK your key

Every SDK looks for the key in the LLMJURY_API_KEY environment variable — set it once in your shell, CI secret store, deployment environment, or .env file, and the client picks it up automatically, so the key never lives in your code:

export LLMJURY_API_KEY=llmj_pk_...

Second option — pass the key explicitly. If your platform injects secrets another way, or you're in a browser where environment variables don't exist, hand the key to the constructor: Client(api_key="llmj_pk_...") in Python, new Client({ apiKey: 'llmj_pk_...' }) in TypeScript, LlmjuryClient.builder("llmj_pk_...") in Java.

Either way, the SDK defaults to the production API — no localhost, no base URL to configure. If no key is found, the client fails fast at construction with a message telling you exactly that.

4. Create an experiment

In the dashboard, create an experiment with two variants (control, treatment) and pick your metrics — built-ins like quality and latency, or a custom judge metric. Give the experiment a unique name (like checkout-prompt): that name is how your code addresses it — no UUIDs. You can also configure each variant's prompt and custom variables (model, temperature, …) right in the dashboard, so iterating never needs a deploy. The recommended statistical test per metric is pre-selected; see method selection.

5. Integrate

The recommended integration is two calls on the request path: get_prompt resolves the user's variant prompt from client memory (your in-code default keeps the app working through a full LLMJury outage), and the setup-once wrap() intercepts your provider client so every model call records latency, tokens, and errors with no call-site code. The only event you track explicitly is your business outcome:

from llmjury import Client

# ---- setup, once at startup -------------------------------------------------
# Client() needs your publishable API key (dashboard -> Settings -> API keys).
# Easiest: `export LLMJURY_API_KEY=llmj_pk_...` — it's read automatically.
# Or pass it in code: Client(api_key="llmj_pk_...", ...)
client = Client(experiments=["checkout-prompt"])          # prefetch by experiment NAME
llm = client.wrap(anthropic_client, "checkout-prompt")    # every model call is now traced

# ---- per request ------------------------------------------------------------
with client.as_user(user_id):
    # Variant prompt from client memory; your in-code default survives an outage.
    p = client.get_prompt("checkout-prompt", user_id, default="You are a helpful assistant.")
    # Call your provider client DIRECTLY — latency/tokens/errors are intercepted.
    response = llm.messages.create(model="claude-haiku-4-5", max_tokens=1024,
                                   system=p.prompt,
                                   messages=[{"role": "user", "content": user_input}])

# The ONLY explicit metric: your business outcome.
client.track("business_event", {"experiment_id": "checkout-prompt", "user_id": user_id,
                                "variant": p.variant, "business_metric": "conversion", "value": 1})

Every call is deterministic and sticky — the same user gets the same variant in every language — and nothing here ever blocks or throws into your app. The same surface exists in TypeScript and Java, and the lower-level assign + track calls remain available if you want full manual control.

6. Read the result

Results appear after the daily pipeline runs with enough judged events. Every result shows the effect with a confidence interval, raw and FDR-corrected p-values, the test used, and the SRM check status. Significance is judged on the corrected p-value.