All JevBench numbers are on the public easy, standard and hard tiers (231 items). The sealed judge tier is not included, and the Jev and Kev numbers are restricted to the same public items.
Without thinking the same checkpoint scores 0.804 on our test split (2,962 items), against 0.840 with it.
Requirements: Python 3.12 and a CUDA GPU.
pip install -r requirements.txt
Download the released weights and serve them:
hf download PostHog/jeeves –local-dir jeeves-weights python -m inference.serve –model jeeves-weights –drafter jeeves-weights/drafter_k4.safetensors –port 8009
Or fuse your own trained checkpoint into a standalone model and serve it with a drafter:
python export.py runs/cispo/final –out runs/fused python -m inference.serve –model runs/fused –drafter runs/drafter_k4/drafter.safetensors –port 8009
Then send a request in Jev’s format:
curl -s localhost:8009/v1/systemone -H ‘content-type: application/json’ -d ‘{ “state”: “Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.”, “questions”: { “department”: {“type”: “choice”, “instructions”: “Which team should handle this?”, “criteria”: {“returns”: “Exchanges, refunds, wrong or damaged items”, “shipping”: “Delivery status, delays, lost packages”, “billing”: “Charges, invoices, payment problems”}}, “escalate”: {“type”: “noul”, “instructions”: “Does this need urgent human attention?”}, “frustration”: {“type”: “score”, “instructions”: “How frustrated is the customer?”, “criteria”: [“Calm”, “Frustrated”, “Very angry”]} }, “options”: {“max_think”: 512}}’
Response on one H100 (FP8), with the three questions thinking in parallel:
{ “model”: “jeeves-latest”, “answers”: { “department”: { “type”: “choice”, “choice”: “billing”, “confidence”: 0.19, “probabilities”: { “returns”: 0.4, “shipping”: 0.14, “billing”: 0.46 } }, “escalate”: { “type”: “noul”, “noul”: 0.72 }, “frustration”: { “type”: “score”, “score”: 1.5, “legend”: { “0”: “Calm”, “1”: “Frustrated”, “2”: “Very angry” }, “probabilities”: { “0”: 0.04, “1”: 0.43, “2”: 0.54 }, “confidence”: 0.75 } }, “usage”: { “input_tokens”: 129, “output_tokens”: 160, “reasoning_tokens”: 1536 }, “latency_ms”: 8141.6 }
Python
sdk/ is a drop-in replacement for Jev’s Python SDK (typesafe-sdk):
pip install ./sdk
from jeeves_sdk import Choice, Noul, Score, TypeSafeClient
with TypeSafeClient() as client:
result = client.system_one(
state="I was charged twice. Please help.",
questions={
"billing": Noul(instructions="Is this about billing?"),
"tone": Choice(instructions="What is the tone?", criteria={"calm": None, "angry": None}),
"urgency": Score(instructions="How urgent is this?", criteria=["can wait", "this week", "today"]),
},
max_think=768,
return_reasoning=True,
)
print(result.nouls["billing"].noul, result.choices["tone"].choice, result.scores["urgency"].score)
print(result.reasoning["tone"].text)
The client connects to http://127.0.0.1:8009 by default (or JEEVES_BASE_URL), needs no API key, and waits up to 120s.
options is optional and ignored by Jev clients that don’t send it. Server-wide defaults are set with the matching serve flags.
Questions, states and answers are loaded into the Qwen chat template like
<state> …state… <q> instructions <opt> option 1 </opt> <opt> option 2 </opt> … <think>
The model then rolls out its reasoning chain, and after the </think> token we append
`
instructions
A pointer head scores each option with a scaled dot product between a query projection of the hidden state at <decide> and a key projection of the hidden state at that option’s </opt>, where
<state>, <q>, <opt>, </opt>, <decide> = "<|fim_prefix|>", "<|fim_middle|>", "<|box_start|>", "<|box_end|>", "<|fim_suffix|>"
These are rare, largely unused tokens in the Qwen tokenizer. Ablations found that using plain text like “State” in the prompt instead worsened performance. Likewise, not repeating the questions after the reasoning block also decreases performance. The final probabilities are a softmax over the option scores, divided by a temperature fitted on the dev set.
- SFT (2 epochs, 596 steps on 8 GPUs). LoRA r=16 on all projections of Qwen3.5-9B plus the pointer head, trained on 19,126 questions from 12 public datasets and synthetic policy data. Half the questions carry a reasoning chain sampled from the base model.
- CISPO (a 624-step schedule stopped at step 402). 9,992 RL questions, 8 rollouts each at temperature 1, capped at 2,560 thinking tokens.
- Calibration. A single temperature fitted on dev, stored with the checkpoint.
Stopping at step 402 keeps the best calibration and dev score. Past it, the head over-sharpens on the saturated RL pool.
A diffusion view of the frozen model (drafter/), inspired by Orthrus.
Unlike Orthrus, which supports attention-only models, it supports Qwen3.5’s Gated DeltaNet layers by letting mask tokens cross-attend to those layers’ post-convolution keys and values.