Quality

Jers tells you how often it is right on your own decisions, and tunes its confidence to them. Everything here works from labels: the right answer for a question, given after the fact.

Labels

Every decision has a decision_id. When you learn the right answer, send it:

POST /v1/feedback
{"decision_id": "dec_4b637aef332bc18f111236e0", "question": "route", "correct": "billing"}

correct is an option name for a choice, a level number counted from 0 (the lowest level) for a score, and true or false (or "yes" or "no") for a noul. The answer says whether Jers was right and how many labels the question has. To match labels against, Jers keeps each decision's question ids, a hash of each question and the probabilities for 30 days, never the state or the question text. Labels and a question's own temperature belong to its id and wording: change its type, instructions or options and it starts again from zero labels. POST /v1/decisions/delete removes every record and label of your tenant.

The quality report

GET /v1/quality?question=route&days=30

Per question and model: labels, accuracy, calibration_error (how far the stated probabilities are from how often they come true), brier, log_loss, reliability bands, and automate_at: for target accuracies such as 0.9, 0.95 and 0.99, the probability threshold that reaches the target on your labels and the share of decisions above it. That share is what you can automate at that accuracy; the rest goes to a person.

Calibration per tenant

When a question has 50 labels (and again at 100, 200, 400, 800, 1,600 and 3,200) Jers fits a temperature for it on your labels and keeps it only if it lowers the calibration error and does not raise the log loss on labels it was not fitted on. The same fit sets a temperature for the question's group (same model, type and number of options), which other questions of that group use until they have one of their own. POST /v1/calibration/fit does the same on demand (50 labels by default; min_labels may lower that to 20), per question and per group. A kept temperature changes only the probabilities you receive, for that model only; each answer carries calibration and the raw_ values before it. GET /v1/calibration lists what is in use.

Measured on a constructed 60-case support set (2026-09-23, measurements/jers-gateway-live-2026-09-23.txt, item 8): the English checkpoint was 83% right with a calibration error of 0.17 on held-out folds, lowered to 0.13; typed-decisions 85% right, 0.34 lowered to 0.10. On that set the engine was under-confident. Your data will differ, which is the point of fitting on it.

Golden sets and regressions

A golden set is a list of cases with known answers that you keep and re-run: before changing a model, a question's wording, or a memory.

POST /v1/golden
{"cases": [{"id": "refund-1", "state": "We were charged twice.", "questions": {...}, "expected": {"route": "billing", "refund": true}}]}

POST /v1/golden/run   {"model": "jers-english"}

A run answers every case with the run's model (default jers) and reports accuracy per question and changed_since_last_run: every answer that differs from the previous run of the same model, so a change in behaviour shows up as a list, not a surprise. Cases may name a subject, values, derive, memory, robust and windows, so memories and rules are tested too; a case with an existing id replaces it. A case that fails, or is not run because the credit ran out, is left out of the accuracy. Unlike decisions, a golden case keeps its state until you delete it (POST /v1/golden/delete with ids, or with no ids for every case; an empty list deletes nothing). Runs through the API are billed like decisions, in one request: give a large set a longer client timeout; Jers can also run your set for you without billing, and load a large set from a file; ask. Up to 5,000 cases per tenant.

The loop

  1. Decide, and keep the decision_id with your record.
  2. Send feedback when the outcome is known.
  3. Read quality: automate above the automate_at threshold for the accuracy your process needs.
  4. Keep a golden set, and run it before every change.