LLM as a Judge: Typed Verdicts From One API Call
Run LLM-as-a-judge checks with yes/no, choice and score questions that return probabilities, in one call per item, billed on input tokens only.
Get a pass or fail, a category and a severity for every output, in one prompt. Paste this into your agent:
Set up https://monid.ai/SKILL.md, and then use Monid to run typesafe /systemone on each of these support replies with three questions: is the answer correct (yes or no), which category does it fall into, and how severe is the problem on a five-level scale. Give me one table with the three verdicts and their probabilities.
One prompt judges a whole list. 1 min setup · 10-24 hrs saved.
You have outputs to check: model answers, support replies, scraped records, candidate matches. Reading them yourself does not scale, so you use a language model as the judge. Most guides tell you to prompt a general model for JSON and parse it. This guide shows the other route, a model that only judges: you send a state and typed questions, and you get typed answers with probabilities back. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.
What is the easiest way to run LLM as a judge through an API?
Monid. Install it once, add a key, and your agent calls a judging endpoint with a state and a map of questions, then gets typed verdicts back, with no prompt to write and no output to parse.
- Pay for what you run. The endpoint is billed on input tokens only, at $0.042 per million tokens, and output tokens are free (Verified 2026-10-16 on the AI tools page). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
- One install, every API. The same key and balance call the judge and every other tool on Monid. When the thing you are judging comes from a scraper or a search endpoint, it is another call on the same balance, not another vendor.
- Lives inside your agent. Install the CLI, add your key, run the prompt above, and the judge is there whenever your agent needs a verdict. Your agent reads
SKILL.mdand learns the rest, so setup is about a minute.
At that price, a 1,000-token state costs $0.000042 to judge, however many questions you ask about it (Verified 2026-10-16).
How do you run an LLM judge over many outputs?
Send one call per item with every question you need about that item, and loop over the list. The endpoint is /systemone from the provider typesafe. It takes a state, a model and a map of questions.
| Route | What you get | Cost shape | Fits |
|---|---|---|---|
| Prompt a general model for JSON | Free text you parse, a number the model chose | Generation tokens, output included | A one-off decision in a conversation |
| Build an evaluation harness | Whatever your rubric and glue code cover | Your hours | A pipeline you will maintain |
| Labelling by people | The most trusted verdict, slowly | Per label | A gold set to check the judge against |
| Monid judging endpoint | Typed answers with probabilities, many questions per call | Per input token, output free | Thousands of items against fixed questions |
Monid fits a list because the work is the same call repeated, and you pay for the tokens you send and nothing between runs.
For agents
Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:
set up https://monid.ai/SKILL.md
It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.
For humans
npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main
Inspect, then run one
monid inspect -p typesafe -e /systemone
The body is JSON, so a run passes it with -i. state is what you are judging: a string, or better an object so each part has a name. questions is a map whose keys become the keys of the answer.
monid run -p typesafe -e /systemone -i '{
"state": { "question": "How do I reset my password?", "reply": "Click Forgot password on the login page.", "rating": 4 },
"model": "jev-latest",
"questions": {
"is_correct": { "type": "noul",
"instructions": "Does the reply answer the question?",
"criteria": { "true": "It gives a usable answer.", "false": "It is off topic or wrong." } },
"category": { "type": "choice",
"instructions": "What kind of reply is this?",
"criteria": { "answer": "Gives the steps.", "deflection": "Sends the user elsewhere.", "refusal": "Declines to help." } },
"severity": { "type": "score",
"instructions": "How serious is any problem with this reply?",
"criteria": ["none", "minor", "noticeable", "serious", "harmful"] } } }' -w
Batch, because the state is the cost
The endpoint's pricing note says the state is ingested once per call and that batching many questions into one call is the cheap path. Its description says every question is evaluated in parallel against the same state. So the pattern is one call per item, many questions per call, and a loop over the items:
mkdir -p out
for f in items/*.json; do
monid run -p typesafe -e /systemone -i "$(cat "$f")" -w > "out/$(basename "$f")"
done
Or hand the whole job to an agent
Give this to your agent
Set up https://monid.ai/SKILL.md, and then use Monid to for each of these 500 support replies, decide whether it answers the question, classify the reply into answer, deflection or refusal, score how serious any problem is, and give me the three most common problem types with counts.Can you build an LLM judge yourself with a prompt?
Yes, and for a handful of outputs it is the right start. You write a prompt that states the rubric, ask a general model to answer in JSON, parse the reply, and keep the prompt in version control. Many evaluation frameworks and libraries wrap exactly this loop.
The cost is everything around the prompt. You are paying generation prices for a decision. The reply is free text that is occasionally not valid JSON, so you add a repair step and a retry. The number the model gives you is a token it chose, not a probability you can put a threshold on. Asking for a second judgment means a second call that sends the same state again. Moving to a different model changes the behaviour of the whole rubric, and you find out when your scores shift.
A judging endpoint removes those steps by making the shape a contract. You declare each question as a type, and the answer comes back in that type. The Monid route is the choice for anything you will run more than once: the endpoint takes instructions and criteria instead of a prompt you maintain, and it bills on input tokens at $0.042 per million (Verified 2026-10-16). For a few outputs, prompt the model you already use.
What does a typed verdict look like, and what can you build with it?
A typed verdict is an answer whose shape you chose before the call, so your code can branch on it without parsing. The endpoint supports three question types, and you can mix them freely in one call.
The three question types
noulis a yes/no.instructionsis the question, andcriteriais an optional pair that says what a yes (true) and a no (false) mean. It answers with a calibrated probability of yes.choiceclassifies into your own options.criteriais an object that maps each option name to a description, up to 255 options. It answers with the chosen option, a probability per option and a confidence.scorerates along your rubric.criteriais an ordered array of 2 to 10 level descriptions, low to high. It answers with a probability-weighted value that can land between levels, a probability per level and a confidence.
Every question has an id you pick, and the answers come back under the same ids.
What you can build with it
- A guardrail. Ask whether an output contains a claim the source does not support, and block or flag anything above a probability you choose. The endpoint's own description lists LLM guardrails, citation and claim verification among its uses.
- A triage router. Classify tickets or messages by intent and urgency in one call, then send low-confidence ones to a person. The documented uses include triage, intent routing and confidence-gated escalation.
- A re-ranker or deduper. Put two candidate records in the state and ask whether they are the same entity, or which of a set fits a query best.
- An evaluation run. Score every answer your agent produced last night against a rubric, and track the average per day. The score can land between levels, so an average moves smoothly.
Where the state comes from is your choice. Search results from a free web search API for agents and the cited answers described in the two Perplexity API surfaces both work as state, as do scraped reviews and extracted pages.
What goes wrong when you use an LLM as a judge?
Most failures come from reading the verdict wrongly, not from the call. Five habits prevent them, and each follows from how the endpoint is built.
Why should you read the probability and not just the label?
Because the label throws away how sure the model was. A choice answer carries a probability per option and a confidence, and a noul answer is a probability, not a boolean. A rule such as "block when the probability of unsupported is above 0.8" uses the information you paid for. A rule that rounds first uses half of it.
Why is a score between two levels valid?
Because the score is probability-weighted. The endpoint states that it can land between levels, so a value between your level 1 and level 2 is a real answer, not an error. Rank on the value, and set thresholds on the per-level probabilities when a decision depends on one level. Look up how your five labels map to numbers in the answer before you build a dashboard on it, rather than assuming the first level is 1.
Why should low confidence go to a person?
Because confidence is the cheapest routing key you have. Choose a floor, send everything under it to a human, and automate the rest. The queue is small when the judge is sure about most items, and it grows exactly when the cases get unclear, which is when you want eyes on them.
Why should you check the judge against labels you trust?
Because any judge, human or model, needs a gold set. Label a few dozen items yourself, run them through the judge and compare. If the two disagree on a question, the fix is usually in the criteria text, so tighten the definitions of the options and run the set again. Re-run the same set whenever you change a rubric or a model value.
Why pin the model once your thresholds are tuned?
Because your thresholds were tuned against one model. The model field accepts jev-latest, jev-preview and the pinned jev-1.13.0, and the endpoint's schema says the output reports the versioned model that answered. Store that version beside every verdict, and you can tell later whether a shift came from your data or from a new release.
What does a batch cost?
Tokens in, nothing out. Each item costs its input tokens times $0.042 per million (Verified 2026-10-16). A 600-token state is $0.0000252, so 10,000 of them is $0.252. Adding a fourth or fifth question to the same call does not resend the state. The listed median run time is about 1.6 seconds and the 95th percentile about 2.0 seconds, so a loop of 10,000 takes hours when run one after another. Run several at once.
Which endpoint should I use for which job?
| Endpoint | What it does | Input | Billing | Price (Verified 2026-10-16) |
|---|---|---|---|---|
typesafe/systemone | Typed judgments on a state: yes/no, choice, score | state, questions, optional model | Per input token, output free | $0.042 per million tokens |
Inspect it for free before you design a schema: monid inspect -p typesafe -e /systemone. The price is the same on all three model values.
The endpoint judges and does not fetch. The things you judge come from somewhere else, such as a scraper, a search or your own database, and those are other calls on the same key and balance. The AI tools page lists the neighbours, and current prices are on monid.ai/tools.
When is another route a better fit?
The output is prose. If you need an explanation, a summary or a rewritten sentence, a judging endpoint is the wrong shape. It returns values, not text.
The question is open-ended. "What are the themes in these reviews" is not a typed question. Find the categories with a general model first, then encode them as a choice and run the volume through the judge.
You need one judgment, once. The advantage is per-item cost at scale and a parseable contract. For a single decision in a conversation, the general model you are already calling is simpler.
You cannot list the options. choice takes your categories. If the set is unknown, you are extracting rather than classifying, and that is a different job.
You need a human's verdict. For anything with legal or safety weight, a model judge is a filter in front of a person, not a replacement.
And the disclosure: we resell this endpoint. For prose, exploration and one-off decisions, use something else. What it does well is the case where you have thousands of things and a fixed question.
Conclusion
The quickest way to run LLM as a judge over a list is one call per item through Monid: $0.042 per million input tokens, output free (Verified 2026-10-16), one key for this and every other tool, about a minute to set up and ready inside your agent whenever you need it. Writing your own prompt works for a few outputs and costs you the parsing, the retries and the rubric upkeep.
Two things matter more than the endpoint you pick. Read the probability and the confidence, not just the label, and route low confidence to a person. And check the judge against a small set of items you labelled yourself before you trust it on the rest.
How we estimated the time saved: building a judging harness with a rubric prompt, output parsing, retries and a first calibration set takes roughly 6 to 12 hours, plus 4 to 12 hours over the first month for rubric changes and re-checks. That is 10 to 24 hours. The 1 minute is the install, key and first call once you already have a Monid key.
Free next step: run monid inspect -p typesafe -e /systemone, then make one call on three outputs whose right verdict you already know and compare. Start at monid.ai.
FAQ
What does LLM as a judge mean?
It means using a language model to evaluate something instead of writing something: score this answer, classify this ticket, decide whether two records match. It became standard because evaluation is the step that scales worst with human reviewers.
Is there an API for LLM as a judge?
Yes. The typesafe/systemone endpoint on Monid takes a state and typed yes/no, choice and score questions and returns typed answers with probabilities. It is billed on input tokens at $0.042 per million (Verified 2026-10-16), and output is free.
How do I use a judge for many items at once?
Make one call per item with all your questions in it, and loop over the items with the shell loop above. The state is ingested once per call, so extra questions barely change cost or latency.
How do I calibrate an LLM judge?
Label a small set yourself, run it through the judge and compare. Then tighten the criteria text where they disagree, and send answers under a confidence floor to a person. Re-run the set whenever you change the rubric or the model.
Should I pin the judge model version?
Yes, once your thresholds are tuned. The model field accepts jev-latest, jev-preview and the pinned jev-1.13.0, and the output reports the versioned model that answered, so you can store it with each verdict.
Published September 2026.
