free_tool
Is your eval judge biased?
When you score models with an LLM judge, every eval number inherits the judge's biases: it favors the answer shown first, rewards longer responses, and scores an undefined 1-to-10 differently each run. Paste your judge prompt and get a graded report on what's tilting the ruler.
The prompt you paste runs entirely in your browser. It is never uploaded, sent to a server, or stored. (Anonymous usage metrics, never your prompt text, are sent to analytics.)
Judge rubric reliability
27/100
2 to fix · 4 warnings · 0 passed · 2 notes · pairwise judging
Grade F, score 27 out of 100, 2 to fix, 4 warnings, 0 passed.The rubric names explicit evaluation criteria
highThe prompt asks the judge to score or pick a winner without naming the criteria. 'Which answer is better?' means the model decides, per call, what better even means, so scores are inconsistent between runs and don't track what you care about. List the specific dimensions to judge on and, ideally, their relative weight.
Evaluate each response on: (1) factual accuracy, (2) relevance to the question, (3) completeness, (4) safety. Weight accuracy highest.Pairwise judging mitigates position bias
highThis is a pairwise comparison with no guard against position bias. LLM judges reliably prefer the first (sometimes the second) answer regardless of quality, so a raw A-vs-B verdict partly measures order, not quality. Run each pair in both orders and only count a win when the answer wins both, and/or instruct the judge that order is random and must not influence the verdict.
Evaluate the pair in both orders (A,B) and (B,A); only declare a winner if it wins both. The order is randomized and must not affect your judgment.The score scale is defined with anchors
highThe prompt asks for a numeric score but doesn't define what the levels mean. 'Rate 1 to 10' leaves the judge to invent thresholds, so the same answer scores 6 one run and 8 the next, and the numbers aren't comparable across items. Anchor each level, or use a short ordinal scale (1–4) with a one-line description per point. Fewer, well-defined levels beat a wide undefined range.
Score 1–4 where 1 = wrong or unsafe, 2 = partially correct, 3 = correct but incomplete, 4 = fully correct and complete.The judge reasons before giving a verdict
mediumThe prompt asks for a score or verdict without first asking the judge to reason. A snap verdict is noisier and amplifies bias, and you get no explanation to inspect when a score looks wrong. Ask for a brief rationale before the final answer, and put the verdict last so the reasoning conditions it.
First, briefly explain your reasoning against each criterion. Then, on a new line, give the final verdict.The verdict comes back in a parseable format
mediumNo output format is specified, so the judge answers in free prose and you're left extracting the score with a fragile regex. When the wording drifts, the parser silently grabs the wrong number or none. Ask for a fixed structure (JSON with a reasoning field and a score/verdict field, or a 'Verdict:' line) so extraction is exact.
Respond with JSON only: {"reasoning": "…", "verdict": "A" | "B" | "tie"}The rubric guards against length bias
mediumNo guard against length bias. LLM judges systematically prefer longer, more verbose responses regardless of whether the extra text adds value, so a padded answer can beat a correct concise one. Instruct the judge to evaluate substance and correctness independent of length, and to not reward verbosity.
Judge on correctness and substance only. Do not reward longer or more elaborate answers; a concise correct answer is better than a verbose one.The judge is grounded on a reference answer
No reference answer is provided to the judge. That's unavoidable for open-ended quality judging, but for factual or closed-ended tasks a judge comparing to a gold answer is far more reliable than one relying on its own knowledge. Provide the reference where you have one.
Pairwise judging allows a tie
The pairwise prompt appears to force a winner with no tie option. When two answers are genuinely equal, forcing a pick adds noise and can mask that a change made no real difference. Allow a 'tie' verdict so equal cases are recorded as equal.
If the judge is biased, every eval number built on it is biased the same way, and you're tuning a model against a crooked ruler. Calibrating the judge against human labels is the step most teams skip. That's the kind of review I do.
Get your eval judge calibrated: book a callThis is a heuristic linter of the prompt's wording, not a parser or a runtime check: it looks for the presence or absence of the mitigations it grades (criteria, position-bias handling, scale anchors, a reasoning step, a length-bias guard, a structured output). It executes no prompts, runs entirely in your browser, and uploads nothing.
why_it_matters
A biased judge biases every number built on it
LLM-as-judge is how most teams score open-ended output at scale, and it has well-measured failure modes. Position bias: judges favor whichever answer is shown first, so an A-vs-B verdict partly measures order. Verbosity bias: they reward longer answers even when a concise one is better. Undefined scales: a bare 1-to-10 with no anchors scores the same answer differently each run.
If the judge is tilted, every eval, every A/B, every model comparison built on it is tilted the same way, and you're tuning against a crooked ruler. This linter flags the missing mitigations so you can straighten it before you trust the numbers.
faq
Questions & answers
- What does the LLM-as-Judge Rubric Linter check?
- It reads your LLM-as-judge prompt (the instructions that tell a model how to score or compare other models' outputs) and grades it for the biases and ambiguities that make judge scores unreliable: whether it names explicit evaluation criteria instead of an undefined 'better', whether pairwise comparisons guard against position bias, whether a numeric score scale is anchored (levels defined) rather than a bare 1-to-10, whether the judge reasons before giving a verdict, whether it guards against rewarding longer answers, and whether it asks for a parseable output. Each finding explains the bias and gives a snippet to add.
- What is position bias in an LLM judge?
- When you ask a model to compare two answers (A vs B), it has a strong, well-measured tendency to favor whichever one it sees first, regardless of quality. So a raw A-vs-B verdict is partly measuring presentation order, not the answers. The standard fix is to run each pair in both orders and only count a win when the answer wins both, and/or to tell the judge explicitly that the order is randomized and must not influence the verdict. The linter flags any pairwise prompt that has no such guard as a high-severity issue.
- Why does an undefined 1-to-10 scale hurt?
- If you ask the judge to 'rate 1 to 10' without defining what the levels mean, it invents the thresholds on each call, so the same answer scores a 6 one run and an 8 the next and the numbers aren't comparable across items. Anchoring each level (or using a short ordinal scale like 1–4 with a one-line description per point) makes the judge apply the same standard every time. Fewer, well-defined levels are more reliable than a wide undefined range.
- Why should the judge reason before scoring?
- Asking the judge to write its reasoning against each criterion before giving the final verdict (chain-of-thought) measurably improves agreement with human judgments and reduces the noise and bias of a snap decision. It also gives you an audit trail: when a score looks wrong, you can read why the judge decided that, instead of staring at an unexplained number. Put the reasoning first and the verdict last so the explanation conditions the score.
- Does linting the prompt make my judge reliable?
- It removes the obvious biases, but the step that actually makes a judge trustworthy is calibration: run it against a set of human-labeled examples and confirm its scores track the human ones (and check inter-run agreement). A clean prompt with no calibration can still disagree with what you actually care about. Treat this linter as the first pass that fixes the wording, then calibrate against real labels before you trust the numbers.
- Is my judge prompt uploaded anywhere?
- No. This is a heuristic text linter that runs entirely in your browser: it scans your prompt for the presence or absence of the mitigations it grades and never executes it, calls a model, or needs an API key. Because it reads wording rather than runtime behavior, it can miss a mitigation phrased in an unusual way or over-credit one mentioned but not really applied. Nothing you paste is uploaded or stored; only anonymous usage metrics are sent to analytics.
Want the judge calibrated against real labels?
A cleaner prompt is the floor. The step most teams skip is calibrating the judge against a set of human-labeled examples so you know its scores actually track what you care about. Book a call, or leave your email.
Prefer proof first? See how this plays out in real case studies →