Teach the model by example: supply 2-3 input to output samples, then have it apply the pattern to your task.
Prompts / Techniques / LLM-as-Judge Rubric Builder with Calibration Anchors
LLM-as-Judge Rubric Builder with Calibration Anchors
Designs an evaluation rubric with scored dimensions and anchor examples so an LLM judge scores outputs consistently.
ROLE: You are an evaluation designer who builds calibrated rubrics for scoring model outputs reliably.
CONTEXT: We are judging outputs for the task [TASK], where quality means [QUALITY_DEFINITION] and the audience is [AUDIENCE].
TASK:
1. Decompose quality into 3-5 independent scoring dimensions, each with a short definition and why it matters.
2. For each dimension, define a 1-5 scale and write a one-line descriptor for scores 1, 3, and 5.
3. Provide one anchor example output per dimension at the score-3 and score-5 level to calibrate the judge.
4. Specify how dimension scores combine into an overall verdict, including any veto rule.
CONSTRAINTS: Dimensions must not overlap conceptually. Avoid subjective words without an anchor. The rubric must let two judges reach the same score from the same output. No more than 5 dimensions.
OUTPUT FORMAT: A rubric table [Dimension, Definition, 1/3/5 Descriptors, Weight], a calibration anchors section, and the aggregation + veto rule.