Rubric
Autograders and scoring
The three graders, how each scores, and the fields in a result.
An autograder takes a generate_fn and returns an EvaluationReport.
| Autograder | Model calls | Model returns | generate_fn returns |
|---|---|---|---|
PerCriterionGrader | One per criterion, in parallel | MET or UNMET for one criterion | PerCriterionOutput |
PerCriterionOneShotGrader | One | MET or UNMET for each criterion | OneShotOutput |
RubricAsJudgeGrader | One | A score from 0 to 100 | RubricAsJudgeOutput |
from rubric.autograders import (
PerCriterionGrader,
PerCriterionOneShotGrader,
RubricAsJudgeGrader,
)Per-criterion scoring
PerCriterionGrader and PerCriterionOneShotGrader score the same way. A MET
criterion adds its weight; an UNMET criterion adds nothing.
raw_score = sum of w_i over criteria judged METWith normalize=True, the default, the score falls between 0 and 1:
score = clamp(raw_score / sum(max(0, w_i)), 0, 1)If all weights are negative, as in an error-detection rubric, a response that avoids all the errors scores 1.0 and one that makes all of them scores 0.0:
score = clamp(1 + raw_score / sum(|w_i|), 0, 1)Pass normalize=False for the raw weighted sum.
Holistic scoring
RubricAsJudgeGrader asks the model for one score from 0 to 100 and keeps it
as llm_raw_score. It converts that score to a weighted sum, so you can
compare results across graders:
raw_score = llm_raw_score / 100 * sum(max(0, w_i))
score = clamp(llm_raw_score / 100, 0, 1) # normalize=True
score = raw_score # normalize=FalseResult fields
| Field | Description |
|---|---|
score | 0 to 1 with normalize=True, otherwise raw_score. |
raw_score | The weighted sum, comparable across graders. |
llm_raw_score | The model's 0–100 score for RubricAsJudgeGrader; otherwise raw_score. |
report | Per-criterion verdict, requirement, and reason. None for RubricAsJudgeGrader. |
result = await rubric.grade(query="Input query...", to_grade="Output to evaluate...", autograder=grader)
print(f"Score: {result.score:.2f}")
for criterion in result.report:
print(f" [{criterion.verdict}] {criterion.requirement}")
print(f" → {criterion.reason}")