Getting started
Grade model output against a weighted rubric with an LLM judge.
Install
Requires Python 3.10 or later.
uv add rubricQuick start
The built-in generate functions call Gemini, so set GEMINI_API_KEY first.
import asyncio
from rubric import Rubric, default_per_criterion_generate_fn
from rubric.autograders import PerCriterionGrader
async def main():
rubric = Rubric.from_dict([
{"weight": 10.0, "requirement": "Response mentions Paris"},
{"weight": 5.0, "requirement": "Response is concise"},
])
grader = PerCriterionGrader(generate_fn=default_per_criterion_generate_fn)
result = await rubric.grade("Paris is the capital of France.", autograder=grader)
print(f"Score: {result.score}")
asyncio.run(main())rubric.grade() takes the text to grade, an optional query, and an
autograder. Without an autograder it uses PerCriterionGrader with the Gemini
generate function. To use another model, pass your own generate function; see
Customization.
Writing rubrics
A criterion has a weight and a requirement. A positive weight rewards a
response that meets the requirement. A negative weight penalizes it, which
suits error checks.
from rubric import Criterion, Rubric
rubric = Rubric([
Criterion(weight=10.0, requirement="States Q4 2023 base margin as 17.2%"),
Criterion(weight=8.0, requirement="Explicitly uses Shapley attribution for decomposition"),
Criterion(weight=-15.0, requirement="Uses total deliveries instead of cash-only deliveries"),
])You can also load a rubric from a dictionary, JSON, YAML, or a file:
rubric = Rubric.from_dict([{"weight": 10.0, "requirement": "Example requirement"}])
rubric = Rubric.from_json('[{"weight": 10.0, "requirement": "Example requirement"}]')
rubric = Rubric.from_yaml('- weight: 10.0\n requirement: "Example requirement"')
rubric = Rubric.from_file("rubric.json") # or rubric.yaml- weight: 10.0
requirement: "States Q4 2023 base margin as 17.2%"
- weight: 8.0
requirement: "Explicitly uses Shapley attribution for decomposition"
- weight: -15.0
requirement: "Uses total deliveries instead of cash-only deliveries"