Customization

Use any model provider, override the grading prompts, or write your own autograder.

Bring your own model

A generate_fn is an async function that takes a system prompt and a user prompt and returns the autograder's Pydantic output model. You can call any provider. Use its structured outputs so responses validate.

import os
from openai import AsyncOpenAI
from rubric import PerCriterionOutput
from rubric.autograders import PerCriterionGrader

client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])

async def generate_with_openai(system_prompt: str, user_prompt: str) -> PerCriterionOutput:
    response = await client.chat.completions.create(
        model="gpt-5-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        response_format={"type": "json_schema", "json_schema": {
            "name": "criterion_output",
            "schema": PerCriterionOutput.model_json_schema(),
        }},
    )
    return PerCriterionOutput.model_validate_json(response.choices[0].message.content or "{}")

grader = PerCriterionGrader(generate_fn=generate_with_openai)

Each autograder expects its own return type:

PerCriterionGrader → PerCriterionOutput (criterion_status, explanation)
PerCriterionOneShotGrader → OneShotOutput (a list of CriterionEvaluation)
RubricAsJudgeGrader → RubricAsJudgeOutput

Errors and retries

Pydantic validates what your generate_fn returns and raises a ValidationError on invalid data. Put retries inside your generate_fn:

from pydantic import ValidationError
from rubric import PerCriterionOutput

async def generate_with_retries(system_prompt: str, user_prompt: str, max_retries: int = 3) -> PerCriterionOutput:
    for attempt in range(max_retries):
        try:
            response = await your_llm_call(system_prompt, user_prompt)
            return PerCriterionOutput.model_validate_json(response)
        except ValidationError:
            if attempt == max_retries - 1:
                raise

Grading prompts

Each autograder has a default system prompt. Override it with system_prompt:

grader = PerCriterionGrader(
    generate_fn=your_function,
    system_prompt="Your custom system prompt here",
)

Autograders wrap the response in <response> tags and the optional query in <query> tags. Write custom prompts to expect them:

<response>
{content}
</response>

The default prompts live in the source: per_criterion_grader.py, per_criterion_one_shot_grader.py, and rubric_as_judge_grader.py.

Your own autograder

Subclass Autograder and implement two methods:

judge() evaluates the submission and returns raw results.
aggregate() turns those results into an EvaluationReport.

A custom autograder can call models directly, use several, or use none.

Bring your own model

Errors and retries

Grading prompts

Your own autograder

On this page