https://github.com/auraoneai/rubric-spec
Portable rubric schema, validator, linter, diff, adapters, and conformance tests for AI evaluation.
https://github.com/auraoneai/rubric-spec
ai-evaluation evals json-schema rubric
Last synced: 6 days ago
JSON representation
Portable rubric schema, validator, linter, diff, adapters, and conformance tests for AI evaluation.
- Host: GitHub
- URL: https://github.com/auraoneai/rubric-spec
- Owner: auraoneai
- License: mit
- Created: 2026-05-12T01:32:24.000Z (22 days ago)
- Default Branch: main
- Last Pushed: 2026-05-12T04:02:53.000Z (22 days ago)
- Last Synced: 2026-05-12T04:29:39.754Z (22 days ago)
- Topics: ai-evaluation, evals, json-schema, rubric
- Language: Python
- Homepage: https://auraone.ai/open
- Size: 11.7 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# rubric-spec
`rubric-spec` is the reference implementation for AuraOne Rubric Schema v1, a portable rubric format that can move between Inspect AI, PromptFoo, DeepEval, LangSmith, and EvalKit without losing criterion ids, anchors, weights, judge prompt contracts, or provenance.
## Quickstart
```bash
python -m venv .venv
. .venv/bin/activate
pip install rubric-spec
rubric-spec validate examples/minimal_rubric.json
rubric-spec conformance
```
## Adapter Migration Guides
- Inspect AI: export scorer criteria as JSON, then run `rubric-spec convert --from inspect_ai --to rubric_spec inspect.json`.
- PromptFoo: map `assert`/grading configs into criteria; `from_spec` preserves weights and anchors.
- DeepEval: test case metrics become rubric criteria.
- LangSmith: feedback schema entries become criteria.
- EvalKit: JSONL rows round-trip into the canonical v1 object.
## FAQ
### Why a spec?
Rubrics are operational contracts. A named schema makes reviews portable, diffable, lintable, and conformance-testable.
### Why not OpenEval?
This package focuses on the human-judgment layer: anchors, tie-break rules, reviewer-facing examples, judge prompt contracts, and provenance. It can be used alongside broader eval orchestration standards.
### How is this different from OpenAI Evals YAML?
OpenAI Evals YAML configures a run. AuraOne Rubric Schema v1 defines the rubric artifact itself so it can be reused across runners.
## What This Is Not
This is not a leaderboard, a benchmark claim, or a source of real customer rubrics. All bundled examples are synthetic.
## Conformance
The conformance suite is defined in `spec/conformance-tests.md` and can be surfaced with `rubric-spec conformance`.