https://github.com/auraoneai/rubric-pr-bot
Rubric diffs and lint feedback for pull requests that change evaluation criteria.
https://github.com/auraoneai/rubric-pr-bot
ai-evaluation evals github-app rubric
Last synced: 6 days ago
JSON representation
Rubric diffs and lint feedback for pull requests that change evaluation criteria.
- Host: GitHub
- URL: https://github.com/auraoneai/rubric-pr-bot
- Owner: auraoneai
- License: mit
- Created: 2026-05-12T01:33:51.000Z (22 days ago)
- Default Branch: main
- Last Pushed: 2026-05-12T06:54:55.000Z (22 days ago)
- Last Synced: 2026-05-12T08:19:05.146Z (22 days ago)
- Topics: ai-evaluation, evals, github-app, rubric
- Language: TypeScript
- Homepage: https://auraone.ai/open
- Size: 25.4 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# rubric-pr-bot
A GitHub App that watches PRs touching `*.rubric.json`, runs rubric-spec diff/lint, comments with downstream impact, and can expose required-check failures for error findings. Public install target: `github.com/apps/auraone-rubric-pr-bot`.
## What This Is Not
No customer rubrics are included; examples are synthetic.
## Review Behavior
- Handles pull request open and synchronize events for changed `*.rubric.json` files.
- The GitHub webhook helper fetches changed files and base/head rubric contents from the GitHub API.
- `npm start` runs a deployable `/webhook` server with GitHub webhook HMAC verification and GitHub App installation-token exchange.
- Summarizes added, removed, and changed criteria with links back to `rubric-spec` criteria docs.
- Posts or updates a marked PR comment with diff and lint findings.
- Creates a `rubric-pr-bot` check run and returns a `failure` conclusion when lint findings include severity `error`, allowing teams to wire the app as a required check.
## Deployment
```bash
npm install
npm run build
RUBRIC_PR_BOT_APP_ID=123 \
RUBRIC_PR_BOT_PRIVATE_KEY="$(cat private-key.pem)" \
RUBRIC_PR_BOT_WEBHOOK_SECRET=change-me \
npm start
```
The server listens on `PORT` or `3000` and expects GitHub webhooks at `/webhook`.
## App Permissions
The manifest requests `contents: read`, `pull_requests: read`, `issues: write`, and `checks: write`. `issues: write` is used for PR conversation comments, and `checks: write` is used for optional merge-blocking check runs.