{"id":50733737,"url":"https://github.com/synapt-dev/eval","last_synced_at":"2026-06-10T11:31:05.914Z","repository":{"id":356384141,"uuid":"1232297819","full_name":"synapt-dev/eval","owner":"synapt-dev","description":"Eval framework for AI quality discipline. 3-loop discipline (PR-gate, sprint, quarter), reviewer-framework SDK, suggestion engine. Library + GitHub Actions adapter.","archived":false,"fork":false,"pushed_at":"2026-05-07T21:29:04.000Z","size":54,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-07T21:37:19.000Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/synapt-dev.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-07T19:41:41.000Z","updated_at":"2026-05-07T21:29:08.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/synapt-dev/eval","commit_stats":null,"previous_names":["synapt-dev/eval"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/synapt-dev/eval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synapt-dev%2Feval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synapt-dev%2Feval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synapt-dev%2Feval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synapt-dev%2Feval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/synapt-dev","download_url":"https://codeload.github.com/synapt-dev/eval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synapt-dev%2Feval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34151271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-10T11:31:05.671Z","updated_at":"2026-06-10T11:31:05.908Z","avatar_url":"https://github.com/synapt-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# @synapt/eval\n\n[![PyPI](https://img.shields.io/pypi/v/synapt-eval)](https://pypi.org/project/synapt-eval/)\n[![Python](https://img.shields.io/pypi/pyversions/synapt-eval)](https://pypi.org/project/synapt-eval/)\n[![License](https://img.shields.io/github/license/synapt-dev/eval)](LICENSE)\n[![CI](https://img.shields.io/github/actions/workflow/status/synapt-dev/eval/ci.yml?label=CI)](https://github.com/synapt-dev/eval/actions)\n\nDomain-agnostic eval framework for AI applications. Measure retrieval quality, generation accuracy, and policy compliance across any vertical.\n\n## Install\n\n```bash\npip install synapt-eval\n```\n\nOr install from source:\n\n```bash\npip install git+https://github.com/synapt-dev/eval.git\n```\n\n## Quick Start\n\n```python\nimport asyncio\nfrom synapt_eval import Fixture, EvalResult, CategoryMetrics\nfrom synapt_eval.adapters import RetrievalAdapter, RetrievalCandidate\nfrom synapt_eval.scoring import precision_at_k, recall_at_k\nfrom synapt_eval.report_card import compose_report_card, generate_markdown\n\nclass MyRetrieval(RetrievalAdapter):\n    async def retrieve(self, query: str, k: int = 10) -\u003e list[RetrievalCandidate]:\n        # Connect your vector store here\n        return [RetrievalCandidate(id=\"doc1\", score=0.95)]\n\n# Run eval and generate report\nresults = [EvalResult(\n    category=\"retrieval\",\n    metrics=CategoryMetrics(p_at_5=0.85, r_at_10=0.72, n=50),\n)]\ncard = compose_report_card(results, run_id=\"my-first-eval\")\nprint(generate_markdown(card))\n```\n\nSee [docs/quickstart.md](docs/quickstart.md) for a complete walkthrough and [examples/](examples/) for runnable code.\n\n## Architecture\n\nsynapt-eval separates the **eval framework** (scoring, review, reporting) from **domain-specific adapters** (your retrieval backend, your generation pipeline, your fixtures).\n\n```\nLayer              Module                      Purpose\n-------            ------                      -------\nTypes              synapt_eval.types           Core data types (Fixture, EvalResult, CategoryMetrics)\nScoring            synapt_eval.scoring         Precision@K, Recall@K, Kendall's Tau\nAdapters           synapt_eval.adapters        Customer-facing ABCs (Retrieval, Generation, Judge, Fixture)\nRunner             synapt_eval.runner          Eval execution, orchestration, PR gate\nReviewer           synapt_eval.reviewer        Verdict framework, predicate chains, LLM judge bridge\nSuggestion Engine  synapt_eval.suggestion_engine  Rule-based actionable recommendations\nReport Card        synapt_eval.report_card     Markdown + JSON report generation\nTrending           synapt_eval.trending        Self-hosted JSON history store + delta computation\nCLI                synapt_eval.cli             Command-line viewer (synapt-eval trending)\nActions            synapt_eval.actions         GitHub Actions PR-gate adapter\n```\n\n## Features\n\n| Feature | Description |\n|---------|-------------|\n| **Scoring primitives** | Precision@K, Recall@K, Kendall's Tau rank correlation |\n| **Adapter pattern** | Plug in any retrieval/generation backend via ABCs |\n| **Reviewer SDK** | Composable predicate chains + LLM judge integration |\n| **Suggestion engine** | 10 baseline rules with decorator pattern for custom rules |\n| **Report card** | Markdown + JSON output with schema versioning |\n| **PR gate** | Regression detection with configurable thresholds |\n| **Trending** | Self-hosted history store with CLI viewer |\n| **GitHub Action** | `uses: synapt-dev/eval@v0.1.0` for CI integration |\n\n## GitHub Action\n\nAdd eval gating to your PR workflow:\n\n```yaml\n- name: Run eval\n  run: python my_eval_script.py --output results.json\n\n- name: PR Gate\n  uses: synapt-dev/eval@v0.1.0\n  with:\n    results-path: results.json\n    baseline-path: baseline.json\n    threshold: \"0.05\"\n    fail-on: error\n```\n\nThe action posts a report card comment on the PR and fails the workflow on regressions. See [docs/pr-gate.md](docs/pr-gate.md) for full configuration.\n\n## CLI\n\n```bash\n# View eval trending history\nsynapt-eval trending --path .synapt-eval/history --format text\n\n# Output as markdown or JSON\nsynapt-eval trending --format markdown\nsynapt-eval trending --format json --limit 5\n```\n\n## Documentation\n\n| Guide | Description |\n|-------|-------------|\n| [Quickstart](docs/quickstart.md) | End-to-end retrieval eval in 60 lines |\n| [Adapter API](docs/adapter-api.md) | Writing custom adapters |\n| [Reviewer Framework](docs/reviewer-framework.md) | Custom reviewers + judge integration |\n| [PR Gate](docs/pr-gate.md) | GitHub Actions CI integration |\n| [Suggestions](docs/suggestions.md) | Writing custom suggestion rules |\n| [Trending](docs/trending.md) | Self-hosted trending CLI |\n\n## Examples\n\nRunnable examples in [examples/](examples/):\n\n- **[retrieval-eval](examples/retrieval-eval/)** -- mock retrieval backend + fixtures + report card\n- **[generation-eval](examples/generation-eval/)** -- mock generation pipeline + judge\n- **[full-pipeline](examples/full-pipeline/)** -- combined retrieval + generation + reviewer + suggestions\n\n## Pro Tier\n\nWant vertical-specific eval packs, a hosted dashboard, or SOC2 attestations? Visit [synapt.dev](https://synapt.dev) for synapt-eval Pro.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsynapt-dev%2Feval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsynapt-dev%2Feval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsynapt-dev%2Feval/lists"}