{"id":50770462,"url":"https://github.com/chahine-tech/promptci","last_synced_at":"2026-06-11T18:00:59.018Z","repository":{"id":352388912,"uuid":"1214944307","full_name":"Chahine-tech/PromptCI","owner":"Chahine-tech","description":"Pytest for LLM prompts — write tests, detect regressions, run in CI","archived":false,"fork":false,"pushed_at":"2026-04-20T10:47:38.000Z","size":50,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-20T12:38:55.986Z","etag":null,"topics":["ci","prompt-engineering","pytest","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Chahine-tech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-19T09:07:24.000Z","updated_at":"2026-04-20T10:47:14.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Chahine-tech/PromptCI","commit_stats":null,"previous_names":["chahine-tech/check","chahine-tech/promptci"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/Chahine-tech/PromptCI","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Chahine-tech%2FPromptCI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Chahine-tech%2FPromptCI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Chahine-tech%2FPromptCI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Chahine-tech%2FPromptCI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Chahine-tech","download_url":"https://codeload.github.com/Chahine-tech/PromptCI/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Chahine-tech%2FPromptCI/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34211067,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","prompt-engineering","pytest","testing"],"created_at":"2026-06-11T18:00:57.186Z","updated_at":"2026-06-11T18:00:59.011Z","avatar_url":"https://github.com/Chahine-tech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pytest-prompts\n\n\u003e **pytest for LLM prompts.** Write tests, detect regressions, run in CI.\n\n```python\nfrom pytest_prompts import prompt_test\n\n@prompt_test()\ndef test_summary_is_concise(runner):\n    result = runner.run(\n        prompt=\"prompts/summarizer.txt\",\n        input=\"The Eiffel Tower is a wrought-iron lattice tower in Paris...\",\n    )\n    verdict = runner.judge(result, \"The summary is under 50 words and factually correct\")\n    assert verdict.verdict, verdict.reasoning\n```\n\n```bash\n$ pytest-prompts run\n                                pytest-prompts results\n Test                                           Model             Tokens  Latency  Status\n tests/test_summarizer.py::test_summary...      claude-sonnet-4-6    174    1.2s    PASS\n tests/test_summarizer.py::test_qa_knows...     claude-sonnet-4-6     48    0.9s    PASS\n\n2 passed, 0 failed — 222 tokens total — $0.0009\n```\n\nChange a prompt → `pytest-prompts diff main` → see exactly what regressed.\n\n---\n\n## Install\n\n```bash\nuv add pytest-prompts\nexport ANTHROPIC_API_KEY=sk-ant-...\n```\n\n## Quickstart\n\n```bash\n# 1. Write a test\ncat \u003e tests/test_summarizer.py \u003c\u003c 'EOF'\nfrom pytest_prompts import prompt_test\n\n@prompt_test()\ndef test_summary_is_concise(runner):\n    result = runner.run(\n        prompt=\"prompts/summarizer.txt\",\n        input=\"Long article here...\",\n    )\n    verdict = runner.judge(result, \"The summary is under 50 words\")\n    assert verdict.verdict, verdict.reasoning\nEOF\n\n# 2. Run\npytest-prompts run\n\n# 3. Change your prompt, then detect regressions\npytest-prompts diff main\n```\n\n## Write a test\n\nAny `pytest` file. Decorate with `@prompt_test()`, declare a `runner` fixture, assert on the result.\n\n```python\nfrom pytest_prompts import prompt_test\n\n@prompt_test(model=\"claude-sonnet-4-6\")\ndef test_json_extraction(runner):\n    result = runner.run(\n        prompt=\"prompts/extractor.txt\",\n        input='{\"name\": \"Alice\", \"age\": 30}',\n    )\n    import json\n    data = json.loads(result.output)\n    assert data[\"name\"] == \"Alice\"\n    assert result.tokens_used \u003c 500\n```\n\n`result` exposes `output`, `input_tokens`, `output_tokens`, `tokens_used`, `latency_ms`, `model`, `cost_usd`.\n\n## LLM-as-judge\n\nString matching is fragile — use `runner.judge()` to evaluate outputs semantically:\n\n```python\n@prompt_test()\ndef test_qa_knows_capital_of_france(runner):\n    result = runner.run(prompt=\"prompts/qa.txt\", input=\"What is the capital of France?\")\n    verdict = runner.judge(result, \"The answer correctly identifies Paris as the capital of France\")\n    assert verdict.verdict, verdict.reasoning\n```\n\n`verdict` exposes `verdict` (bool), `reasoning` (one sentence), `criterion`, `input_tokens`, `output_tokens`, `cost_usd`. Judge calls are recorded in snapshots alongside the run result.\n\n## Detect regressions\n\n```bash\n# One command — runs tests on main, then on HEAD, compares automatically\npytest-prompts diff main\n\n# Scope to a specific test directory\npytest-prompts diff main --path tests/prompts/\n```\n\nOutput:\n\n```\npytest-prompts diff\n Test                             main           HEAD           Status\n test_summary_is_concise          ✓ 342t 1.2s    ✓ 891t 3.1s    REGRESSION\n test_qa_knows_capital_of_france  ✓ 48t 0.9s     ✓ 48t 0.8s     ok\n\n❌ REGRESSION  test_summary_is_concise\n   tokens 342 → 891 (+160%)\n```\n\nExit code `1` on any regression. Wire it into CI and you're done.\n\n## CI (GitHub Actions)\n\n```yaml\n- uses: actions/checkout@v4\n  with:\n    fetch-depth: 0  # required for base-ref diff mode\n- uses: chahine-tech/pytest-prompts@v0.1\n  with:\n    path: tests/prompts\n    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}\n    base-ref: main\n    github-token: ${{ secrets.GITHUB_TOKEN }}\n```\n\nOn pull requests, the action runs your tests against both `main` and your branch,\ncompares them, fails the job on regressions, and posts a summary comment on the PR.\n\n| Input                | Default | Description                                                               |\n| -------------------- | ------- | ------------------------------------------------------------------------- |\n| `path`               | `.`     | Test path (file or directory)                                             |\n| `anthropic-api-key`  | —       | **Required.** Anthropic API key                                           |\n| `python-version`     | `3.13`  | Python version                                                            |\n| `base-ref`           |         | Base git ref (e.g. `main`). Tests run on base + head and are compared.    |\n| `threshold`          | `0.05`  | Regression threshold as a fraction (5% by default)                        |\n| `github-token`       |         | When set on `pull_request` events, posts the diff as a PR comment         |\n| `fail-on-regression` | `true`  | Fail the job if a regression is detected                                  |\n\nOutputs: `passed`, `failed`, `total-tokens`, `total-cost-usd`, `regressions`.\n\n## What's included\n\n- `@prompt_test` decorator with `pytest` integration\n- `Runner` for the Anthropic API (Claude Sonnet 4.6 default)\n- `runner.judge()` — LLM-as-judge for semantic assertions\n- `pytest-prompts run` — run tests, summarize tokens/latency/cost\n- `pytest-prompts diff \u003cref\u003e` — run on a git ref + HEAD, detect regressions automatically\n\n**Not here yet:** OpenAI/Gemini adapters, static prompt analysis, HTML reports. Open an issue — priorities come from usage, not a roadmap.\n\n## License\n\nMIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchahine-tech%2Fpromptci","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchahine-tech%2Fpromptci","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchahine-tech%2Fpromptci/lists"}