{"id":30459665,"url":"https://github.com/geminimir/promptproof-demo-project","last_synced_at":"2026-04-07T09:31:12.386Z","repository":{"id":309456018,"uuid":"1035970931","full_name":"geminimir/promptproof-demo-project","owner":"geminimir","description":"Support \u0026 RAG Assistant demo using PromptProof SDK + Action. Records LLM interactions to fixtures and validates contracts (PII, schema, budgets, disclaimers) locally and in GitHub Actions.","archived":false,"fork":false,"pushed_at":"2025-08-16T17:18:37.000Z","size":87,"stargazers_count":3,"open_issues_count":4,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-23T21:49:05.214Z","etag":null,"topics":["demo","express","fixtures","github-actions","llm","node","prompt-engineering","rag","testing"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geminimir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-11T11:17:22.000Z","updated_at":"2025-08-16T17:18:40.000Z","dependencies_parsed_at":"2025-08-12T02:36:08.429Z","dependency_job_id":"b4e39056-f187-4d7f-b64e-943045fd46d0","html_url":"https://github.com/geminimir/promptproof-demo-project","commit_stats":null,"previous_names":["geminimir/promptproof-demo-project"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/geminimir/promptproof-demo-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geminimir%2Fpromptproof-demo-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geminimir%2Fpromptproof-demo-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geminimir%2Fpromptproof-demo-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geminimir%2Fpromptproof-demo-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geminimir","download_url":"https://codeload.github.com/geminimir/promptproof-demo-project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geminimir%2Fpromptproof-demo-project/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31507916,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T03:10:19.677Z","status":"ssl_error","status_checked_at":"2026-04-07T03:10:13.982Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["demo","express","fixtures","github-actions","llm","node","prompt-engineering","rag","testing"],"created_at":"2025-08-23T18:48:53.040Z","updated_at":"2026-04-07T09:31:12.381Z","avatar_url":"https://github.com/geminimir.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PromptProof Demo - Support \u0026 RAG Assistant\n\nA demonstration application showcasing PromptProof's capabilities to catch common LLM failure modes.\n\n[![CI](https://img.shields.io/github/actions/workflow/status/geminimir/promptproof-demo-project/promptproof.yml?branch=main)](https://github.com/geminimir/promptproof-demo-project/actions)\n[![Action](https://img.shields.io/badge/Marketplace-promptproof--action-blue?logo=github)](https://github.com/marketplace/actions/promptproof-eval)\n\n\u003e Website: https://promptproof.io\n\n## Features\n\n- **Support Reply Endpoint**: Generates support responses with required disclaimers and PII protection\n- **RAG Answer Endpoint**: Provides document-based Q\u0026A with proper citations\n- **Tool Calling**: Calendar event scheduling with argument validation\n- **Multi-language Support**: English and French locales\n\n## See it in action\n\n- Regression fail PR · Cost gate PR · Assertion fail PR\n  \n  [Links to live PRs and GIFs to be inserted after publishing]\n\n## Setup\n\n1. Install dependencies (includes PromptProof SDK and CLI):\n```bash\nnpm install\n```\n\n2. Set up environment variables:\n```bash\ncp .env.example .env\n# Add your OpenAI API key\n```\n\n3. Run the development server:\n```bash\nnpm run dev\n```\n\n## PromptProof Integration\n\nThis demo showcases the complete PromptProof workflow using the official npm packages and GitHub Action:\n\n### **SDK Integration** ✅\n- **`promptproof-sdk-node@beta`**: Automatically records LLM interactions as fixtures\n- **One-line integration**: `withPromptProofOpenAI()` wrapper\n- **Automatic recording**: All LLM calls recorded to `fixtures/support-replies/outputs.jsonl`\n\n### **CLI Integration** ✅\n- **`promptproof-cli@beta`**: Validates fixtures against contracts\n- **NPM scripts**: `npm run test:promptproof` for easy validation\n- **Multiple formats**: Console, HTML, JSON reports\n- **New**: `--regress`, `--seed`, `--runs` flags; `snapshot` command for baselines\n\n### **GitHub Action** ✅\nAlready included as `.github/workflows/promptproof.yml` (fixtures-only, no secrets). It runs in `report-only` so forks stay green, and can be made required via branch rules.\n```yaml\nname: PromptProof (demo)\non: [pull_request]\npermissions:\n  contents: read\n  pull-requests: write\njobs:\n  eval:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: geminimir/promptproof-action@v0\n        with:\n          config: promptproof.yaml\n          format: html\n          mode: report-only\n      - name: Create snapshot on success\n        if: github.ref == 'refs/heads/main' \u0026\u0026 success()\n        run: |\n          npx promptproof snapshot promptproof.yaml --promote\n```\n\n### **Environment Variables**\n```bash\nPP_RECORD=1              # Enable recording (default: 1 in dev, 0 in prod)\nPP_SUITE=support-replies # Fixture suite name\nPP_SAMPLE_RATE=1.0       # Record 100% of calls\n```\n\n### **Complete Workflow**\n1. **SDK records** LLM interactions automatically\n2. **CLI validates** fixtures against contracts in `promptproof.yaml`\n3. **CI/CD integration** via GitHub Actions\n4. **Violation detection** prevents regressions\n\n## Endpoints\n\n### POST /support/reply\nGenerates a support response with PII protection and required disclaimers.\n\n```bash\ncurl -X POST localhost:3000/support/reply \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"email\":\"user@example.com\",\"locale\":\"en\",\"message\":\"How do I get a refund?\"}'\n```\n\n### POST /rag/answer\nProvides document-based answers with proper citations.\n\n```bash\ncurl -X POST localhost:3000/rag/answer \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"locale\":\"en\",\"question\":\"What is your refund policy?\"}'\n```\n\n## Testing with PromptProof\n\nRun contract validation:\n```bash\n# Using npm script\nnpm run test:promptproof\n\n# Or directly with CLI\nnpx promptproof eval -c promptproof.yaml\n\n# Generate HTML report\nnpm run test:promptproof:html\n\n# Compare against baseline (if snapshot exists)\nnpx promptproof eval -c promptproof.yaml --regress\n\n# Create a baseline snapshot after green runs\nnpx promptproof snapshot promptproof.yaml --promote\n```\n\n## Demo Value\n\nThis demo proves PromptProof's effectiveness by showcasing:\n\n### **Real-World Failure Modes**\n1. **PII Leakage**: Email/phone numbers in responses\n2. **Schema Drift**: Invalid JSON structure from extractors\n3. **Missing Disclaimers**: Required legal text omitted\n4. **Bogus Citations**: Fake or missing source URLs\n5. **Tool Argument Errors**: Invalid calendar event parameters\n6. **Cost/Latency Budgets**: Excessive token usage or response time\n7. **Multilingual Regressions**: Inconsistent behavior across locales\n\n### **Red → Green Demonstrations**\n- **Red**: Fixtures with intentional failures → Action goes red with HTML report\n- **Green**: Quick fix → re-run → green check\n\n## One-click template\n\nThis repository is designed to be a template. Enable \"Template repository\" in Settings, then click \"Use this template\" to spin up a new project with PromptProof pre-wired. The default workflow uses fixtures-only and report-only so your first PR is green without any secrets.\n\n### **Production-Ready Integration**\n- **Zero network calls in CI**: Tests run on recorded fixtures\n- **Deterministic**: Same input = same output, no flaky tests\n- **Privacy-safe**: Built-in PII redaction\n- **Cost-effective**: No API costs during validation\n\n\u003e Automated PR test: 2025-08-16 17:17:26Z\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeminimir%2Fpromptproof-demo-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeminimir%2Fpromptproof-demo-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeminimir%2Fpromptproof-demo-project/lists"}