https://github.com/geminimir/promptproof-demo-project
Support & RAG Assistant demo using PromptProof SDK + Action. Records LLM interactions to fixtures and validates contracts (PII, schema, budgets, disclaimers) locally and in GitHub Actions.
https://github.com/geminimir/promptproof-demo-project
demo express fixtures github-actions llm node prompt-engineering rag testing
Last synced: 3 months ago
JSON representation
Support & RAG Assistant demo using PromptProof SDK + Action. Records LLM interactions to fixtures and validates contracts (PII, schema, budgets, disclaimers) locally and in GitHub Actions.
- Host: GitHub
- URL: https://github.com/geminimir/promptproof-demo-project
- Owner: geminimir
- Created: 2025-08-11T11:17:22.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2025-08-16T17:18:37.000Z (10 months ago)
- Last Synced: 2025-08-23T21:49:05.214Z (10 months ago)
- Topics: demo, express, fixtures, github-actions, llm, node, prompt-engineering, rag, testing
- Language: TypeScript
- Homepage:
- Size: 85 KB
- Stars: 3
- Watchers: 1
- Forks: 1
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# PromptProof Demo - Support & RAG Assistant
A demonstration application showcasing PromptProof's capabilities to catch common LLM failure modes.
[](https://github.com/geminimir/promptproof-demo-project/actions)
[](https://github.com/marketplace/actions/promptproof-eval)
> Website: https://promptproof.io
## Features
- **Support Reply Endpoint**: Generates support responses with required disclaimers and PII protection
- **RAG Answer Endpoint**: Provides document-based Q&A with proper citations
- **Tool Calling**: Calendar event scheduling with argument validation
- **Multi-language Support**: English and French locales
## See it in action
- Regression fail PR · Cost gate PR · Assertion fail PR
[Links to live PRs and GIFs to be inserted after publishing]
## Setup
1. Install dependencies (includes PromptProof SDK and CLI):
```bash
npm install
```
2. Set up environment variables:
```bash
cp .env.example .env
# Add your OpenAI API key
```
3. Run the development server:
```bash
npm run dev
```
## PromptProof Integration
This demo showcases the complete PromptProof workflow using the official npm packages and GitHub Action:
### **SDK Integration** ✅
- **`promptproof-sdk-node@beta`**: Automatically records LLM interactions as fixtures
- **One-line integration**: `withPromptProofOpenAI()` wrapper
- **Automatic recording**: All LLM calls recorded to `fixtures/support-replies/outputs.jsonl`
### **CLI Integration** ✅
- **`promptproof-cli@beta`**: Validates fixtures against contracts
- **NPM scripts**: `npm run test:promptproof` for easy validation
- **Multiple formats**: Console, HTML, JSON reports
- **New**: `--regress`, `--seed`, `--runs` flags; `snapshot` command for baselines
### **GitHub Action** ✅
Already included as `.github/workflows/promptproof.yml` (fixtures-only, no secrets). It runs in `report-only` so forks stay green, and can be made required via branch rules.
```yaml
name: PromptProof (demo)
on: [pull_request]
permissions:
contents: read
pull-requests: write
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: geminimir/promptproof-action@v0
with:
config: promptproof.yaml
format: html
mode: report-only
- name: Create snapshot on success
if: github.ref == 'refs/heads/main' && success()
run: |
npx promptproof snapshot promptproof.yaml --promote
```
### **Environment Variables**
```bash
PP_RECORD=1 # Enable recording (default: 1 in dev, 0 in prod)
PP_SUITE=support-replies # Fixture suite name
PP_SAMPLE_RATE=1.0 # Record 100% of calls
```
### **Complete Workflow**
1. **SDK records** LLM interactions automatically
2. **CLI validates** fixtures against contracts in `promptproof.yaml`
3. **CI/CD integration** via GitHub Actions
4. **Violation detection** prevents regressions
## Endpoints
### POST /support/reply
Generates a support response with PII protection and required disclaimers.
```bash
curl -X POST localhost:3000/support/reply \
-H 'Content-Type: application/json' \
-d '{"email":"user@example.com","locale":"en","message":"How do I get a refund?"}'
```
### POST /rag/answer
Provides document-based answers with proper citations.
```bash
curl -X POST localhost:3000/rag/answer \
-H 'Content-Type: application/json' \
-d '{"locale":"en","question":"What is your refund policy?"}'
```
## Testing with PromptProof
Run contract validation:
```bash
# Using npm script
npm run test:promptproof
# Or directly with CLI
npx promptproof eval -c promptproof.yaml
# Generate HTML report
npm run test:promptproof:html
# Compare against baseline (if snapshot exists)
npx promptproof eval -c promptproof.yaml --regress
# Create a baseline snapshot after green runs
npx promptproof snapshot promptproof.yaml --promote
```
## Demo Value
This demo proves PromptProof's effectiveness by showcasing:
### **Real-World Failure Modes**
1. **PII Leakage**: Email/phone numbers in responses
2. **Schema Drift**: Invalid JSON structure from extractors
3. **Missing Disclaimers**: Required legal text omitted
4. **Bogus Citations**: Fake or missing source URLs
5. **Tool Argument Errors**: Invalid calendar event parameters
6. **Cost/Latency Budgets**: Excessive token usage or response time
7. **Multilingual Regressions**: Inconsistent behavior across locales
### **Red → Green Demonstrations**
- **Red**: Fixtures with intentional failures → Action goes red with HTML report
- **Green**: Quick fix → re-run → green check
## One-click template
This repository is designed to be a template. Enable "Template repository" in Settings, then click "Use this template" to spin up a new project with PromptProof pre-wired. The default workflow uses fixtures-only and report-only so your first PR is green without any secrets.
### **Production-Ready Integration**
- **Zero network calls in CI**: Tests run on recorded fixtures
- **Deterministic**: Same input = same output, no flaky tests
- **Privacy-safe**: Built-in PII redaction
- **Cost-effective**: No API costs during validation
> Automated PR test: 2025-08-16 17:17:26Z