{"id":50955658,"url":"https://github.com/shwetank/promptecho","last_synced_at":"2026-06-18T06:30:39.285Z","repository":{"id":363781918,"uuid":"1261423438","full_name":"shwetank/promptecho","owner":"shwetank","description":"Record \u0026 replay for LLM API calls. Like vcrpy / nock, but built for the way LLM traffic actually behaves.","archived":false,"fork":false,"pushed_at":"2026-06-10T09:17:34.000Z","size":115,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-10T11:13:26.330Z","etag":null,"topics":["anthropic","llm","openai","pytest","record-replay","testing","testing-tools","vcr"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/promptecho/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shwetank.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-06T17:01:45.000Z","updated_at":"2026-06-10T09:17:38.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/shwetank/promptecho","commit_stats":null,"previous_names":["shwetank/promptecho"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/shwetank/promptecho","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shwetank%2Fpromptecho","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shwetank%2Fpromptecho/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shwetank%2Fpromptecho/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shwetank%2Fpromptecho/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shwetank","download_url":"https://codeload.github.com/shwetank/promptecho/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shwetank%2Fpromptecho/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34479552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-18T02:00:06.871Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic","llm","openai","pytest","record-replay","testing","testing-tools","vcr"],"created_at":"2026-06-18T06:30:38.240Z","updated_at":"2026-06-18T06:30:39.280Z","avatar_url":"https://github.com/shwetank.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# promptecho\n\n[![CI](https://github.com/shwetank/promptecho/actions/workflows/ci.yml/badge.svg)](https://github.com/shwetank/promptecho/actions/workflows/ci.yml)\n[![PyPI](https://img.shields.io/pypi/v/promptecho)](https://pypi.org/project/promptecho/)\n\n**Record \u0026 replay for LLM API calls.** Like [`vcrpy`](https://github.com/kevin1024/vcrpy) / [`nock`](https://github.com/nock/nock), but built for the way LLM traffic actually behaves.\n\nYour LLM tests have three problems: they're **flaky** (non-deterministic outputs), **slow** (real network round-trips), and **expensive** (burning tokens in CI on every run). promptecho records each real API call once to a cassette file, then replays it forever — deterministically, instantly, for free.\n\n```python\nimport promptecho\nfrom anthropic import Anthropic\n\n@promptecho.use_cassette(\"cassettes/summarize.yaml\")\ndef test_summarize():\n    client = Anthropic()\n    msg = client.messages.create(\n        model=\"claude-opus-4-8\",\n        max_tokens=100,\n        messages=[{\"role\": \"user\", \"content\": \"Summarize: the cat sat on the mat.\"}],\n    )\n    assert \"cat\" in msg.content[0].text.lower()\n```\n\nFirst run: one real call, recorded to `cassettes/summarize.yaml` — this needs the provider SDK installed (`pip install anthropic`) and a real `ANTHROPIC_API_KEY` in the environment.\nEvery run after: replayed from disk. No network, no tokens, no API key, no flake.\n\n\u003e **Proof, not marketing.** The end-to-end test that gates every release records against a local server, **shuts the server down**, then replays. Same response, zero network. If the response can come back with the upstream gone, the cassette is genuinely doing the work — not a partial proxy. See [`tests/test_record_replay.py`](tests/test_record_replay.py).\n\n---\n\n## Why not just use vcrpy?\n\nYou can — at the HTTP layer, vcrpy works on LLM calls today. promptecho exists because LLM traffic breaks vcrpy's assumptions in five specific ways:\n\n1. **Matching.** vcrpy matches on raw request bytes. LLM bodies carry volatile fields (client-injected IDs, reordered tools, whitespace) that change the bytes without changing the *meaning* — so byte-matching misses on replay. promptecho matches on a **normalized fingerprint** of the fields that determine the response, and **canonicalizes across providers**: it knows `content: \"hi\"` equals `content: [{\"type\":\"text\",\"text\":\"hi\"}]`, an Anthropic top-level `system` equals an OpenAI `system`-role message, and an Anthropic `input_schema` tool def equals an OpenAI `function.parameters`. A raw-bytes VCR can't.\n2. **Streaming.** Most LLM calls are SSE streams. promptecho records the event stream and faithfully re-emits it on replay, so `stream=True` and token-by-token iteration work identically against a cassette — including reasoning deltas.\n3. **Binary / multimodal responses.** vcrpy's text-based cassettes silently corrupt raw `image/*` / `audio/*` / `octet-stream` bodies. promptecho detects them by `Content-Type` and base64-encodes them in the cassette, so image-out and audio-out responses round-trip byte-exact.\n4. **Debuggable CI failures.** When a vcrpy cassette miss happens, you get *\"no match\"*. promptecho prints the exact path that changed: `messages[1].content: recorded \"summarize the cat\" / incoming \"summarize the dog\"`. Test failures are actionable, not detective work.\n5. **Secrets.** API keys live in headers on every call. promptecho redacts them by default — a cassette is safe to commit.\n\n## What promptecho is *not*\n\n- **Not a cache.** Replay matching is exact/normalized and deterministic, on purpose. It does **not** semantically match \"different prompt, close enough\" — that would put non-determinism back into the harness you're using to remove it. (A separate opt-in fuzzy mode is on the roadmap as a dev-loop convenience; it will never be the default and never used in CI.)\n- **Not an eval.** It freezes a response so your *surrounding code* is testable. Judging whether the response is *good* is a different tool (see roadmap: `toMatchLLMSnapshot()`).\n\n---\n\n## What it covers\n\npromptecho intercepts at the `httpx` transport layer. **If the SDK uses httpx, promptecho sees the call** — which is almost everything modern.\n\n| You're calling | Covered? |\n|---|---|\n| Anthropic, OpenAI, Mistral, Cohere, `google-genai` SDKs | ✅ |\n| **OpenAI SDK with custom `base_url`** → OpenRouter, Together, Fireworks, Cerebras, Groq, DeepInfra, Perplexity | ✅ |\n| **Self-hosted vLLM / TGI / SGLang / LM Studio / Ollama** (OpenAI-compatible mode) | ✅ |\n| Your **own fine-tune** behind any of the above | ✅ |\n| **Reasoning models** — o1/o3, Claude extended thinking, DeepSeek-R1 | ✅ (incl. `reasoning_effort` / `thinking` in default match-on) |\n| **Multimodal** — base64-in-JSON (vision, Claude image-out, GPT-4o) and raw binary (`image/*`, `audio/*`) | ✅ (byte-exact round-trip) |\n| Bedrock via boto3, HF `InferenceClient`, in-process `transformers` | ❌ (see workarounds in [SUPPORT.md](SUPPORT.md)) |\n\nFull matrix with caveats and workarounds: [**SUPPORT.md**](SUPPORT.md). For practical recipes by scenario (startup / enterprise / research), see [**TUTORIAL.md**](TUTORIAL.md).\n\n### Hosted open-source via the OpenAI SDK\n\nThis is the dominant pattern for non-Anthropic/non-OpenAI usage, and it Just Works:\n\n```python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"https://openrouter.ai/api/v1\", api_key=\"...\")\n\n@promptecho.use_cassette(\"cassettes/openrouter.yaml\")\ndef test_via_openrouter():\n    r = client.chat.completions.create(\n        model=\"meta-llama/llama-3.1-70b-instruct\",\n        messages=[{\"role\": \"user\", \"content\": \"hi\"}],\n    )\n    assert r.choices[0].message.content\n```\n\nDetection falls back to body shape when the host is unknown, so localhost gateways, in-house proxies, and self-hosted vLLM/TGI behave the same way as the brand-name hosts.\n\n---\n\n## Install\n\n```bash\npip install promptecho\n```\n\nRequires Python ≥ 3.9 and `httpx ≥ 0.24`. To work on promptecho itself:\n\n```bash\ngit clone https://github.com/shwetank/promptecho \u0026\u0026 cd promptecho\npip install -e \".[dev]\" \u0026\u0026 pytest\n```\n\n---\n\n## Usage\n\n### Decorator\n```python\n@promptecho.use_cassette(\"cassettes/foo.yaml\")\ndef test_foo(): ...\n```\n\n### Context manager\n```python\nwith promptecho.use_cassette(\"cassettes/foo.yaml\"):\n    client.messages.create(...)\n```\n\n### pytest fixture (auto-named per test)\n```python\ndef test_bar(promptecho_cassette):   # records to cassettes/test_bar.yaml\n    client.messages.create(...)\n```\n\nThe fixture defaults to `mode=\"once\"` locally and `mode=\"none\"` when `CI=true` — so a forgotten recording fails the build instead of making a live call. Configure it per test with the marker:\n\n```python\n@pytest.mark.promptecho(match_on=[\"model\", \"messages\", \"temperature\"], mode=\"new_episodes\")\ndef test_bar(promptecho_cassette): ...\n```\n\n### Record modes\nBorrowed from vcrpy, so the mental model is free:\n\n| mode | absent cassette | present cassette | use for |\n|------|-----------------|------------------|---------|\n| `once` *(default)* | record | replay | normal dev |\n| `none` | **error** | replay | **CI** — guarantees no live calls |\n| `new_episodes` | record | replay + record new | evolving tests |\n| `all` | record | re-record everything | refreshing fixtures |\n\n```python\n@promptecho.use_cassette(\"cassettes/foo.yaml\", mode=\"none\")\n```\n\nPrompts changed and a pile of cassettes went stale? Re-record the whole suite without touching code — the env var overrides every cassette's mode:\n\n```bash\nPROMPTECHO_MODE=all pytest\n```\n\n### Choosing what to match on\n\nDefaults to `[\"model\", \"messages\", \"system\", \"tools\", \"tool_choice\", \"reasoning_effort\", \"reasoning\", \"thinking\"]` — everything that determines the response for a chat-shaped call, including reasoning-model knobs.\n\n```python\n@promptecho.use_cassette(\n    \"cassettes/foo.yaml\",\n    match_on=[\"model\", \"messages\", \"system\", \"temperature\"],  # add temperature\n)\n```\n\nFor non-chat shapes (raw TGI `/generate`, embeddings) you'll want to override, e.g. `match_on=[\"model\", \"input\"]` for an embeddings endpoint. See [SUPPORT.md → Request shapes](SUPPORT.md#request-shapes).\n\n### Async\n\nWorks identically with `httpx.AsyncClient` and the async surfaces of Anthropic / OpenAI / Mistral SDKs — the async transport is patched the same way as sync.\n\n---\n\n## Cassette format\n\nHuman-readable YAML, designed to diff cleanly in PRs:\n\n```yaml\nversion: 2\nmatch_on: [model, messages, system, tools, tool_choice, reasoning_effort, reasoning, thinking]\ninteractions:\n  - request:\n      method: POST\n      url: https://api.anthropic.com/v1/messages\n      match_key: 7d206bed48a0bc0c        # fingerprint of method + URL path + matched fields\n      matched_on: [model, messages, system, tools, tool_choice]\n      body:                              # canonical (provider-normalized) body\n        model: claude-opus-4-8\n        messages:\n          - {role: user, content: \"Summarize: the cat sat on the mat.\"}\n    response:\n      status: 200\n      headers: {content-type: application/json}\n      streaming: false\n      body:\n        content: [{type: text, text: \"A cat sat on a mat.\"}]\n        usage: {input_tokens: 14, output_tokens: 8}\n```\n\n- **Streamed** responses store the ordered SSE events under `response.events` with `streaming: true`; replay re-emits them in order.\n- **Binary** responses (image/audio/octet-stream) get `binary: true` and the body is base64-encoded; replay decodes and returns the original bytes.\n- **The stored body is the canonical, provider-normalized shape** — not the raw provider JSON. That makes cassettes provider-agnostic and easier to skim in code review.\n\nAuto-redacted on record: the `authorization`, `x-api-key`, `openai-organization`, and `set-cookie` headers, plus **every URL query-string value** (query-param auth like `?key=…` never reaches disk). Configurable. Secrets *inside prompt text* are not auto-detected — don't put credentials in prompts.\n\nSee [`examples/cassettes/example.yaml`](examples/cassettes/example.yaml) for a real one.\n\n---\n\n## Status\n\n**Pre-1.0, working core** — on [PyPI](https://pypi.org/project/promptecho/), CI-tested on Python 3.9–3.13 (see badge for the current state; [CHANGELOG](CHANGELOG.md) for what's changed).\n\nRecords and replays real httpx traffic — sync, async, SSE streaming, binary responses, cross-provider request shapes — verified end-to-end against a local server that gets shut down between record and replay. Pre-1.0 means the API can still change; breaking changes are flagged in the changelog.\n\n### Roadmap (build-in-public)\n\nDone:\n- [x] httpx sync + async transport interception\n- [x] SSE streaming record/replay\n- [x] pytest plugin + auto-naming\n- [x] Per-provider request normalizers (Anthropic / OpenAI / generic)\n- [x] Reasoning-model match defaults (`reasoning_effort`, `thinking`, `reasoning`)\n- [x] Binary response round-trip (image/audio/octet-stream — base64 in cassette)\n- [x] Field-level diff on cassette miss (CI `mode=none` errors pinpoint the changed path, not just the field name)\n- [x] `on_record_error` policy (`warn` / `raise` / `record`) — prevents silently baking transient 4xx/5xx into cassettes\n- [x] Cassette format v2 — method + URL path in the match key; non-JSON bodies keyed by raw-byte hash (no silent collisions)\n- [x] Secret-safe cassettes — header **and** URL query-string redaction\n- [x] `PROMPTECHO_MODE=all pytest` suite-wide re-record; `@pytest.mark.promptecho` fixture config\n\nNext:\n- [ ] `requests` / `urllib3` interception backend — unlocks boto3-Bedrock and HF `InferenceClient`\n- [ ] `promptecho lint` — find un-recorded calls in a test suite\n- [ ] **`toMatchLLMSnapshot()` sibling** — semantic snapshot assertions on top of recorded calls\n\n## FAQ\n\n### \"If you replay a frozen response, aren't you testing nothing? The model is the risky part.\"\n\nYou're testing everything *except* the model — which is most of your code: response parsing, tool-call dispatch, streaming UI rendering, retry/fallback logic, prompt construction (a changed prompt is a cassette miss, so drift gets caught, not masked). That layer is deterministic and belongs in fast, free CI. Judging whether the model's *output is good* is an eval — a genuinely different job, run on a different cadence with a different budget (see deepeval, promptfoo, braintrust). You need both; promptecho is deliberately only the first. The roadmap `toMatchLLMSnapshot()` is the bridge between them.\n\n### Why does `CassetteMiss` inherit from `BaseException`?\n\nBecause the OpenAI / Anthropic / Mistral SDKs all wrap any `Exception` raised inside their transport into their own connection-error type (`openai.APIConnectionError(\"Connection error.\")`), which would bury the field-level diff — the most useful thing promptecho produces — under a generic message at the top of your pytest failure. Inheriting from `BaseException` (the same trick `pytest.fail`'s internal exception uses) lets the diagnostic pass through `except Exception:` blocks intact. The trade-off is deliberate: your own `except Exception:` won't catch it either — but a test-fixture failure should never be silently swallowed. `except CassetteMiss:` and `pytest.raises(CassetteMiss)` both still work. Full rationale in [DESIGN.md](DESIGN.md).\n\n### Can I run cassettes concurrently?\n\nOne cassette at a time per process — promptecho patches httpx process-wide, and a nested or concurrent `use_cassette` raises `RuntimeError` immediately rather than interleaving recordings. `pytest-xdist` is fine (workers are separate processes). Note that while a cassette is active it intercepts **all** httpx traffic in the process, not just LLM calls.\n\n## Design\n\nFor the why-not-the-other-way decisions — fingerprint vs raw bytes, why semantic matching is fenced off, how SSE re-emission works, how cross-provider normalization is structured — see [DESIGN.md](DESIGN.md).\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshwetank%2Fpromptecho","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshwetank%2Fpromptecho","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshwetank%2Fpromptecho/lists"}