{"id":49594649,"url":"https://github.com/zer0contextlost/llmtape","last_synced_at":"2026-05-04T03:10:44.181Z","repository":{"id":354465418,"uuid":"1223765607","full_name":"zer0contextlost/llmtape","owner":"zer0contextlost","description":"Record and replay LLM API calls in tests — zero cost, zero flakiness","archived":false,"fork":false,"pushed_at":"2026-04-28T16:37:41.000Z","size":30,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-28T18:27:43.971Z","etag":null,"topics":["anthropic","cassette","developer-tools","llm","openai","python","testing","vcr"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zer0contextlost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-28T16:25:13.000Z","updated_at":"2026-04-28T16:37:46.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/zer0contextlost/llmtape","commit_stats":null,"previous_names":["zer0contextlost/llmtape"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/zer0contextlost/llmtape","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zer0contextlost%2Fllmtape","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zer0contextlost%2Fllmtape/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zer0contextlost%2Fllmtape/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zer0contextlost%2Fllmtape/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zer0contextlost","download_url":"https://codeload.github.com/zer0contextlost/llmtape/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zer0contextlost%2Fllmtape/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32592746,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T22:12:39.696Z","status":"online","status_checked_at":"2026-05-04T02:00:06.625Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic","cassette","developer-tools","llm","openai","python","testing","vcr"],"created_at":"2026-05-04T03:10:43.659Z","updated_at":"2026-05-04T03:10:44.168Z","avatar_url":"https://github.com/zer0contextlost.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# llmtape\n\n**Record and replay LLM API calls in tests — zero cost, zero flakiness.**\n\n```python\nimport llmtape\n\n@llmtape.tape\ndef answer(question: str) -\u003e str:\n    return openai_client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[{\"role\": \"user\", \"content\": question}],\n    )\n```\n\nRecord once. Replay forever. No API calls in CI.\n\n---\n\n## The problem\n\nEvery test run of an LLM app costs money, hits rate limits, takes 2–10s per call, and\nproduces non-deterministic output that makes assertions fragile. Hand-written mocks \"fix\"\nthis but diverge from reality the moment a prompt changes.\n\n## How it works\n\n`@llmtape.tape` intercepts calls to `openai.chat.completions.create` and\n`anthropic.messages.create` made inside the decorated function, keyed by a hash of the\nnormalized request (model, messages, temperature, tools).\n\n**Three modes via `LLMTAPE_MODE`:**\n\n| Mode | Behavior |\n|---|---|\n| `replay` (default) | Return saved cassette. Raise `CassetteNotFoundError` on miss. |\n| `record` | Call through, save cassette. |\n| `record-missing` | Replay if cassette exists, record if not. Use this day-to-day. |\n\n---\n\n## Install\n\n```bash\npip install llmtape\n```\n\n## Quickstart\n\n```python\nimport os\nimport openai\nimport llmtape\n\nclient = openai.OpenAI()\n\n@llmtape.tape\ndef classify(text: str) -\u003e str:\n    resp = client.chat.completions.create(\n        model=\"gpt-4o-mini\",\n        messages=[{\"role\": \"user\", \"content\": f\"Classify as positive or negative: {text}\"}],\n    )\n    return resp.choices[0].message.content\n\n# First run: LLMTAPE_MODE=record python script.py\n# Subsequent runs (CI): python script.py\nresult = classify(\"This movie was fantastic!\")\n```\n\nCassette saved to `.cassettes/classify_\u003cfingerprint\u003e.yaml`.\n\n---\n\n## Cassette format\n\n```yaml\ncassette_version: 1\nprovider: openai\nrequest:\n  fingerprint: sha256:7f5924ae...\n  normalized:\n    messages:\n      - content: |\n          Classify as positive or negative: This movie was fantastic!\n        role: user\n    model: gpt-4o-mini\nresponse:\n  raw:\n    choices:\n      - finish_reason: stop\n        index: 0\n        message:\n          content: positive\n          role: assistant\n          tool_calls: null\n    model: gpt-4o-mini\n    usage:\n      completion_tokens: 1\n      prompt_tokens: 22\n      total_tokens: 23\nmetadata:\n  function_name: classify\n  latency_ms: 412\n  recorded_at: '2026-04-28T14:22:11+00:00'\n  sdk_version: openai==1.55.0\n```\n\nCassettes are human-readable, diffable in PRs, and safe to commit — **review them before\ncommitting** since they contain your prompts and responses (which may include system\nprompts, proprietary instructions, or user data).\n\n---\n\n## Async support\n\nWorks identically with `async def`:\n\n```python\n@llmtape.tape\nasync def answer_async(question: str):\n    return await async_client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[{\"role\": \"user\", \"content\": question}],\n    )\n```\n\n---\n\n## Anthropic support\n\n```python\nimport anthropic\nimport llmtape\n\nclient = anthropic.Anthropic()\n\n@llmtape.tape\ndef summarize(text: str):\n    return client.messages.create(\n        model=\"claude-3-5-haiku-20241022\",\n        max_tokens=256,\n        messages=[{\"role\": \"user\", \"content\": f\"Summarize: {text}\"}],\n    )\n```\n\n---\n\n## Tool calls and structured outputs\n\nFull provider response objects are stored verbatim — tool calls, `finish_reason`,\n`logprobs`, structured outputs all round-trip correctly through the cassette.\n\n---\n\n## Configuration\n\n`pyproject.toml`:\n```toml\n[tool.llmtape]\ncassette_dir = \".cassettes\"   # where cassettes are stored\nmode = \"replay\"               # default mode (overridden by LLMTAPE_MODE env var)\nmax_age_days = 30             # for llmtape check\nredact = [\"authorization\", \"api_key\", \"x-api-key\"]  # keys to redact before saving\n```\n\nOr `.llmtape.toml` with the same keys (without the `[tool.]` wrapper).\n\n---\n\n## CLI\n\n```\nllmtape list                  # list all cassettes: age, function, model, tokens\nllmtape show \u003cname\u003e           # pretty-print a cassette\nllmtape check                 # flag cassettes older than max_age_days\nllmtape delete \u003cpattern\u003e      # delete cassettes by name or glob\n```\n\n---\n\n## GitHub Actions\n\n```yaml\n- name: Run tests\n  run: pytest\n  # No LLMTAPE_MODE set → defaults to replay\n  # Cassettes are committed to the repo\n  # Zero network calls, zero cost, runs in under 1s\n```\n\n---\n\n## Streaming\n\n`stream=True` is not supported in v0. LLMTape raises `TapeStreamingError` if a taped\nfunction makes a streaming call. Use `vcrpy` for HTTP-level recording of streaming\nresponses, or remove `stream=True` from calls inside taped functions.\n\n---\n\n## Compared to vcrpy\n\n`vcrpy` records at the HTTP level — cassettes are HTTP wire format (headers, status codes,\nraw bytes). It works against any HTTP client but cassettes are hard to read for LLM content.\n\nLLMTape records at the function level — cassettes are structured YAML with readable prompts\nand responses. It's provider-agnostic (any client, any transport) and cassettes are\ndesigned to be reviewed in pull requests as LLM content, not as HTTP traffic.\n\nUse `vcrpy` if you need HTTP-level recording or streaming support.\nUse LLMTape if you want cassettes that are readable, reviewable, and structured as LLM calls.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzer0contextlost%2Fllmtape","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzer0contextlost%2Fllmtape","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzer0contextlost%2Fllmtape/lists"}