{"id":48355051,"url":"https://github.com/rylinjames/litmus","last_synced_at":"2026-04-21T01:01:21.112Z","repository":{"id":346973907,"uuid":"1191345574","full_name":"rylinjames/litmus","owner":"rylinjames","description":"Record and deterministically replay AI agent executions. Flight recorder for LLM agents. Fault injection, reliability scoring, CI gating.","archived":false,"fork":false,"pushed_at":"2026-03-26T07:07:31.000Z","size":981,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-27T02:32:25.224Z","etag":null,"topics":["ai-agents","cli","developer-tools","fault-injection","llm","observability","python","reliability","replay","testing"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/litmus-trace/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rylinjames.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-25T06:35:27.000Z","updated_at":"2026-03-26T07:07:34.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rylinjames/litmus","commit_stats":null,"previous_names":["rylinjames/litmus"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/rylinjames/litmus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rylinjames%2Flitmus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rylinjames%2Flitmus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rylinjames%2Flitmus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rylinjames%2Flitmus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rylinjames","download_url":"https://codeload.github.com/rylinjames/litmus/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rylinjames%2Flitmus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32072323,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T21:26:33.338Z","status":"ssl_error","status_checked_at":"2026-04-20T21:26:22.081Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","cli","developer-tools","fault-injection","llm","observability","python","reliability","replay","testing"],"created_at":"2026-04-05T11:00:25.772Z","updated_at":"2026-04-21T01:01:21.015Z","avatar_url":"https://github.com/rylinjames.png","language":"Python","funding_links":[],"categories":["Software"],"sub_categories":["AI \u0026 LLM Testing"],"readme":"# Litmus\n\n**Record and deterministically replay AI agent executions.**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"demo.gif\" alt=\"Litmus demo — record, replay, fault inject\" width=\"600\"\u003e\n\u003c/p\u003e\n\nLitmus captures every LLM and tool call your agent makes, saving structured trace files you can inspect, share, and replay.\n\n```bash\npip install litmus-trace\n```\n\n## Quick Start — Zero Code Changes\n\n```bash\n# Record your agent (wraps the process, captures all LLM calls)\nlitmus run python my_agent.py\n\n# View the trace\nlitmus view ./traces/lt-abc123.trace.json\n```\n\nYour agent code stays completely unchanged. Litmus patches the SDK transport layer at runtime.\n\n## What It Does\n\n### Free (works offline, no account needed)\n\n**Record** — Intercepts every HTTP call to LLM APIs (Anthropic, OpenAI, Mistral, 14+ providers). Saves the full request and response as a trace file. API keys are automatically redacted.\n\n**View** — Pretty-print traces with step-by-step details, latency, and model info.\n\n### Coming Soon (Litmus Cloud)\n\n**Replay** — Feed recorded responses back to your agent. Same code path, same output, no real API calls.\n\n**Fault Injection** — Mutate recorded responses to test resilience. What happens when Claude refuses? When GPT returns a 500? When the API times out?\n\n**CI Gating** — Score your trace corpus for reliability and block deploys that drop below a threshold.\n\nJoin the [Discord](https://discord.gg/fA2SHvHb2D) to get notified when these features launch.\n\n## Three Ways to Record\n\n### 1. CLI Wrapper (recommended — zero code changes)\n\n```bash\nlitmus run python my_agent.py\n```\n\n### 2. One-Line Python API\n\n```python\nimport litmus\n\nlitmus.record()\n# ... your existing agent code, unchanged ...\nlitmus.stop()\n```\n\n### 3. Proxy Mode (any language, advanced use)\n\n```bash\nlitmus proxy --mode record\n# Then point your SDK:\nANTHROPIC_BASE_URL=http://localhost:8787/anthropic python my_agent.py\n```\n\n## Supported Providers\n\nWorks with any LLM API out of the box:\n\n| Provider | Status |\n|----------|--------|\n| Anthropic (Claude) | Tested |\n| OpenAI (GPT) | Tested |\n| Google (Gemini) | Supported |\n| Mistral | Supported |\n| Cohere | Supported |\n| Groq | Supported |\n| Together AI | Supported |\n| Fireworks AI | Supported |\n| DeepSeek | Supported |\n| Perplexity | Supported |\n| OpenRouter | Supported |\n| Ollama (local) | Supported |\n| vLLM (local) | Supported |\n| LM Studio (local) | Supported |\n\n**Custom/self-hosted models:**\n\n```bash\nlitmus proxy --provider my-model=https://my-finetuned-llama.example.com/v1\n```\n\n## CLI Reference\n\n```\nlitmus run          Wrap a command to record (zero code changes)\nlitmus view         Pretty-print a trace file\nlitmus proxy        Start the recording proxy server\nlitmus providers    List all supported providers\nlitmus replay       Replay a trace (coming soon — requires Litmus Cloud)\nlitmus ci           Score traces and gate deploys (coming soon — requires Litmus Cloud)\n```\n\n## How It Works\n\nLitmus monkey-patches the `httpx` transport layer used by both Anthropic and OpenAI Python SDKs. When you call `client.messages.create(...)`, Litmus intercepts the HTTP request before it leaves your machine.\n\n**Record mode:** The real API call goes through. Litmus captures the request and response, then saves them to a trace file. API keys are automatically redacted.\n\n**Replay mode:** The real API is never called. Litmus serves the recorded response directly from the trace file. Your agent gets the exact same response it got during recording — same tool calls, same content, same stop reason.\n\n## Security\n\n- API keys (`Authorization`, `x-api-key`) are **automatically redacted** from trace headers\n- Use `--compact` to strip request bodies for smaller trace files\n- Note: message content in request/response bodies is NOT redacted — don't include secrets in your prompts\n\n## Limitations\n\n- **Python only** — the monkey-patch approach (`litmus run`, `litmus.record()`) requires Python. Use proxy mode for other languages.\n- **httpx-based SDKs** — works with SDKs that use `httpx` under the hood (Anthropic, OpenAI, Mistral, Cohere, etc). SDKs using `requests` or `aiohttp` are not intercepted.\n- **Sequential replay** — responses are served in recorded order. Agents that make calls in a different order on replay will get mismatched responses.\n- **No tool call recording** — only LLM API calls are captured. External tool calls (database, HTTP APIs) are not recorded.\n\n## Community\n\n- [Discord](https://discord.gg/Nmr6tBx4xQ) — fastest way to get help, share traces, and request features\n- [GitHub Issues](https://github.com/rylinjames/litmus/issues) — bug reports and feature requests\n- [PyPI](https://pypi.org/project/litmus-trace/) — package\n\n## Talk to Me\n\nI'm building Litmus in the open and I want to hear from you — whether it's a bug, a feature idea, or just telling me about your agent setup. I personally respond to everything.\n\n- **Email:** romirj@gmail.com\n- **Discord:** romirj ([join the server](https://discord.gg/Nmr6tBx4xQ))\n- **Twitter/X:** [@romir_jain](https://twitter.com/romir_jain)\n\nIf you're running agents in production and want to use Litmus, I'll personally help you set it up. DM me anywhere.\n\n## Why Litmus?\n\n**Observability tools** (LangSmith, Langfuse) tell you what happened. They log traces.\n\n**Litmus captures the full picture.** Every LLM call, every response, every token — in a structured trace file you can inspect, share, and (soon) replay deterministically with fault injection.\n\nLangSmith is the dashcam. Litmus is building the crash test facility.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frylinjames%2Flitmus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frylinjames%2Flitmus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frylinjames%2Flitmus/lists"}