{"id":39654693,"url":"https://github.com/deepankarm/agent-chaos","last_synced_at":"2026-01-18T09:15:26.072Z","repository":{"id":331357605,"uuid":"1121150892","full_name":"deepankarm/agent-chaos","owner":"deepankarm","description":"Chaos engineering for AI agents","archived":false,"fork":false,"pushed_at":"2026-01-02T10:44:57.000Z","size":1658,"stargazers_count":12,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-06T22:40:45.853Z","etag":null,"topics":["agents","chaos-engineering","llm","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepankarm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-22T14:16:14.000Z","updated_at":"2026-01-06T06:32:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/deepankarm/agent-chaos","commit_stats":null,"previous_names":["deepankarm/agent-chaos"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/deepankarm/agent-chaos","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepankarm%2Fagent-chaos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepankarm%2Fagent-chaos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepankarm%2Fagent-chaos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepankarm%2Fagent-chaos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepankarm","download_url":"https://codeload.github.com/deepankarm/agent-chaos/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepankarm%2Fagent-chaos/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28534156,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","chaos-engineering","llm","testing"],"created_at":"2026-01-18T09:15:25.380Z","updated_at":"2026-01-18T09:15:26.064Z","avatar_url":"https://github.com/deepankarm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# agent-chaos\n\n\u003cimg src=\"src/agent_chaos/ui/static/favicon.svg\" width=\"150\" height=\"150\" alt=\"agent-chaos logo\"\u003e\n\n[![CI](https://github.com/deepankarm/agent-chaos/actions/workflows/ci.yml/badge.svg)](https://github.com/deepankarm/agent-chaos/actions/workflows/ci.yml)\n[![codecov](https://codecov.io/gh/deepankarm/agent-chaos/graph/badge.svg)](https://codecov.io/gh/deepankarm/agent-chaos)\n[![PyPI version](https://badge.fury.io/py/agent-chaos.svg)](https://pypi.org/project/agent-chaos/)\n\n**Chaos engineering for AI agents.**\n\n\u003e *\"Introduce a little anarchy. Upset the established order, and everything becomes chaos. I'm an agent of chaos. Oh, and you know the thing about chaos? It's fair!\"*\n\n\u003c/div\u003e\n\n---\n\nYour agent works in demos. It passes evals. Then it hits production: the LLM sends a 500, the tool returns garbage, the stream cuts mid-response. The agent fails silently, returns wrong answers, or loops forever.\n\n**agent-chaos** breaks your agent on purpose, before production does. For teams building agents for production, not demos.\n\n```bash\npip install agent-chaos\n```\n\n---\n\n## Why does this exist?\n\nLLM APIs are unreliable. They claim certain rate limits, then behave differently. They accept a stream request, then start sending tokens 10 seconds later. They reject mid-stream. They hang for 20 seconds before returning a 500. We've seen providers return `\"Sorry about that\"` as an error message.\n\nProduction agent backends run multiple LLMs with retry and fallback because things break randomly. What worked last week might not work today. You often don't realize it until production.\n\nBut the chaos isn't just at the transport layer. There's a semantic layer that's harder to catch.\n\nTools fail in obvious ways (timeouts, errors), but also in subtle ways: empty responses, partial data, wrong data types, malformed JSON, stale information, or data for the wrong entity entirely. A tool might return a 200 OK with an error message buried in the response body. An LLM-backed tool might hallucinate. With MCP, your agent calls tools you don't control, with schemas that can change without notice.\n\nTraditional chaos engineering tools (Chaos Monkey, Gremlin) operate at infrastructure: network partitions, pod failures. They can't corrupt a tool result or cut an LLM stream mid-response.\n\n**agent-chaos** injects these failures so you can test how your agent handles them before users find out. It integrates with LLM-as-judge evaluation frameworks like [DeepEval](https://github.com/confident-ai/deepeval) and [Pydantic Evals](https://ai.pydantic.dev/evals/), so you can inject chaos and judge the quality of your agent's response.\n\n---\n\n## Core concepts\n\n### Scenarios: baseline + variants\n\nA baseline scenario defines a conversation with your agent. A variant adds chaos:\n\n```python\nfrom agent_chaos import BaselineScenario, Turn\nfrom agent_chaos.chaos import llm_rate_limit, tool_error\n\n# Baseline: happy path\nbaseline = BaselineScenario(\n    name=\"order-inquiry\",\n    agent=my_agent,\n    turns=[\n        Turn(\"What's the status of order #123?\"),\n        Turn(\"Can I get a refund?\"),\n    ],\n)\n\n# Variant: what happens when the LLM rate-limits?\nbaseline.variant(\n    name=\"llm-rate-limit\",\n    chaos=[llm_rate_limit().after_calls(1)],\n)\n\n# Variant: what happens when the refund API fails?\nbaseline.variant(\n    name=\"refund-api-down\",\n    chaos=[tool_error(\"Service unavailable\").for_tool(\"check_refund\")],\n)\n```\n\n### Chaos and assertions\n\nagent-chaos provides chaos injectors for LLM failures (`llm_rate_limit`, `llm_server_error`, `llm_timeout`), tool failures (`tool_error`, `tool_timeout`), data corruption (`tool_mutate`), and more. These are composable and support targeting specific tools, turns, or call counts.\n\nBuilt-in assertions include `MaxTotalLLMCalls`, `AllTurnsComplete`, `TokenBurstDetection`, among others. For semantic evaluation, agent-chaos integrates with [DeepEval](https://github.com/confident-ai/deepeval) and [Pydantic Evals](https://ai.pydantic.dev/evals/), letting you use LLM-as-judge metrics as assertions.\n\nBoth chaos and assertions can be applied per-scenario or per-turn using the `at()` helper:\n\n```python\nfrom agent_chaos import at\nfrom agent_chaos.chaos import tool_error\nfrom agent_chaos.scenario import CompletesWithin\n\n# With DeepEval\nfrom agent_chaos.integrations.deepeval import as_assertion\nfrom deepeval.metrics import GEval\n\nbaseline.variant(\n    name=\"check-refund-fails\",\n    turns=[\n        at(\n            2,\n            chaos=[tool_error(\"Service unavailable\").for_tool(\"check_refund\")],\n            assertions=[\n                CompletesWithin(60.0),\n                as_assertion(GEval(name=\"task-completion\", criteria=\"Did the agent complete the task?\")),\n            ],\n        ),\n    ],\n)\n\n# Or with Pydantic Evals\nfrom agent_chaos.integrations.pydantic_evals import as_assertion\nfrom pydantic_evals.evaluators import LLMJudge\n\nbaseline.variant(\n    name=\"check-refund-fails-pydantic\",\n    turns=[\n        at(\n            2,\n            chaos=[tool_error(\"Service unavailable\").for_tool(\"check_refund\")],\n            assertions=[\n                CompletesWithin(60.0),\n                as_assertion(LLMJudge(rubric=\"Did the agent complete the task?\"), threshold=0.7),\n            ],\n        ),\n    ],\n)\n```\n\n---\n\n## Fuzzing\n\nIt's difficult to define every failure mode upfront. `fuzz_chaos` generates random chaos combinations based on a `ChaosSpace` configuration, so you can explore how your agent behaves under varied conditions.\n\n```python\nfrom agent_chaos import fuzz_chaos, ChaosSpace, LLMFuzzConfig, ToolFuzzConfig\n\nvariants = fuzz_chaos(\n    baseline, \n    n=10, \n    space=ChaosSpace(\n        llm=LLMFuzzConfig(probability=0.3),\n        tool=ToolFuzzConfig(probability=0.5, targets=[\"get_order\", \"process_refund\"]),\n    ),\n)\n```\n\nFuzzing is for exploration, not CI. See [`examples/ecommerce-support-agent/scenarios/fuzzing.py`](examples/ecommerce-support-agent/scenarios/fuzzing.py) for more.\n\n---\n\n## Examples\n\nThe [`examples/ecommerce-support-agent/`](examples/ecommerce-support-agent/) directory contains a complete example with an e-commerce support agent built with `pydantic-ai`, including:\n\n- [`scenarios/quickstart.py`](examples/ecommerce-support-agent/scenarios/quickstart.py) - baseline scenarios with chaos variants\n- [`scenarios/resilience.py`](examples/ecommerce-support-agent/scenarios/resilience.py) - comprehensive resilience testing\n- [`scenarios/fuzzing.py`](examples/ecommerce-support-agent/scenarios/fuzzing.py) - automated, random chaos generation\n\n```bash\ncd examples/ecommerce-support-agent\nuv sync\nuv run agent-chaos run scenarios/quickstart.py\n\n# on another terminal\nuv run agent-chaos ui .agent_chaos_runs\n```\n\n**Scenario overview showing baselines, chaos variants, and assertion results:**\n\n![All scenarios](.github/images/all%20scenarios.png)\n\n**LLM rate limit injected on turn 1. Agent failed to respond, caught by turn-coherence assertion:**\n\n![Rate limit injected](.github/images/rate-limit-injected.png)\n\n**Tool error injected. Agent gracefully handles the failure and offers alternatives:**\n\n![Tool error injected](.github/images/tool-error-injected.png)\n\n---\n\n## Status\n\nUnder active development.\n\n**Supported:**\n- Anthropic models (via `anthropic` SDK)\n- Multi-turn conversations\n- LLM chaos (rate limits, server errors, timeouts, stream cut/hang, slow chunks)\n- Tool chaos (errors, timeouts, mutations)\n- User input chaos (prompt injections)\n- LLM-as-judge assertions via [DeepEval](https://github.com/confident-ai/deepeval) and [Pydantic Evals](https://ai.pydantic.dev/evals/)\n- Scenario fuzzing\n\n**Planned:**\n- OpenAI, Gemini models\n- More chaos types\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepankarm%2Fagent-chaos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepankarm%2Fagent-chaos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepankarm%2Fagent-chaos/lists"}