{"id":51940308,"url":"https://github.com/comet-ml/adversarial-benchmark-agent","last_synced_at":"2026-07-28T18:01:53.701Z","repository":{"id":355402179,"uuid":"1164939366","full_name":"comet-ml/adversarial-benchmark-agent","owner":"comet-ml","description":null,"archived":false,"fork":false,"pushed_at":"2026-02-24T13:36:37.000Z","size":216,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-08T10:41:42.561Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/comet-ml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-23T16:46:54.000Z","updated_at":"2026-02-24T13:36:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/comet-ml/adversarial-benchmark-agent","commit_stats":null,"previous_names":["comet-ml/adversarial-benchmark-agent"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/comet-ml/adversarial-benchmark-agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comet-ml%2Fadversarial-benchmark-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comet-ml%2Fadversarial-benchmark-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comet-ml%2Fadversarial-benchmark-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comet-ml%2Fadversarial-benchmark-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/comet-ml","download_url":"https://codeload.github.com/comet-ml/adversarial-benchmark-agent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comet-ml%2Fadversarial-benchmark-agent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36002533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-28T02:00:06.341Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-28T18:01:52.568Z","updated_at":"2026-07-28T18:01:53.682Z","avatar_url":"https://github.com/comet-ml.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The Babadook Observability Test\n\nAdversarial Benchmark Agent for LLM observability instrumentation tools.\n\nBabadook is a functional multi-step research agent that is intentionally structured to defeat auto-instrumentation. It uses real frameworks and makes real LLM calls, but every integration is hidden behind layers of indirection.\n\n## What It Does\n\nTakes a query, searches for information (via TypeScript embeddings), summarizes findings (via LangChain), optionally runs a multi-step reasoning flow (via LangGraph), optionally fact-checks (via CrewAI), and synthesizes a final answer. It can also use Google ADK for grounded search.\n\n## Anti-Instrumentation Patterns\n\n| # | Pattern | Where | Why It's Hard |\n|---|---------|-------|---------------|\n| 1 | Dynamic imports (`__import__`, `importlib`) | All providers, all tools | Import scanning finds nothing |\n| 2 | Factory/registry with `__init_subclass__` | `core/_registry.py`, `providers/_base.py` | No direct `OpenAI()` or `Anthropic()` calls visible |\n| 3 | Mixed Python + TypeScript | `tools/search.ts` called via `tools/search_bridge.py` | LLM calls cross a subprocess boundary |\n| 4 | Misleading filenames | `core/agent.py` = data models, `core/models.py` = orchestrator | Static analysis follows wrong file |\n| 5 | Fake instrumentation already present | `tools/_instrument.py` mimics Opik's API | Looks already instrumented but does nothing |\n| 6 | Monkey-patching at import time | `core/__init__.py` adds `.run()` to Agent class | Behavior changes when you import the package |\n| 7 | Client creation inside decorators | `core/middleware.py` `@with_llm()` injects client as kwarg | No module-level client to wrap |\n| 8 | Client creation inside descriptors | `core/middleware.py` `_LLMDescriptor.__get__` | Client created on first attribute access |\n| 9 | Proxy objects wrapping real clients | `core/_proxy.py` `LLMProxy` with `__getattr__` | Instrumentation can't see through the proxy |\n| 10 | Multiple entry points | `cli.py`, `serve.py`, `run.sh` | No single \"main\" to instrument |\n| 11 | Environment-driven provider switching | `providers/_router.py` reads `$PROVIDER` / `$LLM_BACKEND` | Provider unknown until runtime |\n| 12 | `cached_property` for client creation | `providers/oai.py` | Client created lazily, not at import |\n| 13 | Class-level client cache in method body | `providers/claude.py` | Import + instantiation hidden in method |\n| 14 | Async generators yielding intermediate results | `core/models.py`, `tools/summarize.py` | Complex control flow hard to wrap |\n| 15 | LangGraph via dynamic import + provider indirection | `tools/graph_flow.py` | 3 layers between code and actual LLM call |\n| 16 | Google ADK with runtime config | `tools/adk_agent.py` | Conditional import, `SimpleNamespace` config |\n| 17 | CrewAI via `__import__` + `getattr` for all classes | `tools/crew.py` | No scannable CrewAI imports at all |\n\n## Frameworks Used (All Hidden)\n\n| Framework | File | How It's Hidden |\n|-----------|------|-----------------|\n| **OpenAI** (Python) | `providers/oai.py` | `__import__(\"openai\")` + `cached_property` — no `import openai` anywhere |\n| **Anthropic** | `providers/claude.py` | `__import__(\"anthropic\")` inside method body, class-level cache |\n| **LangChain** | `tools/summarize.py` | `importlib.import_module(\"langchain_openai\")` + `getattr` |\n| **LangGraph** | `tools/graph_flow.py` | `importlib.import_module(\"langgraph.graph\")` + provider registry for LLM |\n| **Google ADK** | `tools/adk_agent.py` | Conditional `__import__(\"google.adk\")`, `SimpleNamespace` config object |\n| **CrewAI** | `tools/crew.py` | `__import__(\"crewai\")` + `getattr` for Agent, Task, Crew, Process |\n| **OpenAI** (TypeScript) | `tools/search.ts` | Separate language, invoked as subprocess via `search_bridge.py` |\n\n## File Structure\n\n```\nbabadook/\n├── cli.py                      # CLI entry point (Click)\n├── serve.py                    # HTTP entry point (FastAPI)\n├── run.sh                      # Shell entry point — calls Python AND TypeScript\n├── main.py                     # Decoy entry point\n├── core/\n│   ├── __init__.py             # Monkey-patches Agent class on import\n│   ├── agent.py                # MISLEADING: just Pydantic data models\n│   ├── models.py               # MISLEADING: the actual agent orchestrator\n│   ├── _registry.py            # Provider registry — dynamic import + factory\n│   ├── _proxy.py               # Proxy object that wraps any LLM client\n│   └── middleware.py           # Decorators/descriptors that create LLM clients\n├── tools/\n│   ├── __init__.py\n│   ├── search.ts               # TypeScript tool — uses OpenAI for embeddings\n│   ├── search_bridge.py        # Python subprocess caller for search.ts\n│   ├── summarize.py            # LangChain via importlib\n│   ├── graph_flow.py           # LangGraph workflow via dynamic import\n│   ├── adk_agent.py            # Google ADK agent — conditional, unusual setup\n│   ├── crew.py                 # CrewAI via __import__ + getattr\n│   └── _instrument.py          # FAKE instrumentation — mimics Opik but isn't\n├── providers/\n│   ├── __init__.py\n│   ├── _base.py                # Abstract base with __init_subclass__ registration\n│   ├── oai.py                  # OpenAI — __import__ + cached_property\n│   ├── claude.py               # Anthropic — lazy import in method body\n│   └── _router.py              # Env-var-based provider selection\n├── package.json\n├── tsconfig.json\n├── pyproject.toml\n└── .env.example\n```\n\n## Setup\n\n```bash\ncp .env.example .env\n# Add OPENAI_API_KEY (required), ANTHROPIC_API_KEY (optional), GOOGLE_API_KEY (optional)\n\nuv sync\nnpm install\n```\n\n## Usage\n\n```bash\n# CLI\nuv run python cli.py \"What is quantum computing?\" --provider=openai -v\n\n# With LangGraph multi-step flow\nuv run python cli.py \"What is CRISPR?\" --provider=openai --graph -v\n\n# With CrewAI fact-checking\nuv run python cli.py \"What is CRISPR?\" --provider=anthropic --verify -v\n\n# HTTP server\nuv run python serve.py\n```\n\n## Testing Instrumentation Tools\n\n**Delete `README.md` before running instrumentation tools** — it gives away all the patterns.\n\n```bash\nrm README.md\n# Run your instrumentation tool here\ngit checkout README.md\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomet-ml%2Fadversarial-benchmark-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcomet-ml%2Fadversarial-benchmark-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomet-ml%2Fadversarial-benchmark-agent/lists"}