{"id":51460828,"url":"https://github.com/amitpatole/agent-vision","last_synced_at":"2026-07-06T05:01:33.023Z","repository":{"id":365737016,"uuid":"1273455251","full_name":"amitpatole/agent-vision","owner":"amitpatole","description":"Eyes for AI coding agents 👁️ — render → perceive → report → fix → re-render. A machine-graded visual feedback loop (DOM/contrast/OCR-grounded + optional vision LLM) agents consume to self-correct before claiming done.","archived":false,"fork":false,"pushed_at":"2026-06-25T22:04:32.000Z","size":3926,"stargazers_count":5,"open_issues_count":0,"forks_count":3,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-25T23:12:16.847Z","etag":null,"topics":["accessibility","agent-tools","ai-agents","claude-code","computer-vision","dom","llm","mcp","ocr","playwright","python","screenshot","vision-llm","visual-regression","wcag"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/agentvision/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amitpatole.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-18T14:38:22.000Z","updated_at":"2026-06-25T22:04:36.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/amitpatole/agent-vision","commit_stats":null,"previous_names":["amitpatole/agent-vision"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/amitpatole/agent-vision","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitpatole%2Fagent-vision","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitpatole%2Fagent-vision/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitpatole%2Fagent-vision/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitpatole%2Fagent-vision/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amitpatole","download_url":"https://codeload.github.com/amitpatole/agent-vision/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitpatole%2Fagent-vision/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35178403,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-06T02:00:07.184Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accessibility","agent-tools","ai-agents","claude-code","computer-vision","dom","llm","mcp","ocr","playwright","python","screenshot","vision-llm","visual-regression","wcag"],"created_at":"2026-07-06T05:01:32.062Z","updated_at":"2026-07-06T05:01:32.985Z","avatar_url":"https://github.com/amitpatole.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AgentVision — Eyes for AI Agents 👁️\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/banner.png\" alt=\"AgentVision — Eyes for AI Agents: render, see, report, fix — before claiming done\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/agentvision/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/agentvision?color=8b5cf6\u0026label=pip%20install%20agentvision\" alt=\"PyPI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://amitpatole.github.io/agent-vision/\"\u003e\u003cimg src=\"https://img.shields.io/badge/docs-amitpatole.github.io-5ad1e6\" alt=\"Docs\"\u003e\u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/badge/license-MIT-blue\" alt=\"MIT\"\u003e\n\u003c/p\u003e\n\n\u003e **Problem:** AI coding agents are *blind* — they write a UI, chart, SVG or PDF and never *see* the result, shipping breakage they can't perceive.  \n\u003e **Result:** AgentVision gives them eyes — render → see → report → fix — catching overflow, low contrast, clipped/truncated text (incl. SVG labels), broken images and typos.  \n\u003e So your agent **self-corrects before it claims done.**\n\nAgentVision is a provider-agnostic framework that closes the visual feedback loop for AI\ncoding agents:\n\n```\nrender → perceive → report → (agent fixes) → re-render → diff\n```\n\nIt is **not** human-reviewed visual regression (Percy/Applitools/Argos) and **not** browser\nautomation (browser-use/Playwright). It is a **machine-graded visual critique loop an agent\nconsumes to self-correct before claiming done** — with a verdict (`pass`/`warn`/`fail`) and\nactionable, coordinate-grounded issues.\n\n## The 60-second pitch\n\n```bash\npip install \"agentvision[render]\"\nplaywright install chromium     # see `agentvision doctor` if Chromium won't launch\nagentvision demo                # no API key required\n```\n\n`agentvision demo` renders a deliberately broken page, prints a **FAIL** report (overflow +\nlow-contrast + a 404 image — all DOM/CV-grounded, no LLM key needed), then loops against the\nfixed version and prints *\"what changed: 3 issues resolved → PASS.\"* That command *is* the\nproduct.\n\n## What makes it trustworthy\n\nFindings are grounded in sources we can actually trust:\n\n- **DOM geometry** (`getBoundingClientRect` + scroll offset) — precise element boxes.\n- **Computed-style contrast** (`getComputedStyle`) — real WCAG ratios, with a `confidence`\n  flag (it degrades honestly over gradients/images/pseudo-elements rather than lying).\n- **OCR word boxes** (Tesseract) — precise text locations.\n- **Console / network / 4xx capture** — the #1 \"looks fine in code, broken live\" cause.\n\nA vision LLM (Claude/OpenAI/Gemini) adds semantic critique on top. Its pixel boxes are\ntreated as **advisory** (`bbox_precise: false`), never marketed as pixel-accurate.\n\n**Full-coverage vision.** On a large artifact the model gets a downscaled overview *plus*\nfull-resolution tiles covering it, so fine detail and small text aren't lost to downscaling.\nIt's pixel-based and source-agnostic — the same coverage applies to HTML, a flat image, or a\nPDF page, not just elements the DOM enumerates.\n\n## Documents, decks \u0026 confidential inputs\n\nPoint any command at a PDF or Office/OpenDocument file (`.docx/.pptx/.xlsx/.odt/…`) and it's\nrasterized per page and graded like a screenshot. PowerPoint decks also get an **offline slide\ninspector** — `agentvision check deck.pptx` runs key-free and no-egress, flagging unreadable\ntext (low / dark-on-dark contrast), clipped/truncated text, off-slide shapes and overlapping\nboxes, each tagged `[slide N]`.\n\nProcessing something confidential? Add **`--no-cache`** to any source command to render in a\nthrowaway temp dir that's wiped on exit — nothing touches `~/.cache/agentvision`:\n\n```bash\nagentvision check confidential-deck.pptx --no-cache --backend local   # nothing cached, nothing leaves the box\n```\n\nSee [docs/security.md](docs/security.md) for the full model (path confinement, `file_root`, the\nephemeral cache, and the renderer trust boundary).\n\n## Match the intent, not just avoid defects\n\nA typo-free, well-laid-out artifact can still be **the wrong thing** — an infographic that\nshows the wrong stages, a page missing the panel you asked for, a generated image that\nignored half the prompt. Give AgentVision the **intent** and it grades the render against it,\nso **PASS means \"matches what I set out to build,\"** not merely \"defect-free\":\n\n```bash\n# Does the render match the thought? (text claims grade deterministically via OCR)\nagentvision conform ./infographic.png \\\n  --brief \"launch infographic for AgentVision\" \\\n  --expect 'must: title reads \"AgentVision\"' \\\n  --expect 'should: shows 4 stages left to right'\n```\n\nFor **AI-generated** artifacts the fix is a better *prompt*, not code — so the generative loop\n**generate → see → grade vs intent → refine prompt → regenerate** runs until it matches. The\nimage generator is a hook you supply; AgentVision never bundles an image-gen dependency:\n\n```bash\nagentvision generate --generator mypkg.gen:make_image \\\n  --brief \"minimalist infographic, dark background, no typos\" --max-iter 4 -o final.png\n```\n\nSee [docs/conformance.md](docs/conformance.md). Express intent three ways — a free-text\n**brief** (eyes extract the checklist), an **explicit checklist** (`--expect`, deterministic),\nor a **reference image** (`--reference`). Claims are `must:` / `should:` / `nice:`.\n\n## Eyes → brain: the handoff\n\nIn anatomy the eyes are only the *afferent* half — the retina perceives, the optic nerve\ncarries the signal to the brain, the brain decides, the hand acts, the eyes look again.\nAgentVision is that afferent pathway for an agent: it perceives and hands a clean signal back\nto **the brain** (whatever does your reasoning/planning/memory) — it deliberately doesn't\ndecide for you. Any perception call distills to a **`Handoff`**:\n\n```bash\nagentvision analyze ./page.html --handoff\n```\n```jsonc\n{ \"perceived\": \"fail\", \"next_action\": \"revise\", \"matches_intent\": false,\n  \"todo\": [\"[overflow] hero text overflows on the right\",\n           \"[intent/must] a \\\"Checkout\\\" button is visible\"],\n  \"open_questions\": [\"Verify: uses the brand's dark theme\"] }\n```\n\n`next_action` (`done` / `revise` / `review`) drives the brain's loop; `todo` is the work-list;\n`open_questions` is what perception couldn't confirm (never dropped). Available as\n`report.to_handoff()`, the MCP `perceive_handoff` tool, `POST /handoff`, and a `handoff.json`\nper loop iteration — provider- and brain-agnostic. See [docs/handoff.md](docs/handoff.md).\n\n## Eyes \u0026 Brain — AgentVision × Verel\n\nAgentVision is the **eyes**. It pairs with **[Verel](https://github.com/amitpatole/verel)**,\nthe **brain** — an agent framework where *nothing is \"done\" until a grader returns a verdict.*\nThe eyes perceive and grade intent; the brain decides with attestation and **compounds only\nverified work** into memory; then the eyes look again.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/unified-architecture.png\" alt=\"Eyes \u0026 Brain — AgentVision perceives and grades intent; Verel decides and compounds verified work into memory\" width=\"100%\"\u003e\n\u003c/p\u003e\n\nThey ship and version independently (`pip install agentvision`, `pip install verel`) yet work\nin sync: AgentVision plugs into Verel as its `verel.senses` perception organ — mapped onto a\nunified verdict bus (vision *alongside* tests, lint and types), with intent conformance\nrecorded in the brain's memory each iteration. Since `0.9.0` both speak one language: the\n`Report`/`Handoff` types come from the shared [`agentsensory`](https://pypi.org/project/agentsensory/)\ncontract, so a graded `Report` drops onto that bus with no per-organ translation. AgentVision\nstays brain-agnostic; Verel is the reference brain. See [docs/handoff.md](docs/handoff.md).\n\n## Many faces, one core\n\n| Surface | Who it's for |\n|---|---|\n| **Library** (`import agentvision`) | Python apps, custom harnesses |\n| **CLI** (`agentvision …`) | Any agent that can run a shell command; CI |\n| **Claude Code Skill** | Claude agents — auto-invokes the loop *before claiming done* |\n| **MCP server** (`agentvision-mcp`) | Cursor, Claude, any MCP-capable host |\n| **REST service** (`agentvision-serve`) | Non-MCP / networked / CI agents |\n| **Integration recipes** | Cursor rules, Aider, generic \"agent contract\" |\n\n\u003e ⚠️ \"Provider-agnostic\" describes the **API surface**, not behavior. The framework can't\n\u003e *force* a non-Claude agent into the loop — it gives every agent the *means*. The Claude\n\u003e Code Skill is the one surface that makes an agent use it proactively; MCP is the\n\u003e first-class cross-host path; the recipes cover the rest.\n\n## Many agents, one set of eyes\n\nOne agent with eyes self-corrects. A **swarm** of agents sharing one set of eyes is the real\nprize — dozens of workers each rendering UIs, charts, decks or PDFs, every output graded against\nthe same contract before it counts as done. Run the eyes as a horizontally-scaled **service**\n(`agentvision serve`) or embed the library per worker; the single-shot endpoints\n(`analyze`/`check`/`conform`) are stateless and scale with zero coordination. The one piece of\nstate to mind is the **loop session** — kept in-process, so behind multiple workers keep loops\nclient-side or sticky-route them. And because every worker returns the same `agentsensory`\n`Report`/`Handoff`, a coordinator (or a brain like [Verel](https://github.com/amitpatole/verel))\naggregates all the verdicts on **one bus** — vision graded alongside tests, lint and types.\n\nSee **[Swarms \u0026 scaling](https://amitpatole.github.io/agent-vision/scaling/)** for the\ntopologies, the stateless/stateful split, and a fan-out example.\n\n## Vision backends\n\nPluggable and selectable via `--backend` / `AGENTVISION_VISION_BACKEND`:\n\n- **`anthropic`** (default model `claude-haiku-4-5`, upgradable to Sonnet/Opus)\n- **`openai`**, **`gemini`**\n- **`local`** — CV/OCR heuristics only, **no API key, no egress** (great for CI / air-gapped)\n\n## Install\n\n```bash\npip install \"agentvision[all]\"          # everything\npip install \"agentvision[render]\"       # just rendering + the no-key local loop\npip install \"agentvision[render,anthropic]\"  # + Claude analysis\n```\n\nSystem dependencies (Chromium, Tesseract, poppler) and a `doctor` that checks them:\n\n```bash\nagentvision doctor          # attempts a real Chromium launch; lists every missing lib\nagentvision doctor --fix    # installs the Chromium browser binary\n```\n\nOn a bare RHEL/CentOS box, `playwright install-deps` does **not** work (apt-only). See\n[docs/quickstart.md](docs/quickstart.md) for the `dnf` line, or use the bundled\n**Dockerfile** which bakes the deps in.\n\n## Usage\n\n```bash\n# Analyze a file/URL/HTML string and print a structured report\nagentvision analyze ./index.html --backend local --json\n\n# Run the self-correcting loop\nagentvision loop ./dashboard.html --max-iter 3\n\n# Responsive contact sheet across breakpoints\nagentvision sheet ./index.html --breakpoints 375,768,1280,1920\n\n# Visual regression against a named baseline\nagentvision baseline ./index.html --name home\nagentvision regress  ./index.html --name home\n```\n\n**Live pages, SPAs \u0026 dashboards** (polling, websockets, canvas/WebGL):\n\n```bash\n# localhost dev server, wait for the data to render, freeze animation, machine output\nagentvision analyze http://localhost:5173 --allow-local \\\n  --wait-for \"#dashboard\" --settle-ms 800 --quiet\n```\n\n**Streaming / video / over-time behavior** — watch, don't just glance:\n\n```bash\n# Is the video actually playing? Did loading finish? Are captions on?\nagentvision watch https://app.example.com/player --frames 6 --interval-ms 500 \\\n  --expect 'must: the video is playing'\n```\n\n`watch` reads deterministic `\u003cvideo\u003e` state (currentTime/readyState/captions) + pixel\nliveness/stall/black-frame detection, then adds a time-aware vision pass. See\n[docs/use-cases/streaming.md](docs/use-cases/streaming.md).\n\n`--nav-wait` defaults to `load` (polling pages never go idle); `--freeze` (default on) pauses\nanimations + `requestAnimationFrame` so canvas/WebGL pages capture without hanging; `--quiet`\nprints only JSON (logs to stderr, exit codes 0 pass/warn · 2 fail · 3 error).\n\nLibrary:\n\n```python\nimport asyncio\nfrom agentvision import load_settings\nfrom agentvision.core.loop import LoopSession\n\nasync def main():\n    settings = load_settings(vision_backend=\"local\")\n    session = LoopSession(\"examples/broken_layout.html\", settings=settings)\n    result = await session.iterate()\n    print(result.report.verdict, [i.message for i in result.report.issues])\n\nasyncio.run(main())\n```\n\n## Drop it into your workflow \u0026 your agents\n\n```yaml\n# CI gate (GitHub Action): fails the build on a visual FAIL verdict\n- uses: amitpatole/agent-vision@v0.11.0\n  with: { source: dist/index.html, command: check, args: --full-page }\n```\n\n- **CI / pre-commit / Makefile** — shell out; exit codes `0 pass/warn · 2 fail · 3 error`,\n  `--quiet` for JSON-only output. Reusable **GitHub Action** + **pre-commit hook** included.\n- **Your agents** — drop [`integrations/agent-contract.md`](integrations/agent-contract.md)\n  into the system prompt, use the **Claude Code Skill**, or the **MCP** tools (Cursor/Claude/any host).\n\nFull guide: [docs/integrations.md](docs/integrations.md).\n\n## Documentation\n\n📖 **Full docs site: [amitpatole.github.io/agent-vision](https://amitpatole.github.io/agent-vision/)**\n\n- [Quickstart](docs/quickstart.md) · [The Loop](docs/the-loop.md) ·\n  [Conformance](docs/conformance.md) · [Handoff (eyes→brain)](docs/handoff.md) ·\n  [Streaming / temporal](docs/use-cases/streaming.md) · [Backends](docs/backends.md) ·\n  [Adapters](docs/adapters.md) · [Integrations](docs/integrations.md) · [Vision](docs/VISION.md)\n\n## What we do **not** claim (honesty)\n\n- Pixel-accurate *vision-model* bounding boxes (they're advisory).\n- WCAG verdicts on rasterized non-HTML (heuristic only).\n- Bit-reproducible screenshots / deterministic LLM reports.\n- Uniform provider-agnostic *behavior* (only the API surface is uniform).\n\n## License\n\nMIT © Amit Patole\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famitpatole%2Fagent-vision","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famitpatole%2Fagent-vision","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famitpatole%2Fagent-vision/lists"}