{"id":51351364,"url":"https://github.com/farhanic017/vision-tool","last_synced_at":"2026-07-02T16:07:11.948Z","repository":{"id":358813098,"uuid":"1243196836","full_name":"farhanic017/vision-tool","owner":"farhanic017","description":"MCP vision server for AI agents, Claude, Cursor, OpenCode, VS Code, local LLMs, Ollama, LM Studio, OpenRouter, Gemini, image analysis, and video understanding.","archived":false,"fork":false,"pushed_at":"2026-06-07T12:26:25.000Z","size":20606,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-07T13:21:42.005Z","etag":null,"topics":["ai-agent","ai-agents","claude-code","claude-desktop","computer-vision","continue-dev","cursor","gemini","image-analysis","lm-studio","local-llm","mcp","mcp-server","multimodal-ai","ollama","opencode","openrouter","video-analysis","vision-ai","vscode"],"latest_commit_sha":null,"homepage":"https://github.com/farhanic017/vision-tool","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/farhanic017.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"patreon":"Farhanic"}},"created_at":"2026-05-19T06:14:10.000Z","updated_at":"2026-06-07T12:52:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/farhanic017/vision-tool","commit_stats":null,"previous_names":["farhanic017/vision-for-opencode","farhanic017/vision-tool"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/farhanic017/vision-tool","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farhanic017%2Fvision-tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farhanic017%2Fvision-tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farhanic017%2Fvision-tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farhanic017%2Fvision-tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/farhanic017","download_url":"https://codeload.github.com/farhanic017/vision-tool/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farhanic017%2Fvision-tool/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35053618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agent","ai-agents","claude-code","claude-desktop","computer-vision","continue-dev","cursor","gemini","image-analysis","lm-studio","local-llm","mcp","mcp-server","multimodal-ai","ollama","opencode","openrouter","video-analysis","vision-ai","vscode"],"created_at":"2026-07-02T16:07:10.518Z","updated_at":"2026-07-02T16:07:11.877Z","avatar_url":"https://github.com/farhanic017.png","language":"Python","funding_links":["https://patreon.com/Farhanic"],"categories":[],"sub_categories":[],"readme":"# vision-tool\n\n![license GPLv3](https://img.shields.io/badge/license-GPLv3-8A2BE2)\n![platform Python 3.8+](https://img.shields.io/badge/platform-Python%203.8%2B-22C55E)\n![author Farhan Dhrubo](https://img.shields.io/badge/author-Farhan%20Dhrubo-F97316)\n![version v8](https://img.shields.io/badge/version-v8-2563EB)\n![tests 591 passed](https://img.shields.io/badge/tests-591%20passed-16A34A)\n![MCP server](https://img.shields.io/badge/MCP-server-06B6D4)\n![AI agents](https://img.shields.io/badge/AI-agents-22C55E)\n![local LLMs](https://img.shields.io/badge/local-LLMs-F59E0B)\n![vision models](https://img.shields.io/badge/vision-models-A855F7)\n\n\u003e Created by [Farhan Dhrubo](https://github.com/farhanic017) — [Submit an issue](https://github.com/farhanic017/vision-tool/issues)\n\n**MCP vision server for AI agents, coding assistants, local LLMs, and text-only models.**\n\nvision-tool gives **Claude Desktop, Claude Code, Cursor, OpenCode, VS Code,\nContinue.dev, Windsurf, Ollama, LM Studio, llama.cpp, OpenRouter, Gemini,\nOpenAI, Anthropic, and local AI agents** the ability to understand screenshots,\ndiagrams, UI clips, images, and videos.\n\nIt works as a **Model Context Protocol (MCP) server**, CLI tool, OpenCode skill,\nor Python library. Text-only models and local LLMs can call `analyze_image` or\n`analyze_video`, then receive a normal text description they can reason over.\n\nIf this saves you from \"I can't view images\" responses, star the repo so more\nAI agent builders can find it.\n\n## Why developers star it\n\n- **Adds vision to text-only agents** - Claude, Cursor, OpenCode, Continue,\n  VS Code agents, local LLMs, and terminal coding assistants can inspect images.\n- **Works with MCP** - exposes `analyze_image` and `analyze_video` as standard\n  MCP tools for any MCP-compatible client.\n- **Supports images and videos** - screenshots, diagrams, UI mockups, web pages,\n  app flows, screen recordings, MP4 clips, and animated GIFs.\n- **Auto-detects working vision backends** - checks local VLMs, cloud keys,\n  OpenRouter, Gemini, Ollama, LM Studio, and MCP/CLI configs before asking for\n  new API keys.\n- **Saves quota with memory** - remembers rate-limit, token-limit, and quota\n  failures for 24 hours, then retries them after cooldown.\n- **Keeps itself fresh** - refreshes free/paid capability status every 2 days\n  in the background when the agent starts.\n- **Routes across 23 vision backends** - Gemini first, then provider fallback\n  through Azure, Groq, HuggingFace, Mistral, Fireworks, ZAI, and more.\n\n## Popular use cases\n\n- Add an **MCP vision server** to Claude Desktop, Claude Code, Cursor, VS Code,\n  OpenCode, Continue.dev, Windsurf, or any MCP-compatible AI coding assistant.\n- Give **local LLMs** such as Ollama, LM Studio, and llama.cpp image analysis\n  without switching away from local text models.\n- Route **OpenRouter vision models**, Gemini vision, OpenAI vision, Anthropic\n  Claude vision, and free vision APIs through one fallback chain.\n- Convert screenshots, UI mockups, architecture diagrams, and web pages into\n  plain text an AI agent can understand.\n- Summarize MP4 screen recordings, app demos, UI clips, and videos into\n  keyframe-based descriptions for agent reasoning.\n\n## Demo Video\n\n![vision-tool animated demo](docs/demo/vision_tool_how_it_works.gif)\n\nThe preview above is embedded directly in the README from `docs/demo/vision_tool_how_it_works.gif`.\n\n## Features\n\n- **Images** — PNG, JPG, WebP, BMP, animated GIF\n- **Videos** — MP4, WebM, MOV, AVI, MKV, FLV, WMV, M4V (via ffmpeg keyframe extraction)\n- **23 fallback backends** — Gemini first, then Azure, Groq, HF, Mistral, Fireworks, ZAI all in parallel\n- **Full parallel fire** — ALL backends run simultaneously, first success wins, rest cancelled\n- **Fast — typical analysis in 2-5s**, worst case ~19s (no backends available)\n- **Smart file search** — checks direct path → known user dirs → shallow recursive scan\n- **Natural language prompts** — default prompts are conversational, not robotic checklists\n- **Auto JPEG compression** — progressive quality down to 15 for large images\n- **Zero hardcoded secrets** — API keys in `config.json` (gitignored) or env vars\n- **Works everywhere** — CLI, MCP server, opencode skill, or direct Python import\n\n\n## 🔄 Always-on mode\n\nvision-tool is designed to be **always-on** — not a triggered skill that\nonly activates on certain keywords. Once installed:\n\n1. **`ALWAYS_ON.md`** is added to your AI client's permanent system\n   instructions. The model is told in every session: \"You MUST use\n   vision-tool for ALL images and videos. Never say you can't view images.\"\n\n2. **The MCP server** (`vision_mcp_server.py`) exposes `analyze_image` and\n   `analyze_video` as first-class tools available at all times.\n\n3. **The invisible watchdog** (`vision_watchdog.vbs` / `vision_watchdog.exe`)\n   monitors for ALL 13 supported AI tool process names (opencode, Claude,\n   Cursor, Windsurf, Aider, Continue, VSCode, VSCodium, Antigravity 1.x/2.x,\n   GitHub Copilot CLI, and more) and starts/stops the vision server\n   automatically — no manual steps.\n\n4. **The dynamic-skill-loader** integration marks vision-tool as\n   `alwaysOn`, so it's never filtered by keyword triggers.\n\n### What this means for users\n\n- Your AI will **never say** \"I can't view images\" or \"please describe\n  what you see\" — it will just analyze the file.\n- No need to remember trigger keywords — just provide the file path.\n- Works across all major AI coding assistants.\n\n## Quick start\n\n### Drop-in install (tell your AI)\n\nJust send this URL to your AI assistant:\n\n```\nhttps://github.com/farhanic017/vision-tool\n```\n\nYour AI will clone, install deps, set up API keys, and configure everything\nautomatically as **always-on** — the model will never say \"I can't view\nimages\" again. The `SKILL.md` and `ALWAYS_ON.md` files contain the\nstep-by-step instructions any AI agent reads and follows.\n\n### Manual install\n\n```bash\n# 1. Clone\ngit clone https://github.com/farhanic017/vision-tool.git\ncd vision-tool\n\n# 2. Install deps\npip install pillow\n\n# 3. Run setup (choose: enter keys now or add later)\npython setup.py\n\n# 4. Analyse anything\npython vision_proxy.py screenshot.png\npython vision_proxy.py demo.mp4 \"Describe the UI flow\"\n```\n\n### Auto-installer\n\n```bash\n# Interactive (asks questions)\npython install.py\n\n# Non-interactive (best for automation)\npython install.py --auto\n```\n\n## Vision backends\n\nGoogle Gemini models are tried **first** (2 fast attempts at 8s each). All remaining backends fire **simultaneously** (12s per backend) — the first successful response wins, the rest are cancelled. Typical analysis completes in **2-5 seconds**.\n\n| # | Model | Provider | Cost |\n|---|-------|----------|------|\n| 1 | Gemini 2.5 Flash | Google Gemini | Free tier |\n| 2 | Gemini 3 Flash Preview | Google Gemini | Free tier |\n| 3 | Gemini 2.0 Flash | Google Gemini | Free tier |\n| 4 | Gemini 2.0 Flash Lite | Google Gemini | Free tier |\n| 5 | Gemini 2.5 Pro | Google Gemini | Free tier |\n| 6 | Gemini 3 Pro Preview | Google Gemini | Free tier |\n| 7 | Azure DeepSeek-V4-Pro | Azure AI Foundry | Free (Azure credits) |\n| 8 | Azure gpt-4.1 | Azure AI Foundry | Free (Azure credits) |\n| 9 | Azure gpt-4.1-mini | Azure AI Foundry | Free (Azure credits) |\n| 10 | Azure gpt-4.1-nano | Azure AI Foundry | Free (Azure credits) |\n| 11 | Azure gpt-4o | Azure AI Foundry | Free (Azure credits) |\n| 12 | Azure gpt-4o-mini | Azure AI Foundry | Free (Azure credits) |\n| 13 | Azure gpt-5.1 | Azure AI Foundry | Free (Azure credits) |\n| 14 | Azure gpt-5.4 | Azure AI Foundry | Free (Azure credits) |\n| 15 | Azure gpt-5.4-mini | Azure AI Foundry | Free (Azure credits) |\n| 16 | Azure gpt-5.4-nano | Azure AI Foundry | Free (Azure credits) |\n| 17 | Azure Kimi-K2.6 | Azure AI Foundry | Free (Azure credits) |\n| 18 | Azure Phi-4 multimodal | Azure AI Foundry | Free (Azure credits) |\n| 19 | Groq Llama 4 Scout 17B | Groq | Free |\n| 20 | HF Qwen3-VL-8B | HuggingFace Inference Providers | Free tier |\n| 21 | Mistral pixtral-large | Mistral AI | Free tier |\n| 22 | Fireworks Llama 3.2 90B Vision | Fireworks AI | Free tier |\n| 23 | ZAI Glm-4.5-Flash | Zhipu AI (Z.AI) | Free tier |\n\n\u003e First 2 backends tried sequentially (8s timeout each), then rest fire in parallel (12s timeout each) — first success cancels all remaining. Total operation timeout: 25s.\n\u003e Only backends with configured API keys are launched. Missing keys are skipped instantly.\n\u003e Gemini, Azure, Groq, HuggingFace, and Mistral all offer free tiers.\n\n## Capabilities \u0026 Limitations\n\n**Images** — Describes visible content, layout, colors, text, and UI elements. The image is downscaled to **max 1024px**, so tiny details/fine text may blur.\n\n**Videos** — Extracts **up to 8 evenly-spaced keyframes** via ffmpeg, analyzes them for UI flow, actions, scene changes, layout, text.\n\n**What determines quality** — Gemini models are tried first (2 fast sequential attempts, 8s each). All remaining backends fire in parallel (12s each). The first backend to respond wins (typically 2-5s). Gemini 2.5 Pro and Flash give the best balance of speed and quality.\n\n**Caveats:**\n- Image capped at 1024px → small UI text/icons may be unreadable\n- Video limited to 8 frames → fast transitions get missed\n- First successful backend wins — not necessarily the most detailed\n\n**Getting better results** — Speak naturally. The default prompts are conversational, but you can be more specific:\n\n```bash\npython vision_proxy.py screenshot.png \"What's the main layout here? Describe the colors and buttons.\"\npython vision_proxy.py screenshot.png \"Read all the text on this page and describe the UI structure.\"\n```\n\n## Getting API keys\n\nYou need at least **one** of these:\n\n| Key | Get it | Powers |\n|-----|--------|--------|\n| **Gemini API key** ⭐ | https://aistudio.google.com/apikey | Gemini 2.5 Flash / 3 Pro / 2.0 Flash (free tier, tried first) |\n| **Azure AI key** | https://ai.azure.com | Azure AI Foundry (12 models, free credits) |\n| **Groq API key** | https://console.groq.com/keys | Groq Llama 4 Scout (free tier) |\n| **Mistral AI API key** | https://console.mistral.ai/api-keys | Mistral pixtral-large (free tier) |\n| **HuggingFace token** | https://huggingface.co/settings/tokens | HF Qwen3-VL (free tier) |\n| **Fireworks AI API key** | https://fireworks.ai/api-keys | Fireworks Llama 3.2 90B Vision (free tier) |\n| **Zhipu AI (Z.AI) key** | https://z.ai | ZAI Glm-4.5-Flash (free tier) |\n| **Cloudflare API key** | https://dash.cloudflare.com/profile/api-tokens | Cloudflare Workers AI (free tier, for --model flag) |\n| **OpenRouter API key** | https://openrouter.ai/keys | Multi-model access (free + paid) |\n\nRun `python setup.py` — choose to enter keys now or add later.\nAdd keys later anytime with: `python setup.py --add-key`\n\n## Integration guides\n\n### 1. CLI (any terminal)\n\nWorks with any AI coding assistant that can run shell commands.\n\n```bash\npython /path/to/vision_proxy.py image.png\npython /path/to/vision_proxy.py video.mp4 \"Describe the gameplay\"\n```\n\nYour AI just needs to call this as a bash/terminal command.\n\n### 2. MCP server (OpenCode, Claude Desktop, Cursor, Windsurf, Continue.dev, VSCode, VSCodium, Antigravity 1.x/2.x)\n\nAdd the MCP server to your client's config. This exposes `analyze_image` and\n`analyze_video` as first-class MCP tools that any agent can call directly.\n\n#### OpenCode (`opencode.jsonc`)\n\n```jsonc\n{\n  \"mcp\": {\n    \"vision-tool\": {\n      \"type\": \"local\",\n      \"command\": [\"python\", \"path/to/vision_mcp_server.py\"],\n      \"enabled\": true\n    }\n  },\n  \"instructions\": [\n    \"path/to/ALWAYS_ON.md\"   // \u003c-- ensures model never says \"can't view\"\n  ]\n}\n```\n\nFor **always-on behavior**, add `ALWAYS_ON.md` to your `instructions` array.\nThis injects the mandatory vision-tool usage rules into every session so\nthe model automatically analyzes any image or video without being asked to.\n\n#### Claude Desktop (`claude_desktop_config.json`)\n\n```json\n{\n  \"mcpServers\": {\n    \"vision-tool\": {\n      \"command\": \"python\",\n      \"args\": [\"path/to/vision_mcp_server.py\"]\n    }\n  }\n}\n```\n\n#### VSCode (`mcp.json`)\n\nVSCode uses the `\"servers\"` key (not `\"mcpServers\"`). Add to your user or workspace `mcp.json`:\n\n```json\n{\n  \"servers\": {\n    \"vision-tool\": {\n      \"type\": \"stdio\",\n      \"command\": \"python\",\n      \"args\": [\"path/to/vision_mcp_server.py\"]\n    }\n  }\n}\n```\n\n**Locations:**\n- User (global): `%APPDATA%\\Code\\User\\mcp.json` (Windows), `~/Library/Application Support/Code/User/mcp.json` (macOS), `~/.config/Code/User/mcp.json` (Linux)\n- Workspace: `.vscode/mcp.json` in your project root\n\nOpen via Command Palette: `MCP: Open User Configuration` or `MCP: Open Workspace Folder MCP Configuration`.\n\n#### VSCodium (open-source VS Code fork)\n\nVSCodium uses the same `mcp.json` format as VSCode:\n\n```json\n{\n  \"servers\": {\n    \"vision-tool\": {\n      \"type\": \"stdio\",\n      \"command\": \"python\",\n      \"args\": [\"path/to/vision_mcp_server.py\"]\n    }\n  }\n}\n```\n\n**Config path:** `%APPDATA%\\VSCodium\\User\\mcp.json` (Windows) or `~/.config/VSCodium/User/mcp.json` (macOS/Linux).\n\n#### Antigravity 1.x (VS Code fork)\n\nAntigravity 1.x (the older VS Code fork with full extension support) uses\nthe same native MCP format as VSCode:\n\n```json\n{\n  \"servers\": {\n    \"vision-tool\": {\n      \"type\": \"stdio\",\n      \"command\": \"python\",\n      \"args\": [\"path/to/vision_mcp_server.py\"]\n    }\n  }\n}\n```\n\n**Config path:** `%APPDATA%\\Antigravity\\User\\mcp.json` (Windows) or `~/.config/Antigravity/User/mcp.json` (macOS/Linux).\n\n#### Antigravity 2.x (Google AI-first IDE)\n\nAntigravity 2.x uses standard `mcpServers` format. Add to `mcp_config.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"vision-tool\": {\n      \"command\": \"python\",\n      \"args\": [\"path/to/vision_mcp_server.py\"]\n    }\n  }\n}\n```\n\n**Config path:** `%USERPROFILE%\\.gemini\\antigravity\\mcp_config.json` (Windows) or `~/.gemini/antigravity/mcp_config.json` (macOS/Linux).\n\nOpen via Agent Panel → `...` → Manage MCP Servers → View raw config.\n\n#### Cursor\n\nIn Cursor's MCP server settings:\n\n```\nName: vision-tool\nType: command\nCommand: python path/to/vision_mcp_server.py\n```\n\nOnce added, your AI can call `analyze_image` or `analyze_video` with any\nfile path — no shell commands needed.\n\n### 3. OpenCode skill (always-on)\n\nAdd to your `opencode.jsonc`:\n\n```jsonc\n{\n  \"instructions\": [\n    \"path/to/ALWAYS_ON.md\"    // permanent system instruction\n  ],\n  \"skills\": {\n    \"paths\": [\n      \"path/to/vision-tool\"\n    ]\n  }\n}\n```\n\nThe `ALWAYS_ON.md` file tells the model in every session: use vision-tool\nfor all images, never say you can't view them. This is what makes it\n**always-on** rather than trigger-dependent.\n\nFor **dynamic-skill-loader** users, vision-tool is also configured as\n`alwaysOn` so it loads on every session regardless of trigger keywords.\n\n### 4. Local models (Ollama, LM Studio, llama.cpp)\n\nLocal models don't have vision hardware. **This tool is designed for exactly\nthis case.** The AI runs locally, but calling `vision_proxy.py` sends the\nimage/video to cloud vision APIs for analysis and returns a text description\nthat your local model can read.\n\nWorks identically with any local model in any MCP client:\n\n```jsonc\n{\n  \"model\": \"ollama/llama3.2\",\n  \"mcp\": {\n    \"vision-tool\": {\n      \"type\": \"local\",\n      \"command\": [\"python\", \"path/to/vision_mcp_server.py\"],\n      \"enabled\": true\n    }\n  }\n}\n```\n\n### 5. Invisible background watchdog (Windows)\n\nFor a zero-setup experience, the watchdog auto-starts the vision MCP server\nwhenever **any** supported AI coding tool runs and kills it when all tools\nexit — all hidden, no windows, no taskbar icons.\n\n**Monitored tools (13 process names):** opencode.exe, claude.exe, cursor.exe,\nwindsurf.exe, aider.exe, continue.exe, code.exe (VSCode), vscode.exe,\ncodium.exe (VSCodium), studio.exe, antigravity.exe (Antigravity 1.x/2.x),\nclaudecode.exe, ghcopilot.exe (GitHub Copilot CLI)\n\n**How it starts with Windows:**\n\n- **Startup folder** (recommended): A `.lnk` shortcut is added to `shell:startup`\n  pointing to `wscript.exe \"path\\to\\vision_watchdog.vbs\"` — reliable on all\n  Windows systems, no admin needed\n- **Task Scheduler** (secondary): The watchdog can run at user login via\n  `schtasks /create /tn \"vision-tool-watchdog\" /tr \"...\" /sc onlogon /delay 0000:30`\n- **Zero-flash EXE**: The C# version (`vision_watchdog.exe`) has no console,\n  no window, no taskbar icon — compiled with `csc.exe /target:winexe`\n\n**How it works:**\n\n```\nWindows starts\n  │\n  ▼\nvision_watchdog.vbs launches (invisible via wscript.exe)\n  │\n  ▼\nEvery 10s polls WMI: \"Is any AI coding tool running?\"\n  │\n  ├── Yes → Launch vision_mcp_server.py as hidden process\n  │         (writes PID to %TEMP%\\vision_watchdog.pid)\n  │\n  └── No  → Kill child process, delete PID file\n```\n\n#### Quick start\n\n```cmd\n:: Start the watchdog (double-click)\nwscript.exe //nologo \"C:\\path\\to\\vision_watchdog.vbs\"\n```\n\nAdd to startup folder (`shell:startup`) so it runs every time you log in:\n\n```powershell\n$ws = New-Object -ComObject WScript.Shell\n$s = $ws.CreateShortcut(\"$env:APPDATA\\Microsoft\\Windows\\Start Menu\\Programs\\Startup\\vision-tool-watchdog.lnk\")\n$s.TargetPath = \"wscript.exe\"\n$s.Arguments = \"C:\\path\\to\\vision_watchdog.vbs\"\n$s.WorkingDirectory = \"C:\\path\\to\\vision-tool\"\n$s.Description = \"vision-tool watchdog\"\n$s.Save()\n```\n\nOr create a Task Scheduler task (user login, not system boot):\n\n```cmd\nschtasks /create /tn \"vision-tool-watchdog\" /tr \"wscript.exe C:\\path\\to\\vision_watchdog.vbs\" /sc onlogon /delay 0000:30 /ru %USERNAME% /f\n```\n\n#### Zero-flash option (no wscript icon)\n\nFor absolute invisibility (no wscript.exe taskbar icon), compile the C# version:\n\n```cmd\n:: Install .NET Framework or dotnet, then:\ncsc.exe /target:winexe /out:vision_watchdog.exe vision_watchdog.cs\n\n:: Run the compiled EXE instead\nvision_watchdog.exe\n```\n\nThe compiled EXE has zero presence — no console, no window, no icon.\n\n#### Custom command\n\nBy default the watchdog launches `vision_mcp_server.py`. You can point it at\nany process:\n\n```cmd\nwscript.exe //nologo vision_watchdog.vbs \"notepad.exe\"\nwscript.exe //nologo vision_watchdog.vbs \"python my_script.py\" \"my_pid.pid\"\n```\n\n### 6. Python import (programmatic)\n\n```python\nfrom vision_proxy import analyze\n\n# Analyse an image\ndescription = analyze(\"screenshot.png\")\nprint(description)\n\n# Analyse a video with custom prompt\ndescription = analyze(\"demo.mp4\", \"Describe the UI flow step by step\")\nprint(description)\n\n# Analyse with custom prompt\ndescription = analyze(\"diagram.jpg\", \"Extract all visible text and explain the architecture\")\nprint(description)\n```\n\n## Model compatibility\n\nThe vision tool works with **any AI model** — it doesn't matter if the model\nhas vision or not. The model never processes the image/video directly; the\nvision proxy handles that externally and returns plain text.\n\n| Model / Client | How it connects | Verified |\n|----------------|-----------------|----------|\n| **OpenCode** (`big-pickle`, `DeepSeek`, etc.) | MCP server or skill | ✅ Yes |\n| **Claude Desktop** / **Claude Code** | MCP server | ✅ Yes |\n| **Cursor** | MCP server | ✅ Yes |\n| **Windsurf** | MCP server | ✅ Yes |\n| **Continue.dev** | MCP server | ✅ Yes |\n| **VSCode** (native MCP via Copilot Agent) | MCP server (`.vscode/mcp.json` or user `mcp.json`) | ✅ Yes |\n| **Antigravity** (Google AI-first IDE) | MCP server (`mcp_config.json`) | ✅ Yes |\n| **Hermes** (NousResearch) | MCP server or CLI | ✅ Compatible (standard MCP) |\n| **OpenClaw** | MCP server or CLI | ✅ Compatible (standard MCP) |\n| **Ollama** (any local model) | MCP server + `\"model\": \"ollama/...\"` | ✅ Yes |\n| **LM Studio** | MCP server | ✅ Yes |\n| **llama.cpp** | MCP server | ✅ Yes |\n| **Any terminal** | CLI (`python vision_proxy.py`) | ✅ Yes |\n\nAll MCP-compatible tools use the same protocol — if your client supports\nMCP, it works.\n\n## How it works\n\n```\nUser: \"What's in this image?\"  or  \"describe this naturally\"\n        │\n        ▼\n  AI model (no vision)\n        │\n        ▼\n  CLI / MCP / Skill\n        │\n        ▼\n  vision_proxy.py analyze()\n        │\n        ├── Images → resize to 1024px → JPEG @75 quality\n        └── Videos → ffmpeg extracts 8 keyframes\n        │\n        ▼\n  Fire ALL configured backends:\n    ☆ Gemini models       (first, 2 fast sequential attempts)\n    ☆ Azure models        \n    ☆ Groq Llama 4 Scout\n    ☆ HuggingFace Qwen3-VL\n    ☆ Mistral pixtral-large\n    ☆ Fireworks Llama 3.2 90B Vision\n    ☆ ZAI Glm-4.5-Flash\n        │\n        ▼\n  First success wins → rest cancelled\n        │\n        ▼\n  Returns natural text description\n```\n\n## File structure\n\n```\nvision-tool/\n├── README.md                 # This file\n├── SKILL.md                  # opencode skill def. + always-on rules (AI reads to install)\n├── ALWAYS_ON.md              # Permanent system instruction: never say \"can't view\"\n├── install.py                # Auto-installer (one command setup)\n├── vision_proxy.py           # Core analysis engine (CLI + Python API)\n├── vision_mcp_server.py      # MCP server (stdio + HTTP modes)\n├── vision_watchdog.vbs       # Invisible background process manager (WMI)\n├── vision_watchdog.cs        # C# source for zero-flash compiled EXE\n├── setup.py                  # First-run API key wizard (10 providers: Gemini, OpenRouter, Cloudflare, Azure, OpenAI, Anthropic, Mistral, Groq, HF, Vertex AI)\n├── config.json.example       # Example config (safe to commit)\n├── config.json               # Your actual keys (gitignored)\n├── requirements.txt          # pip dependencies\n├── .gitignore                # Ignores config.json, __pycache__\n├── NOTICE                    # Legal notice\n└── LICENSE                   # GPL-3.0\n```\n\n## Requirements\n\n- **Python 3.8+**\n- **`pillow`** — image resize/resample (`pip install pillow`)\n- **`ffmpeg`** — video keyframe extraction ([download](https://ffmpeg.org/download.html))\n\n## Security\n\n- **No API keys in code.** All keys go into `config.json` (in `.gitignore`) or\n  environment variables.\n- **No telemetry.** This script never phones home. It only talks to the API\n  providers you configure.\n- **No data storage.** Images/videos are never saved or logged; keyframes are\n  written to a temp directory and immediately cleaned up.\n\n## Version\n\n### v8 (Current) - Capability memory, auto-detection, and provider expansion\n- Added persistent backend memory for quota, token, rate-limit, and health states.\n- Limited models are skipped on later runs, then retried after the 24-hour cooldown.\n- Added first-install capability profiling to detect whether working access is local, free/included, paid/metered, quota-limited, or payment/plan-limited.\n- Added background capability refresh every 2 days at startup, so new provider/model access is discovered without blocking the agent.\n- Added auto-detection for local VLMs, cloud credentials, MCP environment blocks, and CLI/provider setups before asking for API keys.\n- Expanded provider routing and detection across local runtimes, OpenRouter, Gemini, OpenAI, Anthropic, Together, DeepInfra, Cohere, xAI, Mistral, Groq, HuggingFace, Fireworks, ZAI, Cloudflare, Azure AI, Ollama, and LM Studio.\n- Added `--refresh-profile` for manual capability refresh and JSON profile output.\n- Hardened setup/install so non-interactive installs do not hang on prompts.\n- Verified with syntax checks, focused capability tests, aggressive tests, and fuzz/stress tests.\n\n### v7 - Local/cloud vision auto-setup\n- Installer checks for already-working vision models before asking for API keys.\n- Local runtimes such as Ollama and LM Studio are probed with a tiny image test.\n- Existing CLI, MCP, and provider credentials can be reused when they pass a real vision request.\n- OpenRouter can be scanned for vision-capable models even when the active CLI model is text-only.\n- Manual provider additions remain simple config/env updates.\n\n### v6 — Fireworks \u0026 Fuzz Hardening\n- Added **Fireworks AI** backend (Llama 3.2 90B Vision) — 22 total backends\n- Fixed 4 pre-existing **fuzz test failures** (312/312 passing):\n  - MCP int path crash — `_resolve_path()` type guard\n  - Resource leak false positive — `_INITIAL_TMP_COUNT` baseline\n  - `show_keys` crash on closed stdout — `_is_tty()` / `_safe_print()` wrappers\n  - Corrupted config crash — same print hardening\n- **Gemini-first priority** — swapped backend order so Gemini is always tried first\n- **Secure config save** — `securesave()` auto-merges env vars into config.json\n- **Bytes JSON serialization fix** — `call_gemini_multi()` tuple frame handling\n- `test_*.py` / `list_*.py` / `assets/` added to `.gitignore`\n\n### v5 — Fuzz Testing \u0026 Security Hardening\n- Added `fuzz_stress_test.py` — 312 tests covering encoding attacks, path traversal, concurrency, memory pressure, corrupted inputs, protocol violations, subprocess failures, environment corruption, type confusion, resource leaks\n- Config corruption protection — `load_config()` resilient to all JSON attack formats\n- MCP server fuzzing — type confusion, HTTP attacks, tool call attacks\n- Stdio wrapping safety — safe `_REAL_STDOUT` preservation across all subprocesses\n- GraphQL introspection blocked — `/mcp` endpoint hardened\n\n### v4 — Gemini Backend \u0026 Parallel Fire\n- Added **Google Gemini** as primary backend (6 models: 2.5 Flash, 3 Flash Preview, 2.0 Flash, 2.0 Flash Lite, 2.5 Pro, 3 Pro Preview)\n- Added **OpenRouter** support\n- **Parallel fire mode** — first 2 backends sequential (8s), rest simultaneous (12s)\n- **Fireworks AI** (planned, fully stubbed)\n- `DEFAULT_MODEL` config support\n- `requirements.txt` added\n\n### v3 — MCP Server \u0026 Always-On Mode\n- `vision_mcp_server.py` — stdio + HTTP MCP server\n- `ALWAYS_ON.md` — permanent system instruction for never saying \"can't view\"\n- `vision_watchdog.vbs` + `vision_watchdog.cs` — invisible background process manager\n- `install.py` — interactive and non-interactive auto-installer\n- OpenCode skill integration (`SKILL.md`)\n- Dynamic-skill-loader `alwaysOn` support\n\n### v2 — Multi-Provider \u0026 Video\n- Added **Groq**, **HuggingFace**, **Mistral AI** backends\n- **Video support** — ffmpeg keyframe extraction, 8 evenly-spaced frames\n- `setup.py` — interactive API key wizard with validation\n- `_has_key()` provider detection for runtime backend filtering\n- `get_mime()` MIME detection for unknown file types\n- `extract_video_frames()` GIF support (no ffmpeg needed)\n\n### v1 — Initial Release\n- Basic image analysis via **Cloudflare Workers AI** + **Azure AI Foundry**\n- CLI entry point (`vision_proxy.py main()`)\n- Pillow-based resize/JPEG compression\n- File search across Desktop, Downloads, Pictures, Documents\n- `first_success` sequential backend strategy\n- `config.json` with gitignored secrets\n- `setup.py` initial version with Cloudflare + Azure only\n\n## License\n\nGNU General Public License v3.0 — see [LICENSE](./LICENSE).\n\nThis program is free software: you can redistribute and/or modify it under the terms of the GPLv3.\nModified versions must be licensed under GPLv3 with clear attribution to the original author.\n\n© 2026 Farhan Dhrubo.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffarhanic017%2Fvision-tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffarhanic017%2Fvision-tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffarhanic017%2Fvision-tool/lists"}