{"id":50396390,"url":"https://github.com/arizawan/vidlizer","last_synced_at":"2026-05-30T21:01:42.174Z","repository":{"id":353288115,"uuid":"1215712141","full_name":"arizawan/vidlizer","owner":"arizawan","description":"Point it at a video, image, or PDF — get structured JSON. Runs local (Ollama, LM Studio, oMLX) or cloud (OpenRouter). CLI + MCP server for Claude Code, Cursor, and Claude Desktop.","archived":false,"fork":false,"pushed_at":"2026-04-23T09:07:20.000Z","size":18926,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-23T09:35:52.592Z","etag":null,"topics":["ai","apple-silicon","cli","ffmpeg","image-analysis","json","llm","lm-studio","mcp","mcp-server","mlx-whisper","ollama","omlx","openrouter","pdf","video","vision"],"latest_commit_sha":null,"homepage":"https://theatechsolutions.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arizawan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-20T07:21:53.000Z","updated_at":"2026-04-23T09:08:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/arizawan/vidlizer","commit_stats":null,"previous_names":["arizawan/vidlizer"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/arizawan/vidlizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arizawan%2Fvidlizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arizawan%2Fvidlizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arizawan%2Fvidlizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arizawan%2Fvidlizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arizawan","download_url":"https://codeload.github.com/arizawan/vidlizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arizawan%2Fvidlizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33709269,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","apple-silicon","cli","ffmpeg","image-analysis","json","llm","lm-studio","mcp","mcp-server","mlx-whisper","ollama","omlx","openrouter","pdf","video","vision"],"created_at":"2026-05-30T21:01:39.796Z","updated_at":"2026-05-30T21:01:42.160Z","avatar_url":"https://github.com/arizawan.png","language":"Python","funding_links":["https://buymeacoffee.com/arizawan"],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# vidlizer\n\n**Point it at a video, image, or PDF. Get structured JSON — scene by scene.**\n\n[![PyPI](https://img.shields.io/pypi/v/vidlizer.svg)](https://pypi.org/project/vidlizer/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)\n[![macOS](https://img.shields.io/badge/platform-macOS-lightgrey.svg)](#requirements)\n[![CI](https://github.com/arizawan/vidlizer/actions/workflows/ci.yml/badge.svg)](https://github.com/arizawan/vidlizer/actions)\n[![Tests](https://img.shields.io/badge/tests-248%20passing-brightgreen.svg)](#testing)\n[![Buy Me a Coffee](https://img.shields.io/badge/Buy%20Me%20a%20Coffee-☕-FFDD00?style=flat\u0026logo=buy-me-a-coffee\u0026logoColor=black)](https://buymeacoffee.com/arizawan)\n[![Author](https://img.shields.io/badge/author-arizawan.com-0A66C2?style=flat)](https://arizawan.com)\n[![Company](https://img.shields.io/badge/by-TheaTechSolutions-6E40C9?style=flat)](https://theatechsolutions.com)\n\n![demo](https://raw.githubusercontent.com/arizawan/vidlizer/main/assets/demo.gif)\n\n\u003c/div\u003e\n\n---\n\nvidlizer pulls frames out of any video, image, or PDF using ffmpeg, sends them to a vision LLM, and returns a `flow` array — one entry per scene. Each entry tells you what happened, who was on screen, what text was visible, and what changed. If the video has audio, it transcribes it with Apple MLX Whisper and merges the speech into each step.\n\nRuns fully local via [Ollama](https://ollama.com) or any OpenAI-compatible server (LM Studio, vLLM, oMLX) — no API key, no data leaving your machine. Or connect [OpenRouter](https://openrouter.ai) for cloud models. `vidlizer setup` detects what you have installed and writes your config in under a minute.\n\n```bash\nvidlizer demo.mp4\nvidlizer \"https://youtube.com/watch?v=...\"\nvidlizer screenshot.png\nvidlizer document.pdf\n```\n\n---\n\n## ✨ Features\n\n- **Any input** — local video, image (jpg/png/webp/…), PDF, or URL (YouTube, Loom, Vimeo, Twitter)\n- **4 providers** — Ollama (fully offline), LM Studio (port 1234), oMLX (Apple Silicon, port 8000), OpenRouter (cloud) — auto-detected in that order\n- **Cross-provider fallback** — primary model fails → automatically switches provider (e.g. oMLX → OpenRouter)\n- **JSON repair** — malformed model output is re-sent to the model to fix before skipping; recovers from partial JSON\n- **Free-model guard** — `:free` OpenRouter models auto-force `concurrency=1` to stay within rate limits\n- **3 output formats** — `--format json` (default), `summary` (plain text by phase), `markdown` (step-per-section doc)\n- **Usage tracking** — `--stats` shows per-model token + cost breakdown across all runs; `get_usage_stats()` MCP tool\n- **Auto transcript** — detects audio, transcribes with Apple MLX Whisper (Neural Engine), merges speech into each flow step\n- **Perceptual dedup** — removes near-duplicate frames before sending (saves tokens)\n- **analyze_moment** — `--start`/`--end` flags to focus on a time range\n- **In-memory cache** — repeat runs on the same file skip the API call\n- **Cost guard** — aborts if spend exceeds `MAX_COST_USD` (default $1.00)\n- **Live progress** — Rich streaming indicator shows elapsed time and token count per batch\n- **MCP server** — use from Claude Code, Cursor, Claude Desktop; provider/model locked via env vars; result includes `model_used` + `provider_used`\n- **Auto-install** — missing `ffmpeg` is brew-installed; `mlx-whisper` bundled in default install (macOS)\n- **`doctor --fix`** — interactive repair wizard: installs missing ffmpeg/Ollama/LM Studio via Homebrew, re-runs `vidlizer setup` for `.env`, upgrades mlx-whisper\n- **`mcp-setup`** — one-command MCP config wizard: detects `vidlizer-mcp`, reads `.env`, writes editor config or shows a `claude mcp add-json` one-liner\n- **Mac-native** — file picker dialog, Apple MLX transcription, handles macOS Unicode filenames (e.g. \"11:26 AM\")\n\n---\n\n## 📦 Requirements\n\n- macOS (Apple Silicon recommended for transcription speed)\n- Python 3.10+\n- **Ollama mode**: [Ollama](https://ollama.com) installed + a vision model pulled (5 GB+ RAM)\n- **LM Studio mode**: [LM Studio](https://lmstudio.ai) 0.3.16+ with a vision model loaded\n- **Cloud mode**: An [OpenRouter API key](https://openrouter.ai/keys)\n\n`ffmpeg` is installed automatically via Homebrew on first run if missing.\n\n---\n\n## 🚀 Install\n\n### Option 1 — uvx (no install, run directly)\n\n```bash\nuvx vidlizer setup\n```\n\n### Option 2 — pipx (isolated, globally available)\n\n```bash\npipx install vidlizer\nvidlizer setup    # interactive wizard: detects providers, writes .env\n```\n\n### Option 3 — pip / virtualenv\n\n```bash\npip install vidlizer\nvidlizer setup\n```\n\n### Option 4 — from source\n\n```bash\ngit clone https://github.com/arizawan/vidlizer.git\ncd vidlizer\npython -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -e .\nvidlizer setup    # or: cp env.sample .env\n```\n\n### First-run wizard\n\n`vidlizer setup` detects all installed providers, lets you pick primary + fallback, and writes a `.env` for you. It also offers to pull a vision model for Ollama if none is installed.\n\n```\n$ vidlizer setup\n  Detected providers:\n    1.  Ollama        → qwen2.5vl:3b\n    2.  OpenRouter    → google/gemma-3-27b-it:free\n\n  Primary provider (1–2): 1\n  Fallback (1–1, Enter to skip): 2\n\n✓  .env written → /your/project/.env\n```\n\n### Health check\n\n```bash\nvidlizer doctor          # shows ffmpeg, .env, provider, mlx-whisper status\nvidlizer doctor --fix    # interactive repair: brew-installs ffmpeg/Ollama/LM Studio, re-runs setup\n```\n\n### Manual provider setup\n\n**Ollama** (fully offline, no API key):\n\n```bash\n# Install Ollama from https://ollama.com, then:\nollama pull qwen2.5vl:3b   # ~3.2 GB, requires 5 GB+ RAM (recommended)\nollama pull qwen2.5vl:7b   # ~6.0 GB, requires 10 GB+ RAM (best quality)\n```\n\n**LM Studio** (GPU-accelerated local inference):\n\n```bash\n# In LM Studio: load a vision model (e.g. Qwen2.5-VL 7B), enable the local server\n# Set PROVIDER=openai and OPENAI_BASE_URL=http://localhost:1234/v1 in .env\n```\n\n**OpenRouter** (cloud):\n\n```bash\n# Paste your OpenRouter key in .env:\n# OPENROUTER_API_KEY=sk-or-v1-...\n```\n\n---\n\n## ⚡ Quick start\n\n```bash\n# Ollama — fully local, no API key\nvidlizer demo.mp4 --provider ollama --model qwen2.5vl:3b\n\n# LM Studio (or any OpenAI-compat server)\nvidlizer demo.mp4 --provider openai --model qwen/qwen2.5-vl-7b-instruct\n\n# OpenRouter (cloud)\nvidlizer demo.mp4 --provider openrouter --model google/gemini-2.5-flash\n\n# Analyze a YouTube video\nvidlizer \"https://www.youtube.com/watch?v=dQw4w9WgXcQ\"\n\n# Analyze a single image or PDF\nvidlizer screenshot.png\nvidlizer report.pdf\n\n# Focus on a time range\nvidlizer demo.mp4 --start 30 --end 90\n\n# Output as Markdown or plain-text summary\nvidlizer demo.mp4 --format markdown -o result.md\nvidlizer demo.mp4 --format summary -o result.txt\n```\n\nRun with no arguments for an interactive file picker + provider/model selector.\n\n---\n\n## 📄 Output\n\nThree formats via `--format`:\n\n| Format | Flag | Default extension | Description |\n|---|---|---|---|\n| JSON | `--format json` | `.analysis.json` | Full structured flow array (default) |\n| Markdown | `--format markdown` | `.analysis.md` | Step-per-section document with scene/action/speech |\n| Summary | `--format summary` | `.analysis.txt` | Plain text grouped by phase |\n\nDefault output path is `\u003cnormalized-name\u003e.analysis.json` (or matching extension), or pass `-o path`.\n\n```json\n{\n  \"flow\": [\n    {\n      \"step\": 1,\n      \"timestamp_s\": 0.0,\n      \"phase\": \"Introduction\",\n      \"scene\": \"Title card with product logo on dark background.\",\n      \"subjects\": [\"Logo\", \"Text overlay\"],\n      \"action\": \"Static title card displayed.\",\n      \"text_visible\": \"vidlizer — analyze any video\",\n      \"context\": \"Opening of a product demo.\",\n      \"observations\": \"Clean minimal design.\",\n      \"next_scene\": \"Screen recording of the CLI.\",\n      \"speech\": \"Welcome to the vidlizer demo.\"\n    }\n  ],\n  \"transcript\": [\n    { \"start\": 0.0, \"end\": 2.4, \"text\": \"Welcome to the vidlizer demo.\" }\n  ],\n  \"model_used\": \"gemma-4-E2B-it-MLX-4bit\",\n  \"provider_used\": \"openai\"\n}\n```\n\n`model_used` and `provider_used` reflect the model that actually produced the result — including after fallback. Surfaced in MCP `analyze_video` response so agents know which provider ran.\n\n### Flow step fields\n\n| Field | Description |\n|---|---|\n| `step` | Sequential integer |\n| `timestamp_s` | Approximate time in seconds (from frame label) |\n| `phase` | Logical section — Introduction, Demo, Action, Conclusion… |\n| `scene` | What is currently visible |\n| `subjects` | Key people, objects, UI elements present |\n| `action` | What is happening — interaction, movement, narration, event |\n| `text_visible` | All readable text on screen |\n| `context` | Persistent state — timer, score, topic, brand… |\n| `observations` | Errors, anomalies, emotions, key facts |\n| `next_scene` | Brief description of what follows |\n| `speech` | Transcript text spoken during this step (audio videos only) |\n\n---\n\n## 🤖 Models\n\nAll providers send **one frame per request** (batch_size=1) for maximum compatibility with context-limited models. Thinking-mode output (`\u003cthink\u003e` tags) is stripped automatically.\n\n### Local — Ollama\n\nFully offline, no API key. Install with `ollama pull \u003cname\u003e`:\n\n| Model | Disk | RAM | Notes |\n|---|---|---|---|\n| `qwen2.5vl:3b` ★ | 3.2 GB | ~5 GB | **Recommended** — 128K ctx, strong JSON, multi-image |\n| `qwen2.5vl:7b` ★ | 6.0 GB | ~9 GB | Best Ollama quality — 128K ctx, needs 10+ GB RAM |\n| `minicpm-v:8b` | 5.5 GB | ~8 GB | Strong OCR + visual reasoning, 32K ctx |\n| `llava-onevision:7b` | 5.5 GB | ~8 GB | Strong multi-image + video frames, reliable JSON |\n\nFallback order (if configured model unavailable): `qwen2.5vl:7b` → `qwen2.5vl:3b` → `minicpm-v:8b` → `llava-onevision:7b` → `llava:13b` → `llava:7b`\n\nUses Ollama's native `/api/chat` with `format: json` for reliable structured output.\n\n### Local — LM Studio / oMLX / vLLM / LocalAI (OpenAI-compatible)\n\nGPU-accelerated inference via any OpenAI-compatible server. Set `PROVIDER=openai` and point `OPENAI_BASE_URL` at your server.\n\n| Model | VRAM | Notes |\n|---|---|---|\n| `qwen/qwen2.5-vl-7b-instruct` ★ | ~8 GB | **Recommended** — 128K ctx, reliable JSON, multi-image |\n| `qwen/qwen3-vl-8b` ★ | ~10 GB | Latest Qwen vision — thinking tags stripped automatically |\n| `qwen/qwen2.5-vl-3b-instruct` | ~5 GB | Lightweight — fast, 5–6 GB VRAM |\n| `google/gemma-4-e4b-it` | ~6 GB | Google MoE — LM Studio 0.3.16+ native support |\n| `google/gemma-4-9b-it` | ~10 GB | Stronger Gemma 4 — 128K ctx, better instruction following |\n| `zai-org/glm-4.6v-flash` | ~8 GB | ZhipuAI MoE — 128K ctx, strong JSON, low latency |\n| `openbmb/minicpm-v-4.5` | ~8 GB | 8B Qwen3-based — strong OCR, multi-image, vLLM ready |\n\n**oMLX** (Apple Silicon native, [omlx.ai](https://omlx.ai)) — MLX-format models from HuggingFace, **auto-detected on port 8000** (distinct from LM Studio). Model IDs are HuggingFace paths:\n\n| oMLX Model | Unified RAM | Notes |\n|---|---|---|\n| `mlx-community/Qwen2.5-VL-7B-Instruct-8bit` ★ | ~8 GB | Best Apple Silicon pick — fast, strong JSON |\n| `mlx-community/Qwen2.5-VL-3B-Instruct-8bit` | ~4 GB | Lightweight — 4–5 GB RAM |\n| `mlx-community/Qwen3-VL-8B-8bit` | ~9 GB | Latest Qwen vision — thinking tags stripped |\n| `mlx-community/MiniCPM-V-2_6-8bit` | ~8 GB | Strong OCR + reasoning |\n\nModel IDs are as shown in LM Studio's model browser, oMLX's admin panel, or your vLLM config. LM Studio / oMLX serve one model at a time (no fallback needed). vLLM with multiple models loaded uses the same fallback sequence.\n\nFallback fragment order (vLLM / oMLX with multiple models): `qwen2.5-vl-7b` → `qwen2.5-vl-3b` → `qwen3-vl` → `gemma-4` → `glm-4` → `minicpm-v` → `llava-onevision` → `llava`\n\n### Cloud — OpenRouter\n\nModels fetched live with current pricing. Run `vidlizer --list-models` to see the live list.\n\n| Model | Input / 1M tokens | Notes |\n|---|---|---|\n| `google/gemini-2.5-flash` ★ | $0.15 | **Recommended** — 1M ctx, fast, accurate |\n| `google/gemini-2.5-flash-lite` | $0.075 | Cheaper — slightly less accurate |\n| `google/gemini-2.5-pro` | $1.25 | Best quality, higher cost |\n| `openai/gpt-4o` | $2.50 | OpenAI flagship |\n| `openai/gpt-4o-mini` | $0.15 | OpenAI budget option |\n| `nvidia/nemotron-nano-12b-v2-vl:free` | free ⚡ | Rate-limited (8K/req), 128K ctx |\n| `google/gemma-4-31b-it:free` | free ⚡ | Rate-limited, 128K ctx |\n\nFree models auto-fallback to the cheapest available paid model on failure.\n\n---\n\n## 🎙️ Transcription\n\nFor videos with an audio track, vidlizer automatically:\n\n1. Extracts a mono 16kHz WAV with ffmpeg\n2. Transcribes with **Apple MLX Whisper** (Neural Engine + GPU on M-series)\n3. Merges each transcript segment into the nearest flow step as `speech`\n\n`mlx-whisper` is included in the default install on macOS. The base Whisper model (~150 MB) is downloaded on first transcription.\n\nTo opt out: `--no-transcript`\n\n---\n\n## 🛠️ CLI reference\n\n### Subcommands\n\n```bash\nvidlizer setup              # interactive wizard: detects providers, writes .env\nvidlizer doctor             # health check: ffmpeg, .env, providers, mlx-whisper\nvidlizer doctor --fix       # auto-repair: brew-installs missing deps, re-runs setup\nvidlizer mcp-setup          # generate MCP config for Claude Code / Cursor / Claude Desktop\n```\n\n### Analysis options\n\n```\nvidlizer [video] [options]\n\npositional:\n  video                 Path to file or URL (YouTube/Loom/Vimeo/Twitter)\n\noptions:\n  -o, --output PATH     Output path (default: \u003cname\u003e.analysis.json/.md/.txt)\n  --format FORMAT       Output format: json (default), summary, markdown\n  --provider PROVIDER   ollama | openai | openrouter  (openai covers LM Studio + oMLX + vLLM)\n  --model MODEL         Model slug — Ollama name, OpenAI-compat ID, or OpenRouter slug\n  --max-frames N        Max frames to send (default 60, hard cap 200)\n  --start SECONDS       Analyze from this timestamp\n  --end SECONDS         Analyze up to this timestamp\n  --scene THRESHOLD     Scene-change sensitivity 0–1 (default 0.1)\n  --min-interval SECS   Minimum seconds between frames (default 2)\n  --fps FPS             Extract at fixed FPS instead of scene-change\n  --scale PX            Frame width in pixels (default 512)\n  --batch-size N        Frames per API call (0 = auto, default 1 for all providers)\n  --concurrency N       Parallel batch workers (default: 4 for OpenRouter, 1 for local)\n  --dedup-threshold N   Perceptual dedup Hamming distance (default 8, 0 = off)\n  --no-transcript       Skip audio transcription\n  --max-cost USD        Abort if spend exceeds this (default 1.00)\n  --timeout SECS        Per-request timeout (default 600)\n  -v, --verbose         Debug output\n  --stats               Show token + cost usage stats by model, then exit\n  --clear-stats         Reset usage log, then exit\n```\n\n---\n\n## 🔧 Environment variables\n\nCopy `env.sample` to `.env`:\n\n```bash\n# ── Provider ────────────────────────────────────────────────────────────────\n# ollama    — local Ollama server (no API key, no cost)\n# openai    — any OpenAI-compatible server (LM Studio, oMLX, vLLM, LocalAI, real OpenAI)\n# openrouter — cloud inference via OpenRouter\nPROVIDER=ollama\n\n# ── Ollama (local, no API key) ──────────────────────────────────────────────\nOLLAMA_HOST=http://localhost:11434\nOLLAMA_MODEL=qwen2.5vl:3b\n\n# ── OpenAI-compatible (LM Studio, oMLX, vLLM, LocalAI, real OpenAI) ─────────\n#   LM Studio default port: 1234  |  oMLX default port: 8000  |  vLLM: 8000\nOPENAI_BASE_URL=http://localhost:1234/v1\nOPENAI_API_KEY=lm-studio        # \"not-needed\" for oMLX/vLLM, real key for OpenAI\nOPENAI_MODEL=                   # exact model ID as shown in server (required)\n\n# ── OpenRouter (cloud) ──────────────────────────────────────────────────────\nOPENROUTER_API_KEY=sk-or-v1-...\nOPENROUTER_MODEL=google/gemini-2.5-flash\n\n# ── Frame extraction ────────────────────────────────────────────────────────\nSCENE_THRESHOLD=0.1     # lower = more frames\nMIN_INTERVAL=2          # seconds between forced frames\nMAX_FRAMES=60\nFRAME_WIDTH=512\nMAX_COST_USD=1.00\nBATCH_SIZE=0            # 0 = auto (defaults to 1 per request for all providers)\nREQUEST_TIMEOUT=600\n\n# ── Fallback ────────────────────────────────────────────────────────────────\n# Same-provider fallback: set FALLBACK_MODEL only.\n#   Blank = auto-detect installed models (Ollama) or available models (OpenAI-compat).\n# Cross-provider fallback: set FALLBACK_PROVIDER + FALLBACK_MODEL.\n#   FALLBACK_BASE_URL and FALLBACK_API_KEY default to the primary provider's values\n#   if not set — override only when the fallback uses a different server/key.\nFALLBACK_PROVIDER=      # ollama | openai | openrouter  (blank = same as PROVIDER)\nFALLBACK_MODEL=         # model ID for fallback  (blank = auto-detect same-provider)\nFALLBACK_BASE_URL=      # base URL for fallback openai/ollama server (optional)\nFALLBACK_API_KEY=       # API key for fallback provider (optional)\n```\n\n---\n\n## 📊 Usage tracking\n\nEvery successful run appends a record to `~/.cache/vidlizer/usage.jsonl`. Test runs (`pytest`) are excluded.\n\n```bash\nvidlizer --stats          # print per-model breakdown\nvidlizer --clear-stats    # reset log\n```\n\nExample output:\n\n```\nUsage statistics  (/Users/you/.cache/vidlizer/usage.jsonl)\n\n  Total runs:       12\n  Total tokens in:  58,240\n  Total tokens out: 6,102\n  Total cost:       ~$0.0312\n  Total steps:      94\n\n  Model                               Provider     Runs   Tokens in  Tokens out  Cost USD\n  gemma-4-E2B-it-MLX-4bit             openai          9      45,800       4,900      free\n  google/gemini-2.5-flash             openrouter       3      12,440       1,202  ~$0.0312\n```\n\nAlso available as MCP tool: `get_usage_stats()` → same data as JSON. `clear_usage_stats()` resets the log.\n\n---\n\n## 🔌 MCP Server\n\nUse vidlizer from any MCP-compatible agent — Claude Code, Cursor, Claude Desktop, Gemini CLI.\n\n**Model and provider are set via env vars and cannot be overridden by the AI agent.** This prevents agents from switching to unexpected or expensive models mid-session.\n\n### Install\n\n```bash\n# uvx (no install — run directly)\nuvx \"vidlizer[mcp]\" vidlizer-mcp\n\n# pipx (isolated, globally available)\npipx install \"vidlizer[mcp]\"\n\n# If vidlizer is already installed via pipx — inject mcp into the same venv\npipx inject vidlizer mcp\n\n# pip / virtualenv\npip install \"vidlizer[mcp]\"\n```\n\n`vidlizer-mcp` is added to your PATH automatically. Verify with `which vidlizer-mcp`.\n\n\u003e **Note on pipx**: `pipx install vidlizer` creates the `vidlizer-mcp` entry point but the `mcp` package is not installed. Run `pipx inject vidlizer mcp` to add it, or use `pipx install \"vidlizer[mcp]\"` upfront.\n\n### One-command MCP setup\n\n```bash\nvidlizer mcp-setup\n```\n\nInteractive wizard — detects `vidlizer-mcp`, reads your `.env`, and either writes the config to your editor's config file or shows a one-liner to copy-paste. Supports Claude Code, Cursor, Claude Desktop, Windsurf.\n\n### Configure manually\n\n#### Quick start — uvx (no install required)\n\nNo `pip install` or `pipx` needed. uvx downloads and runs vidlizer on the fly:\n\n```json\n{\n  \"mcpServers\": {\n    \"vidlizer\": {\n      \"type\": \"stdio\",\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"vidlizer[mcp]\", \"vidlizer-mcp\"],\n      \"env\": {\n        \"PROVIDER\": \"ollama\",\n        \"OLLAMA_HOST\": \"http://localhost:11434\",\n        \"OLLAMA_MODEL\": \"gemma4:2b\"\n      }\n    }\n  }\n}\n```\n\nSwap the `env` block for any provider (OpenRouter, LM Studio, oMLX) — the `command`/`args` stay the same.\n\n---\n\nUse the **absolute path** to the binary — `which vidlizer-mcp` gives it. All configs below use JSON (works in Claude Code, Cursor, Claude Desktop, Gemini CLI).\n\nFor Claude Code via CLI:\n```bash\nclaude mcp add-json vidlizer '{\"type\":\"stdio\",\"command\":\"/path/to/vidlizer-mcp\",\"env\":{\"PROVIDER\":\"openrouter\",\"OPENROUTER_API_KEY\":\"sk-or-v1-...\",\"OPENROUTER_MODEL\":\"google/gemini-2.5-flash\"}}'\n# Add --scope project for per-project config instead of global\n```\n\n---\n\n#### Single provider — OpenRouter (cloud)\n\n```json\n{\n  \"mcpServers\": {\n    \"vidlizer\": {\n      \"type\": \"stdio\",\n      \"command\": \"/absolute/path/to/.venv/bin/vidlizer-mcp\",\n      \"env\": {\n        \"PROVIDER\": \"openrouter\",\n        \"OPENROUTER_API_KEY\": \"sk-or-v1-...\",\n        \"OPENROUTER_MODEL\": \"google/gemini-2.5-flash\"\n      }\n    }\n  }\n}\n```\n\n\u003e Free models (`:free` suffix) automatically fall back to the cheapest paid model if rate-limited — no extra config needed.\n\n---\n\n#### Single provider — Ollama (local, no API key)\n\n```json\n{\n  \"mcpServers\": {\n    \"vidlizer\": {\n      \"type\": \"stdio\",\n      \"command\": \"/absolute/path/to/.venv/bin/vidlizer-mcp\",\n      \"env\": {\n        \"PROVIDER\": \"ollama\",\n        \"OLLAMA_HOST\": \"http://localhost:11434\",\n        \"OLLAMA_MODEL\": \"gemma4:2b\"\n      }\n    }\n  }\n}\n```\n\n\u003e `FALLBACK_MODEL` is optional — omitting it auto-detects installed models and tries them in preferred order.\n\n---\n\n#### Single provider — LM Studio (OpenAI-compatible, port 1234)\n\n```json\n{\n  \"mcpServers\": {\n    \"vidlizer\": {\n      \"type\": \"stdio\",\n      \"command\": \"/absolute/path/to/.venv/bin/vidlizer-mcp\",\n      \"env\": {\n        \"PROVIDER\": \"openai\",\n        \"OPENAI_BASE_URL\": \"http://localhost:1234/v1\",\n        \"OPENAI_API_KEY\": \"lm-studio\",\n        \"OPENAI_MODEL\": \"qwen/qwen2.5-vl-7b-instruct\"\n      }\n    }\n  }\n}\n```\n\n---\n\n#### Single provider — oMLX (Apple Silicon, port 8000)\n\n```json\n{\n  \"mcpServers\": {\n    \"vidlizer\": {\n      \"type\": \"stdio\",\n      \"command\": \"/absolute/path/to/.venv/bin/vidlizer-mcp\",\n      \"env\": {\n        \"PROVIDER\": \"openai\",\n        \"OPENAI_BASE_URL\": \"http://localhost:8000/v1\",\n        \"OPENAI_API_KEY\": \"not-needed\",\n        \"OPENAI_MODEL\": \"mlx-community/Qwen2.5-VL-7B-Instruct-8bit\"\n      }\n    }\n  }\n}\n```\n\n---\n\n#### Cross-provider fallback — oMLX → OpenRouter\n\nPrimary runs on local oMLX; if it fails, switches to OpenRouter cloud automatically.\n\n```json\n{\n  \"mcpServers\": {\n    \"vidlizer\": {\n      \"type\": \"stdio\",\n      \"command\": \"/absolute/path/to/.venv/bin/vidlizer-mcp\",\n      \"env\": {\n        \"PROVIDER\": \"openai\",\n        \"OPENAI_BASE_URL\": \"http://localhost:8000/v1\",\n        \"OPENAI_API_KEY\": \"not-needed\",\n        \"OPENAI_MODEL\": \"mlx-community/Qwen2.5-VL-7B-Instruct-8bit\",\n        \"FALLBACK_PROVIDER\": \"openrouter\",\n        \"FALLBACK_MODEL\": \"google/gemini-2.5-flash\",\n        \"FALLBACK_API_KEY\": \"sk-or-v1-...\"\n      }\n    }\n  }\n}\n```\n\n---\n\n#### Cross-provider fallback — oMLX → Ollama\n\nBoth local; oMLX fails (model missing / server down) → Ollama takes over.\n\n```json\n{\n  \"mcpServers\": {\n    \"vidlizer\": {\n      \"type\": \"stdio\",\n      \"command\": \"/absolute/path/to/.venv/bin/vidlizer-mcp\",\n      \"env\": {\n        \"PROVIDER\": \"openai\",\n        \"OPENAI_BASE_URL\": \"http://localhost:8000/v1\",\n        \"OPENAI_API_KEY\": \"not-needed\",\n        \"OPENAI_MODEL\": \"mlx-community/Qwen2.5-VL-7B-Instruct-8bit\",\n        \"FALLBACK_PROVIDER\": \"ollama\",\n        \"FALLBACK_MODEL\": \"qwen2.5vl:7b\"\n      }\n    }\n  }\n}\n```\n\n\u003e `FALLBACK_BASE_URL` defaults to `OLLAMA_HOST` (or `http://localhost:11434`) — only set it if Ollama is on a non-standard host.\n\n---\n\n#### Cross-provider fallback — Ollama → OpenRouter\n\nLocal-first; cloud kicks in if no model is installed or inference fails.\n\n```json\n{\n  \"mcpServers\": {\n    \"vidlizer\": {\n      \"type\": \"stdio\",\n      \"command\": \"/absolute/path/to/.venv/bin/vidlizer-mcp\",\n      \"env\": {\n        \"PROVIDER\": \"ollama\",\n        \"OLLAMA_HOST\": \"http://localhost:11434\",\n        \"OLLAMA_MODEL\": \"qwen2.5vl:7b\",\n        \"FALLBACK_PROVIDER\": \"openrouter\",\n        \"FALLBACK_MODEL\": \"google/gemini-2.5-flash\",\n        \"FALLBACK_API_KEY\": \"sk-or-v1-...\"\n      }\n    }\n  }\n}\n```\n\n---\n\n#### Same-provider fallback — pin explicit fallback model\n\nWhen auto-detection isn't wanted; both models use the same provider.\n\n```json\n{\n  \"mcpServers\": {\n    \"vidlizer\": {\n      \"type\": \"stdio\",\n      \"command\": \"/absolute/path/to/.venv/bin/vidlizer-mcp\",\n      \"env\": {\n        \"PROVIDER\": \"ollama\",\n        \"OLLAMA_MODEL\": \"qwen2.5vl:7b\",\n        \"FALLBACK_MODEL\": \"qwen2.5vl:3b\"\n      }\n    }\n  }\n}\n```\n\n---\n\n**Fallback env var reference:**\n\n| Var | Purpose | Default |\n|---|---|---|\n| `FALLBACK_PROVIDER` | Provider for fallback (`ollama`/`openai`/`openrouter`) | same as `PROVIDER` |\n| `FALLBACK_MODEL` | Model ID for fallback | auto-detect (same provider) |\n| `FALLBACK_BASE_URL` | Base URL for fallback `openai`/`ollama` server | `OPENAI_BASE_URL` or `OLLAMA_HOST` |\n| `FALLBACK_API_KEY` | API key for fallback provider | `OPENAI_API_KEY` or `OPENROUTER_API_KEY` |\n\n### Logs\n\nAll activity (frame extraction, API calls, errors) written to:\n```bash\ntail -f ~/.cache/vidlizer/mcp.log\n```\n\n### Tools\n\n| Tool | Returns | Tokens out |\n|---|---|---|\n| `analyze_video(path, **opts)` | `analysis_id` + meta | ~100 |\n| `list_analyses()` | all stored analyses (meta only) | ~50/entry |\n| `get_summary(id, level)` | brief/medium/full text summary | ~200–2K |\n| `get_step(id, step)` | single flow step | ~150 |\n| `get_steps(id, start, end)` | step range | scaled |\n| `get_phase(id, phase)` | all steps in named phase | scaled |\n| `search_analysis(id, query)` | steps matching text (`query` or `keyword`) | only hits |\n| `get_transcript(id, start_s, end_s)` | transcript slice | scaled |\n| `get_full_analysis(id)` | full flow + transcript | full |\n| `delete_analysis(id)` | confirmation | ~10 |\n| `get_usage_stats()` | per-model token + cost breakdown | ~50/model |\n| `clear_usage_stats()` | reset usage log | ~10 |\n\n### Token-efficient workflow\n\n`analyze_video` stores the full result on disk and returns only `analysis_id` + metadata (~100 tokens). The LLM pulls specific parts on demand — a 60-step video costs ~100 tokens to register but only loads what's needed per query.\n\n```\nagent: analyze_video(\"demo.mp4\")\n→ { \"analysis_id\": \"abc123\", \"step_count\": 42, \"phases\": [\"Intro\", \"Demo\", \"Outro\"] }\n\nagent: get_summary(\"abc123\", level=\"brief\")\n→ \"Intro: Title card displayed | Demo: CLI recording | Outro: Results shown\"\n\nagent: get_phase(\"abc123\", \"Demo\")\n→ [ { step, timestamp_s, action, scene }, … ]\n\nagent: search_analysis(\"abc123\", \"error\")\n→ [ { step: 17, matched_field: \"observations\", matched_value: \"Stack trace visible\" } ]\n```\n\n### MCP resources\n\n| URI | Content |\n|---|---|\n| `vidlizer://analyses` | All analyses (meta JSON) |\n| `vidlizer://analyses/{id}` | Full analysis JSON |\n| `vidlizer://analyses/{id}/summary` | Medium text summary |\n\n---\n\n## 🧪 Testing\n\nFully automated test suite — **248 unit + integration tests, 3 e2e tests**.\n\n```bash\nmake install-dev    # installs pytest, pytest-html, pytest-mock\nmake test           # runs unit + integration (no network) → reports/test-report.html\nmake test-e2e       # also runs YouTube download + full pipeline e2e\nmake smoke          # real-provider smoke test against all detected providers\n```\n\nUnit/integration report: `reports/test-report.html`. Tests cover:\n\n- Frame extraction (ffmpeg), perceptual dedup, cache TTL\n- Audio detection, transcript merge (no duplicates)\n- PDF → frames, image encoding, URL detection\n- Output formatter: json/summary/markdown correctness\n- Full pipeline with mocked OpenRouter (fake HTTP server)\n- HTTP layer: SSE parsing, 429 handling, cost cap, Ollama streaming\n- Batch: JSON repair retry, free-model concurrency guard\n- Models: pricing lookup, fallback sequences, format helpers\n- Usage tracking: record/stats/clear lifecycle\n- Real CLI subprocess invocations against real media\n- Real YouTube download + full analysis (opt-in `-m e2e`)\n\n### Smoke test\n\n`make smoke` runs the full real-provider pipeline against every detected provider in order (Ollama → LM Studio → oMLX → OpenRouter). Before tests start, it interactively prompts for each local provider:\n\n- Vision model found → `\"Test ollama with qwen2.5vl:3b? [Y/n]\"`\n- No vision model (Ollama) → `\"Download qwen2.5vl:3b (~2.3 GB)? [y/N]\"`\n- No vision model (LM Studio/oMLX) → instructions to load one + retry prompt\n\nEach provider is tested in isolation with its own output directory (prevents cache contamination). Local models are unloaded from VRAM after each provider's tests. Results land in `reports/smoke-TIMESTAMP.html` with per-provider pass/fail scorecards.\n\n```bash\nmake smoke                                  # auto-detect all providers\nmake smoke ARGS=\"--provider ollama\"         # force single provider\nmake smoke ARGS=\"--provider openrouter\"     # OpenRouter only (no local needed)\n```\n\n---\n\n## 🤝 Contributing\n\nContributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, code style, and PR guidelines.\n\n- **Bug report**: [open an issue](https://github.com/arizawan/vidlizer/issues) — include provider, model, macOS version, full error\n- **Feature request**: open an issue describing the use case\n- **PR**: branch from `main`, add tests, `make test` passes, `ruff check vidlizer/` clean\n\n---\n\n## ❤️ Support\n\nvidlizer is free and open source. If it saves you time:\n\n[![Buy Me a Coffee](https://img.shields.io/badge/Buy%20Me%20a%20Coffee-%E2%98%95-FFDD00?style=for-the-badge\u0026logo=buy-me-a-coffee\u0026logoColor=black)](https://buymeacoffee.com/arizawan)\n\n---\n\n## 📝 License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farizawan%2Fvidlizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farizawan%2Fvidlizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farizawan%2Fvidlizer/lists"}