{"id":34071050,"url":"https://github.com/byjlw/video-analyzer","last_synced_at":"2026-04-01T19:32:17.690Z","repository":{"id":262139080,"uuid":"886299924","full_name":"byjlw/video-analyzer","owner":"byjlw","description":"Analyze videos using LLMs, Computer Vision and Automatic Speech Recognition","archived":false,"fork":false,"pushed_at":"2026-03-19T06:03:18.000Z","size":375,"stargazers_count":1321,"open_issues_count":2,"forks_count":183,"subscribers_count":19,"default_branch":"main","last_synced_at":"2026-03-19T21:21:09.552Z","etag":null,"topics":["asr","llms","video","video-processing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/byjlw.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-10T17:49:24.000Z","updated_at":"2026-03-17T13:47:25.000Z","dependencies_parsed_at":"2024-11-10T19:40:08.825Z","dependency_job_id":"0f9d411f-2d88-4ddc-9975-cce805493d69","html_url":"https://github.com/byjlw/video-analyzer","commit_stats":null,"previous_names":["byjlw/video-analyzer"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/byjlw/video-analyzer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/byjlw%2Fvideo-analyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/byjlw%2Fvideo-analyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/byjlw%2Fvideo-analyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/byjlw%2Fvideo-analyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/byjlw","download_url":"https://codeload.github.com/byjlw/video-analyzer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/byjlw%2Fvideo-analyzer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31291147,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asr","llms","video","video-processing"],"created_at":"2025-12-14T07:48:39.121Z","updated_at":"2026-04-01T19:32:17.682Z","avatar_url":"https://github.com/byjlw.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Video Analysis using vision models like Llama3.2 Vision and OpenAI's Whisper Models\n\nA video analysis tool that combines vision models like Llama's 11B vision model and Whisper to create a description by taking key frames, feeding them to the vision model to get details. It uses the details from each frame and the transcript, if available, to describe what's happening in the video. \n\n## Table of Contents\n- [Features](#features)\n- [Requirements](#requirements)\n  - [System Requirements](#system-requirements)\n  - [Installation](#installation)\n  - [Ollama Setup](#ollama-setup)\n  - [OpenAI-compatible API Setup](#openai-compatible-api-setup-optional)\n- [Usage](#usage)\n  - [Quick Start](#quick-start)\n  - [Sample Output](#sample-output)\n  - [Complete Usage Guide](docs/USAGES.md)\n- [Design](#design)\n  - [Detailed Design Documentation](docs/DESIGN.md)\n- [Project Structure](#project-structure)\n- [Configuration](#configuration)\n- [Output](#output)\n- [Prompt Tuning](#prompt-tuning)\n- [Uninstallation](#uninstallation)\n- [License](#license)\n- [Contributing](#contributing)\n\n## Features\n- 💻 Can run completely locally - no cloud services or API keys needed\n- ☁️  Or, leverage any OpenAI API compatible LLM service (openrouter, openai, etc) for speed and scale\n- 🎬 Intelligent key frame extraction from videos\n- 🔊 High-quality audio transcription using OpenAI's Whisper\n- 👁️ Frame analysis using Ollama and Llama3.2 11B Vision Model\n- 📝 Natural language descriptions of video content\n- 🔄 Automatic handling of poor quality audio\n- 📊 Detailed JSON output of analysis results\n- ⚙️ Highly configurable through command line arguments or config file\n\n## Design\nThe system operates in three stages:\n\n1. Frame Extraction \u0026 Audio Processing\n   - Uses OpenCV to extract key frames\n   - Processes audio using Whisper for transcription\n   - Handles poor quality audio with confidence checks\n\n2. Frame Analysis\n   - Analyzes each frame using vision LLM\n   - Each analysis includes context from previous frames\n   - Maintains chronological progression\n   - Uses frame_analysis.txt prompt template\n\n3. Video Reconstruction\n   - Combines frame analyses chronologically\n   - Integrates audio transcript\n   - Uses first frame to set the scene\n   - Creates comprehensive video description\n\n![Design](docs/design.png)\n\n## Requirements\n\n### System Requirements\n- Python 3.11 or higher\n- FFmpeg (required for audio processing)\n- When running LLMs locally (not necessary when using openrouter)\n  - At least 16GB RAM (32GB recommended)\n  - GPU at least 12GB of VRAM or Apple M Series with at least 32GB\n\n### Installation\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/byjlw/video-analyzer.git\ncd video-analyzer\n```\n\n2. Create and activate a virtual environment:\n```bash\npython3 -m venv .venv\nsource .venv/bin/activate  # On Windows: .venv\\Scripts\\activate\n```\n\n3. Install the package:\n```bash\npip install .  # For regular installation\n# OR\npip install -e .  # For development installation\n```\n\n4. Install FFmpeg:\n- Ubuntu/Debian:\n  ```bash\n  sudo apt-get update \u0026\u0026 sudo apt-get install -y ffmpeg\n  ```\n- macOS:\n  ```bash\n  brew install ffmpeg\n  ```\n- Windows:\n  ```bash\n  choco install ffmpeg\n  ```\n\n### Ollama Setup\n\n1. Install Ollama following the instructions at [ollama.ai](https://ollama.ai)\n\n2. Pull the default vision model:\n```bash\nollama pull llama3.2-vision\n```\n\n3. Start the Ollama service:\n```bash\nollama serve\n```\n\n### OpenAI-compatible API Setup (Optional)\n\nIf you want to use OpenAI-compatible APIs (like OpenRouter or OpenAI) instead of Ollama:\n\n1. Get an API key from your provider:\n   - [OpenRouter](https://openrouter.ai)\n   - [OpenAI](https://platform.openai.com)\n\n2. Configure via command line:\n   ```bash\n   # For OpenRouter\n   video-analyzer video.mp4 --client openai_api --api-key your-key --api-url https://openrouter.ai/api/v1 --model gpt-4o\n\n   # For OpenAI\n   video-analyzer video.mp4 --client openai_api --api-key your-key --api-url https://api.openai.com/v1 --model gpt-4o\n   ```\n\n   Or add to config/config.json:\n   ```json\n   {\n     \"clients\": {\n       \"default\": \"openai_api\",\n       \"openai_api\": {\n         \"api_key\": \"your-api-key\",\n         \"api_url\": \"https://openrouter.ai/api/v1\"  # or https://api.openai.com/v1\n       }\n     }\n   }\n   ```\n\nNote: With OpenRouter, you can use llama 3.2 11b vision for free by adding :free to the model name\n\n## Design\nFor detailed information about the project's design and implementation, including how to make changes, see [docs/DESIGN.md](docs/DESIGN.md).\n\n## Usage\n\nFor detailed usage instructions and all available options, see [docs/USAGES.md](docs/USAGES.md).\n\n### Quick Start\n\n```bash\n# Local analysis with Ollama (default)\nvideo-analyzer video.mp4\n\n# Cloud analysis with OpenRouter\nvideo-analyzer video.mp4 \\\n    --client openai_api \\\n    --api-key your-key \\\n    --api-url https://openrouter.ai/api/v1 \\\n    --model meta-llama/llama-3.2-11b-vision-instruct:free\n\n# Analysis with custom prompt\nvideo-analyzer video.mp4 \\\n    --prompt \"What activities are happening in this video?\" \\\n    --whisper-model large\n```\n\n## Output\n\nThe tool generates a JSON file (`output\\analysis.json`) containing:\n- Metadata about the analysis\n- Audio transcript (if available)\n- Frame-by-frame analysis\n- Final video description\n\n### Sample Output\n```\nThe video begins with a person with long blonde hair, wearing a pink t-shirt and yellow shorts, standing in front of a black plastic tub or container on wheels. The ground appears to be covered in wood chips.\\n\\nAs the video progresses, the person remains facing away from the camera, looking down at something inside the tub. ........\n```\nfull sample output in `docs/sample_analysis.json`\n## Configuration\n\nThe tool uses a cascading configuration system with command line arguments taking highest priority, followed by user config (config/config.json), and finally the default config. See [docs/USAGES.md](docs/USAGES.md) for detailed configuration options.\n\n\n## Prompt Tuning\n\nThe prompts that drive frame analysis and video reconstruction can be automatically optimized for your specific content and use case using [video-analyzer-tune](video-analyzer-tune/README.md).\n\n```bash\npip install video-analyzer-tune\n```\n\nRun `video-analyzer` on a few representative videos, edit the outputs to show what ideal results look like, then let DSPy MIPROv2 find better prompt instructions automatically. The tuned prompts are written as new files you point to via your config — the main package is unaffected.\n\nSee [video-analyzer-tune/README.md](video-analyzer-tune/README.md) for full instructions.\n\n## Uninstallation\n\nTo uninstall the package:\n```bash\npip uninstall video-analyzer\n```\n\n## License\n\nApache License\n\n## Contributing\n\nWe welcome contributions! Please see [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md) for detailed guidelines on how to:\n- Review the project design\n- Propose changes through GitHub Discussions\n- Submit pull requests\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbyjlw%2Fvideo-analyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbyjlw%2Fvideo-analyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbyjlw%2Fvideo-analyzer/lists"}