{"id":51302238,"url":"https://github.com/jit-roy/prompt2clip","last_synced_at":"2026-06-30T21:01:16.657Z","repository":{"id":331762760,"uuid":"1131732566","full_name":"Jit-Roy/Prompt2Clip","owner":"Jit-Roy","description":"Turn long videos into viral clips automatically. Long-form is a drag, but clipping shouldn't be. ✂️ Just drop an instruction like \"Extract the funniest moments\" or \"Only clip Speaker B,\" and our LLM does the rest. It’s giving unmatched control.","archived":false,"fork":false,"pushed_at":"2026-06-27T13:54:59.000Z","size":2508,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-27T15:22:36.854Z","etag":null,"topics":["clip","clips","llm","shorts","shorts-maker","video-clip"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jit-Roy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-10T15:38:13.000Z","updated_at":"2026-06-27T13:55:03.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Jit-Roy/Prompt2Clip","commit_stats":null,"previous_names":["jit-roy/clipz","jit-roy/prompt2clip"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/Jit-Roy/Prompt2Clip","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jit-Roy%2FPrompt2Clip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jit-Roy%2FPrompt2Clip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jit-Roy%2FPrompt2Clip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jit-Roy%2FPrompt2Clip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jit-Roy","download_url":"https://codeload.github.com/Jit-Roy/Prompt2Clip/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jit-Roy%2FPrompt2Clip/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34983171,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-30T02:00:05.919Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","clips","llm","shorts","shorts-maker","video-clip"],"created_at":"2026-06-30T21:01:15.354Z","updated_at":"2026-06-30T21:01:16.642Z","avatar_url":"https://github.com/Jit-Roy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Clipz - Turn long videos into viral clips automatically.\n\n![Viral Clip Extractor](docs/img.png)\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eAI-powered instruction-driven multimodal video clip extraction\u003c/b\u003e\u003cbr/\u003e\n  Audio • Visual • Speech • LLM Reasoning\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Python-3.8+-blue\" alt=\"Python 3.8+\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/LLM-Powered-purple\" alt=\"LLM Powered\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Multimodal-Audio%20%7C%20Video%20%7C%20Text-orange\" alt=\"Multimodal\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Status-Active-success\" alt=\"Active\"/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#quick-start\"\u003e⚡ Quick Start\u003c/a\u003e •\n  \u003ca href=\"#-why-clipz\"\u003e🤔 Why Clipz?\u003c/a\u003e •\n  \u003ca href=\"#features\"\u003e✨ Features\u003c/a\u003e •\n  \u003ca href=\"#api-reference\"\u003e📚 API\u003c/a\u003e •\n  \u003ca href=\"#future-roadmap\"\u003e🚀 Roadmap\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## 🤔 Why Clipz?\n\nMany existing clip tools already do a great job with:\n- audio-based excitement detection  \n- visual motion \u0026 scene analysis  \n- even basic LLM-assisted highlight detection  \n\n**Clipz goes one step further — it’s instruction-driven.**\n\nInstead of passively finding “hot moments,” you tell the system *what you want*:\n- *“Extract the funniest moments”*\n- *“Only clip Speaker B”*\n- *“Find emotionally intense reactions”*\n\nAn LLM interprets your intent and grounds it using:\n✔ audio cues \u0026 prosody  \n✔ visual signals \u0026 scene context  \n✔ sentence-aware transcription  \n\nSo clips aren’t just *popular* — they’re **exactly aligned with your instruction**.\n\n---\n\n\n## Quick Start\n\n👉 **New to this project?** Check out the [Quick Start Guide](docs/QUICKSTART.md) for installation and first run in 5 minutes!\n\n## Features\n\n### Core Capabilities\n\n- 🎵 **Multi-Scale Audio Analysis**: Detects excitement through loudness, spectral novelty, rhythm, prosody, and semantic events (laughter, applause)\n- 🎬 **Advanced Visual Analysis**: Tracks motion, semantic surprise (CLIP), composition quality, shot boundaries, and face detection\n- 🗣️ **Speech Transcription**: Word-level timestamps using Whisper, respects sentence boundaries for natural clips\n- 🤖 **LLM-Powered Intelligence**: Instruction-driven clip extraction with semantic merging and context-aware ranking\n- ⚡ **Parallel Processing**: Multi-threaded feature extraction with automatic caching\n- 🎯 **Smart Clip Boundaries**: Never cuts mid-sentence, aligns to natural speech segments\n\n## API Reference\n\n### Main Pipeline (`main.py`)\n\n```python\nfrom main import ViralClipExtractor\n\n# Initialize with custom weights\nextractor = ViralClipExtractor(\n    audio_weight=0.5,      # 0-1, weight for audio excitement\n    video_weight=0.5,      # 0-1, weight for visual excitement\n    use_cache=True,        # Cache features for faster re-runs\n    output_dir=\"output\"    # Output directory\n)\n\n# Process video end-to-end\nresults = extractor.process(\n    video_path=\"video.mp4\",\n    user_query=\"give me 10 interesting clips\",  # Natural language query\n    target_fps=2,          # Video analysis FPS (lower=faster)\n    min_duration=5,        # Minimum clip length (seconds)\n    max_duration=60,       # Maximum clip length (seconds)\n    export=True            # Export video files\n)\n\n# Access results\nfor clip in results[\"clips\"]:\n    print(f\"Time: {clip['start']:.1f}s - {clip['end']:.1f}s\")\n    print(f\"Transcript: {clip['transcript']}\")\n    print(f\"Score: {clip['llm_interest_score']}/10\")\n    print(f\"Reason: {clip['reason']}\")\n    print(f\"Tags: {clip['tags']}\")\n```\n\n### Individual Modules\n\n#### Audio Analysis (`Audio/audio.py`)\n\n**Features Extracted:**\n- Multi-scale loudness (short/long-term RMS)\n- Spectral novelty via MFCC delta\n- Rhythm variance and onset strength\n- Silence contrast and dramatic pauses\n- Structural boundaries (change-point detection)\n- Semantic events (laughter, applause, cheering) via YAMNet\n\n```python\nfrom Audio.audio import ClipAudio\n\ndetector = ClipAudio(sr=16000)  # 16kHz optimized for speed\ntimestamps, scores = detector.compute_audio_scores(\n    audio_path=\"audio.wav\",\n    use_cache=True\n)\n```\n\n#### Video Analysis (`video/video.py`)\n\n**Features Extracted:**\n- Optical flow motion magnitude\n- CLIP semantic surprise detection\n- Composition scoring (rule of thirds)\n- Shot boundary detection\n- Face detection and tracking\n- Temporal rhythm analysis\n\n```python\nfrom video.video import ClipVideo\n\ndetector = ClipVideo()\ntimestamps, scores = detector.compute_visual_scores(\n    video_path=\"video.mp4\",\n    target_fps=2,\n    use_cache=True\n)\n```\n\n#### Transcription (`Transcription/transcribe.py`)\n\n**Returns:** List of sentence segments with timestamps\n\n```python\nfrom Transcription.transcribe import Transcriber\n\nsegments = Transcriber.transcribe_with_timestamps(\n    audio_path=\"audio.wav\",\n    model_size=\"base\",  # tiny/base/small/medium/large\n    verbose=False\n)\n# [{\"start\": 0.0, \"end\": 3.5, \"text\": \"Hello world\"}, ...]\n```\n\n#### LLM Integration (`LLM/llm.py`)\n\n**Uses:** OpenRouter API for GPT-4o-mini\n\n```python\nfrom LLM.llm import LLM\n\nllm = LLM()\nresponse = llm.generate_text(\n    prompt=\"Your prompt here\",\n    model=\"openai/gpt-4o-mini\",\n    max_tokens=2000,\n    temperature=0.3\n)\n```\n\n## Command Line Options\n\n```\nusage: main.py [-h] [--query QUERY] [--audio-weight AUDIO_WEIGHT]\n               [--video-weight VIDEO_WEIGHT] [--fps FPS]\n               [--min-duration MIN_DURATION] [--max-duration MAX_DURATION]\n               [--output-dir OUTPUT_DIR] [--no-export]\n               video_path\n\npositional arguments:\n  video_path            Path to input video file\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --query QUERY         Query for clip selection (default: \"give me 10 interesting clips\")\n  --audio-weight AUDIO_WEIGHT\n                        Weight for audio scores 0-1 (default: 0.5)\n  --video-weight VIDEO_WEIGHT\n                        Weight for video scores 0-1 (default: 0.5)\n  --fps FPS             Target FPS for video analysis (default: 2)\n  --min-duration MIN_DURATION\n                        Minimum clip duration in seconds (default: 5)\n  --max-duration MAX_DURATION\n                        Maximum clip duration in seconds (default: 60)\n  --output-dir OUTPUT_DIR\n                        Output directory for clips (default: \"output\")\n  --no-export           Skip exporting video files\n```\n\n## Output\n\nThe system generates:\n\n### Video Clips\n- **Location**: `output/clips_\u003ctimestamp\u003e/`\n- **Format**: Individual MP4 files (`clip_001.mp4`, `clip_002.mp4`, etc.)\n- **Content**: Extracted video segments ready to use\n\n### Metadata \u0026 Cache\n- **Clip Metadata**: `.cache/metadata/` - Individual JSON files for each clip\n- **Analysis Report**: `.cache/analysis/` - Complete analysis metadata\n- **Feature Cache**: `.cache/audio/` and `.cache/video/` - Cached features for faster re-runs\n- **Transcription Cache**: `.cache/transcription/` - Cached transcripts\n\nExample clip metadata:\n```json\n{\n  \"clip_number\": 1,\n  \"video_file\": \"output/clips_20260110_192101/clip_001.mp4\",\n  \"start_time\": 45.2,\n  \"end_time\": 58.7,\n  \"duration\": 13.5,\n  \"transcript\": \"...\",\n  \"interest_score\": 9.5,\n  \"reason\": \"Emotional storytelling with dramatic pause\",\n  \"tags\": [\"emotional\", \"dramatic\"]\n}\n```\n\n**Clean Output**: Your `output/` folder only contains the video clips - all metadata and cache files are organized in `.cache/` to keep things tidy!\n\n## Advanced Usage\n\n### Individual Module Testing\n\nTest audio analysis:\n```bash\npython audio.py path/to/audio.wav\n```\n\nTest video analysis:\n```bash\npython video.py path/to/video.mp4\n```\n\nTest transcription:\n```bash\npython transcribe.py path/to/audio.wav\n```\n\n### Caching\n\nThe system automatically caches expensive computations in the `.cache/` directory:\n- **Audio features**: `.cache/audio/audio_cache_\u003chash\u003e.npz`\n- **Video features**: `.cache/video/visual_cache_\u003chash\u003e.npz`\n- **Transcriptions**: `.cache/transcription/transcript_\u003chash\u003e.json`\n- **Metadata**: `.cache/metadata/` and `.cache/analysis/`\n\nThis makes subsequent runs much faster! To disable caching:\n```python\nextractor = ClipExtractor(use_cache=False)\n```\n\n## Performance Tips\n\n- **GPU Acceleration**: Install CUDA-enabled PyTorch for faster processing\n- **Lower FPS**: Use `--fps 1` for faster video analysis (less accurate)\n- **Smaller Models**: Whisper uses \"base\" model by default (good balance)\n- **Cache Results**: Re-runs on the same video are much faster with caching\n\n## Dependencies\n\nKey dependencies:\n- **ultralytics**: YOLOv8 object detection\n- **transformers**: CLIP model for semantic analysis\n- **whisper**: Speech transcription\n- **librosa**: Audio analysis\n- **opencv-python**: Video processing\n- **torch**: Deep learning framework\n- **dlib**: Face detection\n- **praat-parselmouth**: Prosody analysis\n\nSee `requirements.txt` for complete list.\n\n## Troubleshooting\n\n### YOLO Model Download\n\nThe YOLOv8 model (`yolov8n.pt`) is **automatically downloaded** on first run by the Ultralytics package. You don't need to manually download it.\n\nIf you encounter issues:\n- Ensure you have internet connection on first run\n- The model (~6MB) downloads to Ultralytics cache\n- Check firewall settings if download fails\n\n### FFmpeg Not Found\n\nInstall FFmpeg:\n- **Windows**: Download from https://ffmpeg.org/download.html\n- **macOS**: `brew install ffmpeg`\n- **Linux**: `sudo apt-get install ffmpeg`\n\n### OpenRouter API Errors\n\nCheck your `.env` file has a valid `OPENROUTER_API_KEY`.\n\n### Out of Memory\n\nTry:\n- Lowering `target_fps` (default is 2)\n- Processing shorter videos\n- Closing other applications\n\n## Known Limitations\n\n### Multi-Language Videos\nVideos containing multiple languages may produce unexpected clips because Whisper translates everything into English by default. This can result in:\n- Loss of context from non-English speech\n- Incorrect clip boundaries due to translation timing differences\n- Mixed language content being merged incorrectly\n\n**Workaround**: For better results with multi-language content, process each language segment separately or use the `task=\"transcribe\"` parameter to keep original language.\n\n### Long Video Processing\nVideos longer than 1 hour may consume significant processing time (30-60+ minutes depending on hardware):\n- Audio feature extraction scales with video duration\n- Video analysis requires processing thousands of frames\n- LLM analysis has context window limits for very long videos\n\n**Tips for long videos**:\n- Use lower `target_fps` (1 instead of 2) for faster processing\n- Enable caching to avoid re-processing if you need to re-run\n- Consider splitting very long videos into smaller segments\n- Ensure sufficient RAM (16GB+ recommended for 1-hour videos)\n\n## Future Roadmap\n\n#### 1️⃣ Speaker-Aware Extraction\n- **Speaker Diarization**: Integrate `pyannote.audio` or `SpeechBrain` to segment clips per speaker\n- **Speaker Queries**: Enable queries like \"Give me all clips where speaker X is explaining something\" or \"Combine all funny reactions of speaker Y\"\n- **Speaker Embeddings**: Integrate speaker identity into LLM scoring for intelligent semantic merges based on who's speaking\n- **Multi-speaker Analysis**: Track speaker transitions and dialogue patterns for better clip boundaries\n\n#### 2️⃣ Emotion / Excitement Detection\n- **Emotion Recognition**: Train or integrate pre-trained models for emotion/intensity detection beyond generic audio peaks\n- **Engagement Scoring**: Guide LLM to rank clips not only by volume/motion but by perceived emotional engagement\n- **Sentiment Analysis**: Combine audio emotion with transcript sentiment for deeper understanding\n- **Facial Expression Analysis**: Detect smiles, laughter, surprise in video frames to enhance excitement scoring\n\n#### 3️⃣ Adaptive Clip Duration\n- **Platform Presets**: User-specified clip length preferences (short for TikTok/Reels, longer for podcasts/YouTube)\n- **Intelligent Merging**: LLM can merge multiple peaks while respecting target duration constraints\n- **Dynamic Segmentation**: Automatically adjust clip boundaries based on content density and pacing\n- **Custom Templates**: Save and reuse clip duration strategies for different content types\n\n#### 4️⃣ Content-Type Tuning\n- **Auto-Classification**: Detect content type (comedy, sports, interview, tutorial, etc.) and adjust fusion weights accordingly\n  - Stand-up comedy → Audio-heavy (0.7 audio, 0.3 video)\n  - Sports/Gaming → Video-heavy (0.3 audio, 0.7 video)\n  - Interviews/Podcasts → Balanced (0.5 audio, 0.5 video)\n- **Genre-Specific Models**: Fine-tune excitement scoring for different video genres\n- **Context-Aware Features**: Enable/disable specific features based on content type\n\n#### 5️⃣ Real-Time / Streaming Mode\n- **Live Stream Support**: Extract highlights on-the-fly from live streams or ongoing recordings\n- **Streaming Inference**: Fast scoring models optimized for real-time processing\n- **Incremental LLM Prompts**: Streaming-friendly LLM prompt design for progressive clip selection\n- **Buffer Management**: Smart windowing for continuous audio/video analysis\n\n#### 6️⃣ Auto-Subtitle / Captioning Integration\n- **Forced Alignment**: Combine transcript with precise word-level timestamps\n- **Subtitle Generation**: Auto-generate SRT/VTT files for each extracted clip\n- **Multi-Language Support**: Transcribe and caption in multiple languages\n- **Styling Options**: Customizable subtitle appearance for different platforms (TikTok, YouTube Shorts, Instagram)\n- **Accessibility**: Ensure all clips are accessible with proper closed captions\n\n### 🚀 Community Contributions Welcome!\n\nWe're excited about these features and welcome contributions! If you're interested in implementing any of these enhancements, please:\n1. Open an issue to discuss your approach\n2. Fork the repository and create a feature branch\n3. Submit a pull request with comprehensive tests\n\n## License\n\n[MIT License](LICENSE) - see the LICENSE file for details.\n\n## Contributing\n\nContributions welcome! Please read our [Contributing Guidelines](docs/CONTRIBUTING.md) for details on how to submit pull requests, report issues, and contribute to the project.\n\n## Acknowledgments\n\n- YOLOv8 by Ultralytics\n- CLIP by OpenAI\n- Whisper by OpenAI\n- OpenRouter for LLM API access\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjit-roy%2Fprompt2clip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjit-roy%2Fprompt2clip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjit-roy%2Fprompt2clip/lists"}