{"id":46215453,"url":"https://github.com/sysadmindoc/stock-video-collector","last_synced_at":"2026-06-28T04:01:02.430Z","repository":{"id":339876532,"uuid":"1163687789","full_name":"SysAdminDoc/Stock-Video-Collector","owner":"SysAdminDoc","description":"Headless browser crawler with a PyQt6 GUI for discovering, cataloging, and downloading stock video clips from Artlist, Pexels, Pixabay, Storyblocks, and more.","archived":false,"fork":false,"pushed_at":"2026-06-13T00:07:04.000Z","size":1187,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-13T01:12:55.281Z","etag":null,"topics":["crawler","gui","pyqt6","python","stock-video","video"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SysAdminDoc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-22T01:39:55.000Z","updated_at":"2026-06-13T00:07:07.000Z","dependencies_parsed_at":"2026-05-24T18:03:49.700Z","dependency_job_id":null,"html_url":"https://github.com/SysAdminDoc/Stock-Video-Collector","commit_stats":null,"previous_names":["sysadmindoc/stock-video-collector"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/SysAdminDoc/Stock-Video-Collector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SysAdminDoc%2FStock-Video-Collector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SysAdminDoc%2FStock-Video-Collector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SysAdminDoc%2FStock-Video-Collector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SysAdminDoc%2FStock-Video-Collector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SysAdminDoc","download_url":"https://codeload.github.com/SysAdminDoc/Stock-Video-Collector/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SysAdminDoc%2FStock-Video-Collector/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34876271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","gui","pyqt6","python","stock-video","video"],"created_at":"2026-03-03T10:10:00.354Z","updated_at":"2026-06-28T04:01:02.416Z","avatar_url":"https://github.com/SysAdminDoc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Stock Video Collector\n\n![Version](https://img.shields.io/badge/version-0.7.4-blue)\n![Python](https://img.shields.io/badge/Python-3.9+-3776AB?logo=python\u0026logoColor=white)\n![PyQt6](https://img.shields.io/badge/PyQt6-GUI-41CD52?logo=qt\u0026logoColor=white)\n![Playwright](https://img.shields.io/badge/Playwright-Headless_Browser-2EAD33?logo=playwright\u0026logoColor=white)\n![SQLite](https://img.shields.io/badge/SQLite-FTS5-003B57?logo=sqlite\u0026logoColor=white)\n![Platform](https://img.shields.io/badge/platform-Windows%20%7C%20Linux%20%7C%20macOS-lightgrey)\n![License](https://img.shields.io/badge/license-MIT-green)\n\n\u003e Headless browser crawler with a dark-themed PyQt6 desktop GUI for discovering, cataloging, and downloading stock video clips from multiple sites — with full metadata extraction, FTS5 keyword search, and a concurrent download manager.\n\n---\n\n## Quick Start\n\n```bash\ngit clone https://github.com/SysAdminDoc/Stock-Video-Collector.git\ncd Stock-Video-Collector\npy -3 -m venv .venv\n.\\.venv\\Scripts\\python -m pip install -r requirements.txt\n.\\.venv\\Scripts\\python -m playwright install chromium\n.\\.venv\\Scripts\\python artlist_scraper.py\n```\n\nThe setup installs:\n\n1. Python packages (`PyQt6`, `playwright`, `imageio-ffmpeg`)\n2. Chromium for Playwright crawling\n3. The dark PyQt6 desktop GUI\n\n\u003e **Requirements:** Python 3.9+ — no other prerequisites. Works on Windows, Linux, and macOS.\n\n### Build EXE\n\n```bash\n.\\.venv\\Scripts\\python -m pip install pyinstaller\n.\\.venv\\Scripts\\python -m PyInstaller --onefile --windowed --name Stock-Video-Collector --icon icon.ico --add-data \"icon.png;.\" --runtime-hook build_hooks/runtime_hook_mp.py artlist_scraper.py\n```\n\n---\n\n## Features\n\n### Multi-Site Crawling\n\n| Site | Video Types | Metadata | Pagination |\n|------|-------------|----------|------------|\n| **Artlist** | M3U8 HLS streams | Clip ID, resolution, duration, FPS, camera, formats, creator, collection, tags | Infinite scroll |\n| **Adobe Stock** | Watermarked MP4/WebM/HLS/DASH previews | OpenGraph, JSON-LD, asset cards, preview metadata | Search/video grids + infinite scroll |\n| **Shutterstock** | MP4/WebM/HLS/DASH previews | OpenGraph, JSON-LD, clip cards, preview metadata | Video/search grids + infinite scroll |\n| **Envato Elements** | MP4/WebM/HLS/DASH previews | OpenGraph, JSON-LD, item cards, preview metadata | Stock-video grids + infinite scroll |\n| **Motion Array** | MP4/WebM/HLS/DASH previews | OpenGraph, JSON-LD, product cards, preview metadata | Stock-video grids + infinite scroll |\n| **Vimeo** | HLS, MP4, WebM, DASH previews | OpenGraph, JSON-LD, channel/card metadata | Channel/group/showcase grids + infinite scroll |\n| **Pexels** | MP4 direct (SD/HD/UHD via Canva CDN) | OpenGraph + JSON-LD, URL slug titles | Load More button (up to 15 clicks) |\n| **Pixabay** | MP4, WebM | OpenGraph + JSON-LD | Infinite scroll |\n| **Storyblocks** | M3U8, MP4, WebM | OpenGraph + JSON-LD | Infinite scroll |\n| **Generic** | M3U8, MP4, WebM, DASH, MOV | Auto-detect (OG, JSON-LD, DOM) | Infinite scroll |\n\nThe **Generic** profile works on any site — it intercepts all video network requests and extracts whatever metadata is available.\n\n### Browser Automation \u0026 Anti-Detection\n\n| Feature | Description |\n|---------|-------------|\n| Stealth mode | Hides `navigator.webdriver` flag, spoofs plugin array and WebGL vendor/renderer |\n| Challenge detection | Auto-detects Cloudflare, CAPTCHA, and challenge pages |\n| Manual solve mode | Switches to visible browser for CAPTCHA solving, resumes automatically on clearance |\n| Persistent profile | Browser session cookies, localStorage, and tokens persist across runs |\n| Request interception | Blocks heavy HLS `.ts` segments during crawl to save bandwidth |\n| Configurable delays | Page delay, scroll delay, M3U8 wait, timeout — all adjustable per-run |\n\n### Video Discovery\n\nThe crawler uses four complementary strategies to find video URLs on every page:\n\n```\n┌───────────────────────────────────────────────────────────────────┐\n│                         Page Load                                 │\n├───────────────┬─────────────────┬─────────────┬───────────────────┤\n│  XHR/Fetch    │   DOM Observer  │  Response   │   HTML Regex      │\n│  Intercept    │   (MutationObs) │  Body Scan  │   Fallback        │\n│               │                 │             │                   │\n│  Hooks into   │  Watches for    │  Scans all  │  Regex sweep for  │\n│  XMLHttpReq \u0026 │  \u003cvideo src\u003e    │  HTTP resp  │  M3U8/MP4/WebM    │\n│  fetch() API  │  injections     │  bodies     │  + Canva partner  │\n│               │                 │             │  links (Pexels)   │\n└───────┬───────┴────────┬────────┴──────┬──────┴────────┬──────────┘\n        │                │               │               │\n        └────────────────┴───────┬───────┴───────────────┘\n                                 ▼\n                    ┌─────────────────────┐\n                    │  Quality Comparison  │\n                    │  UHD \u003e HD \u003e SD       │\n                    │  Dedup by clip ID    │\n                    └──────────┬──────────┘\n                               ▼\n                    ┌─────────────────────┐\n                    │   SQLite Database    │\n                    │   + FTS5 Index       │\n                    └─────────────────────┘\n```\n\n### Database \u0026 Search\n\n| Feature | Description |\n|---------|-------------|\n| SQLite with WAL mode | Concurrent reads, crash-safe writes |\n| FTS5 full-text search | Search across title, creator, collection, tags, resolution, camera, duration |\n| AND/OR search modes | Toggle between inclusive and exclusive multi-term search |\n| Column filters | Filter by source site, resolution, creator, collection — all combinable with text search |\n| Duration filter | Quick filter by clip length range |\n| Saved searches | Save and recall frequent search + filter combos |\n| FTS index rebuild | One-click repair if search results drift out of sync |\n\n### Asset Management\n\n| Feature | Description |\n|---------|-------------|\n| Star ratings | 1–5 star rating per clip |\n| Favorites | Quick-toggle favorite flag for any clip |\n| Notes | Free-text notes per clip |\n| User tags | Custom tag system independent of source tags |\n| Collections | Organize clips into named collections with color coding |\n| Bulk operations | Context menu actions on any card in the grid |\n\n### Download Manager\n\n| Feature | Description |\n|---------|-------------|\n| Concurrent downloads | Configurable parallel download workers (default: 2) |\n| ffmpeg HLS→MP4 | Automatic M3U8-to-MP4 conversion via ffmpeg |\n| Retry with backoff | Exponential backoff retry (configurable max attempts) |\n| Speed \u0026 ETA tracking | Real-time download speed and estimated completion time |\n| Bandwidth limiting | Optional download speed cap |\n| Filename templates | Customizable output filenames: `{title}`, `{clip_id}`, `{creator}`, `{collection}`, `{resolution}` |\n| Sidecar metadata | JSON metadata file written alongside each downloaded MP4 |\n| Thumbnail extraction | Auto-extracts a thumbnail frame from downloaded videos |\n\n### Export Formats\n\n| Format | Contents |\n|--------|----------|\n| `.txt` | Plain list of M3U8/MP4 URLs |\n| `.json` | Full metadata for all clips (title, creator, tags, URLs, timestamps) |\n| `.m3u` | Media player playlist — uses local path if downloaded, M3U8 URL otherwise |\n| `.csv` | Spreadsheet-ready with all metadata columns |\n| **Batch** | Export all four formats at once |\n\n### GUI\n\n| Feature | Description |\n|---------|-------------|\n| Dark theme | Catppuccin-inspired deep dark palette |\n| Card grid view | Visual thumbnail grid with configurable card sizes (S/M/L) |\n| Hover video preview | Mouse-over any card to preview the video inline |\n| Detail panel | Always-visible side panel with full metadata, ratings, notes, tags, collections |\n| System tray | Minimize to tray, continue crawling/downloading in background |\n| Toast notifications | Non-blocking status notifications |\n| Live crawl log | Real-time scrolling log with verbose/quiet toggle |\n| Clipboard monitor | Opt-in URL detection from clipboard (auto-fills crawl URL input) |\n\n### Keyboard Shortcuts\n\n| Key | Action |\n|-----|--------|\n| `Ctrl+F` | Focus search bar |\n| `F5` | Refresh search results |\n| `Ctrl+1` through `Ctrl+6` | Switch between tabs |\n\n---\n\n## Usage\n\n### Basic Workflow\n\n1. **Select a site profile** — check one or more profiles in the Crawl tab (Artlist, Pexels, Pixabay, Storyblocks, or Generic)\n2. **Set the start URL** — auto-populated per profile, or paste any URL for Generic mode\n3. **Configure crawl settings** — batch size, depth, delays, headless mode\n4. **Start crawling** — the crawler discovers pages, extracts metadata, and intercepts video URLs\n5. **Browse results** — switch to the Library tab to search, filter, rate, tag, and organize clips\n6. **Download** — select clips and download with the built-in manager, or export URL lists for external tools\n\n### Configuration\n\nAll settings persist automatically in a JSON config file. Key options:\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Batch size | 50 | Pages per crawl batch |\n| Page delay | 2s | Wait between page loads |\n| Scroll delay | 1s | Wait between scroll steps |\n| M3U8 wait | 5s | Time to wait for video URLs to appear |\n| Scroll steps | 10 | Number of scroll-down actions per page |\n| Timeout | 30s | Page load timeout |\n| Max pages | 0 (unlimited) | Stop after N pages |\n| Max depth | 3 | Link-following depth |\n| Headless | On | Run browser without visible window |\n| Concurrent DLs | 2 | Parallel download workers |\n| Max retries | 3 | Download retry attempts |\n| Bandwidth limit | 0 (unlimited) | Download speed cap in KB/s |\n| Clipboard monitor | Off | Auto-detect URLs from clipboard |\n\n### Filename Templates\n\nCustomize download filenames using template variables:\n\n```\n{title}                      → Beautiful_Sunset.mp4\n{clip_id}_{title}            → abc123_Beautiful_Sunset.mp4\n{creator}/{collection}/{title} → JohnDoe/Nature/Beautiful_Sunset.mp4\n```\n\nAvailable variables: `{title}`, `{clip_id}`, `{creator}`, `{collection}`, `{resolution}`\n\n---\n\n## How It Works\n\n```\n┌─────────────────────────────────────────────────────────────────────────┐\n│                            PyQt6 GUI                                    │\n│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐ │\n│  │  Crawl   │  │  Library  │  │  Detail   │  │ Download │  │  Export  │ │\n│  │  Tab     │  │  Tab      │  │  Panel    │  │  Tab     │  │  Tab    │ │\n│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬────┘ │\n└───────┼──────────────┼──────────────┼──────────────┼──────────────┼─────┘\n        │              │              │              │              │\n        ▼              ▼              ▼              ▼              ▼\n┌──────────────┐  ┌──────────────────────────┐  ┌──────────────────────┐\n│   Crawler    │  │      SQLite + FTS5       │  │   Download Worker    │\n│   Worker     │──│                          │──│                      │\n│  (QThread)   │  │  clips, crawl_queue,     │  │  ThreadPoolExecutor  │\n│              │  │  crawled_pages,           │  │  + ffmpeg HLS→MP4   │\n│  Playwright  │  │  collections,            │  │  + retry backoff     │\n│  Chromium    │  │  saved_searches           │  │  + speed tracking    │\n└──────────────┘  └──────────────────────────┘  └──────────────────────┘\n```\n\n**Crawler Worker** — Runs Playwright in an async event loop on a dedicated QThread. Navigates pages, injects JavaScript hooks for XHR/fetch/DOM video interception, extracts metadata via regex selectors + OpenGraph + JSON-LD, and manages the crawl queue with depth/priority.\n\n**Database Layer** — Thread-safe SQLite with WAL mode and a dedicated `threading.Lock`. FTS5 external content table indexes title, creator, collection, tags, resolution, camera, and duration. Quality-aware M3U8 URL upgrades prefer UHD over HD over SD.\n\n**Download Worker** — Persistent queue on a QThread with a `ThreadPoolExecutor` for concurrent downloads. Handles M3U8→MP4 conversion via ffmpeg, exponential backoff retry, real-time speed/ETA calculation, sidecar JSON metadata, and thumbnail extraction.\n\n---\n\n## Troubleshooting\n\n**\"Chromium not found\"** — Click the \"Install Browser\" button on the Crawl tab. This runs `playwright install chromium` automatically.\n\n**Search results seem wrong or incomplete** — Click the \"🔄 Rebuild Index\" button on the Crawl tab to rebuild the FTS5 search index from scratch.\n\n**Bot challenge / CAPTCHA detected** — Uncheck \"Headless\" mode and restart the crawl. The browser will open visibly so you can solve the challenge manually. The crawler pauses and resumes automatically once the challenge clears.\n\n**Downloads fail repeatedly** — Check that ffmpeg is installed and on your PATH. The scraper auto-detects ffmpeg in common locations, but if it can't find it, downloads that require HLS→MP4 conversion will fail.\n\n**Clipboard monitor not working** — The clipboard monitor is opt-in. Enable it in your config by adding `\"clipboard_monitor\": true`, or toggle it programmatically. On Linux/Wayland, clipboard access may require additional permissions.\n\n---\n\n## License\n\nMIT License — see [LICENSE](LICENSE) for details.\n\n---\n\n## Contributing\n\nIssues and PRs welcome. If you add support for a new site, submit it as a `SiteProfile.register()` block with documented selectors and test URLs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysadmindoc%2Fstock-video-collector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsysadmindoc%2Fstock-video-collector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysadmindoc%2Fstock-video-collector/lists"}