{"id":50405581,"url":"https://github.com/nikuscs/scrauper","last_synced_at":"2026-05-31T01:30:24.435Z","repository":{"id":357347896,"uuid":"1163913852","full_name":"nikuscs/scrauper","owner":"nikuscs","description":"Multi-threaded ScreenScraper.fr scraper for ES-DE with multi-account rotation, proxy support, and miximage generation","archived":false,"fork":false,"pushed_at":"2026-05-12T11:35:49.000Z","size":311,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-12T12:32:31.856Z","etag":null,"topics":["emulation","es-de","gamelist","miximage","retrogaming","rom","rust","scraper","screenscraper"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nikuscs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-22T10:59:02.000Z","updated_at":"2026-05-12T11:35:53.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/nikuscs/scrauper","commit_stats":null,"previous_names":["nikuscs/scrauper"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/nikuscs/scrauper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikuscs%2Fscrauper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikuscs%2Fscrauper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikuscs%2Fscrauper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikuscs%2Fscrauper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nikuscs","download_url":"https://codeload.github.com/nikuscs/scrauper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikuscs%2Fscrauper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33716338,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["emulation","es-de","gamelist","miximage","retrogaming","rom","rust","scraper","screenscraper"],"created_at":"2026-05-31T01:30:20.686Z","updated_at":"2026-05-31T01:30:24.420Z","avatar_url":"https://github.com/nikuscs.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🕹️ scrauper\n\n![CI](https://github.com/nikuscs/scrauper/actions/workflows/ci.yml/badge.svg)\n![Release](https://img.shields.io/github/v/release/nikuscs/scrauper)\n![License](https://img.shields.io/badge/license-MIT-blue.svg)\n\n**Fast multi-threaded [ScreenScraper.fr](https://www.screenscraper.fr/) scraper for [ES-DE](https://es-de.org/) with multi-account rotation, proxy support, and miximage generation.**\n\n\u003e **Disclaimer:** This project is for **educational purposes and AI automation research only**.\n\u003e The authors are not responsible for any misuse or for any damages resulting from the use of this tool.\n\u003e Users are solely responsible for ensuring compliance with applicable laws and the terms of service\n\u003e of any websites accessed. This software is provided \"as-is\" without warranty of any kind.\n\u003e\n\u003e If you are a rights holder and wish to have this project removed, please [contact me](https://github.com/nikuscs).\n\n\u003e **Note:** This project was partially developed with AI assistance and may contain bugs or unexpected behavior. Use at your own risk.\n\n## Why?\n\nScraping metadata for retro game collections is painful. ScreenScraper.fr is the best source, but the official tools are slow, single-threaded, and hit daily API quotas fast. If you have thousands of ROMs across multiple systems, you're looking at days of scraping.\n\n- **Multi-account rotation** — bypass daily quotas by rotating across multiple ScreenScraper accounts\n- **Pixel-accurate miximage generation** — pure Rust compositing, no ImageMagick dependency\n- **Multi-threaded with per-account rate limiting** — fast scraping that respects API limits\n- **Proxy rotation** — distribute requests across multiple proxies\n- **Intelligent response caching** — skip already-scraped games, resume interrupted runs\n- **Cross-platform** — Linux, macOS, Windows, Android/ARM\n- **Hash-based ROM identification** — CRC32 + MD5 + SHA1 for accurate matching\n- **Interactive mode** — manual selection for ambiguous matches\n\n## Install\n\n```bash\n# From source (requires Rust)\ncargo install --git https://github.com/nikuscs/scrauper\n\n# Or clone and build\ngit clone https://github.com/nikuscs/scrauper\ncd scrauper\ncargo build --release\n```\n\nPre-built binaries available in [Releases](https://github.com/nikuscs/scrauper/releases).\n\n## Usage\n\n```bash\n# Generate a config file\nscrauper init\n\n# Edit scrauper.toml with your ScreenScraper credentials and paths\n$EDITOR scrauper.toml\n\n# Start scraping\nscrauper scrape\n```\n\n### Commands\n\n```\nscrauper \u003cCOMMAND\u003e\n\nCommands:\n  scrape              Scrape game metadata and media from ScreenScraper\n  info                Show user account info and quotas\n  systems             List all supported systems\n  cleanup             Remove orphaned media files\n  generate-miximages  Generate miximages from existing media\n  cache-stats         Show cache statistics\n  init                Create a new config file from the example template\n```\n\n### Scrape options\n\n```bash\nscrauper scrape                     # scrape all detected systems\nscrauper scrape --system snes       # scrape a specific system\nscrauper scrape --force             # re-scrape even for complete games\nscrauper scrape --interactive       # prompt for ambiguous matches\n```\n\n## Configuration\n\nRun `scrauper init` to generate a `scrauper.toml` from the example template. Key sections:\n\n### Credentials and accounts\n\n```toml\n[credentials]\n# Single account\naccounts = \"user1:pass1\"\n# Multi-account rotation (recommended for higher quotas)\naccounts = \"user1:pass1;user2:pass2;user3:pass3\"\n```\n\n### Paths\n\n```toml\n[paths]\nrom_directory = \"~/ROMs\"\nmedia_directory = \"~/.emulationstation/downloaded_media\"\ngamelist_directory = \"~/.emulationstation/gamelists\"\n```\n\n### Proxy rotation\n\n```toml\n[network]\nproxies = \"user:pass@host:port;host2:port2\"\n```\n\n### Miximage generation\n\n```toml\n[miximage]\nenabled = true\nwidth = 1280\nheight = 960\nformat = \"png\"\n```\n\nSee [`scrauper.toml.example`](scrauper.toml.example) for the full configuration reference.\n\n## How It Works\n\n1. **ROM discovery** — scans ROM directories, identifies systems by folder structure\n2. **Hash identification** — computes CRC32, MD5, and SHA1 for accurate game matching\n3. **API lookup** — queries ScreenScraper.fr with hash or filename fallback\n4. **Metadata + media download** — fetches game info, screenshots, box art, videos\n5. **gamelist.xml generation** — writes ES-DE compatible gamelist files\n6. **Miximage compositing** — generates composite images (screenshot + box + marquee)\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnikuscs%2Fscrauper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnikuscs%2Fscrauper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnikuscs%2Fscrauper/lists"}