{"id":24894818,"url":"https://github.com/futureg-lab/mx-scraper","last_synced_at":"2026-05-10T19:15:11.759Z","repository":{"id":252523644,"uuid":"840690169","full_name":"futureg-lab/mx-scraper","owner":"futureg-lab","description":"Download image galleries or metadata accross the web","archived":false,"fork":false,"pushed_at":"2025-02-01T16:49:06.000Z","size":419,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-01T17:32:24.259Z","etag":null,"topics":["beautifulsoup4","cli","cli-application","downloader","graphql-server","image-gallery","metadata-extraction","python","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/futureg-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-10T11:40:55.000Z","updated_at":"2025-02-01T16:49:10.000Z","dependencies_parsed_at":"2024-08-17T15:32:18.457Z","dependency_job_id":"12029661-76ac-493a-b619-4e09eda27793","html_url":"https://github.com/futureg-lab/mx-scraper","commit_stats":null,"previous_names":["futureg-lab/mx-scraper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futureg-lab%2Fmx-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futureg-lab%2Fmx-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futureg-lab%2Fmx-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futureg-lab%2Fmx-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/futureg-lab","download_url":"https://codeload.github.com/futureg-lab/mx-scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245868661,"owners_count":20685689,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup4","cli","cli-application","downloader","graphql-server","image-gallery","metadata-extraction","python","rust"],"created_at":"2025-02-01T19:14:43.083Z","updated_at":"2026-05-10T19:15:11.753Z","avatar_url":"https://github.com/futureg-lab.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mx-scraper\n\nDownload image galleries or metadata on the web.\n\n\u003e This rewrite is expected to support previous implementation's metadata format.\n\u003e\n\u003e The main idea was to separate the core (mx-scraper) from the plugins (user\n\u003e defined) as it was not possible from previous implementations.\n\n# Usage\n\n```bash\n# pip install beautifulsoup4\n\n# Plugins can be specified with -p or --plugin\n# By default, it will be inferred from the args\n# Each plugin may have its own set of dependencies that are independent from mx-scraper\n# Uses bs4\nmx-scraper fetch --plugin images https://www.google.com\n# Uses gallery-dl\nmx-scraper fetch --meta-only -v https://x.com/afmikasenpai/status/1901323062949159354\nmx-scraper fetch -p gallery-dl https://x.com/afmikasenpai/status/1901323062949159354\n\n# Alternatively, to infer batched terms targeting various sources/plugins, prefixing is often required (e.g. id or name)\n# The prefix is plugin specific (refer to plugin_name/__init__.py :: mx_is_supported)\nmx-scraper fetch --meta-only -v img:https://www.google.com https://mto.to/series/68737\nmx-scraper fetch --meta-only -v nh:177013\n```\n\n## Commands\n\n```bash\nmx-scraper engine\n\nUsage: mx-scraper \u003cCOMMAND\u003e\n\nCommands:\n  fetch        Fetch a sequence of terms\n  fetch-files  Fetch a sequence of terms from a collection of files\n  request      Request a url\n  infos        Display various informations\n  server       Spawn a graphql server interfacing mx-scraper\n  help         Print this message or the help of the given subcommand(s)\n\nOptions:\n  -h, --help  Print help\n```\n\nEach fetch strategy will share the same configuration..\n\n# Features\n\n- [x] CLI\n  - [x] Fetch a list of terms\n  - [x] Fetch a list of terms from a collection of files⌈\n  - [x] Generic URL Request\n    - [x] Print as text\n    - [x] Download `--dest` flag\n  - [x] Authentications (Basic, Bearer token)\n\n- [x] Cookies\n  - [x] Loading from a file (Netscape format, key-value)\n  - [x] Loading from the config (key-value)\n\n- [x] Http Client/Downloader\n  - [x] Support of older mx-scraper book schema\n  - [x] Download\n  - [x] Cache support (can be disabled with `--no-cache` or from config)\n  - [x] Configurable Http Client (default, Flaresolverr, cfworker)\n\n- [ ] Plugins\n  - [x] Python plugin\n    - [x] `MxRequest` with runtime context (headers, cookies, auth)\n  - [x] gallery-dl extractors\n  - [ ] Subprocess (e.g. imgbrd-grabber)\n\n- [ ] Send context from an external source (e.g. browser)\n  - [x] Cookies, UA (through `--listen-cookies`, will open a callback url that\n        can receive a `FetchContext` object)\n  - [ ] Rendered HTML page\n\n# GraphQL server\n\nYou can also use the extractors through GraphQL queries. You will have the same\noptions as the command-line interface.\n\n```\nUsage: mx-scraper server [OPTIONS]\n\nOptions:\n      --port \u003cPORT\u003e  Server port\n  -h, --help         Print help\n```\n\n![Playground Screenshot](static/server.png \"Screenshot\")\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffutureg-lab%2Fmx-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffutureg-lab%2Fmx-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffutureg-lab%2Fmx-scraper/lists"}