{"id":50325302,"url":"https://github.com/testy-cool/scrape-gateway","last_synced_at":"2026-05-29T05:30:58.853Z","repository":{"id":356404320,"uuid":"1230150161","full_name":"testy-cool/scrape-gateway","owner":"testy-cool","description":"Unified scraping gateway — 7 providers, cheapest-first routing, content validation, domain memory. One sgw url call, it figures out the rest.","archived":false,"fork":false,"pushed_at":"2026-05-27T08:01:56.000Z","size":266,"stargazers_count":0,"open_issues_count":25,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-27T08:05:41.856Z","etag":null,"topics":["anti-bot","cli","proxy","python","scraping","web-scraping"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/testy-cool.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-05T18:17:09.000Z","updated_at":"2026-05-27T08:02:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/testy-cool/scrape-gateway","commit_stats":null,"previous_names":["testy-cool/scrape-gateway"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/testy-cool/scrape-gateway","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testy-cool%2Fscrape-gateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testy-cool%2Fscrape-gateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testy-cool%2Fscrape-gateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testy-cool%2Fscrape-gateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/testy-cool","download_url":"https://codeload.github.com/testy-cool/scrape-gateway/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testy-cool%2Fscrape-gateway/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33639050,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anti-bot","cli","proxy","python","scraping","web-scraping"],"created_at":"2026-05-29T05:30:58.572Z","updated_at":"2026-05-29T05:30:58.848Z","avatar_url":"https://github.com/testy-cool.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# scrape-gateway (`sgw`)\n\n[![ci](https://github.com/testy-cool/scrape-gateway/actions/workflows/ci.yml/badge.svg)](https://github.com/testy-cool/scrape-gateway/actions/workflows/ci.yml)\n[![version](https://img.shields.io/badge/version-0.3.0-blue)](https://github.com/testy-cool/scrape-gateway/releases/latest)\n[![license](https://img.shields.io/badge/license-Apache--2.0-green)](LICENSE)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/terminal.svg\" alt=\"sgw demo — free providers fail, paid provider succeeds, next time it remembers\" width=\"720\"\u003e\n\u003c/p\u003e\n\nOne command, seven providers. Free ones tried first, paid ones only when needed. Domain memory skips the trial-and-error on repeat visits.\n\n## Quick start\n\n```bash\ngit clone https://github.com/testy-cool/scrape-gateway.git\ncd scrape-gateway\npip install -e .\ncp .env.example .env   # add API keys (optional — free providers work without any)\n\nsgw selftest           # verify installation\nsgw url https://example.com\n```\n\n## Commands\n\n| Command | What it does |\n|---|---|\n| `sgw url \u003curl\u003e` | Scrape one page through the provider chain |\n| `sgw extract \u003curl\u003e` | Pull structured data (JSON/CSV) from listing pages |\n| `sgw detect \u003curl\u003e` | Recon — find repeated elements before extracting |\n| `sgw links \u003curl\u003e` | Index all links on a page |\n| `sgw follow \u003curl\u003e \u003cn\u003e` | Scrape link #n from a page |\n| `sgw recipe \u003cfile\u003e` | Replay a saved YAML workflow |\n| `sgw run \u003cfile\u003e` | Batch scrape URLs from a text file |\n| `sgw meta \u003curl\u003e` | Extract OpenGraph metadata as JSON |\n| `sgw history \u003curl\u003e` | Show scrape timeline and page changes |\n| `sgw telemetry` | Inspect recent scrape reports |\n| `sgw providers` | List all available providers |\n| `sgw extensions` | Browse/install community extensions |\n| `sgw selftest` | Verify installation with known-safe sites |\n\nFull usage and examples: [docs/commands.md](docs/commands.md)\n\n## Providers\n\n7 built-in, 3 free. Router tries cheapest first.\n\n| Provider | Cost | JS | Geo | Anti-bot |\n|---|---|---|---|---|\n| `raw_http` | free | no | no | none |\n| `wreq` | free | no | no | TLS fingerprinting |\n| `curl_cffi` | free | no | no | TLS fingerprinting |\n| `scrapedrive` | paid | yes | yes | full (3 tiers) |\n| `scrape_do` | paid | yes | yes | residential proxies |\n| `scrapingbee` | paid | yes | yes | premium proxies |\n| `scraperapi` | paid | yes | yes | premium proxies |\n\nAdd API keys in `.env` to enable paid providers. Without them, `sgw` uses free providers only.\n\n## Extend it\n\nDrop a `.py` file in `~/.config/scrape-gateway/providers/` or install from the registry with `sgw extensions`. See [docs/extensions.md](docs/extensions.md).\n\n## Python API\n\n```python\nfrom scrape_gateway import ScrapeGateway, ScrapeRequest\n\ngw = ScrapeGateway.from_config()\nresult = await gw.scrape(ScrapeRequest(\"https://example.com\"))\n```\n\nMore: [docs/python-api.md](docs/python-api.md)\n\n## Docs\n\n- [Commands](docs/commands.md) — full reference with examples\n- [Architecture](docs/architecture.md) — how the router, cache, and memory work\n- [Configuration](docs/configuration.md) — YAML config and `.env` setup\n- [Extensions](docs/extensions.md) — writing custom providers\n- [Python API](docs/python-api.md) — using sgw as a library\n- [Providers](docs/providers.md) — provider details and API mapping\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftesty-cool%2Fscrape-gateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftesty-cool%2Fscrape-gateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftesty-cool%2Fscrape-gateway/lists"}