{"id":51547590,"url":"https://github.com/germondai/trawl","last_synced_at":"2026-07-11T20:31:00.506Z","repository":{"id":367642023,"uuid":"1281287479","full_name":"germondai/trawl","owner":"germondai","description":"Self-hosted scraping engine — bypasses any JS challenge \u0026 captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr \u0026 Byparr alternative and drop-in replacement for your *arr stack.","archived":false,"fork":false,"pushed_at":"2026-07-05T16:16:26.000Z","size":376,"stargazers_count":228,"open_issues_count":4,"forks_count":8,"subscribers_count":0,"default_branch":"dev","last_synced_at":"2026-07-05T18:08:22.338Z","etag":null,"topics":["arr","bun","byparr","bypass-cloudflare","camoufox","captcha-bypass","captcha-slover","docker","flaresolverr","jackett","prowlarr","radarr","scraper","scraping","self-hosted","selfhosted","typescript","web-scraping"],"latest_commit_sha":null,"homepage":"https://trawl.germondai.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/germondai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-26T12:01:51.000Z","updated_at":"2026-07-05T16:15:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/germondai/trawl","commit_stats":null,"previous_names":["germondai/trawl"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/germondai/trawl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/germondai%2Ftrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/germondai%2Ftrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/germondai%2Ftrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/germondai%2Ftrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/germondai","download_url":"https://codeload.github.com/germondai/trawl/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/germondai%2Ftrawl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35321224,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arr","bun","byparr","bypass-cloudflare","camoufox","captcha-bypass","captcha-slover","docker","flaresolverr","jackett","prowlarr","radarr","scraper","scraping","self-hosted","selfhosted","typescript","web-scraping"],"created_at":"2026-07-09T20:00:24.166Z","updated_at":"2026-07-11T20:31:00.500Z","avatar_url":"https://github.com/germondai.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003ca href=\"https://trawl.dev\" target=\"_blank\"\u003e\n    \u003cimg align=\"center\" src=\"https://icons.germondai.com/icons?i=bun,elysia,firefox,nuxt,vitepress\" /\u003e\u003cbr/\u003e\u003cbr/\u003e\n    \u003cspan\u003eTRAWL\u003c/span\u003e\n  \u003c/a\u003e\n\u003c/h1\u003e\n\n## **Welcome** to \u003ca href=\"https://trawl.dev\" target=\"_blank\"\u003e**TRAWL**\u003c/a\u003e! 👋\n\nSelf-hosted web scraping engine with adaptive tier execution. Solves Cloudflare challenges natively using [Camoufox](https://github.com/daijro/camoufox) Firefox. Returns cached results in under 500ms. Drop-in FlareSolverr v2 replacement.\n\n## Features\n\n- **4-tier execution** — plain HTTP → cached session → fresh CF solve → residential proxy\n- **Native captcha solving** — CF Turnstile, reCAPTCHA v2 (free STT), hCaptcha, GeeTest v4 Slide\n- **Camoufox Firefox** — fingerprint-patched at the C++/Juggler level; indistinguishable from a real browser\n- **Session cache** — `cf_clearance` cookies stored in Redis; repeat requests to the same domain return in ~500ms\n- **FlareSolverr v2 compatible** — works with Prowlarr, Jackett, Sonarr, Radarr, and the full \\*arr ecosystem out of the box\n- **No external APIs** — reCAPTCHA audio transcription uses Google's free STT endpoint by default; everything else is local\n\n## Quick start\n\n```bash\n# Clone and configure\ngit clone https://github.com/germondai/trawl.git\ncd trawl\ncp .env.example .env\n\n# Start scraper + Redis\ndocker compose up -d\n\n# Verify\ncurl http://localhost:8191/health\n```\n\nFirst boot takes 15–30s while the browser pool warms up. Subsequent starts are fast.\n\n## API\n\n### FlareSolverr-compatible (`/v1`)\n\n```bash\ncurl -X POST http://localhost:8191/v1 \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"cmd\":\"request.get\",\"url\":\"https://nowsecure.nl\",\"maxTimeout\":60000}'\n```\n\n### Native API (`/scrape`)\n\nReturns richer metadata: `tier`, `timings`, `sessionCached`, full cookie list.\n\n```bash\ncurl -X POST http://localhost:8191/scrape \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"url\":\"https://nowsecure.nl\",\"maxTimeout\":60000}'\n```\n\n### Connect Prowlarr / Jackett\n\nSet the FlareSolverr URL to:\n\n```\nhttp://localhost:8191        # running on the same host\nhttp://trawl:8191            # running via Docker Compose on the same network\n```\n\n## Tiers\n\n```\nRequest\n  │\n  ▼\nTier 1: Plain HTTP fetch ────── success ──→ return (\u003c 100ms)\n  │ blocked\n  ▼\nTier 2: Cached session ─────── success ──→ return (~500ms)\n  │ cache miss / expired\n  ▼\nTier 3: Fresh CF solve ─────── success ──→ cache + return (4–15s)\n  │ IP flagged\n  ▼\nTier 4: Residential proxy ──── success ──→ cache + return (15–45s)\n  │ failed\n  ▼\n  error\n```\n\n## Docker Compose files\n\n| File                         | Description                                              |\n| ---------------------------- | -------------------------------------------------------- |\n| `docker-compose.yml`         | Scraper + Redis (default)                                |\n| `docker-compose.minimal.yml` | Scraper only, no Redis                                   |\n| `docker-compose.prod.yml`    | Production: `restart: always`, memory limit, healthcheck |\n| `docker-compose.full.yml`    | Full stack: scraper + web + docs                         |\n\n## Docker images (one GHCR package, two tags)\n\n| Image tag                          | Built from                     | Runtime                       | Use case                                                   |\n| ---------------------------------- | ------------------------------ | ----------------------------- | ---------------------------------------------------------- |\n| `ghcr.io/germondai/trawl:latest`   | `apps/api/Dockerfile`          | Bun 1.3.14 (modern, AVX2)     | Default — modern Linux amd64/arm64                         |\n| `ghcr.io/germondai/trawl:baseline` | `apps/api/Dockerfile.baseline` | Bun 1.3.14 baseline (no AVX2) | Older CPUs / older kernels (Synology NAS, J4125, Atom-era) |\n\nBoth tags live on the same `ghcr.io/germondai/trawl` package — they share the registry but use different Dockerfile sources. Pick whichever tag fits your hardware:\n\n```yaml\n# Modern hardware (most users)\nimage: ghcr.io/germondai/trawl:latest\n\n# Older CPUs without AVX2 / Synology / older kernels\nimage: ghcr.io/germondai/trawl:baseline\n```\n\nSynology note: many Synology NAS units (DSM 7.x on J4125 / older hardware) ship kernel 4.4.x, which Bun's modern runtime can't fully handle. Standard Bun requires kernel 5.1+ (5.6+ recommended); the baseline build degrades gracefully down to kernel 3.10. The `:baseline` tag is published for that case — **confirmed working** on a Synology DS920+ (Celeron J4125, DSM 7.3.2, kernel 4.4.302): the container starts cleanly, `/health` reports healthy, and it solves live Cloudflare challenges via `/v1` (see [#1](https://github.com/germondai/trawl/issues/1)). Published by independent GitHub Actions workflows (`.github/workflows/publish.yml`, `publish-baseline.yml`); tag-triggered releases push matching git tags (e.g. `v1.0.0` → `1.0.0`, `1.0.0-baseline` → `1.0.0-baseline`) and manual `workflow_dispatch` from `main` updates the rolling tag (`latest` and `baseline` respectively).\n\n## Releases \u0026 versioning\n\nTRAWL follows [Semantic Versioning](https://semver.org/). Pushing a `v`-prefixed git tag (e.g.\n`v1.0.0`) triggers `publish.yml`, which builds and pushes the matching un-prefixed Docker tag\n(`ghcr.io/germondai/trawl:1.0.0`) alongside a major-only tag (`:1`). `:latest` always tracks the\ntip of `main`; `:sha-\u003cshortsha\u003e` images are pushed on every `main` commit regardless of tags.\nSee the [Releases page](https://github.com/germondai/trawl/releases) for the full version\nhistory and [CHANGELOG.md](CHANGELOG.md) for what changed in each one.\n\nTo publish a specific past commit that had a major fix without waiting for the next tip-of-`main`\nrelease, tag that exact SHA and push it — `publish.yml` builds off the tag ref, not off `main`'s\ncurrent tip:\n\n```bash\ngit tag -a v1.0.1 \u003csha\u003e -m \"...\"\ngit push origin v1.0.1\n```\n\n## Configuration\n\n| Variable                         | Default                  | Description                                                                         |\n| -------------------------------- | ------------------------ | ----------------------------------------------------------------------------------- |\n| `BROWSER_POOL_SIZE`              | `3`                      | Warm Camoufox Firefox instances                                                     |\n| `BROWSER_ACQUIRE_TIMEOUT_MS`     | `15000`                  | How long `acquire()` polls for a free browser before HTTP 429 is returned           |\n| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8`                      | Recycle a browser after this many `blocked`/`needs-js` outcomes; set `0` to disable |\n| `BROWSER_CONTENT_PROCESSES`      | `2`                      | Cap Firefox content processes per browser (`dom.ipc.processCount`); lowers RAM/CPU  |\n| `SESSION_TTL_SECONDS`            | `3600`                   | Redis session cache TTL (seconds)                                                   |\n| `REDIS_URL`                      | `redis://localhost:6379` | Redis connection string                                                             |\n| `RESIDENTIAL_PROXY_URL`          | —                        | Enables Tier 4 proxy escalation                                                     |\n| `STT_URL`                        | —                        | Local Whisper endpoint for reCAPTCHA (optional)                                     |\n| `PORT`                           | `8191`                   | API listen port                                                                     |\n\n## Stack\n\nBuilt on a modern, fast-by-default stack: Bun + Elysia for the API, Redis for caching,\nCamoufox (hardened Firefox) for browser automation, and Nuxt for the web UI — no legacy\nNode/Express baggage.\n\n| Layer         | Technology                         |\n| ------------- | ---------------------------------- |\n| Runtime       | Bun                                |\n| API           | Elysia                             |\n| Browser       | Camoufox Firefox (via camoufox-js) |\n| Session cache | Redis 8.8                          |\n| Landing page  | Nuxt 4                             |\n| Documentation | VitePress                          |\n\n## License\n\n[AGPL-3.0](LICENSE)\n\n---\n\n\u003cp align=\"center\"\u003e\n    \u003cspan\u003eMade with ❤️ by\u003c/span\u003e\n    \u003ca href=\"https://github.com/germondai\" target=\"_blank\"\u003e@germondai\u003c/a\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgermondai%2Ftrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgermondai%2Ftrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgermondai%2Ftrawl/lists"}