{"id":47711305,"url":"https://github.com/we-be/shoal","last_synced_at":"2026-04-02T18:33:00.470Z","repository":{"id":347872252,"uuid":"1195541883","full_name":"we-be/shoal","owner":"we-be","description":"Browser orchestration platform to scale web automations and stay low key ","archived":false,"fork":false,"pushed_at":"2026-03-29T21:56:35.000Z","size":94,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-29T22:07:57.280Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/we-be.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-29T19:35:33.000Z","updated_at":"2026-03-29T21:56:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/we-be/shoal","commit_stats":null,"previous_names":["we-be/shoal"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/we-be/shoal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/we-be%2Fshoal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/we-be%2Fshoal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/we-be%2Fshoal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/we-be%2Fshoal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/we-be","download_url":"https://codeload.github.com/we-be/shoal/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/we-be%2Fshoal/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31312918,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T12:59:32.332Z","status":"ssl_error","status_checked_at":"2026-04-02T12:54:48.875Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-02T18:32:59.730Z","updated_at":"2026-04-02T18:33:00.418Z","avatar_url":"https://github.com/we-be.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"assets/logo.png\" alt=\"Shoal\" height=\"160\"\u003e\n\n**Scale headless browsers like a school of fish.**\n\n[![CI](https://github.com/we-be/shoal/actions/workflows/ci.yml/badge.svg)](https://github.com/we-be/shoal/actions/workflows/ci.yml)\n[![Go](https://img.shields.io/github/go-mod/go-version/we-be/shoal)](https://go.dev)\n[![License](https://img.shields.io/github/license/we-be/shoal)](LICENSE)\n[![Release](https://img.shields.io/github/v/release/we-be/shoal)](https://github.com/we-be/shoal/releases)\n\n| CF solve | Minnow latency | Parallel speedup | Success rate |\n|:--:|:--:|:--:|:--:|\n| **\u003c 1s** | **~0.2s** | **4.7x** | **10/10** |\n\n\u003c/div\u003e\n\n---\n\nShoal separates **orchestration** from **automation**. One Chrome grouper solves Cloudflare, then a school of minnows ride the earned cookies for fast parallel scraping.\n\n```\n              ┌─────────────┐\n clients ────▶│  Controller │  pool, leases, warm matching, cookie handoff\n              └──────┬──────┘\n                     │\n     ┌───────────────┼───────────────┐\n     ▼                               ▼\n┌──────────┐   cf_clearance    ┌───────────────┐\n│ Grouper  │ ─────────────────▶│    Minnows    │\n│ (Chrome) │   auto-handoff   │  (tls-client)  │\n│  heavy   │                  │    light x N   │\n└──────────┘                  └───────────────┘\n```\n\n## Quick Start\n\n```bash\nmake build\nmake school-cf COUNT=10        # 1 grouper + 10 minnows\nmake school-lp COUNT=5         # 5 lightpanda (JS, no CF)\nmake school-minnow COUNT=20   # 20 tls-client (HTTP only)\nmake school-mixed              # 2 lightpanda + 5 minnows\nmake stop\n```\n\nDashboard at `localhost:8180/dashboard`. Metrics at `localhost:8180/metrics`.\n\n## Client Libraries\n\n### Go\n\n```go\nimport \"github.com/we-be/shoal/pkg/shoal\"\n\nclient := shoal.NewClient(\"http://localhost:8180\")\n\n// One-liner\nresp, _ := client.Fetch(ctx, \"https://example.com\", \"my-scraper\")\n\n// With options\nresp, _ = client.Fetch(ctx, url, \"scraper\",\n    shoal.WithClass(\"heavy\"),\n    shoal.WithCaptureXHR(\"api/v1\"),\n)\n\n// Manual lease control\nlease, _ := client.Lease(ctx, \"scraper\", \"example.com\")\nresp, _ = client.Navigate(ctx, lease.LeaseID, url)\nclient.Release(ctx, lease.LeaseID)\n```\n\n### Python\n\n```python\nfrom shoal import Shoal\n\ns = Shoal(\"http://localhost:8180\")\n\n# One-liner\nresp = s.fetch(\"https://example.com\")\n\n# Session (auto release)\nwith s.session(\"my-scraper\", \"example.com\") as session:\n    page = session.get(\"https://example.com/data\")\n    print(page.json())\n```\n\n## API\n\n```bash\n# Simple one-shot (auto lease/release)\ncurl -X POST localhost:8180/fetch \\\n  -d '{\"url\": \"https://example.com\", \"consumer\": \"my-scraper\"}'\n\n# Manual lease lifecycle\ncurl -X POST localhost:8180/lease \\\n  -d '{\"consumer\": \"my-scraper\", \"domain\": \"example.com\"}'\ncurl -X POST localhost:8180/request \\\n  -d '{\"lease_id\": \"lease-abc\", \"url\": \"https://example.com\"}'\ncurl -X POST localhost:8180/release -d '{\"lease_id\": \"lease-abc\"}'\n\n# Browser actions\ncurl -X POST localhost:8180/request -d '{\n  \"lease_id\": \"lease-abc\",\n  \"url\": \"https://example.com/login\",\n  \"actions\": [\n    {\"type\": \"fill\", \"selector\": \"#user\", \"value\": \"hunter\"},\n    {\"type\": \"submit\", \"selector\": \"#form\"}\n  ]\n}'\n\n# Stateful multi-step (omit URL to stay on page)\ncurl -X POST localhost:8180/request -d '{\n  \"lease_id\": \"lease-abc\",\n  \"actions\": [{\"type\": \"click\", \"selector\": \"#next\"}]\n}'\n\n# XHR capture\ncurl -X POST localhost:8180/request -d '{\n  \"lease_id\": \"lease-abc\",\n  \"url\": \"https://example.com/app\",\n  \"capture_xhr\": true,\n  \"capture_xhr_filter\": \"api/v1\"\n}'\n\n# Force CF renewal\ncurl -X POST localhost:8180/renew -d '{\"domain\": \"example.com\"}'\n```\n\n## Backends\n\n| Backend | Class | Flag |\n|---------|-------|------|\n| Chrome | heavy | `-backend chrome` |\n| [StealthPanda](https://github.com/we-be/stealthpanda) | medium | `-backend stealthpanda` |\n| [Lightpanda](https://github.com/lightpanda-io/browser) | medium | `-backend lightpanda` |\n| CDP (any) | medium | `-backend cdp -cdp-url ws://...` |\n| tls-client | light | `-backend tls-client` |\n| Stub | heavy | `-backend stub` |\n\n## Proxy Support\n\nPer-agent: `-proxy-url http://host:port -proxy-user user -proxy-pass pass`\n\nController pool: `--proxy-file proxies.json` or `--proxy-api http://api/proxies`\n\n## Identity \u0026 Warm Matching\n\nEach agent gets a persistent lowcountry fish name (`redfish-a3b2`, `mullet-8d24`). The controller tracks cookies, CF clearance, and visit history per domain:\n\n| Warmth | Meaning |\n|--------|---------|\n| 3 | Valid `cf_clearance` |\n| 2 | Has cookies |\n| 1 | Visited before |\n| 0 | Cold |\n\n## Reliability\n\n- **Health checks** — dead agents removed after 3 missed polls\n- **Lease TTLs** — abandoned leases auto-expire\n- **Pool persistence** — JSON snapshots, survives controller restarts\n- **Agent reconnection** — same address = same fish, cookies preserved\n- **CF auto-renewal** — clearance refreshed before expiry\n- **Cookie catch-up** — late-joining minnows get cookies on first lease\n- **Tab cleanup** — leaked Chrome tabs closed after every navigation\n- **AcquireWait** — clients can wait for an agent instead of getting rejected\n\n## Configuration\n\nAll flags have `SHOAL_*` env var equivalents for container deployments. CLI flags take precedence.\n\n## Docker\n\n```bash\ndocker compose up  # 1 grouper + 3 minnows\n\n# Or pull from ghcr.io\ndocker pull ghcr.io/we-be/shoal-controller:latest\ndocker pull ghcr.io/we-be/shoal-minnow:latest\ndocker pull ghcr.io/we-be/shoal-grouper:latest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwe-be%2Fshoal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwe-be%2Fshoal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwe-be%2Fshoal/lists"}