{"id":49078273,"url":"https://github.com/scrapfly/scrapfly-cli","last_synced_at":"2026-04-20T11:01:58.391Z","repository":{"id":352592281,"uuid":"1211296621","full_name":"scrapfly/scrapfly-cli","owner":"scrapfly","description":"Agentic CLI for the Scrapfly platform: scrape, extract, crawl, and drive a cloud browser over CDP. Built-in LLM agent with Anthropic, OpenAI, Gemini, and Ollama. JSON-first, pipe-friendly, one static binary.","archived":false,"fork":false,"pushed_at":"2026-04-20T09:50:56.000Z","size":536,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-20T10:39:55.557Z","etag":null,"topics":["agentic","ai-agent","anthropic","anti-bot","anti-detect","antidetect-browser","bot-detection","browser-automation","cdp","cli","data-extraction","golang","headless-browser","llm-agent","openai","scrapfly","screenshot","stealth-browser","web-crawler","web-scraping"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scrapfly.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-15T08:51:01.000Z","updated_at":"2026-04-20T09:51:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/scrapfly/scrapfly-cli","commit_stats":null,"previous_names":["scrapfly/scrapfly-cli"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/scrapfly/scrapfly-cli","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapfly%2Fscrapfly-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapfly%2Fscrapfly-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapfly%2Fscrapfly-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapfly%2Fscrapfly-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scrapfly","download_url":"https://codeload.github.com/scrapfly/scrapfly-cli/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapfly%2Fscrapfly-cli/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32044291,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T10:33:29.490Z","status":"ssl_error","status_checked_at":"2026-04-20T10:32:30.107Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic","ai-agent","anthropic","anti-bot","anti-detect","antidetect-browser","bot-detection","browser-automation","cdp","cli","data-extraction","golang","headless-browser","llm-agent","openai","scrapfly","screenshot","stealth-browser","web-crawler","web-scraping"],"created_at":"2026-04-20T11:01:15.136Z","updated_at":"2026-04-20T11:01:58.379Z","avatar_url":"https://github.com/scrapfly.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://scrapfly.io\"\u003e\u003cimg src=\"https://scrapfly.io/logo.svg\" width=\"120\" alt=\"Scrapfly\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003escrapfly-cli\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eThe agentic CLI for web data and browser automation.\u003c/b\u003e\u003cbr/\u003e\n  Scrape, extract, and drive a cloud browser over CDP from any LLM\n  tool-use loop. JSON-first, pipe-friendly, one static binary.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/scrapfly/scrapfly-cli/releases\"\u003e\u003cimg src=\"https://img.shields.io/github/v/release/scrapfly/scrapfly-cli?label=release\" alt=\"Release\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://scrapfly.io/docs\"\u003e\u003cimg src=\"https://img.shields.io/badge/docs-scrapfly.io-0051ff\" alt=\"Docs\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/scrapfly/scrapfly-cli/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/github/license/scrapfly/scrapfly-cli\" alt=\"License\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n`scrapfly` is the official CLI for the [Scrapfly](https://scrapfly.io)\nplatform. It is built **agent-first**: every verb returns a stable JSON\nenvelope, every product is a tool an LLM can call, and an autonomous\nagent is baked in.\n\n- **Agentic by design**: `scrapfly agent \"\u003ctask\u003e\"` runs a Playwright-MCP\n  style loop against a cloud browser. Providers ship out of the box for\n  Anthropic, OpenAI, Gemini, and any OpenAI-compatible endpoint (Ollama,\n  vLLM). Point any other tool-use loop at the full CLI instead via\n  [`agent-onboarding/SKILL.md`](agent-onboarding/SKILL.md).\n- **Full product surface**: Web Scraping, Screenshot, Extraction, Crawler,\n  and the CDP-driven Browser, each with complete SDK-parity flags.\n- **Pipe-friendly**: stable envelope (`{success, product, data|error}`),\n  `--content-only`/`--data-only` for single-line piping, ndjson for\n  batch runs, `--pretty` for humans.\n- **Composable**: multi-call browser sessions via a Unix-socket daemon\n  (cookies + AXTree refs persist across invocations), WARC/HAR in and\n  out, `scrapfly browser` outputs a CDP URL you can hand to Playwright,\n  Puppeteer, or browser-use.\n\n## Install\n\nOne-liner (macOS / Linux):\n\n```bash\ncurl -fsSL https://scrapfly.io/scrapfly-cli/install | sh\n```\n\nPinned version or custom prefix:\n\n```bash\ncurl -fsSL https://scrapfly.io/scrapfly-cli/install | sh -s -- --version v0.1.0 --prefix ~/.local/bin\n```\n\nOr grab a release tarball directly from\n[Releases](https://github.com/scrapfly/scrapfly-cli/releases) - artifacts\nfollow `scrapfly-{os}-{arch}.tar.gz` (`scrapfly-windows-amd64.zip` on\nWindows). Supported: `darwin-universal`, `darwin-amd64`, `darwin-arm64`,\n`linux-amd64`, `linux-arm64`, `windows-amd64`.\n\nFrom source (Go 1.24+):\n\n```bash\ngo install github.com/scrapfly/scrapfly-cli/cmd/scrapfly@latest\n```\n\nOr from a package manager:\n\n```bash\n# Homebrew (macOS, once the tap is published)\nbrew install scrapfly/tap/scrapfly\n\n# npm / pnpm / yarn\nnpm install -D scrapfly-cli\nnpx scrapfly version\n```\n\n### macOS Gatekeeper\n\nThe released macOS binary is not yet signed with an Apple Developer ID, so\nfirst-run may hit *\"Apple could not verify 'scrapfly' is free of malware\"*.\n\n- `install.sh` strips the quarantine bit automatically.\n- If you downloaded the tarball manually, run once:\n  ```bash\n  xattr -d com.apple.quarantine /path/to/scrapfly\n  ```\n  (or right-click → Open in Finder the first time).\n\nSigning + notarization is on the roadmap.\n\n### Self-update\n\nOnce installed, keep the binary current without re-running the installer:\n\n```bash\nscrapfly update           # upgrade in place to the latest GitHub release\nscrapfly update --check   # just print whether a newer version exists\nscrapfly update --version v0.3.0   # pin to a specific release\n```\n\nThe binary always comes from the GitHub release + `checksums.txt`; the\nrelease-notes feed is only used for the human-readable changelog.\n\n## Authentication\n\n```bash\nexport SCRAPFLY_API_KEY=scp-live-...\n# or persist it\nscrapfly config set-key scp-live-...\n```\n\nResolution order: `--api-key` flag \u003e `SCRAPFLY_API_KEY` env \u003e\n`~/.scrapfly/config.json`.\n\n## Quick start\n\n```bash\n# Scrape a JS-heavy page with anti-bot + markdown output\nscrapfly scrape https://web-scraping.dev/products --render-js --asp --format markdown\n\n# Pipe scrape into extract (two-step: fetch + AI extraction)\nscrapfly scrape https://web-scraping.dev/product/1 --render-js --proxified \\\n  | scrapfly extract --content-type text/html --prompt \"name, price, sku\"\n\n# Screenshot auto-named into a directory\nscrapfly -O ./shots screenshot https://example.com --resolution 1920x1080\n\n# Crawl synchronously, then extract the WARC you downloaded\nscrapfly crawl run https://example.com --max-pages 20 --content-format markdown\nscrapfly crawl artifact \u003cuuid\u003e --type warc -o crawl.warc\nscrapfly crawl parse warc-list crawl.warc --pretty\n```\n\n## Browser \u0026 Agent\n\n`scrapfly browser` gives you a CDP WebSocket URL you can hand to\nPlaywright, Puppeteer, [browser-use](https://github.com/browser-use/browser-use),\nor drive directly through the built-in session daemon:\n\n```bash\nscrapfly browser --session demo start \u0026\nscrapfly browser navigate https://web-scraping.dev/login\nscrapfly browser fill 'input[name=username]' user123\nscrapfly browser fill 'input[name=password]' password\nscrapfly browser click 'button[type=submit]'\nscrapfly browser close\n```\n\nOn Scrapfly's custom browser, `fill` / `click` / `wait` / `slide`\nautomatically go through the **Antibot** CDP domain (human-like timing,\nslider-captcha support) and `content` uses **Page.getRenderedContent**\n(HTML with iframes inlined).\n\nFor fully autonomous runs:\n\n```bash\nANTHROPIC_API_KEY=... scrapfly agent \"Find the first product name and price\" \\\n  --url https://web-scraping.dev/products \\\n  --schema '{\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"},\"price\":{\"type\":\"string\"}},\"required\":[\"name\",\"price\"]}'\n```\n\nProviders: Anthropic (default), OpenAI, Google Gemini, and any\nOpenAI-compatible endpoint (Ollama, vLLM, …).\n\n## Product coverage\n\n| Scrapfly API            | REST                                             | CLI                                    |\n|-------------------------|--------------------------------------------------|----------------------------------------|\n| Web Scraping            | `GET/POST /scrape`                               | `scrapfly scrape \u003curl\u003e`                |\n| Scrape Batch            | `POST /scrape/batch`                             | `scrapfly batch \u003curls…\u003e`               |\n| Screenshot              | `POST /screenshot`                               | `scrapfly screenshot \u003curl\u003e`            |\n| Extraction              | `POST /extraction`                               | `scrapfly extract`                     |\n| Classify                | `POST /classify`                                 | `scrapfly classify`                    |\n| Crawler                 | `POST /crawl` + `/crawl/{uuid}/...`              | `scrapfly crawl {start,run,status,...}`|\n| Browser (CDP)           | `wss://browser.scrapfly.io`                      | `scrapfly browser [start/...]`         |\n| Browser Unblock         | `POST /unblock`                                  | `scrapfly browser \u003curl\u003e --unblock`     |\n| Account                 | `GET /account`                                   | `scrapfly account` / `scrapfly status` |\n\nEvery documented SDK field is exposed as a flag. See\n[`agent-onboarding/SKILL.md`](agent-onboarding/SKILL.md) for the complete\nmap.\n\n## For LLM agents\n\nPoint your tool-use loop at [`agent-onboarding/SKILL.md`](agent-onboarding/SKILL.md):\na compact guide covering the six usage paths, auth, envelope contract,\nand every CLI verb grouped by intent.\n\n## GitHub Actions\n\nUse this repo as a reusable action to install the CLI on a runner:\n\n```yaml\n- uses: scrapfly/scrapfly-cli@v0.2.0\n  with:\n    api-key: ${{ secrets.SCRAPFLY_API_KEY }}\n- run: scrapfly scrape https://example.com --format markdown\n```\n\nInputs: `version` (default `latest`), `api-key` (exported to `SCRAPFLY_API_KEY`\nwith log masking), `prefix` (default `$HOME/.local/bin`).\n\n## MCP server\n\nFor Cursor, Claude Desktop, Claude Code, and any other MCP-compatible\nclient:\n\n```jsonc\n{\n  \"mcpServers\": {\n    \"scrapfly\": {\n      \"command\": \"scrapfly\",\n      \"args\": [\"mcp\", \"serve\"],\n      \"env\": { \"SCRAPFLY_API_KEY\": \"scp-live-...\" }\n    }\n  }\n}\n```\n\nExposes `scrape`, `screenshot`, `extract`, `crawl_run`, and `selector` as\ntools. Schemas are generated from the flag definitions so the client sees\nexactly the surface the CLI does.\n\n## Shell completion\n\n```bash\nscrapfly completion zsh  \u003e \"${fpath[1]}/_scrapfly\"       # zsh\nscrapfly completion bash \u003e ~/.bash_completion.d/scrapfly # bash\nscrapfly completion fish \u003e ~/.config/fish/completions/scrapfly.fish\n```\n\n## Development\n\n```bash\nmake install   # go mod download\nmake dev       # build dist/scrapfly\nmake test      # go test ./...\nmake lint      # go vet ./...\nmake fmt       # gofmt -w .\n\n# cut a release (tags + pushes; CI publishes binaries)\nmake release VERSION=0.2.0 NEXT_VERSION=0.2.1\n```\n\n## Links\n\n- Scrapfly docs: https://scrapfly.io/docs\n- Official SDKs (Python, TypeScript, Go, Rust): https://scrapfly.io/docs/sdk\n- Issues: https://github.com/scrapfly/scrapfly-cli/issues\n\n## License\n\nMIT. See [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrapfly%2Fscrapfly-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscrapfly%2Fscrapfly-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrapfly%2Fscrapfly-cli/lists"}