{"id":50526026,"url":"https://github.com/olostep-api/olostep-cli","last_synced_at":"2026-06-03T08:02:33.837Z","repository":{"id":360248531,"uuid":"1249171389","full_name":"olostep-api/olostep-cli","owner":"olostep-api","description":"CLI for the Olostep API — scrape, map, crawl, answer, batch the web from your terminal. Pure JS rewrite of olostep-cli.","archived":false,"fork":false,"pushed_at":"2026-05-25T15:49:04.000Z","size":127,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-25T17:29:26.625Z","etag":null,"topics":["ai-agents","cli","crawler","mcp","nodejs","npm","olostep","scraping","typescript","web-scraping"],"latest_commit_sha":null,"homepage":"https://www.olostep.com/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/olostep-api.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-25T12:26:45.000Z","updated_at":"2026-05-25T15:49:03.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/olostep-api/olostep-cli","commit_stats":null,"previous_names":["olostep-api/olostep-cli"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/olostep-api/olostep-cli","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olostep-api%2Folostep-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olostep-api%2Folostep-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olostep-api%2Folostep-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olostep-api%2Folostep-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/olostep-api","download_url":"https://codeload.github.com/olostep-api/olostep-cli/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olostep-api%2Folostep-cli/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33854119,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-03T02:00:06.370Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","cli","crawler","mcp","nodejs","npm","olostep","scraping","typescript","web-scraping"],"created_at":"2026-06-03T08:02:30.276Z","updated_at":"2026-06-03T08:02:33.815Z","avatar_url":"https://github.com/olostep-api.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# olostep-cli\n\n[![npm](https://img.shields.io/npm/v/olostep-cli.svg)](https://www.npmjs.com/package/olostep-cli)\n[![Downloads](https://img.shields.io/npm/dm/olostep-cli.svg)](https://www.npmjs.com/package/olostep-cli)\n[![CI](https://github.com/olostep-api/olostep-cli/actions/workflows/ci.yml/badge.svg)](https://github.com/olostep-api/olostep-cli/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](./LICENSE)\n[![Node](https://img.shields.io/node/v/olostep-cli.svg)](https://nodejs.org)\n\nThe official CLI for the [Olostep API](https://www.olostep.com/) — scrape, search, crawl, map, and batch the web from your terminal. Every data command returns **JSON on stdout**, so it pipes cleanly into `jq`, scripts, agents, and CI.\n\nPure JavaScript, Node 18+, no native binaries to download. Installs in under a second, starts in ~200 ms, ships as a single ~100 KB bundle.\n\n---\n\n## Install\n\n```bash\nnpm install -g olostep-cli\nolostep init\n```\n\nRequires Node.js **18+**.\n\n**One-liner alternatives** (no manual `npm` step):\n\n```bash\n# macOS / Linux\ncurl -fsSL https://olostep.com/install.sh | sh\n\n# Windows PowerShell\niwr -useb https://olostep.com/install.ps1 | iex\n```\n\n`olostep init` is the recommended first step — it signs you in, installs the Olostep skills into every detected AI agent, and configures the MCP server, all in one go.\n\nTo **just sign in**: `olostep login` (or `--no-browser` for SSH).\nTo **sign out**: `olostep logout`.\nFor CI/agents: set `OLOSTEP_API_KEY=...`.\nGet a key at \u003chttps://www.olostep.com/dashboard/api-keys\u003e.\n\nOr try without installing: `npx -y olostep-cli@latest --help`.\n\n---\n\n## Quick start\n\n```bash\n# Pull one URL as clean markdown\nolostep scrape \"https://example.com\" --formats markdown\n\n# Live web search\nolostep search \"best web scraping APIs 2025\" --limit 10\n\n# Discover all URLs on a site\nolostep map \"https://example.com\" --top-n 20\n\n# AI-researched answer with citations\nolostep answer \"What does Olostep do?\"\n\n# Crawl every page on a site\nolostep crawl \"https://docs.example.com\" --max-pages 50\n\n# Scrape many URLs from a CSV, in parallel\nolostep batch-scrape urls.csv --formats markdown,html\n```\n\nPipes stay clean — logs go to stderr, JSON to stdout:\n\n```bash\nolostep map \"https://example.com\" | jq '.urls[:10]'\nolostep scrape \"https://example.com\" | jq -r '.result.markdown_content'\nolostep search \"topic\" --json | jq '.links[].url'\n```\n\n---\n\n## Commands\n\nRun `olostep \u003ccommand\u003e --help` for the full flag list.\n\n### Auth\n\n| Command | What it does |\n| --- | --- |\n| `olostep login` | Browser PKCE sign-in |\n| `olostep logout` | Remove saved credentials |\n| `olostep init` | Login + install skills + install MCP server |\n| `olostep status` | Show auth status and config paths |\n| `olostep auth login` | Alias for `olostep login` |\n| `olostep auth logout` | Alias for `olostep logout` |\n| `olostep auth status` | Alias for `olostep status` |\n| `olostep auth set-key \u003ckey\u003e` | Save an API key directly (no browser needed) |\n| `olostep auth set-token \u003ctoken\u003e` | Save a service token (for CI / autonomous agents) |\n\n### Data commands\n\n| Command | What it does |\n| --- | --- |\n| `olostep scrape \u003curl\u003e` | Turn a URL into markdown, HTML, JSON, text, PDF, or screenshot |\n| `olostep scrape-get \u003cid\u003e` | Re-fetch a prior scrape by ID |\n| `olostep search \u003cquery\u003e` | Live web search — returns deduplicated links with title + description |\n| `olostep map \u003curl\u003e` | Discover every URL on a site |\n| `olostep answer \u003cquestion\u003e` | AI-researched, cited answer from live web data |\n| `olostep crawl \u003curl\u003e` | Crawl an entire site (async, polls until done) |\n| `olostep batch-scrape \u003curls.csv\u003e` | Scrape up to 10 000 URLs in parallel |\n| `olostep batch-update \u003cbatch_id\u003e` | Update metadata on a batch job |\n\n**Common flags** on every data command: `--out \u003cpath\u003e` (write JSON to file, default stdout), `--timeout \u003cseconds\u003e`, `--api-key \u003ckey\u003e`.\n\n`olostep search` extra flags: `--limit \u003cn\u003e` (default 12, max 25), `--include-domains \u003clist\u003e`, `--exclude-domains \u003clist\u003e`.\n\n### Skills\n\n```bash\nolostep add skills                    # install all skills into every detected agent\nolostep skills install                # same (alias)\nolostep skills list                   # show what's installed and where\nolostep skills update                 # re-install / overwrite existing skills\nolostep skills uninstall              # remove all skills\n```\n\n### MCP server\n\n```bash\nolostep mcp install                   # detect agents, use hosted endpoint (default)\nolostep mcp install --agent cursor    # target one agent\nolostep mcp install --transport stdio # local `npx olostep-mcp` instead\nolostep mcp install --no-global       # project-scoped config\nolostep mcp install --dry-run --json  # show the plan, don't write\nolostep list mcp                      # see where it's installed\nolostep mcp uninstall                 # remove the olostep entry\n```\n\n### Health \u0026 diagnostics\n\n| Command | What it does |\n| --- | --- |\n| `olostep doctor` | Run health checks — auth, API reachability, MCP endpoint, agent configs |\n| `olostep doctor --json` | NDJSON output (one record per check) — good for CI |\n| `olostep doctor --skip-network` | Auth + config checks only, no network calls |\n| `olostep doctor --fail-on-warn` | Exit 1 on any warning too |\n| `olostep version` | Show CLI version, Node version, channel |\n| `olostep version --json` | Machine-readable: `{ cli, node, channel }` |\n| `olostep update` | Update to the latest version |\n\n---\n\n## Skills for AI agents\n\n`olostep add skills` installs **13 skill files** — `SKILL.md` files that land in Claude Code, Cursor, and other agents so they know what Olostep can do and when to use it.\n\n### Setup\n\n| Skill | What it does |\n| --- | --- |\n| `setup` | Teaches the agent how to configure the Olostep MCP server. Use this first. |\n\n### Core web data\n\n| Skill | What it does |\n| --- | --- |\n| `scrape` | Turn one URL into clean markdown / HTML / JSON / text |\n| `search` | Live web search — results, answers, and in-site URL discovery |\n| `answers` | Cited, structured answers from live web data |\n| `crawl` | Autonomously crawl a whole site |\n| `map` | Discover every URL on a site |\n| `batch` | Scrape up to 10 000 URLs in parallel |\n| `extract-schema` | Scrape a page into structured JSON matching a schema |\n\n### Build \u0026 integrate\n\n| Skill | What it does |\n| --- | --- |\n| `integrate` | Auto-install the Olostep SDK into a project |\n| `docs-to-code` | Scrape API docs and write working code from them |\n| `migrate-code` | Read a migration guide and update local code |\n\n### Research \u0026 debug\n\n| Skill | What it does |\n| --- | --- |\n| `research` | Cited, comparative web research for a decision |\n| `debug-error` | Look up an error message against live GitHub / SO / docs |\n\n```bash\nolostep add skills                              # all 13\nolostep add skills --category usage             # core data skills only\nolostep add skills --skill scrape --skill map   # cherry-pick\nolostep add skills --agent cursor --agent claude\nolostep list skills --json\n```\n\nOther flags: `--exclude \u003cname\u003e`, `--global` / `--no-global`, `--link-mode \u003cauto|symlink|copy\u003e`, `--overwrite` / `--no-overwrite`.\n\n**Supported agents for skills:** Cursor, Claude, Codex, Windsurf, Continue, Augment, Roo, Gemini, Copilot, Factory.\n\n---\n\n## MCP server install\n\nThe CLI writes the Olostep MCP server entry into your agent's config — no JSON editing needed.\n\n**Two transports:**\n\n- **`http`** *(default)* — hosted at `https://mcp.olostep.com/mcp`. No local process, no Node dependency for the agent.\n- **`stdio`** — runs `npx -y olostep-mcp` locally. Useful for offline / corporate-proxy setups.\n\nThe installer merges only the `olostep` key into your existing config without touching other servers. Restart your agent after install.\n\n**Supported agents for MCP install:** Cursor, Claude Code, Claude Desktop, Windsurf, VS Code, Kilo, OpenCode, Continue, Codex.\n\n---\n\n## Auth \u0026 config\n\nAPI key resolution order (first match wins):\n\n1. `--api-key \u003ckey\u003e` flag\n2. `OLOSTEP_API_KEY` env var\n3. `OLOSTEP_API_TOKEN` env var\n4. `.env` file in the current directory\n5. Saved credentials (`olostep login` / `olostep auth set-key`)\n\nCredentials are shared with the Python CLI — existing users keep their login after upgrading.\n\n| OS | Credentials path |\n| --- | --- |\n| macOS | `~/Library/Application Support/olostep-cli/credentials.json` |\n| Linux | `~/.config/olostep-cli/credentials.json` |\n| Windows | `%USERPROFILE%\\AppData\\Roaming\\olostep-cli\\credentials.json` |\n\n**Environment variables:**\n\n| Variable | Effect |\n| --- | --- |\n| `OLOSTEP_API_KEY` | API key |\n| `OLOSTEP_API_TOKEN` | API key (legacy alias) |\n| `OLOSTEP_JSON=1` | Force JSON output on every command (same as passing `--json` globally) |\n| `OLOSTEP_NO_UPDATE_CHECK=1` | Silence the \"update available\" notice |\n| `OLOSTEP_CLI_CONFIG_DIR` | Override the credentials directory |\n\n---\n\n## Tips\n\n**PowerShell** tokenizes `,` and `*` differently — quote arguments:\n\n```powershell\nolostep scrape \"https://example.com\" --formats \"markdown,html\"\nolostep map   \"https://example.com\" --include-url \"/*\"\nolostep answer \"Extract facts\" --json-format '{\"company\":\"\",\"year\":\"\"}'\n```\n\n**CI / agents** — set `OLOSTEP_API_KEY` and optionally `OLOSTEP_JSON=1` to always get machine-readable output:\n\n```bash\nOLOSTEP_API_KEY=sk_... OLOSTEP_JSON=1 olostep scrape \"https://example.com\"\n```\n\n**Health check in CI:**\n\n```bash\nolostep doctor --json --skip-network | jq 'select(.status == \"fail\")'\n```\n\n---\n\n## Links\n\n- **Olostep** — \u003chttps://www.olostep.com\u003e\n- **Docs** — \u003chttps://docs.olostep.com\u003e · [CLI docs](https://docs.olostep.com/sdks/cli)\n- **API keys** — \u003chttps://www.olostep.com/dashboard/api-keys\u003e\n- **npm** — \u003chttps://www.npmjs.com/package/olostep-cli\u003e\n- **GitHub** — \u003chttps://github.com/olostep-api/olostep-cli\u003e\n\n---\n\n## License\n\nMIT — see [`LICENSE`](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folostep-api%2Folostep-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Folostep-api%2Folostep-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folostep-api%2Folostep-cli/lists"}