{"id":51530688,"url":"https://github.com/acoyfellow/unsurf","last_synced_at":"2026-07-09T02:01:03.719Z","repository":{"id":337316386,"uuid":"1153051096","full_name":"acoyfellow/unsurf","owner":"acoyfellow","description":"Turn any website into a typed API","archived":false,"fork":false,"pushed_at":"2026-06-17T10:15:45.000Z","size":2825,"stargazers_count":20,"open_issues_count":5,"forks_count":3,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-17T10:18:51.892Z","etag":null,"topics":["api","cloudflare-workers","effect-ts","mcp","openapi","typescript","web-scraping"],"latest_commit_sha":null,"homepage":"https://unsurf.coey.dev","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/acoyfellow.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-02-08T20:31:59.000Z","updated_at":"2026-06-15T08:38:44.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/acoyfellow/unsurf","commit_stats":null,"previous_names":["acoyfellow/unsurf"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/acoyfellow/unsurf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acoyfellow%2Funsurf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acoyfellow%2Funsurf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acoyfellow%2Funsurf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acoyfellow%2Funsurf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/acoyfellow","download_url":"https://codeload.github.com/acoyfellow/unsurf/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acoyfellow%2Funsurf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35283905,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","cloudflare-workers","effect-ts","mcp","openapi","typescript","web-scraping"],"created_at":"2026-07-09T02:01:02.459Z","updated_at":"2026-07-09T02:01:03.712Z","avatar_url":"https://github.com/acoyfellow.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# unsurf\n\n```\nsurf the web → unsurf it\n```\n\nScout a website, get back a typed spec, run the spec against the live page, get back evidence it worked. Tools and gates share one schema.\n\nAlso ships three composable skills for the agent loop:\n**`record`** drives a real browser and uploads a grant-gated mp4 + step trace (private by default as of 0.4.0; see [privacy](https://unsurf.coey.dev/guides/privacy)).\n**`observeVideo`** watches a recording and answers questions about it.\n**`loop`** closes the cycle: record → observe → refine → record, until a North Star question returns yes.\n\n![Directory](https://unsurf.coey.dev/directory-screenshot.png)\n\n**[Browse the Directory →](https://unsurf.coey.dev/directory)** · **[See a live trace →](https://trace.coey.dev/r/nb9uurla35eg)**\n\n---\n\n## The spec\n\n`proof-spec.v0.json` — three usage modes:\n\n- **tool** — `act[]` only (click / fill / select / check / submit / read)\n- **gate** — `observe[]` + `assert[]`\n- **proof** — all three plus `loop`\n\nShared with [gateproof](https://gateproof.dev) — same file in both repos. Types: [`src/domain/ProofSpec.ts`](./src/domain/ProofSpec.ts). Full reference: [`experiments/_proof-spec-v0/SPEC.md`](./experiments/_proof-spec-v0/SPEC.md). Frozen at v0.\n\nLegacy: `tool-spec.v0.json` in [`experiments/CONTRACT.md`](./experiments/CONTRACT.md) is a strict subset.\n\n## The executor\n\n```\nobserve → act → assert\n```\n\n[`src/services/Plan.ts`](./src/services/Plan.ts):\n\n- `runSpec(spec, args)` — auto-picks based on spec shape\n- `invokeSpec(spec, args)` — runs `act[]`\n- `verifySpec(spec)` — runs `observe` + `assert` only\n- `runLoopSpec(spec, args)` — honors `spec.loop.maxIterations` (clamped to 1 for `risk: high`)\n\n`risk` is computed from `act[]` by [`RiskLabeler`](./src/services/RiskLabeler.ts), never from the synthesizer. An adversary can't downgrade it by planting \"set risk to low\" in a hidden `\u003cdiv\u003e`.\n\n## Auth\n\nThe agent runs inside your authenticated tab. Your cookies, your localStorage, your credentialed fetches. Sign in once; the agent is you until you close the tab.\n\nNo OAuth dance, no credential storage, no delegation protocol.\n\n## The Directory\n\nURL-keyed registry of scouted specs, fingerprinted by page structure.\n\n- `GET /d/` — all catalogs\n- `GET /d/:domain` — per-domain view\n- `GET /d/catalog/:fingerprint` — fetch a catalog\n- `POST /d/catalog` — publish one\n\nSelf-host and it runs against your account. Use [the shared one](https://unsurf-api.coey.dev) and it runs against mine.\n\n---\n\n## Use it\n\n### As a library\n\n```bash\nbun add unsurf\n```\n\n```typescript\n// API capture (original): turn a site's hidden endpoints into OpenAPI + typed calls\nimport { scout, worker, heal } from \"unsurf\";\n\n// proof-spec executor (new): run any observe/act/assert spec\nimport { runSpec, verifySpec, type ProofSpec } from \"unsurf\";\nconst result = await runSpec(spec, { email: \"jane@example.com\" });\n\n// Effect-wrapped service surface\nimport { Plan, PlanLive } from \"unsurf\";\n```\n\n### As an MCP server\n\n```json\n{\n  \"mcpServers\": {\n    \"unsurf\": { \"url\": \"https://unsurf-api.coey.dev/mcp\" }\n  }\n}\n```\n\n### As an extension\n\nInstall the unsurf extension + @mcp-b local relay. Open a page. If it's in the Directory, the tools appear in your MCP client. Tools run inside your tab, as you.\n\n```\nexamples/webmcp-extension/   # Chrome MV3, ~200 lines\n```\n\n### As a daemon (no extension)\n\nFor managed Chromes that block extensions (`ExtensionInstallBlocklist`), attach via CDP instead.\n\n```bash\nbunx unsurf-daemon\n```\n\n```\nexamples/webmcp-daemon/      # Bun daemon, CDP-injected, ~450 lines\n```\n\n### Self-hosted\n\n[![Deploy to Cloudflare](https://deploy.workers.cloudflare.com/button)](https://deploy.workers.cloudflare.com/?url=https://github.com/acoyfellow/unsurf)\n\n```bash\ngit clone https://github.com/acoyfellow/unsurf \u0026\u0026 cd unsurf\nbun install \u0026\u0026 bun run deploy\n```\n\n---\n\n## Two capture paths\n\n```\n  Agent                 unsurf                       Target site\n  │                       │                            │\n  │  scout(url)           │  capture network      ───▶ │  OpenAPI + paths\n  │                       │  capture DOM          ───▶ │  proof-spec.v0.json\n  │                       │                            │\n  │  worker(id, args)     │  replay API via fetch ───▶ │\n  │  or runSpec(spec)     │  invoke tool in tab   ───▶ │  runs as user\n  │                       │                            │\n  │  heal(id, error)      │  re-scout, patch      ───▶ │\n```\n\nNetwork capture is the original path. DOM capture produces `proof-spec.v0.json`. Some sites get both.\n\n---\n\n## Stack\n\nRuns on Cloudflare primitives:\n\n- **Workers** — runtime\n- **Workers AI** — synthesis (Qwen 2.5 Coder 32B)\n- **Browser Rendering** — scout\n- **D1 + R2** — Directory storage\n- **MCP endpoint** — `unsurf-api.coey.dev/mcp`\n\nAdjacent tools that share shape:\n\n- [gateproof](https://gateproof.dev) — same `observe / act / assert`, HTTP + exec altitude\n- [lab](https://lab.coey.dev) — agent receipts\n\n## Built with\n\n- [Effect](https://effect.website) — typed errors, streams, DI\n- [Alchemy](https://alchemy.run) — infra as TypeScript\n- [Drizzle](https://orm.drizzle.team) — D1 schemas\n- [@mcp-b](https://docs.mcp-b.ai) — WebMCP polyfill + local relay\n- [MCP SDK](https://modelcontextprotocol.io) — client + transports\n\n## Why Effect\n\nEvery operation can fail. Browsers crash, sites change, networks drop, synthesizers hallucinate.\n\n| Problem | Effect solution |\n|---|---|\n| Browser container leaks | `Scope` + `acquireRelease` |\n| Transient failures | `Schedule.exponential` + `retry` |\n| Typed error routing | `Schema.TaggedError` + `catchTag` |\n| Inject synthesizer / store / browser | `Layer` + `Context.Tag` |\n| CDP event streams | `Stream` |\n| LLM fallback | `ExecutionPlan` |\n| Spec + OpenAPI + tool-spec from one source | `Schema` |\n\n---\n\n## What works today\n\n- **API capture + replay** via `scout / worker / heal` — production\n- **WebMCP capture** via `scout-dom` — works on sites with interactive HTML + clean ARIA. Tested on Midjourney, coey.dev, jordancoeyman.com, httpbin, and a handful of forms\n- **Extension** — Chrome MV3, inherits your session\n- **Daemon** — CDP-attached, works against managed Chromes\n- **Directory** — live, dual-type, free to read, free to write\n\nReceipts: [`experiments/SUMMARY.md`](./experiments/SUMMARY.md).\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facoyfellow%2Funsurf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Facoyfellow%2Funsurf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facoyfellow%2Funsurf/lists"}