{"id":50294728,"url":"https://github.com/chrisrobison/textweb","last_synced_at":"2026-05-28T08:04:00.265Z","repository":{"id":339500672,"uuid":"1161755380","full_name":"chrisrobison/textweb","owner":"chrisrobison","description":"A text-grid web renderer for AI agents — see the web without screenshots","archived":false,"fork":false,"pushed_at":"2026-03-06T18:36:03.000Z","size":137,"stargazers_count":75,"open_issues_count":0,"forks_count":7,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-06T21:50:56.903Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrisrobison.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-19T13:35:06.000Z","updated_at":"2026-03-06T18:36:06.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/chrisrobison/textweb","commit_stats":null,"previous_names":["chrisrobison/textweb"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chrisrobison/textweb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisrobison%2Ftextweb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisrobison%2Ftextweb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisrobison%2Ftextweb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisrobison%2Ftextweb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrisrobison","download_url":"https://codeload.github.com/chrisrobison/textweb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisrobison%2Ftextweb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33599494,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-28T08:03:58.831Z","updated_at":"2026-05-28T08:04:00.258Z","avatar_url":"https://github.com/chrisrobison.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TextWeb\n\n**A text-grid web renderer for AI agents — see the web without screenshots.**\n\nInstead of taking expensive screenshots and piping them through vision models, TextWeb renders web pages as structured text grids that LLMs can reason about natively. Full JavaScript execution, spatial layout preserved, interactive elements annotated.\n\n📄 [Documentation](https://chrisrobison.github.io/textweb) · 📦 [npm](https://www.npmjs.com/package/textweb) · 🐙 [GitHub](https://github.com/chrisrobison/textweb)\n\n## Why?\n\n| Approach | Size | Requires | Speed | Spatial Layout |\n|----------|------|----------|-------|----------------|\n| Screenshot + Vision | ~1MB | Vision model ($$$) | Slow | Pixel-level |\n| Accessibility Tree | ~5KB | Nothing | Fast | ❌ Lost |\n| Raw HTML | ~100KB+ | Nothing | Fast | ❌ Lost |\n| **TextWeb** | **~2-5KB** | **Nothing** | **Fast** | **✅ Preserved** |\n\n## Quick Start\n\n```bash\nnpm install -g textweb\nnpx playwright install chromium\n```\n\n```bash\n# Render any page\ntextweb https://news.ycombinator.com\n\n# Explicitly request grid mode (same as default)\ntextweb --output grid https://news.ycombinator.com\n\n# Semantic JSON output for agent workflows\ntextweb --output semantic https://example.com\n\n# Hybrid output (grid + semantic metadata)\ntextweb --output hybrid https://example.com\n\n# Interactive mode\ntextweb --interactive https://github.com\n\n# Legacy JSON output (backward compatible)\ntextweb --json https://example.com\n```\n\n## Example Output\n\n```\n[0]Hacker News [1]new | [2]past | [3]comments | [4]ask | [5]show | [6]jobs | [7]submit      [8]login\n\n 1. [9]Show HN: TextWeb – text-grid browser for AI agents (github.com)\n    142 points by chrisrobison 3 hours ago | [10]89 comments\n 2. [11]Why LLMs don't need screenshots to browse the web\n    87 points by somebody 5 hours ago | [12]34 comments\n\n[13:______________________] [14 Search]\n```\n\n~500 bytes. An LLM can read this, understand the layout, and say \"click ref 9\" to open the first link. No vision model needed.\n\n## Integration Options\n\nTextWeb works with any AI agent framework. Pick your integration:\n\n### 🔌 MCP Server (Claude Desktop, Cursor, Windsurf, Cline, OpenClaw)\n\nThe fastest way to add web browsing to any MCP-compatible client.\n\n```bash\n# Install globally\nnpm install -g textweb\n\n# Or run directly\nnpx textweb-mcp\n```\n\n**Claude Desktop** — add to `~/Library/Application Support/Claude/claude_desktop_config.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"textweb\": {\n      \"command\": \"textweb-mcp\"\n    }\n  }\n}\n```\n\n**Cursor** — add to `.cursor/mcp.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"textweb\": {\n      \"command\": \"textweb-mcp\"\n    }\n  }\n}\n```\n\n**OpenClaw** — add to `openclaw.json` skills or MCP config.\n\nThen just ask: *\"Go to hacker news and find posts about AI\"* — the agent uses text grids instead of screenshots.\n\n**New (v0.2.1-style MCP capabilities):**\n- `session_id` on every tool call for isolated parallel workflows\n- `textweb_storage_save` / `textweb_storage_load` for persistent auth/session state\n- `textweb_wait_for` for multi-step async UI transitions\n- `textweb_assert_field` for flow guards before submit\n\n### 🛠️ OpenAI / Anthropic Function Calling\n\nDrop-in tool definitions for any function-calling model. See [`tools/tool_definitions.json`](tools/tool_definitions.json).\n\nPair with the [system prompt](tools/system_prompt.md) to teach the model how to read the grid:\n\n```python\nimport json\n\n# Load tool definitions\nwith open(\"tools/tool_definitions.json\") as f:\n    textweb_tools = json.load(f)[\"tools\"]\n\n# Load system prompt\nwith open(\"tools/system_prompt.md\") as f:\n    system_prompt = f.read()\n\n# Use with OpenAI\nresponse = openai.chat.completions.create(\n    model=\"gpt-4\",\n    messages=[\n        {\"role\": \"system\", \"content\": system_prompt},\n        {\"role\": \"user\", \"content\": \"Go to example.com and click the first link\"},\n    ],\n    tools=textweb_tools,\n)\n```\n\n### 🦜 LangChain\n\n```python\nfrom tools.langchain import get_textweb_tools\n\n# Start the server first: textweb --serve 3000\ntools = get_textweb_tools(base_url=\"http://localhost:3000\")\n\n# Use with any LangChain agent\nfrom langchain.agents import initialize_agent\nagent = initialize_agent(tools, llm, agent=\"zero-shot-react-description\")\nagent.run(\"Find the top story on Hacker News\")\n```\n\n### 🚢 CrewAI\n\n```python\nfrom tools.crewai import TextWebBrowseTool, TextWebClickTool, TextWebTypeTool\n\n# Start the server first: textweb --serve 3000\nresearcher = Agent(\n    role=\"Web Researcher\",\n    tools=[TextWebBrowseTool(), TextWebClickTool(), TextWebTypeTool()],\n    llm=llm,\n)\n```\n\n### 🌐 HTTP API\n\n```bash\n# Start the server\ntextweb --serve 3000\n\n# Navigate\ncurl -X POST http://localhost:3000/navigate \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"url\": \"https://example.com\"}'\n\n# Click, type, scroll\ncurl -X POST http://localhost:3000/click -d '{\"ref\": 3}'\ncurl -X POST http://localhost:3000/type -d '{\"ref\": 7, \"text\": \"hello\"}'\ncurl -X POST http://localhost:3000/scroll -d '{\"direction\": \"down\"}'\n```\n\n### 📦 Node.js Library\n\n```javascript\nconst { AgentBrowser } = require('textweb');\n\nconst browser = new AgentBrowser({ cols: 120 });\nconst { view, elements, semantic, meta } = await browser.navigate('https://example.com');\n\nconsole.log(view);        // The text grid\nconsole.log(elements);    // { 0: { selector, tag, text, href }, ... }\nconsole.log(semantic);    // { mode, url, title, elements: [...] }\nconsole.log(meta.stats);  // { totalElements, interactiveElements, renderMs }\n\nawait browser.click(3);              // Click element [3]\nawait browser.type(7, 'hello');      // Type into element [7]\nawait browser.scroll('down');        // Scroll down\nawait browser.waitFor({ selector: '.step-2.active' }); // Wait for next step\nawait browser.assertField(7, 'hello', { comparator: 'equals' }); // Validate field state\nawait browser.saveStorageState('/tmp/textweb-state.json');\nawait browser.loadStorageState('/tmp/textweb-state.json');\nawait browser.query('nav a');        // Find elements by CSS selector\nawait browser.screenshot();          // PNG buffer (for debugging)\nconsole.log(browser.getCurrentUrl());// Current page URL\nawait browser.close();\n```\n\n## Grid Conventions\n\n| Element | Rendering | Interaction |\n|---------|-----------|-------------|\n| Links | `[ref]link text` | `click(ref)` |\n| Buttons | `[ref button text]` | `click(ref)` |\n| Text inputs | `[ref:placeholder____]` | `type(ref, \"text\")` |\n| Checkboxes | `[ref:X]` / `[ref: ]` | `click(ref)` to toggle |\n| Radio buttons | `[ref:●]` / `[ref:○]` | `click(ref)` |\n| Dropdowns | `[ref:▼ Selected]` | `select(ref, \"value\")` |\n| File inputs | `[ref:📎 Choose file]` | `upload(ref, \"/path\")` |\n| Headings | `═══ HEADING ═══` | — |\n| Separators | `────────────────` | — |\n| List items | `• Item text` | — |\n\n## How It Works\n\n```\n┌─────────────────────────────────────────────┐\n│  Your Agent (any LLM)                        │\n│  \"click 3\" / \"type 7 hello\" / \"scroll down\"  │\n├─────────────────────────────────────────────┤\n│  TextWeb                                     │\n│  Pixel positions → character grid            │\n│  Interactive elements get [ref] annotations  │\n├─────────────────────────────────────────────┤\n│  Headless Chromium (Playwright)              │\n│  Full JS/CSS execution                       │\n│  getBoundingClientRect() for all elements    │\n└─────────────────────────────────────────────┘\n```\n\n1. **Real browser** renders the page (full JS, CSS, dynamic content)\n2. **Extract** every visible element's position, size, text, and interactivity\n3. **Map** pixel coordinates to character grid positions (spatial layout preserved)\n4. **Annotate** interactive elements with `[ref]` numbers for agent interaction\n\n## Selector Strategy\n\nTextWeb builds stable CSS selectors for each interactive element, preferring resilient strategies over brittle positional ones:\n\n| Priority | Strategy | Example |\n|----------|----------|---------|\n| 1 | `#id` | `#email` |\n| 2 | `[data-testid]` | `[data-testid=\"submit-btn\"]` |\n| 3 | `[aria-label]` | `input[aria-label=\"Search\"]` |\n| 4 | `[role]` (if unique) | `[role=\"navigation\"]` |\n| 5 | `[name]` | `input[name=\"email\"]` |\n| 6 | `a[href]` (if unique) | `a[href=\"/about\"]` |\n| 7 | `nth-child` (fallback) | `div \u003e a:nth-child(3)` |\n\nThis means selectors survive DOM changes between snapshots — critical for multi-step agent workflows.\n\n## ATS Workflow Examples (Greenhouse / Lever)\n\nFor multi-step ATS flows, use a stable `session_id` and combine wait/assert guards:\n\n```javascript\n// Keep one session for the whole application\nawait textweb_navigate({ url: 'https://job-boards.greenhouse.io/acme/jobs/123', session_id: 'apply-acme' });\n\n// Fill + continue\nawait textweb_type({ ref: 12, text: 'Christopher', session_id: 'apply-acme' });\nawait textweb_type({ ref: 15, text: 'Robison', session_id: 'apply-acme' });\nawait textweb_click({ ref: 42, session_id: 'apply-acme', retries: 3, retry_delay_ms: 400 });\n\n// Guard transition\nawait textweb_wait_for({ selector: '#step-2.active', timeout_ms: 8000, session_id: 'apply-acme', retries: 2 });\n\n// Validate before submit\nawait textweb_assert_field({ ref: 77, expected: 'San Francisco', comparator: 'includes', session_id: 'apply-acme' });\n\n// Persist auth/session for follow-up flow\nawait textweb_storage_save({ path: '/tmp/ats-state.json', session_id: 'apply-acme' });\n```\n\nUseful session tools:\n- `textweb_session_list` → inspect active sessions\n- `textweb_session_close` → close one session or all\n\n## App Runtime Prototype (Manifest + LARC)\n\nThis repository now includes an early scaffold for a manifest-driven user runtime shell (separate from low-level raw admin tooling):\n\n- Manifest validator: `src/app-runtime/manifest.js`\n- PAN topic contract: `src/app-runtime/topics.js`\n- Runtime shell + left nav + tabbed content: `canvas/app-runtime/app-shell.html`\n- Sample manifest: `canvas/app-runtime/sample-app.json`\n\nThe runtime shell uses [LARC](https://github.com/larcjs/larc) PAN (`@larcjs/core-lite`) for in-page event communication, with a local fallback bus if the module cannot be loaded.\n\nTo open the prototype:\n\n```bash\n# Start API server for integration hooks (save manifest/components)\nnpm run serve\n\n# In another terminal, open the runtime shell\nopen /Users/cdr/Projects/textweb/canvas/app-runtime/app-shell.html\n```\n\nIntegration actions implemented on the API server:\n- `POST /integrations/sync_saved_form`\n- `POST /integrations/save_manifest`\n- `POST /integrations/upsert_nav_item`\n- `POST /integrations/runtime_state`\n\n## Testing\n\n```bash\n# Run all tests (form + live + ATS e2e)\nnpm test\n\n# Form fixture tests\nnpm run test:form\n\n# Live site tests — example.com, HN, Wikipedia\nnpm run test:live\n\n# ATS multi-step fixture test\nnpm run test:ats\n```\n\nTest fixtures are in `test/fixtures/` — includes a comprehensive HTML form and an ATS-style multi-step application fixture.\n\n## Design Principles\n\n1. **Text is native to LLMs** — no vision model middleman\n2. **Spatial layout matters** — flat element lists lose the \"where\"\n3. **Cheap and fast** — 2-5KB per render vs 1MB+ screenshots\n4. **Full web support** — real Chromium runs the JS\n5. **Interactive** — reference numbers map to real DOM elements\n\n## License\n\nMIT © [Christopher Robison](https://cdr2.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisrobison%2Ftextweb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrisrobison%2Ftextweb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisrobison%2Ftextweb/lists"}