{"id":50294726,"url":"https://github.com/chrisrobison/textweb-agent","last_synced_at":"2026-05-28T08:03:59.938Z","repository":{"id":342380888,"uuid":"1173776030","full_name":"chrisrobison/textweb-agent","owner":"chrisrobison","description":null,"archived":false,"fork":false,"pushed_at":"2026-03-05T19:52:18.000Z","size":66,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-03-05T21:51:30.262Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrisrobison.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-05T18:32:08.000Z","updated_at":"2026-03-05T19:52:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/chrisrobison/textweb-agent","commit_stats":null,"previous_names":["chrisrobison/textweb-agent"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/chrisrobison/textweb-agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisrobison%2Ftextweb-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisrobison%2Ftextweb-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisrobison%2Ftextweb-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisrobison%2Ftextweb-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrisrobison","download_url":"https://codeload.github.com/chrisrobison/textweb-agent/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisrobison%2Ftextweb-agent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33599494,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-28T08:03:58.697Z","updated_at":"2026-05-28T08:03:59.922Z","avatar_url":"https://github.com/chrisrobison.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TEXTWEB AGENT\n\nSeller agent service for the Nevermined Autonomous Business Hackathon.\n\nIt exposes a paid HTTP API that renders webpages with TextWeb and returns token-efficient summaries/extractions. Metering follows the Nevermined hackathon pattern (`Payments.getInstance` + `paymentMiddleware`) so each call is an agent-to-agent paid transaction.\n\n## Research summary (required)\n\nThis implementation mirrors the official patterns from:\n\n- Nevermined hackathon repo\n  - `agents/seller-simple-agent/ts/src/server.ts`\n  - `agents/buyer-simple-agent/ts/src/client.ts`\n  - `workshops/getting-started/ts/server.ts`\n  - `workshops/payment-plans/ts/dynamic-pricing.ts`\n- TextWeb repo\n  - `src/browser.js` (`AgentBrowser.navigate()` and text-grid snapshots)\n  - `src/server.js` (`/navigate` flow)\n  - `README.md` (library + CLI usage)\n\n### Exact Nevermined SDK patterns used\n\n- Seller payment init:\n  - `Payments.getInstance({ nvmApiKey, environment })`\n- Seller route metering:\n  - `paymentMiddleware(payments, { \"POST /route\": { planId, credits, agentId? } })`\n- Buyer token acquisition pattern (mirrored in SDK):\n  - `payments.x402.getX402AccessToken(planId, agentId?)`\n\n### Payment validation / credit burn pattern\n\nPer hackathon TS examples, validation + settlement are handled by `paymentMiddleware` at HTTP layer. Handlers only run when payment is valid.\n\n## Architecture\n\n- `src/server.ts`: process bootstrap\n- `src/api/app.ts`: HTTP routes and orchestration\n- `src/textweb/adapter.ts`: TextWeb integration + URL safety + follow-links\n- `src/summarizer/engine.ts`: strict JSON summarization / extraction\n- `src/cache/*`: Redis cache with LRU fallback\n- `src/payments/*`: `PaymentProvider` interface with:\n  - `NeverminedPaymentProvider` (official middleware pattern)\n  - `DummyPaymentProvider` (local testing)\n- `packages/client`: lightweight SDK (`@textweb/client`)\n\n## API\n\n### `POST /v1/summarize`\n\nRequest:\n\n```json\n{\n  \"url\": \"https://example.com\",\n  \"goal\": \"optional user intent\",\n  \"mode\": \"brief\",\n  \"followLinks\": { \"enabled\": true, \"max\": 2 },\n  \"schema\": {\n    \"type\": \"object\",\n    \"properties\": {\n      \"company\": { \"type\": \"string\" }\n    }\n  },\n  \"cache\": true\n}\n```\n\nResponse:\n\n```json\n{\n  \"url\": \"https://example.com/\",\n  \"title\": \"Example Domain\",\n  \"summaryBullets\": [\"...\"],\n  \"keyFacts\": [\"...\"],\n  \"nextActions\": [\"...\"],\n  \"links\": [{ \"text\": \"More information...\", \"href\": \"https://www.iana.org/domains/example\" }],\n  \"extracted\": {},\n  \"cost\": { \"units\": 2, \"credits\": 0.002 },\n  \"meta\": { \"renderMs\": 1200, \"summarizeMs\": 350, \"cached\": false }\n}\n```\n\n### `POST /v1/render`\n\nReturns raw TextWeb representation + links/interactives/text blocks.\n\n### `GET /healthz`\n\nHealth endpoint.\n\n## Local development\n\n```bash\nnpm install\nnpx playwright install chromium\ncp .env.example .env\nPORT=3101 npm run dev\n```\n\nIf port 3000 is busy, keep using `PORT=3101` for demo reliability.\n\n## Hackathon demo quickstart\n\nUse the scripted smoke/demo flow:\n\n```bash\n./scripts/demo-smoke.sh\n```\n\nFor deployed verification (Nevermined token or dummy key):\n\n```bash\nBASE_URL=https://your-domain \\\nPAYMENT_SIGNATURE=\u003cx402-token\u003e \\\n./scripts/deployed-smoke.sh\n```\n\nIt demonstrates:\n\n1. payment gating (`402` when unpaid)\n2. paid summarize call (live)\n3. repeat paid summarize call (cached, lower units)\n4. render endpoint proof\n\nFor full presenter notes and talk track, see `DEMO_TODAY.md`.\n\n## Built-in demo dashboard\n\nWith server running, open:\n\n- `http://localhost:3101/dashboard`\n\nDashboard features:\n\n- submit render/summarize requests from UI\n- view TextWeb text-grid output directly\n- inspect raw JSON response\n- watch live service stats (`/stats`): requests, 402s, cache hits, pages served, units/credits billed\n\n## Environment\n\nSee `.env.example`.\n\nImportant:\n\n- Direct HTTPS on Node is optional. Set `HTTPS_KEY_PATH` and `HTTPS_CERT_PATH` to enable TLS (optionally `HTTPS_CA_PATH`).\n  - Let's Encrypt example: `HTTPS_KEY_PATH=/etc/letsencrypt/live/textweb.net/privkey.pem`, `HTTPS_CERT_PATH=/etc/letsencrypt/live/textweb.net/fullchain.pem`\n\n- `PAYMENT_PROVIDER=nevermined` requires `NVM_API_KEY` and `NVM_PLAN_ID`\n- `PAYMENT_PROVIDER=dummy` requires `DUMMY_API_KEY` and uses `x-api-key` header\n- `URL_ALLOWLIST` supports exact hosts and wildcards (for example `example.com,*.trusted.site`)\n- `BLOCK_PRIVATE_NETWORKS=true` blocks private/loopback IP resolution (SSRF protection)\n- `MAX_SCHEMA_BYTES` caps extraction schema payload size for `/v1/summarize`\n\n## Tests\n\n```bash\nnpm test\n```\n\n## curl examples\n\nHealth:\n\n```bash\ncurl http://localhost:3000/healthz\n```\n\nRender (dummy payment mode):\n\n```bash\ncurl -X POST http://localhost:3000/v1/render \\\n  -H 'content-type: application/json' \\\n  -H 'x-api-key: dev-textweb-key' \\\n  -d '{\"url\":\"https://example.com\",\"cache\":true}'\n```\n\nSummarize:\n\n```bash\ncurl -X POST http://localhost:3000/v1/summarize \\\n  -H 'content-type: application/json' \\\n  -H 'x-api-key: dev-textweb-key' \\\n  -d '{\"url\":\"https://example.com\",\"mode\":\"standard\",\"cache\":true}'\n```\n\nNevermined paid call (buyer-side token):\n\n```bash\n# Token generation pattern follows hackathon buyer example:\n# const { accessToken } = await payments.x402.getX402AccessToken(planId, agentId)\n\ncurl -X POST http://localhost:3000/v1/summarize \\\n  -H 'content-type: application/json' \\\n  -H \"payment-signature: \u003cx402-access-token\u003e\" \\\n  -d '{\"url\":\"https://example.com\",\"mode\":\"brief\"}'\n```\n\n## SDK usage (`@textweb/client`)\n\n```ts\nimport { TextWeb } from '@textweb/client'\n\nconst web = new TextWeb({\n  endpoint: 'http://localhost:3000',\n  apiKey: process.env.TEXTWEB_KEY // dummy mode\n})\n\nconst summary = await web.summarize('https://example.com')\nconst render = await web.render('https://example.com')\nconst extracted = await web.extract('https://example.com', {\n  type: 'object',\n  properties: {\n    headline: { type: 'string' }\n  }\n})\n```\n\nWith Nevermined auto-token generation in SDK:\n\n```ts\nimport { TextWeb } from '@textweb/client'\n\nconst web = new TextWeb({\n  endpoint: 'https://api.textweb.ai',\n  nevermined: {\n    nvmApiKey: process.env.NVM_SUBSCRIBER_API_KEY!,\n    planId: process.env.NVM_PLAN_ID!,\n    agentId: process.env.NVM_AGENT_ID,\n    environment: 'sandbox'\n  }\n})\n\nconst result = await web.summarize({\n  url: 'https://example.com',\n  mode: 'deep'\n})\n\n// Optional: inspect resolved auth headers before a custom fetch call\nconst headers = await web.getAuthHeaders()\n```\n\n## Notes on pricing and cache\n\n- Requests are always metered.\n- Summaries served from cache use lower credit units than live summaries.\n- Cache key dimensions: `url + goal + mode + schema + followLinks`.\n\n## Security controls\n\n- URL validation (`http/https` only, `file://` blocked)\n- Request timeout\n- max follow-links\n- max render chars / response safety cap\n- per-IP rate limiting\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisrobison%2Ftextweb-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrisrobison%2Ftextweb-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisrobison%2Ftextweb-agent/lists"}