{"id":51617535,"url":"https://github.com/parawanderer/window-ml","last_synced_at":"2026-07-12T15:01:36.141Z","repository":{"id":369286546,"uuid":"1289170498","full_name":"parawanderer/window-ml","owner":"parawanderer","description":"Console scripting API for OpenWebUI/Ollama — window.ml in devtools on any page. Multimodal via \u003cimg\u003e nodes, chat history, VRAM management.","archived":false,"fork":false,"pushed_at":"2026-07-11T14:46:44.000Z","size":4015,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-11T15:13:44.355Z","etag":null,"topics":["chrome-extension","devtools","llm","ollama","open-webui","scripting"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/parawanderer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-04T12:12:00.000Z","updated_at":"2026-07-11T14:46:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/parawanderer/window-ml","commit_stats":null,"previous_names":["parawanderer/window-ml"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/parawanderer/window-ml","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parawanderer%2Fwindow-ml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parawanderer%2Fwindow-ml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parawanderer%2Fwindow-ml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parawanderer%2Fwindow-ml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/parawanderer","download_url":"https://codeload.github.com/parawanderer/window-ml/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parawanderer%2Fwindow-ml/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35394859,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-12T02:00:06.386Z","response_time":87,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chrome-extension","devtools","llm","ollama","open-webui","scripting"],"created_at":"2026-07-12T15:01:35.245Z","updated_at":"2026-07-12T15:01:36.135Z","avatar_url":"https://github.com/parawanderer.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# window.ml\n\n[![tests](https://github.com/parawanderer/window-ml/actions/workflows/tests.yml/badge.svg)](https://github.com/parawanderer/window-ml/actions/workflows/tests.yml)\n\n![Screenshot showing extension being used with the gemma4 model in the DevTools console](./docs/fullscreen-and-console-2026-07-06_21-38.png)\n\nPersonal Chrome extension that exposes a scripting API (`window.ml`) on web\npages, bridging them to local LLMs served by [OpenWebUI](https://openwebui.com)\n/ [Ollama](https://ollama.com). Built as devtools-console glue: the deliverable\nis a `window.ml` object you call from any page's console (or from page scripts),\nnot a chat UI.\n\n\n\u003e Honest provenance: this is personal, largely AI-generated glue code, shared\n\u003e so the next person wanting a console-first LLM bridge doesn't have to build\n\u003e it from scratch. It works (and has tests), but there's no roadmap and no\n\u003e support — fork freely. MIT licensed.\n\n```js\nawait ml.chat(\"Summarize this page's title: \" + document.title);\n\n// Multimodal — pass literal \u003cimg\u003e DOM nodes:\nawait ml.chat(\"What's in this image?\", { images: [document.images[0]] });\n\n// Multi-turn:\nconst h = ml.createChat({ system: \"Be terse.\" });\nawait h.chat(\"What is a monad?\");\nawait h.chat(\"Now explain it like I'm five\");\n```\n\n…or let an **agent drive the page** from one line of English —\n`ml.agent(\"Search for cat videos.\")` on YouTube ([example](examples/youtube_search.js)):\n\n\u003cvideo src=\"https://github.com/parawanderer/window-ml/raw/main/docs/cat-videos-demo.mp4\" controls muted playsinline width=\"100%\"\u003e\u003c/video\u003e\n\n## Setup\n\n\u003e **Starting from nothing?** [docs/FULL-SETUP.md](docs/FULL-SETUP.md) stands up the\n\u003e whole backend from scratch with Docker (Ollama + OpenWebUI + web search) and a\n\u003e couple of models — the \"initial setup for max utility\" so `window.ml` can do\n\u003e real tasks immediately.\n\u003e\n\u003e Step-by-step walkthrough (install, minimum config, troubleshooting): [docs/SETUP.md](docs/SETUP.md)\n\n\n1. `chrome://extensions` → Developer mode → **Load unpacked** → this directory.\n2. Recommended: set the extension's **Site access** to **On click** or whitelist specific trusted sites only, so\n   `window.ml` only exists on pages where you've clicked the extension icon.\n3. Open the popup (extension icon):\n   - **Chat completions URL** — e.g. `http://localhost:3000/api/chat/completions`\n   - **API key** — OpenWebUI → Settings → Account → API keys\n   - **Model** — hit **Load** to pick from the server's list\n   - **API format** — must match the URL (see table below)\n   - **Save \u0026 Test** sends a real prompt and shows the extracted reply.\n\n\u003cimg src=\"./docs/extension-config-2026-07-06_21-33.png\" alt=\"Screenshot showing Settings Config Menu For Extension\" width=\"360\"\u003e\n\n\n### Endpoint / format cheatsheet\n\n| URL | API format | Notes |\n| --- | --- | --- |\n| `\u003chost\u003e/api/chat/completions` | OpenAI | OpenWebUI's external API. Broken on OpenWebUI 0.9.5 ([#24550](https://github.com/open-webui/open-webui/issues/24550)), fixed by 0.10.x. |\n| `\u003chost\u003e/ollama/api/chat` | Ollama native | OpenWebUI's raw Ollama passthrough (same bearer key). Bypasses OpenWebUI middleware; needs raw Ollama model names. |\n| `http://ollama-host:11434/api/chat` | Ollama native | Direct Ollama, no OpenWebUI. |\n\nThere is **no** root-level `/v1/chat/completions` on OpenWebUI (tested 0.9.5\nand 0.10.2) — unknown routes return the frontend HTML page.\n\n**Cloud/commercial models** (Claude, GPT, OpenRouter) work with no extension\nchanges — add them as a Connection in OpenWebUI and they appear in the model\nlist. See [docs/CLOUD-MODELS.md](docs/CLOUD-MODELS.md).\n\n## `window.ml` API\n\n| Call | Purpose |\n| --- | --- |\n| `ml.chat(prompt, options?)` | One-shot chat. Returns the reply text. |\n| `ml.chatShort(prompt, options?)` | Same, with a brevity suffix. |\n| `ml.createChat(options?)` | Multi-turn history object (below). |\n| `ml.models()` | Available model ids on the server. |\n| `ml.getModel()` / `ml.setModel(id)` | Read / persistently switch the default model. `setModel` validates against the server list and syncs the popup. |\n| `ml.config()` | Read all exposed configuration parameters for the extension (main model default, OCR model, apiFormat) |\n| `ml.ps()` | Models loaded in VRAM: `[{ model, vramGB, expiresAt }]`. |\n| `ml.unload(model?)` | Evict a model from VRAM (`keep_alive: 0`); no argument = evict all. |\n| `ml.read(image, { model?, prompt? })` | OCR — transcribe baked-in text from an `\u003cimg\u003e` or URL to a plain string, using the configured OCR (vision) model. See [OCR](#ocr). |\n| `ml.step(messages, { tools?, model?, think? })` | One model turn with client-side tools; returns the raw assistant message and hands the loop to you. See [Tools](#tools-agents). |\n| `ml.agent(task, options?)` | Plain-English page agent: runs the whole loop over built-in DOM recon tools (and auto-wired vision) until it acts or answers. See [Agent](#tools-agents). |\n| `ml.chat(prompt, { toolIds })` | Server-side tools: OpenWebUI runs the tools and returns the finished answer. See [Tools](#tools-agents). |\n| `ml.logChat` / `ml.logChatShort` | `console.log` variants. |\n\nOptions (all optional, both for `chat` and `createChat`):\n\n- `system` — system prompt.\n- `model` — model override; doesn't touch the saved default.\n- `think` — `true`/`false` maps to Ollama's native thinking toggle; `null`\n  omits it (server default). Models that support thinking are asked not to\n  by default. Sending images to a model without vision capability fails fast\n  with a clear error (checked via `/api/show`).\n- `cleanup` — strip `\u003cthink\u003e…\u003c/think\u003e` from replies (default on).\n- `images` — (per-call) list of `\u003cimg\u003e` elements and/or URL strings.\n- `maxTokens` — hard cap on generated tokens (OpenAI `max_tokens` / Ollama\n  `num_predict`); `null` omits it. Bounds a runaway generation so it can't peg\n  the model. `ml.agent`'s auto-wired vision tool caps itself this way by default.\n- `onToken` — `(delta, full) =\u003e {}`. Stream the reply token-by-token (for a live\n  \"typing\" effect) while the call still resolves to the full string and history\n  updates as usual. Text-only, so it's ignored when `schema` is set. Works with\n  `toolIds` too (a server-side tool runs first, then its answer streams).\n\n  ```js\n  const out = document.querySelector(\"#out\");\n  await ml.chat(\"Explain the Jevons paradox\", {\n    onToken: (tok) =\u003e { out.textContent += tok; }   // paints as it generates\n  });\n  ```\n- `schema` — a JSON Schema object. Constrains the reply to matching JSON and\n  returns it **parsed** (an object), not a string. Turns `window.ml` into a\n  classifier/extractor — the primitive for DOM-scripting against a policy:\n\n  ```js\n  const verdict = await ml.chat(videoTitle, {\n    system: \"You enforce this feed policy: no rage-bait, no crypto shilling.\",\n    schema: {\n      type: \"object\",\n      properties: {\n        hide: { type: \"boolean\" },\n        rewritten_title: { type: \"string\" }\n      },\n      required: [\"hide\", \"rewritten_title\"]\n    }\n  });\n  if (verdict.hide) tile.style.display = \"none\";\n  ```\n\n  Wire mapping: OpenAI format → `response_format` json_schema; Ollama native →\n  `format`. Support depends on the backend — most reliable against Ollama\n  (directly or via OpenWebUI). In history objects the raw JSON text is stored\n  as context so turns still chain.\n\n### History objects\n\n```js\nconst h = ml.createChat({ system, model, think, cleanup });\nawait h.chat(\"prompt\", { images, model, think, cleanup });  // per-turn overrides\n\nh.messages          // plain [{ role, content, images? }] array — edit freely:\nh.messages.at(-1)   //   last message\nh.messages.pop()    //   drop a turn to retry\nh.fork()            // independent deep copy of the conversation\n```\n\nDesign invariants: assistant replies are stored post-cleanup (thinking blocks\nare never resent as context) and a failed request leaves `messages` untouched.\n\n**Sources.** When a server-side tool or RAG runs (e.g. `toolIds`), OpenWebUI\nattaches provenance, and `window.ml` surfaces it on the stored assistant message\nas `.sources` — the raw OpenWebUI array (`[{ source, metadata, document }]`), so\nyou can render citations. Read it off the turn: `h.messages.at(-1).sources`.\nAbsent on plain chats and the Ollama-native format.\n\n### OCR\n\n`ml.read()` transcribes text that's baked into image pixels — the case where a\nsite renders content as an image so it can't be selected or scraped. It returns\na **plain string**, so it composes with `chat`:\n\n```js\nawait ml.chat(\"Summarize this: \" + await ml.read(document.images[0]));\nconst imgs = [...document.querySelectorAll(\".listing img\")];         // bulk\nconst texts = await Promise.all(imgs.map(img =\u003e ml.read(img)));\n```\n\n**Setup:** OCR needs a vision model. Pull one and set it as the *OCR model* in\nthe popup (kept separate from your chat model, so a text-only reasoning model\nstays the default and never sees image tokens):\n\n```sh\nollama pull qwen2.5vl        # or: docker exec ollama ollama pull qwen2.5vl\n```\n\nTypical bulk flow, using the VRAM controls to avoid holding both models at once:\n\n```js\nconst texts = await Promise.all(imgs.map(img =\u003e ml.read(img)));  // vision model stays warm\nawait ml.unload();                    // purge it\nawait ml.setModel(\"qwen3:235b\");      // load the reasoning model\nawait ml.chat(\"Analyze:\\n\" + texts.join(\"\\n---\\n\"));\n```\n\nThere's no separate OCR server — OCR is just a vision-model call through the\nsame OpenWebUI pipe. For specialized accuracy, point the OCR model at any\nvision/OCR model you've added to OpenWebUI (a GOT-OCR2 or TrOCR GGUF, etc.).\n\n### Tools \u0026 agents\n\nTwo primitives, for the two places tools can run. Both need a model that\nsupports function calling (e.g. qwen3); a model that doesn't just replies with\ntext and no tool calls.\n\n#### Server-Side tools \u0026 agents\n\nOpenWebUI runs its own registered tools (web search,\nMCP servers, community/Python tools) and returns the finished answer. One call,\nno loop on your side. **OpenWebUI only** (the Ollama-native format errors):\n\n```js\nawait ml.chat(\"What's the weather in Amsterdam right now?\", { toolIds: [\"web_search\"] });\n```\n\n\n#### Client-Side tools \u0026 agents\n\n##### Option 1: Build your own agent\n\n`ml.step()` is one model turn that hands the loop back\nto *you*: it returns the raw assistant message, you execute any tool calls\n(in the page), append the results, and call it again.\n\n```js\nconst tools = [{ type: \"function\", function: {\n  name: \"readDom\",\n  description: \"Visible text of elements matching a CSS selector.\",\n  parameters: { type: \"object\", properties: { selector: { type: \"string\" } }, required: [\"selector\"] }\n}}];\n\nconst runners = {\n  readDom: ({ selector }) =\u003e [...document.querySelectorAll(selector)].map(e =\u003e e.innerText).join(\"\\n\")\n};\n\nlet messages = [{ role: \"user\", content: \"Find the menu link text on this page.\" }];\nfor (let i = 0; i \u003c 8; i++) {                      // your loop, your step limit\n  const msg = await ml.step(messages, { tools });\n  if (!msg.tool_calls?.length) { console.log(msg.content); break; }\n  messages.push({ role: \"assistant\", content: msg.content, tool_calls: msg.tool_calls });\n  for (const call of msg.tool_calls) {\n    if (!(call.name in runners)) continue;         // your whitelist\n    const result = await runners[call.name](call.arguments);\n    messages.push({ role: \"tool\", tool_call_id: call.id, content: String(result) });\n  }\n}\n```\n\n`ml.step` works on both OpenWebUI and plain Ollama — the wire differences\n(where `tool_calls` live, string vs object arguments, tool-result shape) are\nnormalized so `{ id, name, arguments }` is the same either way.\n\n##### Option 2: Ready-made extensible agent loop\n\n![An example showing the example.com website running an agent to change the HTML of the web page](./docs/agent-example-2026-07-11_16-10.png)\n\n**An agent, with nothing to configure** — `ml.agent(task)` is the loop above,\nalready assembled. You give it plain English; it discovers the DOM with built-in\nrecon tools, writes **one** rule to act on every match at once, and — when your\nmodel can see — screenshots the page to check its own work. No tool definitions,\nno loop, no step cap to wire up:\n\n```js\n// Edit the page: the model finds the repeating container itself and hides them.\nawait ml.agent(\"Hide every sponsored result on this page.\");\n\n// Or get a live DOM node back — hover it in devtools:\nconst { elements } = await ml.agent(\"Find the top banner ad element.\");\nelements[0];        // a real \u003cdiv\u003e, not a selector string\n```\n\nThat's the whole setup: point the popup at your server, pick a big tool-capable\nmodel, and call `ml.agent`. What's built in:\n\n- **DOM recon tools** — `findByText`, `describeElement`, `ancestors`,\n  `countMatches`, `sampleText`, plus an `exec` escape hatch. Small, structured\n  output (never raw HTML), so context stays cheap.\n- **Eyes, auto-wired — natively when possible.** If your model reports vision\n  capability (or your configured [OCR](#ocr) model does), a `look` tool is added\n  automatically so the agent can orient and *visually verify* its work. **When the\n  agent's own model is vision-capable (e.g. `qwen3.5:122b`), `look` feeds the\n  screenshot straight into that model's own conversation** — it reasons over the\n  real pixels, not a second model's text summary. That's a big reliability jump:\n  the delegated path squashes the page to a paragraph and plans over the paragraph,\n  blind to whatever the summary dropped; native vision removes that lossy step. If\n  only your OCR model can see, it falls back to that delegated `look`. Text-only\n  model? It runs without eyes and says so if a task truly needs them. Turn it off\n  with `vision: false`, or force a model with `vision: \"qwen2.5vl\"`.\n- **A safety gate** — `exec` (arbitrary page JS) is approval-gated; the default\n  is a blocking `confirm()`. Pass your own `approve({ tool, arguments })`.\n- **A step cap** (`maxSteps`, default 10) and a full `transcript`.\n\nReturns `{ summary, steps, transcript, elements }` (`elements` holds any DOM\nnodes the agent designated as its answer). Nudge it without rewriting the prompt\nvia `hints`, and watch every thought and tool call in the console with\n`logDebug` (or pass your own `onStep`):\n\n```js\nconst res = await ml.agent(\"Hide items that can't be delivered today.\", {\n  hints: \"On amazon.nl the delivery line reads 'Wordt vandaag bezorgd'.\",\n  logDebug: true          // one console line per step — thoughts, tool calls, live nodes\n  maxSteps: 45\n});\nconsole.log(res.summary);\n```\n\n`ml.agent` is a *composition* of `ml.step`, not a black box — the loop, tool\nwhitelist, step cap and approval gate are all plain code on top of the same\nprimitive. Drop to `ml.step` when you want to own the loop yourself.\n\n### Using from a userscript\n\n`window.ml` lives on the page's **main-world** `window`, so a userscript that\nruns in page context (Tampermonkey, *User JavaScript and CSS*, …) can call it\ndirectly — no messaging. Since your script and the extension race to load, wait\nfor the readiness signal rather than assuming `window.ml` exists yet:\n\n```js\nconst ml = await (window.ml?.ready\n    ?? new Promise(r =\u003e addEventListener(\"ml:ready\", () =\u003e r(window.ml), { once: true })));\n\n// e.g. flag a lunch menu you'd hate, then nudge the tab title:\nconst menu = await ml.read(document.querySelector(\".menu img\"));\nconst v = await ml.chat(\"I hate sandwiches. Warn me if lunch here is sandwich-only.\\n\\n\" + menu,\n    { schema: { type: \"object\", properties: { warn: { type: \"boolean\" }, why: { type: \"string\" } }, required: [\"warn\", \"why\"] } });\nif (v.warn) document.title = \"🥪 \" + v.why;\n```\n\n`window.ml.ready` is a promise resolving to `ml`; the `ml:ready` event fires once\non injection. (A sandboxed/isolated-world userscript won't see `window.ml` —\nbridge via `window.postMessage` instead.)\n\n\u003e **Worked example:** [`examples/youtube-summarizer.user.js`](./examples/youtube-summarizer.user.js)\n\u003e is a full userscript that injects an in-page **AI Summary** panel into the\n\u003e YouTube watch page — it summarizes the video via an OpenWebUI server-side\n\u003e transcript tool and takes follow-up questions, with a model picker and\n\u003e capability-warning badge. See [`examples/README.md`](./examples/README.md) for\n\u003e setup and a troubleshooting table.\n\n## Architecture\n\n| File | Role |\n| --- | --- |\n| `injected.js` | Runs in the page's main world; defines `window.ml`. Serializes `\u003cimg\u003e`/blob/http images to data URLs. |\n| `content.js` | Dumb relay: `window.postMessage` ⇄ `chrome.runtime.sendMessage`. |\n| `background.js` | Service worker. Owns config, builds per-format request bodies, extracts replies, talks to OpenWebUI/Ollama (CORS-free). |\n| `popup.html/js` | Settings UI (`chrome.storage.sync`), model picker, Save \u0026 Test, VRAM usage readout, Free VRAM. |\n\nSecurity note: config overrides (URL/key) are only accepted from the popup —\nmessages relayed from web pages (`sender.tab` set) cannot repoint the saved\nAPI key at another host, and pages can only ever change the model (validated).\n\n## Tests\n\n```sh\nnpm test\n```\n\nNo dependencies — Node's built-in `node:test` runner (Node ≥ 20). Since the\nextension files are plain scripts, not modules, `tests/helpers.js` loads them\ninto `node:vm` sandboxes with mocked `chrome`/`fetch`/`window` globals, so the\ntests exercise the exact files Chrome runs:\n\n- `tests/background.test.js` — the service-worker message contract: request\n  building per API format, response extraction, error messages, model\n  validation, vision fail-fast, VRAM unload, and the popup-only-overrides\n  security property.\n- `tests/relay.test.js` — `injected.js` + `content.js` wired together in a\n  fake page world; exercises the real postMessage round trip, history\n  semantics, fork, and image conversion.\n- `tests/live.test.js` — **opt-in** checks against a real server, validating\n  the route/capability assumptions everything else relies on:\n\n  ```sh\n  OPENWEBUI_URL=http://localhost:3000 OPENWEBUI_KEY=sk-... npm test\n  ```\n\n  Copy [`.env.example`](.env.example) to `.env` (git-\n  ignored) and fill it in — the suite auto-loads it (`loadDotEnv` in\n  `tests/helpers.js`, no `dotenv` dependency). Real environment variables take\n  precedence over `.env`, so an inline override still works.\n\n  The chat round-trip and structured-output tests additionally require\n  `OPENWEBUI_MODEL=\u003cid\u003e` so a test run never surprise-loads a huge model into\n  VRAM. With only URL + key set, the read-only route/capability checks run and\n  the model-loading ones self-skip.\n\n### VSCode test runner\n\nVSCode has no built-in `node:test` support; install the\n**[nodejs-testing](https://marketplace.visualstudio.com/items?itemName=connor4312.nodejs-testing)**\nextension (`connor4312.nodejs-testing`) and the suite appears in the Testing\nsidebar with per-test run/debug, picked up automatically from the `*.test.js`\nnaming. For quick breakpoint debugging without any extension: open a\n**JavaScript Debug Terminal** (command palette) and run `npm test` in it —\nbreakpoints in both tests and extension source bind through the `vm` loader.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparawanderer%2Fwindow-ml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparawanderer%2Fwindow-ml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparawanderer%2Fwindow-ml/lists"}