Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/parawanderer/window-ml

Console scripting API for OpenWebUI/Ollama — window.ml in devtools on any page. Multimodal via <img> nodes, chat history, VRAM management.
https://github.com/parawanderer/window-ml

chrome-extension devtools llm ollama open-webui scripting

Last synced: 12 days ago
JSON representation

Console scripting API for OpenWebUI/Ollama — window.ml in devtools on any page. Multimodal via <img> nodes, chat history, VRAM management.

Awesome Lists containing this project

README 

          
# window.ml



[![tests](https://github.com/parawanderer/window-ml/actions/workflows/tests.yml/badge.svg)](https://github.com/parawanderer/window-ml/actions/workflows/tests.yml)



![Screenshot showing extension being used with the gemma4 model in the DevTools console](./docs/fullscreen-and-console-2026-07-06_21-38.png)



Personal Chrome extension that exposes a scripting API (`window.ml`) on web

pages, bridging them to local LLMs served by [OpenWebUI](https://openwebui.com)

/ [Ollama](https://ollama.com). Built as devtools-console glue: the deliverable

is a `window.ml` object you call from any page's console (or from page scripts),

not a chat UI.



> Honest provenance: this is personal, largely AI-generated glue code, shared

> so the next person wanting a console-first LLM bridge doesn't have to build

> it from scratch. It works (and has tests), but there's no roadmap and no

> support — fork freely. MIT licensed.



```js

await ml.chat("Summarize this page's title: " + document.title);



// Multimodal — pass literal  DOM nodes:

await ml.chat("What's in this image?", { images: [document.images[0]] });



// Multi-turn:

const h = ml.createChat({ system: "Be terse." });

await h.chat("What is a monad?");

await h.chat("Now explain it like I'm five");

```



…or let an **agent drive the page** from one line of English —

`ml.agent("Search for cat videos.")` on YouTube ([example](examples/youtube_search.js)):



## Setup



> **Starting from nothing?** [docs/FULL-SETUP.md](docs/FULL-SETUP.md) stands up the

> whole backend from scratch with Docker (Ollama + OpenWebUI + web search) and a

> couple of models — the "initial setup for max utility" so `window.ml` can do

> real tasks immediately.

>

> Step-by-step walkthrough (install, minimum config, troubleshooting): [docs/SETUP.md](docs/SETUP.md)



1. `chrome://extensions` → Developer mode → **Load unpacked** → this directory.

2. Recommended: set the extension's **Site access** to **On click** or whitelist specific trusted sites only, so

   `window.ml` only exists on pages where you've clicked the extension icon.

3. Open the popup (extension icon):

   - **Chat completions URL** — e.g. `http://localhost:3000/api/chat/completions`

   - **API key** — OpenWebUI → Settings → Account → API keys

   - **Model** — hit **Load** to pick from the server's list

   - **API format** — must match the URL (see table below)

   - **Save & Test** sends a real prompt and shows the extracted reply.



Screenshot showing Settings Config Menu For Extension



### Endpoint / format cheatsheet



| URL | API format | Notes |

| --- | --- | --- |

| `/api/chat/completions` | OpenAI | OpenWebUI's external API. Broken on OpenWebUI 0.9.5 ([#24550](https://github.com/open-webui/open-webui/issues/24550)), fixed by 0.10.x. |

| `/ollama/api/chat` | Ollama native | OpenWebUI's raw Ollama passthrough (same bearer key). Bypasses OpenWebUI middleware; needs raw Ollama model names. |

| `http://ollama-host:11434/api/chat` | Ollama native | Direct Ollama, no OpenWebUI. |



There is **no** root-level `/v1/chat/completions` on OpenWebUI (tested 0.9.5

and 0.10.2) — unknown routes return the frontend HTML page.



**Cloud/commercial models** (Claude, GPT, OpenRouter) work with no extension

changes — add them as a Connection in OpenWebUI and they appear in the model

list. See [docs/CLOUD-MODELS.md](docs/CLOUD-MODELS.md).



## `window.ml` API



| Call | Purpose |

| --- | --- |

| `ml.chat(prompt, options?)` | One-shot chat. Returns the reply text. |

| `ml.chatShort(prompt, options?)` | Same, with a brevity suffix. |

| `ml.createChat(options?)` | Multi-turn history object (below). |

| `ml.models()` | Available model ids on the server. |

| `ml.getModel()` / `ml.setModel(id)` | Read / persistently switch the default model. `setModel` validates against the server list and syncs the popup. |

| `ml.config()` | Read all exposed configuration parameters for the extension (main model default, OCR model, apiFormat) |

| `ml.ps()` | Models loaded in VRAM: `[{ model, vramGB, expiresAt }]`. |

| `ml.unload(model?)` | Evict a model from VRAM (`keep_alive: 0`); no argument = evict all. |

| `ml.read(image, { model?, prompt? })` | OCR — transcribe baked-in text from an `` or URL to a plain string, using the configured OCR (vision) model. See [OCR](#ocr). |

| `ml.step(messages, { tools?, model?, think? })` | One model turn with client-side tools; returns the raw assistant message and hands the loop to you. See [Tools](#tools-agents). |

| `ml.agent(task, options?)` | Plain-English page agent: runs the whole loop over built-in DOM recon tools (and auto-wired vision) until it acts or answers. See [Agent](#tools-agents). |

| `ml.chat(prompt, { toolIds })` | Server-side tools: OpenWebUI runs the tools and returns the finished answer. See [Tools](#tools-agents). |

| `ml.logChat` / `ml.logChatShort` | `console.log` variants. |



Options (all optional, both for `chat` and `createChat`):



- `system` — system prompt.

- `model` — model override; doesn't touch the saved default.

- `think` — `true`/`false` maps to Ollama's native thinking toggle; `null`

  omits it (server default). Models that support thinking are asked not to

  by default. Sending images to a model without vision capability fails fast

  with a clear error (checked via `/api/show`).

- `cleanup` — strip `…` from replies (default on).

- `images` — (per-call) list of `` elements and/or URL strings.

- `maxTokens` — hard cap on generated tokens (OpenAI `max_tokens` / Ollama

  `num_predict`); `null` omits it. Bounds a runaway generation so it can't peg

  the model. `ml.agent`'s auto-wired vision tool caps itself this way by default.

- `onToken` — `(delta, full) => {}`. Stream the reply token-by-token (for a live

  "typing" effect) while the call still resolves to the full string and history

  updates as usual. Text-only, so it's ignored when `schema` is set. Works with

  `toolIds` too (a server-side tool runs first, then its answer streams).



  ```js

  const out = document.querySelector("#out");

  await ml.chat("Explain the Jevons paradox", {

    onToken: (tok) => { out.textContent += tok; }   // paints as it generates

  });

  ```

- `schema` — a JSON Schema object. Constrains the reply to matching JSON and

  returns it **parsed** (an object), not a string. Turns `window.ml` into a

  classifier/extractor — the primitive for DOM-scripting against a policy:



  ```js

  const verdict = await ml.chat(videoTitle, {

    system: "You enforce this feed policy: no rage-bait, no crypto shilling.",

    schema: {

      type: "object",

      properties: {

        hide: { type: "boolean" },

        rewritten_title: { type: "string" }

      },

      required: ["hide", "rewritten_title"]

    }

  });

  if (verdict.hide) tile.style.display = "none";

  ```



  Wire mapping: OpenAI format → `response_format` json_schema; Ollama native →

  `format`. Support depends on the backend — most reliable against Ollama

  (directly or via OpenWebUI). In history objects the raw JSON text is stored

  as context so turns still chain.



### History objects



```js

const h = ml.createChat({ system, model, think, cleanup });

await h.chat("prompt", { images, model, think, cleanup });  // per-turn overrides



h.messages          // plain [{ role, content, images? }] array — edit freely:

h.messages.at(-1)   //   last message

h.messages.pop()    //   drop a turn to retry

h.fork()            // independent deep copy of the conversation

```



Design invariants: assistant replies are stored post-cleanup (thinking blocks

are never resent as context) and a failed request leaves `messages` untouched.



**Sources.** When a server-side tool or RAG runs (e.g. `toolIds`), OpenWebUI

attaches provenance, and `window.ml` surfaces it on the stored assistant message

as `.sources` — the raw OpenWebUI array (`[{ source, metadata, document }]`), so

you can render citations. Read it off the turn: `h.messages.at(-1).sources`.

Absent on plain chats and the Ollama-native format.



### OCR



`ml.read()` transcribes text that's baked into image pixels — the case where a

site renders content as an image so it can't be selected or scraped. It returns

a **plain string**, so it composes with `chat`:



```js

await ml.chat("Summarize this: " + await ml.read(document.images[0]));

const imgs = [...document.querySelectorAll(".listing img")];         // bulk

const texts = await Promise.all(imgs.map(img => ml.read(img)));

```



**Setup:** OCR needs a vision model. Pull one and set it as the *OCR model* in

the popup (kept separate from your chat model, so a text-only reasoning model

stays the default and never sees image tokens):



```sh

ollama pull qwen2.5vl        # or: docker exec ollama ollama pull qwen2.5vl

```



Typical bulk flow, using the VRAM controls to avoid holding both models at once:



```js

const texts = await Promise.all(imgs.map(img => ml.read(img)));  // vision model stays warm

await ml.unload();                    // purge it

await ml.setModel("qwen3:235b");      // load the reasoning model

await ml.chat("Analyze:\n" + texts.join("\n---\n"));

```



There's no separate OCR server — OCR is just a vision-model call through the

same OpenWebUI pipe. For specialized accuracy, point the OCR model at any

vision/OCR model you've added to OpenWebUI (a GOT-OCR2 or TrOCR GGUF, etc.).



### Tools & agents



Two primitives, for the two places tools can run. Both need a model that

supports function calling (e.g. qwen3); a model that doesn't just replies with

text and no tool calls.



#### Server-Side tools & agents



OpenWebUI runs its own registered tools (web search,

MCP servers, community/Python tools) and returns the finished answer. One call,

no loop on your side. **OpenWebUI only** (the Ollama-native format errors):



```js

await ml.chat("What's the weather in Amsterdam right now?", { toolIds: ["web_search"] });

```



#### Client-Side tools & agents



##### Option 1: Build your own agent



`ml.step()` is one model turn that hands the loop back

to *you*: it returns the raw assistant message, you execute any tool calls

(in the page), append the results, and call it again.



```js

const tools = [{ type: "function", function: {

  name: "readDom",

  description: "Visible text of elements matching a CSS selector.",

  parameters: { type: "object", properties: { selector: { type: "string" } }, required: ["selector"] }

}}];



const runners = {

  readDom: ({ selector }) => [...document.querySelectorAll(selector)].map(e => e.innerText).join("\n")

};



let messages = [{ role: "user", content: "Find the menu link text on this page." }];

for (let i = 0; i < 8; i++) {                      // your loop, your step limit

  const msg = await ml.step(messages, { tools });

  if (!msg.tool_calls?.length) { console.log(msg.content); break; }

  messages.push({ role: "assistant", content: msg.content, tool_calls: msg.tool_calls });

  for (const call of msg.tool_calls) {

    if (!(call.name in runners)) continue;         // your whitelist

    const result = await runners[call.name](call.arguments);

    messages.push({ role: "tool", tool_call_id: call.id, content: String(result) });

  }

}

```



`ml.step` works on both OpenWebUI and plain Ollama — the wire differences

(where `tool_calls` live, string vs object arguments, tool-result shape) are

normalized so `{ id, name, arguments }` is the same either way.



##### Option 2: Ready-made extensible agent loop



![An example showing the example.com website running an agent to change the HTML of the web page](./docs/agent-example-2026-07-11_16-10.png)



**An agent, with nothing to configure** — `ml.agent(task)` is the loop above,

already assembled. You give it plain English; it discovers the DOM with built-in

recon tools, writes **one** rule to act on every match at once, and — when your

model can see — screenshots the page to check its own work. No tool definitions,

no loop, no step cap to wire up:



```js

// Edit the page: the model finds the repeating container itself and hides them.

await ml.agent("Hide every sponsored result on this page.");



// Or get a live DOM node back — hover it in devtools:

const { elements } = await ml.agent("Find the top banner ad element.");

elements[0];        // a real 
, not a selector string

```



That's the whole setup: point the popup at your server, pick a big tool-capable

model, and call `ml.agent`. What's built in:



- **DOM recon tools** — `findByText`, `describeElement`, `ancestors`,

  `countMatches`, `sampleText`, plus an `exec` escape hatch. Small, structured

  output (never raw HTML), so context stays cheap.

- **Eyes, auto-wired — natively when possible.** If your model reports vision

  capability (or your configured [OCR](#ocr) model does), a `look` tool is added

  automatically so the agent can orient and *visually verify* its work. **When the

  agent's own model is vision-capable (e.g. `qwen3.5:122b`), `look` feeds the

  screenshot straight into that model's own conversation** — it reasons over the

  real pixels, not a second model's text summary. That's a big reliability jump:

  the delegated path squashes the page to a paragraph and plans over the paragraph,

  blind to whatever the summary dropped; native vision removes that lossy step. If

  only your OCR model can see, it falls back to that delegated `look`. Text-only

  model? It runs without eyes and says so if a task truly needs them. Turn it off

  with `vision: false`, or force a model with `vision: "qwen2.5vl"`.

- **A safety gate** — `exec` (arbitrary page JS) is approval-gated; the default

  is a blocking `confirm()`. Pass your own `approve({ tool, arguments })`.

- **A step cap** (`maxSteps`, default 10) and a full `transcript`.



Returns `{ summary, steps, transcript, elements }` (`elements` holds any DOM

nodes the agent designated as its answer). Nudge it without rewriting the prompt

via `hints`, and watch every thought and tool call in the console with

`logDebug` (or pass your own `onStep`):



```js

const res = await ml.agent("Hide items that can't be delivered today.", {

  hints: "On amazon.nl the delivery line reads 'Wordt vandaag bezorgd'.",

  logDebug: true          // one console line per step — thoughts, tool calls, live nodes

  maxSteps: 45

});

console.log(res.summary);

```



`ml.agent` is a *composition* of `ml.step`, not a black box — the loop, tool

whitelist, step cap and approval gate are all plain code on top of the same

primitive. Drop to `ml.step` when you want to own the loop yourself.



### Using from a userscript



`window.ml` lives on the page's **main-world** `window`, so a userscript that

runs in page context (Tampermonkey, *User JavaScript and CSS*, …) can call it

directly — no messaging. Since your script and the extension race to load, wait

for the readiness signal rather than assuming `window.ml` exists yet:



```js

const ml = await (window.ml?.ready

    ?? new Promise(r => addEventListener("ml:ready", () => r(window.ml), { once: true })));



// e.g. flag a lunch menu you'd hate, then nudge the tab title:

const menu = await ml.read(document.querySelector(".menu img"));

const v = await ml.chat("I hate sandwiches. Warn me if lunch here is sandwich-only.\n\n" + menu,

    { schema: { type: "object", properties: { warn: { type: "boolean" }, why: { type: "string" } }, required: ["warn", "why"] } });

if (v.warn) document.title = "🥪 " + v.why;

```



`window.ml.ready` is a promise resolving to `ml`; the `ml:ready` event fires once

on injection. (A sandboxed/isolated-world userscript won't see `window.ml` —

bridge via `window.postMessage` instead.)



> **Worked example:** [`examples/youtube-summarizer.user.js`](./examples/youtube-summarizer.user.js)

> is a full userscript that injects an in-page **AI Summary** panel into the

> YouTube watch page — it summarizes the video via an OpenWebUI server-side

> transcript tool and takes follow-up questions, with a model picker and

> capability-warning badge. See [`examples/README.md`](./examples/README.md) for

> setup and a troubleshooting table.



## Architecture



| File | Role |

| --- | --- |

| `injected.js` | Runs in the page's main world; defines `window.ml`. Serializes ``/blob/http images to data URLs. |

| `content.js` | Dumb relay: `window.postMessage` ⇄ `chrome.runtime.sendMessage`. |

| `background.js` | Service worker. Owns config, builds per-format request bodies, extracts replies, talks to OpenWebUI/Ollama (CORS-free). |

| `popup.html/js` | Settings UI (`chrome.storage.sync`), model picker, Save & Test, VRAM usage readout, Free VRAM. |



Security note: config overrides (URL/key) are only accepted from the popup —

messages relayed from web pages (`sender.tab` set) cannot repoint the saved

API key at another host, and pages can only ever change the model (validated).



## Tests



```sh

npm test

```



No dependencies — Node's built-in `node:test` runner (Node ≥ 20). Since the

extension files are plain scripts, not modules, `tests/helpers.js` loads them

into `node:vm` sandboxes with mocked `chrome`/`fetch`/`window` globals, so the

tests exercise the exact files Chrome runs:



- `tests/background.test.js` — the service-worker message contract: request

  building per API format, response extraction, error messages, model

  validation, vision fail-fast, VRAM unload, and the popup-only-overrides

  security property.

- `tests/relay.test.js` — `injected.js` + `content.js` wired together in a

  fake page world; exercises the real postMessage round trip, history

  semantics, fork, and image conversion.

- `tests/live.test.js` — **opt-in** checks against a real server, validating

  the route/capability assumptions everything else relies on:



  ```sh

  OPENWEBUI_URL=http://localhost:3000 OPENWEBUI_KEY=sk-... npm test

  ```



  Copy [`.env.example`](.env.example) to `.env` (git-

  ignored) and fill it in — the suite auto-loads it (`loadDotEnv` in

  `tests/helpers.js`, no `dotenv` dependency). Real environment variables take

  precedence over `.env`, so an inline override still works.



  The chat round-trip and structured-output tests additionally require

  `OPENWEBUI_MODEL=` so a test run never surprise-loads a huge model into

  VRAM. With only URL + key set, the read-only route/capability checks run and

  the model-loading ones self-skip.



### VSCode test runner



VSCode has no built-in `node:test` support; install the

**[nodejs-testing](https://marketplace.visualstudio.com/items?itemName=connor4312.nodejs-testing)**

extension (`connor4312.nodejs-testing`) and the suite appears in the Testing

sidebar with per-test run/debug, picked up automatically from the `*.test.js`

naming. For quick breakpoint debugging without any extension: open a

**JavaScript Debug Terminal** (command palette) and run `npm test` in it —

breakpoints in both tests and extension source bind through the `vm` loader.