{"id":51401513,"url":"https://github.com/koretex-ai/local-browser-use","last_synced_at":"2026-07-04T07:03:12.545Z","repository":{"id":367964481,"uuid":"1282965604","full_name":"koretex-ai/local-browser-use","owner":"koretex-ai","description":"Vision-native, local-first browser-use agent: chat with a local model and have it perform web tasks in your real browser session.","archived":false,"fork":false,"pushed_at":"2026-06-28T12:31:09.000Z","size":181,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-28T14:15:04.989Z","etag":null,"topics":["ai-agent","apple-silicon","browser-agent","chrome-extension","local-llm","ollama","vision-language-model"],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koretex-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-28T11:59:22.000Z","updated_at":"2026-06-28T12:31:13.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/koretex-ai/local-browser-use","commit_stats":null,"previous_names":["koretex-ai/local-browser-use"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/koretex-ai/local-browser-use","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koretex-ai%2Flocal-browser-use","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koretex-ai%2Flocal-browser-use/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koretex-ai%2Flocal-browser-use/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koretex-ai%2Flocal-browser-use/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koretex-ai","download_url":"https://codeload.github.com/koretex-ai/local-browser-use/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koretex-ai%2Flocal-browser-use/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35112709,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agent","apple-silicon","browser-agent","chrome-extension","local-llm","ollama","vision-language-model"],"created_at":"2026-07-04T07:03:12.015Z","updated_at":"2026-07-04T07:03:12.539Z","avatar_url":"https://github.com/koretex-ai.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# local-browser-use\n\n**A vision-native AI agent that lives in your browser sidebar, runs entirely on a local model, and performs web tasks in your real, logged-in session.**\n\nChat with a local model (served via Ollama on your own Mac) and have it *see* your screen and *act* on your behalf — click, type, navigate — inside the browser you already use, with the tabs you're already logged into. No cloud, no credentials handed to a third party, no separate automated browser.\n\nThink Claude Cowork / Operator-style browsing, but **local-first, vision-native, and built from first principles.**\n\n\u003e **Status:** Design + feasibility-spike complete. Phase 0 proved a local 3B vision model can ground and click on real web UI on an 18GB Mac. Implementation (Phase 1+) is next. See [DESIGN.md](DESIGN.md) and [phase0/RESULTS.md](phase0/RESULTS.md).\n\n---\n\n## Why build this from first principles?\n\nThere are already capable open-source browser agents. We're not cloning them, for three reasons:\n\n1. **Vision-native, not vision-bolted-on.** Most existing extensions were architected around *text* LLMs reading the DOM, with vision added awkwardly later. Purpose-built, small, open **vision-grounding** models for the web now exist (they didn't a year ago) — so the agent can be designed around *seeing* the page from day one.\n2. **Local-first economics.** Capable models now fit on consumer Apple Silicon. A vision grounder at 3B runs comfortably on an 18GB Mac. The whole loop can be private and free at the point of use.\n3. **A data flywheel.** Every interaction — screenshot, chosen action, outcome, and especially *human corrections* — is exactly the data used to train the next generation of grounding models. We design the data schema for that from the first commit, so usage compounds into a better model.\n\n## How it works\n\nThe agent core is a set of **decoupled modules behind clean interfaces**, so any model or strategy can be swapped without rewriting the loop:\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  Your real browser (logged-in session)                        │\n│  ┌────────────┐         ┌──────────────────────────────────┐  │\n│  │ Active tab │◄───────►│  Side Panel  — chat · steps ·     │  │\n│  └─────┬──────┘ act+obs │  approve / correct controls       │  │\n│        │                └───────────────┬──────────────────┘  │\n│  ┌─────▼─────────────────────────────────▼─────────────────┐  │\n│  │  AGENT CORE                                              │  │\n│  │  Perception → Planner → Grounder → Executor → Validator  │  │\n│  │                  ▲                      │                │  │\n│  │                  └─────── loop ─────────┘                │  │\n│  │  Trajectory Logger (taps every stage → training data)    │  │\n│  └────────────────┬───────────────────────┬────────────────┘  │\n└───────────────────┼───────────────────────┼───────────────────┘\n            ┌────────▼────────┐     ┌────────▼─────────┐\n            │ Local model      │     │ Local data store  │\n            │ runtime (Ollama) │     │ (trajectories,    │\n            │ vision grounder  │     │  corrections)     │\n            │ + text planner   │     └───────────────────┘\n            └──────────────────┘\n```\n\n- **Perception** builds an observation of the page — DOM / accessibility tree *and* a downscaled screenshot.\n- **Planner** (text model) decides the next high-level action from the task + history.\n- **Grounder** (vision model) maps \"click the Search button\" → an actual element / pixel.\n- **Executor** performs the action in the page.\n- **Validator** checks it worked, and drives retry / replan / ask-the-human.\n- **Trajectory Logger** records every step as training-ready data.\n\n## Key design choices\n\n| Decision | Choice | Why |\n|---|---|---|\n| **Perception** | **Hybrid** — DOM / set-of-marks by default, pixel-vision fallback | DOM is fast and reliable where it works; vision covers what it can't (canvas, custom widgets, iframes). Capturing **both** per action makes the data a *superset* — you can train a pure-vision model later from hybrid-collected trajectories. |\n| **Default grounder** | **Holo1.5-3B** (web-specialized VLM, Apache-2.0) | Purpose-built for web UI localization, fits 18GB with headroom, and is fine-tunable (Qwen2.5-VL backbone — the same base our future training would use). |\n| **Planner** | Small local **text** model, swappable | Pure-text reasoning; can run locally or be dispatched to our local inference network. |\n| **Shell** | Fork an existing MV3 extension for the **side panel + plumbing only** | Don't reinvent Chrome-extension boilerplate; build the agent core fresh where the real value is. |\n| **Reference architecture** | Surfer-H + Holo1 (policy / localizer / validator split) | A proven, vision-native blueprint that maps cleanly onto swappable models. |\n| **Cross-cutting** | Typed action space **==** training-label schema; log trajectories + human corrections from commit #1 | The data flywheel only works if the schema is right from the start. |\n\n### Models considered\n\n- **Holo1.5** (H Company, Apache-2.0) — web-specialized grounding VLM; **our default** (3B).\n- **UI-TARS-1.5-7B** (ByteDance, Apache-2.0) — strong end-to-end GUI agent; evaluated as a single-model benchmark (heavier on 18GB, needs native prompting).\n- **Qwen2.5-VL** — the general-purpose base these are fine-tuned from; useful as a control.\n- **GUI-Actor**, **UGround**, **OmniParser** — alternative grounding approaches kept in the design space.\n\n## Relationship to koretex\n\n[koretex](https://github.com/koretex-ai) is our network for running models locally on Apple Silicon. It serves text/code models — and now **vision-language models too**, over the same OpenAI-compatible `/v1/chat/completions` endpoint (a VLM streams text like any chat model; it's simply kept out of the throughput-based hardware ranking, since image prefill distorts tokens/sec).\n\nThat makes **both halves of this agent first-class koretex workloads**: the **text planner** and the **vision grounder** can each run locally on your own machine *or* ride the koretex network. In practice the grounder benefits from running close to the browser loop for latency, but nothing in the architecture forces it to stay local. Agent and node share the same pinned Ollama engine, so they converge cleanly.\n\n## Proof it works (Phase 0)\n\nWe rendered realistic web pages at a fixed resolution and scored whether the model's predicted click lands inside the target element (ScreenSpot-style). On a **Mac M3 Pro / 18GB**:\n\n- **Holo1.5-3B hit 3/4** grounding cases at **~55–59 tok/s**, including correctly disambiguating the middle of three identical buttons.\n- **Engine ≥ Ollama 0.30.x is required** — older builds can't run Qwen2.5-VL-class models (no M-RoPE support).\n- Latency is **image-prefill bound** (~5s/call) → screenshot downscaling is a first-class lever.\n\nFull method + numbers: [phase0/RESULTS.md](phase0/RESULTS.md). The harness is runnable — see [phase0/](phase0/).\n\n## Roadmap\n\n- **Phase 0 — Feasibility spike.** ✅ Done. Local 3B vision grounding is viable on 18GB.\n- **Phase 1 — Shell + chat loop.** Side panel talking to the local model.\n- **Phase 2 — Perception + executor.** DOM/SoM extraction, downscaled screenshots, action execution.\n- **Phase 3 — Agent loop.** Planner → Grounder → Executor → Validator with bounded retries.\n- **Phase 4 — Modular models + hybrid grounding.** Split grounder/planner; add vision fallback; benchmark and lock defaults.\n- **Phase 5 — Trajectory logging + guardrails.** Training-ready records, human-in-the-loop confirmation for sensitive actions.\n- **Phase 6 (optional) — CDP/native escape hatch** for actions the extension sandbox blocks.\n\n## Repository layout\n\n```\nREADME.md            — this file\nDESIGN.md            — full architecture \u0026 implementation plan\nphase0/              — feasibility-spike harness\n  pages/             — realistic test pages (HTML)\n  shots/             — rendered screenshots (1280×800)\n  ground_truth.json  — target boxes + instructions\n  render.sh          — render pages → PNG (headless Chrome)\n  run.py             — grounding benchmark (stdlib only)\n  RESULTS.md         — Phase-0 findings\n```\n\n## Safety\n\nThe agent is designed to keep a human in the loop for anything irreversible — purchases, sends, deletes, auth changes — and never executes financial transactions autonomously. Bounded autonomy (max steps/retries), a visible stop control, and per-step visibility are core requirements, not add-ons.\n\n## License\n\nTBD.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoretex-ai%2Flocal-browser-use","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoretex-ai%2Flocal-browser-use","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoretex-ai%2Flocal-browser-use/lists"}