{"id":51429633,"url":"https://github.com/segentic-lab/glovebox-mcp","last_synced_at":"2026-07-06T04:00:30.071Z","repository":{"id":368958459,"uuid":"1285665224","full_name":"segentic-lab/glovebox-mcp","owner":"segentic-lab","description":"Sandboxed computer-use MCP — drive a real browser + desktop apps in a nested X11 window.","archived":false,"fork":false,"pushed_at":"2026-07-01T13:25:54.000Z","size":251,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-03T01:27:53.462Z","etag":null,"topics":["ai-agents","automation","browser-automation","computer-use","mcp","model-context-protocol","sandbox","xephyr"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/segentic-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-07-01T03:14:43.000Z","updated_at":"2026-07-02T22:01:24.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/segentic-lab/glovebox-mcp","commit_stats":null,"previous_names":["segentic-lab/glovebox-mcp"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/segentic-lab/glovebox-mcp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segentic-lab%2Fglovebox-mcp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segentic-lab%2Fglovebox-mcp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segentic-lab%2Fglovebox-mcp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segentic-lab%2Fglovebox-mcp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/segentic-lab","download_url":"https://codeload.github.com/segentic-lab/glovebox-mcp/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segentic-lab%2Fglovebox-mcp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35141967,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","automation","browser-automation","computer-use","mcp","model-context-protocol","sandbox","xephyr"],"created_at":"2026-07-05T03:02:28.163Z","updated_at":"2026-07-05T03:02:30.599Z","avatar_url":"https://github.com/segentic-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# glovebox-mcp\n\nA **sandboxed computer-use MCP server** — let an AI agent drive a real browser and desktop apps\n(mouse, keyboard, screenshots, vision grounding), confined to a **nested X11 window** so it can\nnever touch your real screen, files, or other apps.\n\nLike a lab **glovebox**: the agent reaches in and manipulates real applications, sealed off from\neverything else. Bring the sandbox up, log into whatever sites or apps you want to automate *inside\nthat window*, and the agent operates only there — you can watch it live and close it instantly.\n\n\u003e Speaks the [Model Context Protocol](https://modelcontextprotocol.io), so it works with MCP clients\n\u003e like Claude Code. Your host can run Wayland; the sandbox gives the agent a real X server to drive.\n\n![glovebox-mcp — an agent filling a sign-up form inside the sandbox](assets/demo.gif)\n\n\u003csub\u003eAn agent driving a real browser in the sandbox — gliding the cursor, inserting a unicode name (`Nadja Kovačič`), typing, and submitting. All confined to a nested X11 window.\u003c/sub\u003e\n\n## Why a nested X11 sandbox?\n\n- Most desktop automation (xdotool, PyAutoGUI) is **X11-only**, but many modern desktops run **Wayland**.\n- **Xephyr** provides a real X server *inside a single window* (`DISPLAY :1`). Everything the agent\n  does — clicks, typing, screenshots — is **confined to that window**, not your real desktop.\n- You stay in control: watch it live, `pkill Xephyr` to close everything.\n\n## Requirements\n\n- **Linux** — the sandbox nests a real X server (Xephyr), so it works even on **Wayland** hosts\n  (via Xwayland). Not macOS/Windows. Developed on Ubuntu; any modern Linux with the packages below.\n- **Python 3.10+** and [`uv`](https://docs.astral.sh/uv/) (used for the virtualenv).\n- **System packages** — `xserver-xephyr` (Xephyr), `openbox`, `scrot`, `x11-utils`, `xdotool`,\n  `wmctrl`, `xclip` (+ `tesseract-ocr` for `basic`). On Debian/Ubuntu the installer **auto-installs**\n  them via `apt` (sudo); on Fedora/Arch it prints the matching `dnf`/`pacman` command. The MCP server\n  itself is distro-agnostic — any Linux with these tools works.\n- A **browser** in the sandbox (Chromium or Chrome).\n- **NVIDIA GPU (≥6 GB VRAM)** — only for the `local` vision mode.\n\n## Install\n\nPick a vision backend and run its one-liner (clone → install). Each one installs the system packages\n(auto via `apt` on Debian/Ubuntu) and the Python deps for that mode, and writes a ready-to-paste\n`mcp-config.json` with your paths.\n\n**`none`** — no local models; your agent reads screenshots itself (lightest, instant):\n\n```bash\ngit clone https://github.com/segentic-lab/glovebox-mcp \u0026\u0026 cd glovebox-mcp \u0026\u0026 ./install.sh none\n```\n\n**`basic`** — Tesseract OCR grounding (`parse_screen` → text + coordinates, CPU-only):\n\n```bash\ngit clone https://github.com/segentic-lab/glovebox-mcp \u0026\u0026 cd glovebox-mcp \u0026\u0026 ./install.sh basic\n```\n\n**`local`** — OmniParser on an NVIDIA GPU (`parse_screen` → text **+ icons**, pixel-precise; ~4 GB weights, ≥6 GB VRAM):\n\n```bash\ngit clone https://github.com/segentic-lab/glovebox-mcp \u0026\u0026 cd glovebox-mcp \u0026\u0026 ./install.sh local\n```\n\nYour choice is written to `.vision-mode` (override per run with the `GLOVEBOX_VISION` env var).\n\n### Works with any MCP client / harness\n\nClaude Code, Cursor, Codex, or your own agent — it's a standard MCP server, not tied to any one host.\nTwo compatibility notes: **`basic`/`local`** return element **coordinates as text**, so they work even\nwith text-only agents; **`none`** relies on the client passing the tool's **screenshots** to a\nmultimodal model (fine for Claude Code, Cursor, and other image-capable MCP clients).\n\n## Quickstart\n\n1. **Start the sandbox** (leave it running):\n   ```bash\n   ./start-display.sh              # 1440×900 Xephyr window with a browser\n   ./start-display.sh 1920x1080    # …or pass a screen size (or set $RES)\n   ```\n   Log into any sites or apps you want to automate **in that window**.\n2. **Register the server** with your MCP client. `install.sh` already wrote **`mcp-config.json`** with\n   your real install path — copy its `glovebox` block into your client's MCP config:\n   ```jsonc\n   { \"mcpServers\": { \"glovebox\": {\n       \"command\": \"/abs/path/to/glovebox-mcp/.venv/bin/python\",   // filled in by install.sh\n       \"args\":    [\"/abs/path/to/glovebox-mcp/server.py\"],\n       \"env\":     { \"DISPLAY\": \":1\" }\n   } } }\n   ```\n   Restart the client so it loads the server.\n3. Ask the agent to screenshot / click / type — it operates **only** on the `:1` window.\n\n\u003e **Driving it with an AI agent?** Paste [`AGENTS.md`](AGENTS.md) into the agent's system prompt — it\n\u003e teaches the observe → act → verify loop, grounding, the upload/unicode gotchas, and when to stop.\n\n## Tools\n\n| Tool | What |\n|---|---|\n| `parse_screen()` | Vision grounding → JSON of every element (id, type, label, interactive, pixel-center) + a numbered Set-of-Mark image at `/tmp/glovebox_annotated.png`. (`local` mode: OmniParser on GPU, ~2 s.) |\n| `click_element(id)` | Click an element from the last `parse_screen` (no coordinate guessing). |\n| `screenshot()` | Screenshot of an instance. |\n| `click(x,y)` · `move_mouse` · `scroll` · `drag` · `double_click` | Pointer ops. |\n| `type_text(text)` | **Unicode-safe** typing (ASCII via xdotool; anything with č/š/ž… is inserted via the clipboard, because xdotool's synthetic unicode is silently dropped by some GTK apps). |\n| `press_keys(\"ctrl+a\"/\"Return\"/…)` | Keys/combos (xdotool syntax). |\n| `upload_file(filepath, selector?)` | Attach a local file to a page's `\u003cinput type=file\u003e` via the **Chrome DevTools Protocol**. The nested X11 file picker is invisible to automation **and hangs the renderer**, so use this for all uploads — never click an upload button expecting a dialog. Works on Chromium started by `launch_app`/`start-display.sh` (they open a per-instance `--remote-debugging-port`, `9222+N`). **Browser file inputs only** — for native apps see `open_file`. |\n| `open_file(filepath, app?)` | Open a local file in a **native** app on the instance's display (e.g. `app=\"gimp\"`) or via `xdg-open`. GTK apps get the same X11/D-Bus handling as `launch_app`. |\n| `list_files()` | The instance's staging folder `files/\u003cN\u003e/` (under the install dir) + its contents. |\n| `launch_app(command, name?, size?)` · `list_instances()` · `close_instance(n)` | Multi-instance control (see below). |\n| `wait_ms(ms)` · `get_screen_size()` | Timing / sandbox size. |\n\nEvery control tool takes **`instance=N`** and optional **`observe`** / **`settle_ms`** (see below).\nIn `local` mode OmniParser is **lazy-loaded** on first `parse_screen` (~6 s once, then ~2 s/parse).\n\n### Vision backend (selectable)\n\n`GLOVEBOX_VISION` env var, or the `.vision-mode` file, or default `local`:\n\n| Mode | `parse_screen` | Needs | When |\n|---|---|---|---|\n| `none` | disabled (returns a note) — use `screenshot()` + reason | nothing (mcp, mss, xdotool) | lightest; let the agent's own vision do grounding |\n| `basic` | Tesseract OCR → text elements + coords | `tesseract-ocr` + `pytesseract` | no GPU; text-only grounding |\n| `local` | OmniParser → text **+ icons** + coords | torch + CUDA + OmniParser weights | best grounding |\n\nSwitch anytime with `./install.sh \u003cmode\u003e` (installs only what that mode needs).\n\n### Multi-instance (a fleet of app windows)\n\nEvery control tool takes **`instance=N`** (default 1 = the `start-display.sh` sandbox). Spin up more —\neach its own Xephyr display/window on the host desktop:\n\n- `launch_app(command, name?, size?)` → starts the next free `:N` running **any** GUI app\n  (`chromium`, `gimp`, `inkscape`, `xterm`, …). Chromium auto-gets X11 flags, a per-instance profile,\n  a remote-debugging port, and D-Bus isolation. Returns the instance id.\n- `list_instances()` · `close_instance(n)`.\n\nBecause each display has its **own cursor**, multiple agents can drive different instances **in parallel** —\none window each. The only shared resource is the GPU for `local`-mode `parse_screen` (it just queues).\nThe host display for new windows is `GLOVEBOX_HOST_DISPLAY` (default `:0`); `XAUTHORITY` is auto-discovered.\n\n### One-call action + observe\n\n`click` · `click_element` · `type_text` · `press_keys` · `scroll` · `drag` · `double_click` take\n**`observe`** (`none` default · `screenshot` · `parse`) and **`settle_ms`**. With `observe=\"screenshot\"`\nthe action returns its result **and** the resulting screen in a single call (with `settle_ms` to let the\npage update first) — no separate screenshot round-trip. Default `none` keeps routine steps cheap; opt\ninto `screenshot`/`parse` on the steps that change the page (navigations, submits).\n\n### Files \u0026 uploads\n\nEach instance gets a staging folder **`files/\u003cN\u003e/`** inside the install dir — a stable place to drop\nfiles for that instance (readable by native apps and, since it's under `$HOME`, by snap Chromium too).\n`list_files(instance)` shows the folder and its contents.\n\n- **Browser** `\u003cinput type=file\u003e` → **`upload_file(path, instance)`** (via CDP). The nested file\n  picker is invisible to automation and hangs snap Chromium, so never click an upload button expecting\n  a dialog.\n- **Native apps** (GIMP, Inkscape, editors) → **`open_file(path, instance, app=\"gimp\")`**, or just\n  drive the app's own **Open dialog** — unlike the browser's, it's a real visible window you can type a\n  path into (`Ctrl+L` in a GTK file chooser).\n- **Saving / downloads** → apps run as your user, so they can save anywhere you can write. `launch_app`\n  Chromium instances are pre-configured to **download and \"save as\" into `files/\u003cN\u003e/`**; point native\n  apps' Save dialogs there too, then `list_files(instance)` to see the results.\n\n## Maintenance (`local` mode)\n\n`install.sh` clones [OmniParser](https://github.com/microsoft/OmniParser), downloads the v2 weights, and\napplies two patches automatically:\n\n- **PaddleOCR made optional** (this uses easyocr): `OmniParser/util/utils.py`'s\n  `from paddleocr import PaddleOCR` is wrapped in try/except and the module-level `paddle_ocr = PaddleOCR(...)`\n  is guarded with `… if PaddleOCR is not None else None`.\n- **`transformers` is pinned to 4.49.0** — newer releases break Florence-2's remote config.\n\nIf you upgrade OmniParser manually, re-apply the PaddleOCR patch. Weights live in `OmniParser/weights/`.\n\n## Stop\n\n```bash\npkill Xephyr      # closes the sandbox (browser + WM + display)\n```\n\n## Safety\n\n- The agent's input and vision are **scoped to the sandbox display** — it does not see or control your real desktop.\n- The server process runs as your user (shell/file access, like any MCP server); only its **GUI control** is\n  sandboxed to the Xephyr window. For OS-level isolation from your files, run it inside a VM or container.\n- You can watch everything live and close it instantly with `pkill Xephyr`.\n- Automate responsibly — only sites and services you are authorized to use.\n\n## Files\n\n- `server.py` — the MCP server (all tools).\n- `install.sh` — mode-aware installer (`none` / `basic` / `local`).\n- `start-display.sh` — launches the Xephyr sandbox (display + window manager + browser).\n- `AGENTS.md` — drop-in tool-usage instructions for the AI agent (paste into its system prompt).\n- `mcp-config.json` — a ready-to-paste MCP client config snippet.\n\n## Credits\n\n`local` vision mode uses Microsoft's [OmniParser](https://github.com/microsoft/OmniParser)\n(cloned and weights downloaded by `install.sh`, under its own license). Screen capture uses\n[mss](https://github.com/BoboTiG/python-mss); input is driven with\n[xdotool](https://github.com/jordansissel/xdotool). Not affiliated with Microsoft.\n\n## Contributing\n\nShipped **as-is** under MIT. Issues and PRs are welcome, but this is maintained by one person —\n**no support or response time is guaranteed.** If it's useful to you, a ⭐ helps.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsegentic-lab%2Fglovebox-mcp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsegentic-lab%2Fglovebox-mcp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsegentic-lab%2Fglovebox-mcp/lists"}