{"id":51010165,"url":"https://github.com/georgenijo/ghosthands","last_synced_at":"2026-06-21T01:33:41.285Z","repository":{"id":364196815,"uuid":"1266833571","full_name":"georgenijo/ghosthands","owner":"georgenijo","description":"👻 Free, local computer-use for macOS — a small MLX model drives native apps via the Accessibility tree, then replays flows with no model. $0, background, no cursor stolen.","archived":false,"fork":false,"pushed_at":"2026-06-17T09:21:47.000Z","size":292,"stargazers_count":0,"open_issues_count":34,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-21T01:33:40.503Z","etag":null,"topics":["accessibility","ai-agents","apple-silicon","automation","claude","computer-use","local-llm","macos","mcp","mlx"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/georgenijo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-06-12T01:56:45.000Z","updated_at":"2026-06-17T07:24:34.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/georgenijo/ghosthands","commit_stats":null,"previous_names":["georgenijo/ghosthands"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/georgenijo/ghosthands","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/georgenijo%2Fghosthands","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/georgenijo%2Fghosthands/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/georgenijo%2Fghosthands/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/georgenijo%2Fghosthands/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/georgenijo","download_url":"https://codeload.github.com/georgenijo/ghosthands/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/georgenijo%2Fghosthands/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34591166,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-20T02:00:06.407Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accessibility","ai-agents","apple-silicon","automation","claude","computer-use","local-llm","macos","mcp","mlx"],"created_at":"2026-06-21T01:33:40.790Z","updated_at":"2026-06-21T01:33:41.280Z","avatar_url":"https://github.com/georgenijo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# 👻 GhostHands\n\n### Invisible hands any AI can borrow to drive your Mac — **free, local, and without stealing your cursor.**\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-black.svg)](./LICENSE)\n[![Platform: macOS](https://img.shields.io/badge/platform-macOS%2014%2B%20·%20Apple%20Silicon-black.svg)](#requirements)\n[![Cost](https://img.shields.io/badge/cost-%240-brightgreen.svg)](#the-benchmark)\n[![Hands: Cua Driver](https://img.shields.io/badge/hands-Cua%20Driver-blue.svg)](https://github.com/trycua/cua)\n\n*A local computer-use harness that lets an AI agent click, type, and read native macOS apps — driven by a **free local model**, in the **background**, then **replayed with no model at all.***\n\n\u003c/div\u003e\n\n---\n\n## Why\n\nAgents that build UIs need to **test** them. Today that means slow, paid, DOM-only browser automation. GhostHands drives the **real macOS Accessibility tree** instead — so a tiny **local** model (or Claude, or Codex — swappable) can operate any native app, in the background, for **$0**. Run a flow once, and GhostHands records it so every rerun afterward needs **no model**.\n\n```\n          ┌──── brain slot ────┐\n  goal →  │ local · Claude · GPT│ →  Cua Driver (hands) →  macOS apps\n          └─────────────────────┘\n                  swap the brain · shared hands · no cursor stolen\n```\n\n## The three ideas\n\n🧠 **Free local brain.** A small MLX text model (default **Qwen3-4B-Instruct-2507-4bit**) reads the structured **Accessibility tree** and emits the full click plan as compact JSON — *no vision, no API, no tokens*. With a KV prompt cache and JSON early-stop, a warm decision takes **~1.8s**; it drives macOS Calculator to `7 × 6 = 42` at **100% / $0**, ~3.3× faster than the paid ceiling.\n\n👁️ **Vision only when there's no other way.** Canvas/WebGL/game surfaces with no AX tree fall back to a local vision model (MAI-UI-8B) + pixel clicks. It's the *last* resort — the benchmark shows exactly why (below).\n\n🔁 **Record → replay with no model.** Run a flow once with a model in the loop; GhostHands captures **name-targeted** steps (AX id / title). Then `replay` re-runs it **deterministically, $0, no model** — self-healing a moved target with a single model call only when needed. Testing a UI = running the same flow many times. Now those reruns are free.\n\n## 60-second quickstart\n\n```sh\n# Hands: install Cua Driver (MIT) and grant Accessibility once\n/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)\"\ncua-driver permissions grant\n\n# Brain: a local MLX model (Apple Silicon). Pull the default 4B:\npython -m venv .venv \u0026\u0026 .venv/bin/pip install mlx-lm mlx-vlm\n.venv/bin/hf download mlx-community/Qwen3-4B-Instruct-2507-4bit\n\n# Install the CLI on your PATH (editable, with the local brain baked in):\nuv tool install --editable '.[local]'\n\n# Drive a native app with the free local brain, $0:\nghosthands run \"Compute 7 times 6\" --brain local --app com.apple.calculator\n\n# Record a flow once, then replay forever with NO model:\nghosthands record myflow \"\u003cgoal\u003e\" --app \u003cbundle\u003e --url \u003cpage\u003e\nghosthands replay myflow                # deterministic · $0\n```\n\nOther commands: `ghosthands doctor` (verify the environment), `ghosthands smoke` (Calculator 7×6=42), `ghosthands brains` (detect Claude/Codex), `ghosthands run --brain claude|gpt` (subscription agents), `ghosthands scene \"\u003cdiagram\u003e\"` (Excalidraw scene JSON on the routed model), `ghosthands bench`. (`bin/ghosthands` still works zero-install from the repo.)\n\n**For AI agents:** operating instructions ship inside the CLI, always version-matched — `ghosthands skills` lists them, `ghosthands skills get core|web|canvas|tests` prints them. Drop a thin discovery skill into your harness (`~/.claude/skills/ghosthands/`, `~/.codex/skills/ghosthands/`) that tells the agent to run `ghosthands skills get core` first.\n\n## The benchmark\n\nEvery brain, identical task, identical hands. Wall-clock starts at dispatch and stops the instant a **world** check passes — a calculator display value read over AX, or the destination page title in any browser — never the agent's own words. Full per-device results and how to add your own Mac: **[bench/RESULTS.md](./bench/RESULTS.md)**.\n\nMeasured on an **Apple M4 Mac mini · 24 GB · macOS 27** *(local/scripted n=5; claude calc n=5, web n=3)*:\n\n| contender | hands / path | task | success | median | steps | cost\\* |\n|-----------|--------------|------|---------|--------|-------|------|\n| `scripted-ax` | no model (floor) | calc 7×6 | 100% | 4.5s | — | $0 |\n| **`local` 4B** | free MLX · **AX** | calc 7×6 | **100%** | **8.1s** | 6 | **$0** |\n| `mai-ui-pixel` | free vision · **pixel** | calc 7×6 | **0%** | — | 10 | $0 |\n| `claude` | Claude · cua **AX** | calc 7×6 | 100% | 37.8s | 9 | $0.24 |\n| **`local` 4B** | free MLX · **AX** | web | **100%** | **12.1s** | 1 | **$0** |\n| **`local` 4B** | free MLX · **AX** | v2 disambig (5 identical links) | **100%** | **8.2s** | 1 | **$0** |\n| `claude-browser` | Claude · agent-browser **DOM** | web | 100% | 19.2s | 5 | $0.07 |\n| `claude-chrome` | Claude · chrome-devtools-mcp **DOM** | web | 100% | 22.1s | 5 | $0.08 |\n| `claude` | Claude · cua **AX** | web | 100% | 23.1s | 6 | $0.11 |\n| `claude-pixel` | Claude · cua **pixel** | web | **0%** | — | 19 | — |\n\n\u003e **2026-06 speed pass** (same machine, same tasks): swapping the 7B for\n\u003e Qwen3-4B-2507 with a compact click protocol, KV prompt cache, and JSON\n\u003e early-stop cut a warm model decision from **13.4s → 1.8s**; firing actions\n\u003e without waiting for the daemon's ~1.1s padded response cut the no-model\n\u003e floor from **7.2s → 4.5s**.\n\u003e\n\u003e **2026-06 reliability pass**: the brain now answers with button **names**\n\u003e (\"7\", \"Multiply\") instead of volatile element numbers — the harness\n\u003e resolves names against the live tree (with symbol aliases, an honesty\n\u003e guard that refuses plans naming buttons that aren't on screen, and a\n\u003e navigation cut that re-plans after every page change). Calc fell\n\u003e **11.3s → 8.1s**, link disambiguation went **0% → 100%**, and on the\n\u003e model-only gate long ordered plans went **0% → 83%** with fully\n\u003e deterministic replies. Canvas tasks route to a second local model\n\u003e (Qwen3-8B) that generates the whole Excalidraw scene as JSON\n\u003e (`ghosthands scene`, `skills/CANVAS.md`) — the only local model that\n\u003e passed scene validation *and* declined impossible goals in a 6-model\n\u003e bake-off (`bench/model_gate.py`).\n\n\\* Local/scripted contenders burn zero tokens; the Claude `cost` is the metered `total_cost_usd` Claude Code reports (subscription, not out-of-pocket).\n\nTwo results jump out:\n\n1. **The free local 7B on AX is the fastest *model-driven* path on both tasks, at $0** — beating every Claude path (one AX click vs 5–9 tool round-trips).\n2. **Same brain, different hands:** hand *Claude* the pixel path and it still scores **0%** (3× timeout, ~19 flailing clicks). The brain was never the bottleneck — synthetic pixel clicks don't land on backgrounded windows. DOM tools (agent-browser, chrome-devtools-mcp) and AX all work; pixels don't.\n\n### Why AX wins and pixels lose (the whole thesis)\n\n- **AX path** → Cua reads the accessibility tree (`[5] AXButton \"7\"`), a local *text* model picks the element, Cua fires `AXPress`. This is an accessibility *action* aimed at the element itself — no cursor, no focus — so it lands on a **background** window. ✅\n- **Pixel path** → screenshot → vision model *guesses* coordinates → synthetic mouse click. macOS routes synthetic clicks to the **frontmost** window, so they never reach a backgrounded target. → **0%**. The only fix would be to foreground the window and move your cursor — which violates the whole point.\n\nThat's why GhostHands is AX-first and pixels are fallback-only.\n\n## How the hands work\n\nThe hands are [**Cua Driver**](https://github.com/trycua/cua) (MIT) exposed over MCP. GhostHands adds a hardened wrapper that absorbs the prerelease driver's sharp edges:\n\n- **Re-snapshot before every action** — the `element_index` cache expires in seconds.\n- **Match by name, not index** — targets resolve against a fresh snapshot; duplicate-window subtrees are all tried.\n- **Settle until the tree stops changing** before re-reading (a digit press flips a button label a beat before the display updates).\n- **Cursor-less, no focus steal** — apps launch in the background; actions are `AXPress`.\n- **Fire-and-go actions** — an AX action lands in ~250ms but the 0.5.1 daemon\n  pads its success response to ~1.1s; GhostHands dispatches without blocking\n  and reads the truth from the next snapshot (errors return in \u003c50ms and are\n  still caught).\n\n## Requirements\n\n- **macOS 14+ on Apple Silicon.** This is not cross-platform: the local brain uses **MLX** (Metal), and the hands use macOS Accessibility + ScreenCaptureKit. The *architecture* (swappable brain over MCP hands) is portable, but this build is Mac-only.\n- **Cua Driver** pinned to `0.5.1` (prerelease) at `~/.local/bin/cua-driver`; Accessibility granted to `com.trycua.driver`. Screen Recording is optional (only the pixel fallback needs it).\n- **Python 3.10+.** The CLI and AX path are stdlib-only; the local brains add `mlx-lm` / `mlx-vlm` in a venv (lazy-imported).\n\n## Layout\n\n```\nsrc/ghosthands/  driver · ax · actions (hardened wrapper) · ownloop (LocalBrain + loop)\n                 visionloop (pixel fallback) · flows (record/replay) · tasks · cli\nskills/          model-agnostic operating instructions for any agent\nbench/           brain-vs-brain harness + results\nflows/           recorded flows (replay with no model)\n```\n\nSee **[DESIGN.md](./DESIGN.md)** for the full spec, **[ROADMAP.md](./ROADMAP.md)** for milestones + the benchmark, and **[AGENTS.md](./AGENTS.md)** if you're an agent picking this up.\n\n## License \u0026 credits\n\nMIT. The hands are [Cua Driver](https://github.com/trycua/cua) (MIT) — the operating-instruction skill is adapted from its `cua-driver` skill; see [ATTRIBUTION.md](./ATTRIBUTION.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeorgenijo%2Fghosthands","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeorgenijo%2Fghosthands","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeorgenijo%2Fghosthands/lists"}