{"id":50864171,"url":"https://github.com/sumo1/computer-use-harness","last_synced_at":"2026-06-14T23:34:10.362Z","repository":{"id":362087278,"uuid":"1257195405","full_name":"sumo1/computer-use-harness","owner":"sumo1","description":"CLI-first computer-use harness for agents: policy-guarded macOS app automation with a native helper and structured traces.","archived":false,"fork":false,"pushed_at":"2026-06-10T13:51:49.000Z","size":374,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-14T23:34:06.931Z","etag":null,"topics":["ai-agents","automation","cli","computer-use","macos"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sumo1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-02T13:00:56.000Z","updated_at":"2026-06-10T13:52:44.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/sumo1/computer-use-harness","commit_stats":null,"previous_names":["sumo1/computer-use-harness"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sumo1/computer-use-harness","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumo1%2Fcomputer-use-harness","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumo1%2Fcomputer-use-harness/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumo1%2Fcomputer-use-harness/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumo1%2Fcomputer-use-harness/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sumo1","download_url":"https://codeload.github.com/sumo1/computer-use-harness/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumo1%2Fcomputer-use-harness/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34342089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-14T02:00:07.365Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","automation","cli","computer-use","macos"],"created_at":"2026-06-14T23:34:09.147Z","updated_at":"2026-06-14T23:34:10.349Z","avatar_url":"https://github.com/sumo1.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# computer-use-harness\n\nA CLI-first computer-use harness for agents.\n\n让 Claude Code、Codex、shell 脚本和未来 MCP 工具，以同一套协议安全地观察和操作本机应用。\n\n## 核心判断\n\nAI agent 真的要进入桌面环境时，最大问题不是“怎么点一下按钮”。\n\n真正昂贵的是：\n\n**如何让一个 agent 在真实机器上行动时，可解释、可追踪、可拦截、可复现。**\n\n裸调用 Accessibility API 或屏幕坐标点击很容易做出 demo，也很容易把机器点坏。真正能长期使用的 computer-use runtime，需要把动作拆成几件一等公民：\n\n- **Target**：要操作谁，不允许含糊。\n- **Observation**：看到了什么，来源是什么。\n- **Action**：准备做什么，走哪个 adapter。\n- **Policy**：这一步是否允许，为什么。\n- **ActionResult**：结果是什么，失败码是否稳定。\n- **Trace**：整条链路能不能回放和审查。\n\ncomputer-use-harness 的立场很简单：\n\n**电脑操作不是 prompt 技巧，而是一个本地执行系统。**\n\n它必须像工程系统一样，有协议、有边界、有日志、有失败语义。否则今天能点按钮，明天就能误删东西。\n\n## 这个项目是什么\n\ncomputer-use-harness 是一个 Mac 优先的本地 computer-use runtime，第一形态是 CLI。\n\n它不是桌面 App，不是浏览器自动化库，也不是某个 agent 的私有插件。\n\n它要做的是一层可复用的本地执行底座：\n\n```text\nClaude Code / Codex / shell / future MCP\n  -\u003e computer-use CLI\n  -\u003e policy guard\n  -\u003e capability registry\n  -\u003e adapter\n  -\u003e Swift mac helper / browser-harness\n  -\u003e ActionResult\n  -\u003e trace\n```\n\n第一版重点是把协议和安全边界做稳。真实 macOS UI 动作会逐步打开，但不会在 policy、trace、错误码之前打开。\n\n## 现在能做什么\n\n当前仓库已经具备：\n\n- TypeScript CLI skeleton。\n- 机器可读 JSON 输出。\n- 稳定 command error code 和 exit code。\n- Use case harness。\n- 自由 native action runner（observe / open / click / type / key / scroll / drag / hover / extract）。\n- fake runner。\n- JSONL trace runtime。\n- policy guard。\n- app capability registry。\n- TS / Swift helper protocol。\n- Swift native helper 实现。\n- native helper stdio JSON-RPC 长连接。\n- native runner，通过 `--mac-helper \u003cpath\u003e` 把真实 action 执行结果写入 trace。\n\n当前 Swift helper 支持：\n\n- `permissionStatus`\n- `listApps`\n- `listWindows`\n- `getAppState`\n- `open` - 启动 app 或打开文件\n- `click` - 点击 AX 元素或屏幕坐标\n- `type` - 输入文本（标准 AX 输入或 paste fallback）\n- `key` - 按键和快捷键\n- `scroll` - 滚动并重新观察\n- `drag` / `hover` / `secondary-click` - 基础鼠标操作\n\n两条入口并存：\n\n- `computer-use usecases run ...`：跑 `usecases/cases.yaml` 里的预定义回归用例。\n- `computer-use observe/click/type/key/scroll/...`：给 agent 或 shell loop 做一步一决策的通用原子操作，不需要先写 case。\n\n**真实 usecase 已验证**：\n\n- **UC-100**: QQ Music 搜索\"鸭子\"并播放\n- **UC-110**: Sublime Text 编辑文件并保存，含文件系统验证\n\n两个 usecase 都包含完整 trace、policy decision、app-specific fallback 和外部验证证据。\n\n## 任务沉淀\n\n当前活跃任务和双契约入口放在 [`docs/task/260603-multi-app-close-loop/`](./docs/task/260603-multi-app-close-loop/).\n\n## 什么时候用\n\n适合：\n\n- 你正在研究或构建 agent 操作本地电脑的能力。\n- 你需要 CLI-first，而不是先做一个 GUI app。\n- 你关心 trace、policy、错误码和可复现性。\n- 你希望 browser automation 和 native app automation 最终走同一套协议。\n- 你想让 Claude Code / Codex 通过 shell 或 MCP 稳定调用本地 computer-use 能力。\n\n不适合：\n\n- 只想要一个马上能点屏幕坐标的 demo。\n- 只做浏览器自动化，Playwright 已经够用。\n- 不关心本地权限、安全边界和 trace。\n- 想把 macOS Accessibility 调用直接塞进 agent prompt。\n\n最后一种不是工具问题，是事故迟早问题。\n\n## 设计哲学\n\n### 1. CLI 是 runtime SSOT\n\nagent-specific skill 只是薄包装。\n\n真正的能力应该落在 CLI runtime 里，这样 Claude Code、Codex、人类 shell、CI smoke 和未来 MCP 都能复用同一套行为。\n\n```text\nRuntime SSOT = CLI\nAgent UX = skill / plugin / MCP wrapper\n```\n\n### 2. Policy 先于动作\n\n任何真实 action 都必须先经过 policy guard。\n\n受保护目标、当前终端、agent 自身、安全弹窗、系统设置等目标不能靠“agent 自觉”避开。该挡的动作必须在 runtime 层挡住，并写入 trace。\n\n### 3. Trace 不是调试附属品\n\nTrace 是核心产品能力。\n\n每一步都应该能回答：\n\n- 当时 target 是谁？\n- 观察来自哪里？\n- policy 为什么允许或拒绝？\n- action 走哪个 adapter？\n- 失败码是什么？\n- 下一次怎么复现？\n\n没有 trace 的 computer-use，只是一次性魔法。\n\n### 4. Native 能力要隔离\n\nTypeScript 适合写 CLI、协议、trace、policy、adapter 调度。\n\nmacOS Accessibility、Screen Recording、CGEvent、窗口枚举和权限检测应该放在 Swift helper 里。\n\nTS 编排，Swift 触系统 API。边界清楚，才不会把 prompt、业务逻辑和系统调用搅成一锅。\n\n### 5. Browser 不重写\n\n浏览器自动化应该复用成熟的 browser harness。\n\ncomputer-use-harness 只负责把 browser capability 接进统一 Target / Observation / Action / Trace 协议，而不是重新发明一个 Playwright。\n\n## Quick Start\n\n### 1. 安装依赖\n\n```sh\nnpm install\n```\n\n### 2. 构建 CLI\n\n```sh\nnpm run build\n```\n\n### 3. 跑 fake use case\n\n```sh\n./dist/cli/index.js usecases list\n./dist/cli/index.js usecases run UC-030 --fake\n./dist/cli/index.js trace --last\n```\n\n### 4. 跑自由原子 action\n\n```sh\n./dist/cli/index.js observe --app \"Fake Target App\" --fake --pretty\n./dist/cli/index.js click --app \"Fake Target App\" --fake \\\n  --keyword Primary \\\n  --description \"click button named Primary\" \\\n  --pretty\n./dist/cli/index.js scroll --app \"Fake Target App\" --fake down 2 --pretty\n```\n\n每个 action 都会执行 policy、必要的 observe-before、action、observe-after，并写入 trace。\nresult trace event 会附加 `actionTraceStep`，用于审计这一轮闭环：\n\n- `before` / `after`：动作前后的 observation。\n- `execution.inputBackend`：真实输入后端，取值为 `ax-semantic`、`app-targeted-event` 或 `global-hid`。\n- `verification`：动作后观察是否完成、是否命中目标状态。\n- `virtualPointer`：harness 推断的指向位置，不是第二个系统鼠标。\n- `virtualPointerOverlay`：当 observation 带 screenshot 时生成的 SVG 指针 overlay。\n\n### 5. 构建 Swift helper\n\n```sh\ncd native/mac-helper\nswift build\n```\n\n### 6. 跑真实 usecase\n\n```sh\n# QQ Music: 搜索\"鸭子\"并播放\n./dist/cli/index.js usecases run UC-100 \\\n  --mac-helper ./native/mac-helper/.build/debug/computer-use-mac-helper\n\n# Sublime Text: 编辑并保存文件\n./dist/cli/index.js usecases run UC-110 \\\n  --mac-helper ./native/mac-helper/.build/debug/computer-use-mac-helper\n\n# 查看 trace\n./dist/cli/index.js trace --last\n```\n\n预期结果：\n\n- CLI 返回 `ok: true`。\n- run status 为 `passed`（如果 app 已安装且有权限）。\n- trace 包含完整的 observation / action / policy / result 事件，以及 result event 上的 `actionTraceStep`。\n- UC-110 会在 `/tmp/computer-use-harness/uc-110.txt` 留下真实文件。\n\n## CLI\n\n所有命令默认输出 JSON。\n\n```sh\ncomputer-use version\ncomputer-use apps\ncomputer-use capabilities --app Safari\ncomputer-use observe --app Finder --mac-helper ./native/mac-helper/.build/debug/computer-use-mac-helper\ncomputer-use click --app Finder --keyword Downloads --description \"click item named Downloads\" --mac-helper ./native/mac-helper/.build/debug/computer-use-mac-helper\ncomputer-use type --app \"Fake Target App\" --fake --keyword \"Main Input\" --text hello\ncomputer-use key --app Finder --key Enter --mac-helper ./native/mac-helper/.build/debug/computer-use-mac-helper\ncomputer-use scroll --app Finder --direction down --amount 2 --mac-helper ./native/mac-helper/.build/debug/computer-use-mac-helper\ncomputer-use extract --app Finder --query \"visible files\" --fields files --mac-helper ./native/mac-helper/.build/debug/computer-use-mac-helper\ncomputer-use usecases list\ncomputer-use usecases dry-run UC-030\ncomputer-use usecases run UC-030 --fake\ncomputer-use usecases run UC-030 --mac-helper ./native/mac-helper/.build/debug/computer-use-mac-helper\ncomputer-use trace --last\n```\n\n人类可读格式使用：\n\n```sh\n--pretty\n```\n\nCLI 约束：\n\n- stdout 只输出结构化结果。\n- 业务失败也返回 JSON。\n- 日志和非协议噪声不能污染 stdout。\n- error code 必须稳定。\n- action failure 必须进入 trace。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsumo1%2Fcomputer-use-harness","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsumo1%2Fcomputer-use-harness","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsumo1%2Fcomputer-use-harness/lists"}