{"id":51786344,"url":"https://github.com/gregorym/swift-computer-use","last_synced_at":"2026-07-20T18:04:45.841Z","repository":{"id":354336210,"uuid":"1223187814","full_name":"gregorym/swift-computer-use","owner":"gregorym","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-29T01:56:48.000Z","size":42,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-29T03:23:52.492Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Swift","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gregorym.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-28T04:55:45.000Z","updated_at":"2026-05-29T01:56:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/gregorym/swift-computer-use","commit_stats":null,"previous_names":["gregorym/swift-computer-use"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gregorym/swift-computer-use","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gregorym%2Fswift-computer-use","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gregorym%2Fswift-computer-use/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gregorym%2Fswift-computer-use/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gregorym%2Fswift-computer-use/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gregorym","download_url":"https://codeload.github.com/gregorym/swift-computer-use/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gregorym%2Fswift-computer-use/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35695248,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"ssl_error","status_checked_at":"2026-07-20T02:08:09.736Z","response_time":111,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-20T18:04:44.871Z","updated_at":"2026-07-20T18:04:45.834Z","avatar_url":"https://github.com/gregorym.png","language":"Swift","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ComputerUse\n\n`ComputerUse` is a Swift package for controlling visible macOS windows from an agent runtime.\n\nIt provides a list of computer-use tools and implements those tools locally. It does not call an LLM, choose a model, send prompts, or talk to OpenAI/Pi/Anthropic/etc. Your app can pass the tool definitions to any model or agent system, then call `ComputerUseSession.execute(tool:arguments:)` when that system asks to run a tool.\n\n## What It Does\n\n- Lists running apps and windows.\n- Captures a target window and returns semantic Accessibility targets such as `@e1`.\n- Clicks, scrolls, drags, types, presses keys, sets text values, waits, and arranges windows.\n- Prefers macOS Accessibility actions over raw mouse/keyboard events.\n- Returns text plus optional PNG image content when visual fallback is useful.\n- Keeps state IDs and window refs so callers can detect stale screenshots.\n\n## Requirements\n\n- macOS 14 or newer\n- Swift 6.2 or newer\n- Host executable permissions:\n  - Accessibility\n  - Screen Recording\n\nPermissions are granted to the app or executable that imports and runs this package, not to the package itself.\n\n## Install\n\nIn `Package.swift`:\n\n```swift\ndependencies: [\n    .package(url: \"https://github.com/gregorym/swift-computer-use.git\", branch: \"main\")\n],\ntargets: [\n    .target(\n        name: \"YourApp\",\n        dependencies: [\n            .product(name: \"ComputerUse\", package: \"swift-computer-use\")\n        ]\n    )\n]\n```\n\nFor local development:\n\n```swift\n.package(path: \"../swift-computer-use\")\n```\n\n## Quick Start\n\n```swift\nimport ComputerUse\n\nlet session = ComputerUseSession()\n\nlet permissionStatus = await session.checkPermissions()\nif !permissionStatus.accessibility {\n    try await session.openPermissionPane(.accessibility)\n}\nif !permissionStatus.screenRecording {\n    try await session.openPermissionPane(.screenRecording)\n}\n\nlet tools = session.tools\n\nlet screenshot = try await session.execute(tool: \"screenshot\", arguments: [\n    \"image\": \"auto\"\n])\n\nlet click = try await session.execute(tool: \"click\", arguments: [\n    \"ref\": \"@e1\"\n])\n```\n\n`tools` contains names, descriptions, and JSON-schema parameter objects. Pass those to your agent runtime. When the runtime asks to call a tool, pass the tool name and JSON arguments to `execute`.\n\n## Normal Workflow\n\n1. Call `screenshot` first. It selects the current controlled window and returns current UI state.\n2. If `axTargets` contains useful refs such as `@e1`, prefer refs over coordinates.\n3. Use coordinates only when no suitable AX target exists. Coordinates are window-relative screenshot pixels.\n4. Pass `stateId` from the latest result when using coordinates, so stale screenshots are rejected.\n5. Every successful action returns a refreshed semantic state.\n\nExample:\n\n```swift\nlet state = try await session.execute(tool: \"screenshot\", arguments: [\n    \"app\": \"TextEdit\",\n    \"image\": \"auto\"\n])\n\nlet stateId = state.details[\"capture\"]?[\"stateId\"]?.stringValue\n\n_ = try await session.execute(tool: \"set_text\", arguments: [\n    \"ref\": \"@e1\",\n    \"text\": \"Hello from Swift\"\n])\n\n_ = try await session.execute(tool: \"click\", arguments: [\n    \"x\": 120,\n    \"y\": 80,\n    \"stateId\": .string(stateId ?? \"\"),\n    \"image\": \"never\"\n])\n```\n\n## Discovering Windows\n\nUse `list_apps` and `list_windows` when the target app or window is ambiguous.\n\n```swift\nlet apps = try await session.execute(tool: \"list_apps\")\n\nlet windows = try await session.execute(tool: \"list_windows\", arguments: [\n    \"app\": \"Safari\"\n])\n\nlet selected = try await session.execute(tool: \"screenshot\", arguments: [\n    \"window\": \"@w1\"\n])\n```\n\nWindow refs such as `@w1` are stable within the session and can be passed to later tool calls:\n\n```swift\ntry await session.execute(tool: \"click\", arguments: [\n    \"window\": \"@w1\",\n    \"ref\": \"@e1\"\n])\n```\n\n## Handling Results\n\nEach result has two parts:\n\n- `content`: user-facing text and optional PNG images\n- `details`: structured JSON for state, target, execution metadata, AX targets, and config\n\n```swift\nfor item in result.content {\n    switch item {\n    case .text(let text):\n        print(text)\n    case .image(let data, let mimeType):\n        print(\"Image:\", data.count, mimeType)\n    }\n}\n\nif let targets = result.details[\"axTargets\"]?.arrayValue {\n    print(\"AX target count:\", targets.count)\n}\n```\n\nImportant detail fields:\n\n- `target`: app, bundle ID, pid, title, window ID, and window ref\n- `capture`: state ID, width, height, scale factor, timestamp, coordinate space\n- `axTargets`: semantic UI targets with capabilities like `canPress`, `canSetValue`, and `canScroll`\n- `execution`: strategy, AX/fallback status, and strict-mode compatibility\n- `imageReason`: why an image was attached, when one is attached\n\n## Tools\n\n| Tool               | Purpose                                                             |\n| ------------------ | ------------------------------------------------------------------- |\n| `list_apps`        | List running macOS apps.                                            |\n| `list_windows`     | List windows, titles, geometry, focus state, and `@w` refs.         |\n| `screenshot`       | Select or refresh the controlled window.                            |\n| `click`            | Click by AX ref or screenshot coordinate.                           |\n| `double_click`     | Double-click by AX ref or coordinate.                               |\n| `move_mouse`       | Move the pointer to screenshot coordinates.                         |\n| `drag`             | Drag along a coordinate path or adjust an AX target.                |\n| `scroll`           | Scroll by AX ref or screenshot coordinate.                          |\n| `keypress`         | Press keys such as `Enter`, `Tab`, `Escape`, or `Command+L`.        |\n| `type_text`        | Insert text into the focused control.                               |\n| `set_text`         | Replace an AX text value by ref or focused text control.            |\n| `wait`             | Pause and return refreshed state.                                   |\n| `arrange_window`   | Move or resize a window using presets or explicit frame values.     |\n| `navigate_browser` | Navigate supported browser windows directly to a URL/search string. |\n| `computer_actions` | Run 1 to 20 actions as a batch and return one refreshed state.      |\n\n## Configuration\n\n```swift\nlet session = ComputerUseSession(config: .init(\n    browserUse: true,\n    stealthMode: false\n))\n```\n\nOptions:\n\n- `browserUse`: when `false`, known browser windows are refused.\n- `stealthMode`: when `true`, only background-safe AX paths are allowed. Raw mouse, raw keyboard, foreground-focus, and cursor fallbacks are blocked.\n\n## Image Mode\n\nMost tools accept `image`:\n\n- `\"auto\"`: attach an image only when useful for fallback or verification.\n- `\"always\"`: always attach a PNG screenshot.\n- `\"never\"`: suppress image attachment.\n\n```swift\ntry await session.execute(tool: \"screenshot\", arguments: [\n    \"image\": \"always\"\n])\n```\n\n## Batch Actions\n\nUse `computer_actions` when the next actions are obvious and no intermediate screenshot is needed.\n\n```swift\ntry await session.execute(tool: \"computer_actions\", arguments: [\n    \"actions\": [\n        [\"type\": \"click\", \"ref\": \"@e1\"],\n        [\"type\": \"set_text\", \"ref\": \"@e2\", \"text\": \"hello\"],\n        [\"type\": \"keypress\", \"keys\": [\"Enter\"]]\n    ],\n    \"image\": \"auto\"\n])\n```\n\nDo not batch when later actions depend on seeing the result of earlier actions.\n\n## Development\n\nBuild and test:\n\n```bash\nswift build\nswift test\n```\n\nThe default tests use a fake backend and do not require Accessibility or Screen Recording permission.\n\nTo run the gated native smoke test:\n\n```bash\nCOMPUTER_USE_RUN_NATIVE_TESTS=1 swift test\n```\n\n## Notes\n\n- The package currently supports macOS only.\n- Accessibility quality varies by app. When AX coverage is weak, results may include a screenshot image so the caller can fall back to visual reasoning.\n- Browser direct navigation is implemented for Safari and Chromium-family browsers where macOS scripting supports it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgregorym%2Fswift-computer-use","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgregorym%2Fswift-computer-use","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgregorym%2Fswift-computer-use/lists"}