{"id":50508978,"url":"https://github.com/aaronlab/copilot-computer-use","last_synced_at":"2026-06-02T18:32:51.476Z","repository":{"id":346539686,"uuid":"1190425309","full_name":"aaronlab/copilot-computer-use","owner":"aaronlab","description":"The world's first free Computer Use agent powered by GitHub Copilot API. Screenshot → Vision Analysis → Action Execution, all through your Copilot subscription.","archived":false,"fork":false,"pushed_at":"2026-03-24T09:51:37.000Z","size":53,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-09T15:53:22.212Z","etag":null,"topics":["ai-agent","computer-use","copilot","copilot-api","desktop-automation","free","github-copilot","open-source","pyautogui","vision"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aaronlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-24T09:16:12.000Z","updated_at":"2026-03-24T09:55:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/aaronlab/copilot-computer-use","commit_stats":null,"previous_names":["zey413/copilot-computer-use","aaronagent/copilot-computer-use","aaronlab/copilot-computer-use"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/aaronlab/copilot-computer-use","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aaronlab%2Fcopilot-computer-use","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aaronlab%2Fcopilot-computer-use/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aaronlab%2Fcopilot-computer-use/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aaronlab%2Fcopilot-computer-use/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aaronlab","download_url":"https://codeload.github.com/aaronlab/copilot-computer-use/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aaronlab%2Fcopilot-computer-use/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33833277,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-02T02:00:07.132Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agent","computer-use","copilot","copilot-api","desktop-automation","free","github-copilot","open-source","pyautogui","vision"],"created_at":"2026-06-02T18:32:50.612Z","updated_at":"2026-06-02T18:32:51.464Z","avatar_url":"https://github.com/aaronlab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# copilot-computer-use\n\n**English** | [中文](README_zh.md)\n\n\u003e **The world's first free Computer Use agent powered by GitHub Copilot API.**\n\nScreenshot → Vision Analysis → Action Execution — all through your Copilot subscription ($10/mo or free for students).\n\n## Why?\n\nEvery Computer Use agent today requires expensive Vision API calls ($$$). We discovered that **GitHub Copilot API supports base64 image input** via the `copilot-vision-request: true` header. This means you can build a fully functional Computer Use agent at **zero additional cost** beyond your existing Copilot subscription.\n\n## How It Works\n\n```\n┌─────────────────────────────────────────────────┐\n│              Agent Control Loop                  │\n│                                                  │\n│  1. 📸 Capture screenshot (mss + Pillow)         │\n│  2. 🔄 Encode to base64 PNG                      │\n│  3. 👁️ Analyze via Copilot Vision API            │\n│     (copilot-vision-request: true header)        │\n│  4. 🧠 Plan next action via Copilot Text API     │\n│  5. ⚡ Execute action (pyautogui)                │\n│  6. 🔁 Repeat until task complete                │\n└─────────────────────────────────────────────────┘\n         ↕ All API calls go through ↕\n┌─────────────────────────────────────────────────┐\n│        Copilot API Adapter                       │\n│  • GitHub OAuth Device Flow authentication       │\n│  • Automatic JWT token refresh                   │\n│  • VS Code version spoofing                      │\n│  • copilot-vision-request header management      │\n└─────────────────────────────────────────────────┘\n```\n\n## Key Innovation\n\n| Aspect | Traditional Agent | copilot-computer-use |\n|--------|-------------------|---------------------|\n| Vision API Cost | $0.01-0.05/screenshot | **$0 (included in Copilot)** |\n| Text Reasoning Cost | $0.003-0.06/request | **$0 (included in Copilot)** |\n| Monthly Cost | $20-100+ | **$10/mo (Copilot) or free** |\n| Models Available | 1 provider | GPT-4o, Claude, Gemini |\n\n## Prerequisites\n\n- Python 3.11+\n- GitHub account with Copilot subscription (Individual, Business, or Enterprise)\n- macOS or Linux (Windows support planned)\n\n## Quick Start\n\n```bash\n# Clone\ngit clone https://github.com/Zey413/copilot-computer-use.git\ncd copilot-computer-use\n\n# Install\npip install -e .\n\n# Authenticate with GitHub (one-time)\npython -m src.copilot.auth\n\n# Run\npython -m src.main \"Open Chrome and search for the weather\"\n```\n\n## Architecture\n\n```\nsrc/\n├── main.py                 # Entry point + CLI\n├── copilot/\n│   ├── auth.py             # GitHub OAuth Device Flow + JWT refresh\n│   ├── client.py           # Copilot chat/completions API client\n│   └── config.py           # API endpoints, headers, version spoofing\n├── agent/\n│   ├── loop.py             # Core agent control loop\n│   ├── planner.py          # Task planning via text reasoning\n│   └── actions.py          # Action definitions (click, type, scroll...)\n├── screen/\n│   ├── capture.py          # Screenshot capture (mss + Pillow)\n│   └── annotate.py         # Optional: annotate screenshots with markers\n└── executor/\n    ├── base.py             # Abstract executor interface\n    ├── macos.py            # macOS executor (pyautogui + AppleScript)\n    └── linux.py            # Linux executor (pyautogui + xdotool)\n```\n\n## How Copilot Vision Works\n\nThe key discovery: Copilot's `chat/completions` API accepts base64-encoded images when you include the `copilot-vision-request: true` header:\n\n```python\nimport httpx\n\nresponse = httpx.post(\n    \"https://api.githubcopilot.com/chat/completions\",\n    headers={\n        \"Authorization\": f\"Bearer {copilot_jwt}\",\n        \"copilot-vision-request\": \"true\",\n        \"editor-version\": \"vscode/1.104.3\",\n        # ... other headers\n    },\n    json={\n        \"model\": \"gpt-4o\",\n        \"messages\": [{\n            \"role\": \"user\",\n            \"content\": [\n                {\"type\": \"text\", \"text\": \"What do you see on this screen?\"},\n                {\"type\": \"image_url\", \"image_url\": {\n                    \"url\": f\"data:image/png;base64,{screenshot_b64}\"\n                }}\n            ]\n        }]\n    }\n)\n```\n\n## Research Background\n\nThis project is based on extensive research documented in Reviews #001-#013:\n- **Raven** ([nocoo/raven](https://github.com/nocoo/raven)): Proved Copilot API can be proxied with full Anthropic↔OpenAI translation\n- **Copilot Vision**: Confirmed base64 support via 6+ open-source projects (LiteLLM, OpenCode, LobeHub, etc.)\n- **Market Gap**: Zero existing projects combine Copilot proxy + Computer Use\n\n## Supported Models (via Copilot)\n\n- GPT-4o / GPT-4.1 (Vision ✅)\n- Claude Sonnet 4 / Opus 4 (Vision ✅)\n- Gemini 2.5 Pro / 3 Flash (Vision ✅)\n- o4-mini (Reasoning)\n- And more as GitHub adds them\n\n## Limitations\n\n- **Experimental**: This is a research project, not production software\n- **Rate Limits**: Copilot has rate limits; aggressive screenshot loops may be throttled\n- **ToS Gray Area**: Using Copilot API beyond IDE integration may not be explicitly permitted\n- **Image Formats**: PNG is most reliable; JPEG works; WebP may have issues\n- **macOS/Linux only**: Windows executor not yet implemented\n\n## Acknowledgments\n\n- [Raven](https://github.com/nocoo/raven) — Inspiration for Copilot API authentication and format translation\n- [self-operating-computer](https://github.com/OthersideAI/self-operating-computer) — Reference for lightweight Computer Use architecture\n- [copilot-api](https://github.com/ericc-ch/copilot-api) — Early Copilot reverse engineering work\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faaronlab%2Fcopilot-computer-use","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faaronlab%2Fcopilot-computer-use","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faaronlab%2Fcopilot-computer-use/lists"}