{"id":28996416,"url":"https://github.com/loadmill/droid-cua","last_synced_at":"2026-02-10T18:10:10.896Z","repository":{"id":290237986,"uuid":"973753550","full_name":"loadmill/droid-cua","owner":"loadmill","description":"AI agent for controlling Android devices with OpenAI’s computer-use-preview model.","archived":false,"fork":false,"pushed_at":"2026-02-03T19:11:15.000Z","size":213,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-02-03T21:33:15.242Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/loadmill.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-27T17:05:16.000Z","updated_at":"2026-02-03T19:11:20.000Z","dependencies_parsed_at":"2025-06-25T05:09:47.283Z","dependency_job_id":"ff811478-ca62-4c0b-9490-19d2238b2789","html_url":"https://github.com/loadmill/droid-cua","commit_stats":null,"previous_names":["loadmill/droid-cua"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/loadmill/droid-cua","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loadmill%2Fdroid-cua","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loadmill%2Fdroid-cua/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loadmill%2Fdroid-cua/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loadmill%2Fdroid-cua/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/loadmill","download_url":"https://codeload.github.com/loadmill/droid-cua/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loadmill%2Fdroid-cua/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29310114,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-10T17:48:59.043Z","status":"ssl_error","status_checked_at":"2026-02-10T17:45:37.240Z","response_time":65,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-25T05:09:45.097Z","updated_at":"2026-02-10T18:10:10.890Z","avatar_url":"https://github.com/loadmill.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# droid-cua\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@loadmill/droid-cua\"\u003e\u003cimg src=\"https://img.shields.io/npm/v/@loadmill/droid-cua?color=green\" alt=\"npm version\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#what-is-droid-cua\"\u003eWhat is droid-cua?\u003c/a\u003e •\n  \u003ca href=\"#quick-start\"\u003eQuick Start\u003c/a\u003e •\n  \u003ca href=\"#features\"\u003eFeatures\u003c/a\u003e •\n  \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e •\n  \u003ca href=\"#assertions\"\u003eAssertions\u003c/a\u003e •\n  \u003ca href=\"#command-line-options\"\u003eCommand Line Options\u003c/a\u003e •\n  \u003ca href=\"#how-it-works\"\u003eHow It Works\u003c/a\u003e •\n  \u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n**AI-powered mobile testing using OpenAI's computer-use model**\n\nCreate and run automated Android and iOS tests using natural language. The AI explores your app and generates executable test scripts.\n\nhttps://github.com/user-attachments/assets/e6450f45-3700-4cb6-aad5-33ba5f0437c3\n\n---\n\n\u003ch2 id=\"what-is-droid-cua\"\u003e💡 What is droid-cua?\u003c/h2\u003e\n\n`droid-cua` gives you three core components for mobile testing:\n\n* **Interactive Shell** – Design and run tests with real-time feedback and visual status indicators\n* **Test Scripts** – Simple text files with natural language instructions and assertions\n* **AI Agent** – Autonomous exploration powered by OpenAI's computer-use model\n\nTogether, these let you create and execute Android and iOS tests without writing traditional test code.\n\n---\n\n\u003ch2 id=\"quick-start\"\u003e🚀 Quick Start\u003c/h2\u003e\n\n**1. Install**\n\nGlobally (recommended):\n```sh\nnpm install -g @loadmill/droid-cua\n```\n\nOr from source:\n```sh\ngit clone https://github.com/loadmill/droid-cua\ncd droid-cua\nnpm install\nnpm run build\n```\n\n**2. Set your OpenAI API key**\n\nUsing environment variable:\n```sh\nexport OPENAI_API_KEY=your-api-key\n```\n\nOr create a `.env` file:\n```sh\necho \"OPENAI_API_KEY=your-api-key\" \u003e .env\n```\n\n**3. Setup for your platform**\n\nFor Android:\n```sh\nadb version  # Ensure ADB is available\n```\n\nFor iOS (macOS only):\n```sh\n# Install Appium and XCUITest driver\nnpm install -g appium\nappium driver install xcuitest\n```\n\n**4. Run**\n\n```sh\ndroid-cua\n```\n\nAn interactive menu will let you select your platform and device:\n\n```\n┌──────────────────────────────────────┐\n│ Select Platform                      │\n└──────────────────────────────────────┘\n\n❯ Android (1 running) - 2 emulator(s)\n  iOS - 5 simulator(s)\n\n↑/↓ Navigate  Enter Select  q Quit\n```\n\nThe emulator/simulator will auto-launch if not already running.\n\n---\n\n\u003ch2 id=\"features\"\u003e✨ Features\u003c/h2\u003e\n\n- **Design Mode** - Describe what to test, AI explores and creates test scripts\n- **Execution Mode** - Run tests with real-time feedback and assertion handling\n- **Headless Mode** - Run tests in CI/CD pipelines\n- **Test Management** - Create, edit, view, and run test scripts\n- **Smart Actions** - Automatic wait detection and coordinate mapping\n\n---\n\n\u003ch2 id=\"usage\"\u003e📚 Usage\u003c/h2\u003e\n\n### Interactive Commands\n\n| Command | Description |\n|---------|-------------|\n| `/create \u003cname\u003e` | Create a new test |\n| `/run \u003cname\u003e` | Execute a test |\n| `/list` | List all tests |\n| `/view \u003cname\u003e` | View test contents |\n| `/edit \u003cname\u003e` | Edit a test |\n| `/help` | Show help |\n| `/exit` | Exit shell |\n\n### Creating Tests\n\n```sh\ndroid-cua\n\u003e /create login-test\n\u003e Test the login flow with valid credentials\n```\n\nThe AI will explore your app and generate a test script. Review and save it.\n\n### Running Tests\n\nInteractive:\n```sh\ndroid-cua\n\u003e /run login-test\n```\n\nHeadless (CI/CD):\n```sh\ndroid-cua --instructions tests/login-test.dcua\n```\n\n### Test Script Format\n\nOne instruction per line:\n\n```\nOpen the Calculator app\nassert: Calculator app is visible\nType \"2\"\nClick the plus button\nType \"3\"\nClick the equals button\nassert: result shows 5\nexit\n```\n\n\u003ch3 id=\"assertions\"\u003eAssertions\u003c/h3\u003e\n\nAssertions validate the app state during test execution. Add them anywhere in your test script.\n\n**Syntax** (all valid):\n```\nassert: the login button is visible\nAssert: error message appears\nASSERT the result shows 5\n```\n\n**Interactive Mode** - When an assertion fails:\n- `retry` - Retry the same assertion\n- `skip` - Continue to next instruction\n- `stop` - Stop test execution\n\n**Headless Mode** - Assertions fail immediately and exit with code 1.\n\n**Examples**:\n```\nassert: Calculator app is open\nassert: the result shows 8\nassert: error message is displayed in red\nassert: login button is enabled\n```\n\n---\n\n\u003ch2 id=\"command-line-options\"\u003e💻 Command Line Options\u003c/h2\u003e\n\n| Option | Description |\n|--------|-------------|\n| `--avd=NAME` | Specify emulator/simulator name |\n| `--platform=PLATFORM` | Force platform: `android` or `ios` |\n| `--instructions=FILE` | Run test headless |\n| `--record` | Save screenshots |\n| `--debug` | Enable debug logs |\n\n**Examples:**\n```sh\n# Interactive device selection\ndroid-cua\n\n# Android emulator\ndroid-cua --avd Pixel_8_API_35\n\n# iOS Simulator (auto-detected from name)\ndroid-cua --avd \"iPhone 16\"\n\n# iOS Simulator (explicit platform)\ndroid-cua --platform ios --avd \"iPhone 16\"\n\n# Headless CI mode\ndroid-cua --avd \"iPhone 16\" --instructions tests/login.dcua\n```\n\n---\n\n## Requirements\n\n**All platforms:**\n- Node.js 18.17.0+\n- OpenAI API Key (Tier 3 for computer-use-preview model)\n\n**Android:**\n- Android Debug Bridge (ADB)\n- Android Emulator (AVD)\n\n**iOS (macOS only):**\n- Xcode with iOS Simulator\n- Appium (`npm install -g appium`)\n- XCUITest driver (`appium driver install xcuitest`)\n\n---\n\n\u003ch2 id=\"how-it-works\"\u003e🔧 How It Works\u003c/h2\u003e\n\n1. Connects to Android emulator (via ADB) or iOS Simulator (via Appium)\n2. Captures full-screen device screenshots\n3. Scales down the screenshots for OpenAI model compatibility\n4. Sends screenshots and user instructions to OpenAI's computer-use-preview model\n5. Receives structured actions (click, scroll, type, keypress, wait, drag)\n6. Rescales model outputs back to real device coordinates\n7. Executes the actions on the device\n8. Validates assertions and handles failures\n9. Repeats until task completion\n\n---\n\n## 🎞️ Convert Screenshots to Video\n\nIf you run with `--record`, screenshots are saved to:\n```\ndroid-cua-recording-\u003ctimestamp\u003e/\n```\n\nConvert to video with ffmpeg:\n```sh\nffmpeg -framerate 1 -pattern_type glob -i 'droid-cua-recording-*/frame_*.png' \\\n  -vf \"pad=ceil(iw/2)*2:ceil(ih/2)*2\" \\\n  -c:v libx264 -pix_fmt yuv420p session.mp4\n```\n\n---\n\n\u003ch2 id=\"license\"\u003e📄 License\u003c/h2\u003e\n\n© 2025 Loadmill. All rights reserved.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floadmill%2Fdroid-cua","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Floadmill%2Fdroid-cua","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floadmill%2Fdroid-cua/lists"}