{"id":26255065,"url":"https://github.com/openai/openai-cua-sample-app","last_synced_at":"2025-05-15T00:06:50.152Z","repository":{"id":281978648,"uuid":"946708780","full_name":"openai/openai-cua-sample-app","owner":"openai","description":"Learn how to use CUA (our Computer Using Agent) via the API on multiple computer environments.","archived":false,"fork":false,"pushed_at":"2025-04-24T17:36:50.000Z","size":49,"stargazers_count":862,"open_issues_count":32,"forks_count":203,"subscribers_count":70,"default_branch":"main","last_synced_at":"2025-05-08T20:49:36.710Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-11T14:57:29.000Z","updated_at":"2025-05-08T19:30:44.000Z","dependencies_parsed_at":"2025-03-28T01:00:53.490Z","dependency_job_id":"53f9091b-5c81-45cd-8d49-f104b9d67ff4","html_url":"https://github.com/openai/openai-cua-sample-app","commit_stats":null,"previous_names":["openai/openai-cua-sample-app"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fopenai-cua-sample-app","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fopenai-cua-sample-app/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fopenai-cua-sample-app/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fopenai-cua-sample-app/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openai","download_url":"https://codeload.github.com/openai/openai-cua-sample-app/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254249198,"owners_count":22039029,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-13T19:17:08.875Z","updated_at":"2025-05-15T00:06:50.130Z","avatar_url":"https://github.com/openai.png","language":"Python","funding_links":[],"categories":["Openai","HarmonyOS","Runtime"],"sub_categories":["Windows Manager","Tool"],"readme":"# Computer Using Agent Sample App\n\nGet started building a [Computer Using Agent (CUA)](https://platform.openai.com/docs/guides/tools-computer-use) with the OpenAI API.\n\n\u003e [!CAUTION]  \n\u003e Computer use is in preview. Because the model is still in preview and may be susceptible to exploits and inadvertent mistakes, we discourage trusting it in authenticated environments or for high-stakes tasks.\n\n## Set Up \u0026 Run\n\nSet up python env and install dependencies.\n\n```shell\npython3 -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\nRun CLI to let CUA use a local browser window, using [playwright](https://playwright.dev/). (Stop with CTRL+C)\n\n```shell\npython cli.py --computer local-playwright\n```\n\n\u003e [!NOTE]  \n\u003e The first time you run this, if you haven't used Playwright before, you will be prompted to install dependencies. Execute the command suggested, which will depend on your OS.\n\nOther included sample [computer environments](#computer-environments):\n\n- [Docker](https://docker.com/) (containerized desktop)\n- [Browserbase](https://www.browserbase.com/) (remote browser, requires account)\n- [Scrapybara](https://scrapybara.com) (remote browser or computer, requires account)\n- ...or implement your own `Computer`!\n\n## Overview\n\nThe computer use tool and model are available via the [Responses API](https://platform.openai.com/docs/api-reference/responses). At a high level, CUA will look at a screenshot of the computer interface and recommend actions. Specifically, it sends `computer_call`(s) with `actions` like `click(x,y)` or `type(text)` that you have to execute on your environment, and then expects screenshots of the outcomes.\n\nYou can learn more about this tool in the [Computer use guide](https://platform.openai.com/docs/guides/tools-computer-use).\n\n## Abstractions\n\nThis repository defines two lightweight abstractions to make interacting with CUA agents more ergonomic. Everything works without them, but they provide a convenient separation of concerns.\n\n| Abstraction | File                    | Description                                                                                                                                                                                                  |\n| ----------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| `Computer`  | `computers/computer.py` | Defines a `Computer` interface for various environments (local desktop, remote browser, etc.). An implementation of `Computer` is responsible for executing any `computer_action` sent by CUA (clicks, etc). |\n| `Agent`     | `agent/agent.py`        | Simple, familiar agent loop – implements `run_full_turn()`, which just keeps calling the model until all computer actions and function calls are handled.                                                    |\n\n## CLI Usage\n\nThe CLI (`cli.py`) is the easiest way to get started with CUA. It accepts the following arguments:\n\n- `--computer`: The computer environment to use. See the [Computer Environments](#computer-environments) section below for options. By default, the CLI will use the `local-playwright` environment.\n- `--input`: The initial input to the agent (optional: the CLI will prompt you for input if not provided)\n- `--debug`: Enable debug mode.\n- `--show`: Show images (screenshots) during the execution.\n- `--start-url`: Start the browsing session with a specific URL (only for browser environments). By default, the CLI will start the browsing session with `https://bing.com`.\n\n### Run examples (optional)\n\nThe `examples` folder contains more examples of how to use CUA.\n\n```shell\npython -m examples.weather_example\n```\n\nFor reference, the file `simple_cua_loop.py` implements the basics of the CUA loop.\n\nYou can run it with:\n\n```shell\npython simple_cua_loop.py\n```\n\n## Computer Environments\n\nCUA can work with any `Computer` environment that can handle the [CUA actions](https://platform.openai.com/docs/api-reference/responses/object#responses/object-output) (plus a few extra):\n\n| Action                             | Example                         |\n| ---------------------------------- | ------------------------------- |\n| `click(x, y, button=\"left\")`       | `click(24, 150)`                |\n| `double_click(x, y)`               | `double_click(24, 150)`         |\n| `scroll(x, y, scroll_x, scroll_y)` | `scroll(24, 150, 0, -100)`      |\n| `type(text)`                       | `type(\"Hello, World!\")`         |\n| `wait(ms=1000)`                    | `wait(2000)`                    |\n| `move(x, y)`                       | `move(24, 150)`                 |\n| `keypress(keys)`                   | `keypress([\"CTRL\", \"C\"])`       |\n| `drag(path)`                       | `drag([[24, 150], [100, 200]])` |\n\nThis sample app provides a set of implemented `Computer` examples, but feel free to add your own!\n\n| Computer            | Option             | Type      | Description                       | Requirements                                                     |\n| ------------------- | ------------------ | --------- | --------------------------------- | ---------------------------------------------------------------- |\n| `LocalPlaywright`   | local-playwright   | `browser` | Local browser window              | [Playwright SDK](https://playwright.dev/)                        |\n| `Docker`            | docker             | `linux`   | Docker container environment      | [Docker](https://docs.docker.com/engine/install/) running        |\n| `Browserbase`       | browserbase        | `browser` | Remote browser environment        | [Browserbase](https://www.browserbase.com/) API key in `.env`    |\n| `ScrapybaraBrowser` | scrapybara-browser | `browser` | Remote browser environment        | [Scrapybara](https://scrapybara.com/dashboard) API key in `.env` |\n| `ScrapybaraUbuntu`  | scrapybara-ubuntu  | `linux`   | Remote Ubuntu desktop environment | [Scrapybara](https://scrapybara.com/dashboard) API key in `.env` |\n\nUsing the CLI, you can run the sample app with different computer environments using the options listed above:\n\n```shell\npython cli.py --show --computer \u003ccomputer-option\u003e\n```\n\nFor example, to run the sample app with the `Docker` computer environment, you can run:\n\n```shell\npython cli.py --show --computer docker\n```\n\n#### Contributed Computers\n\n| Computer | Option | Type | Description | Requirements |\n| -------- | ------ | ---- | ----------- | ------------ |\n| `tbd`    | tbd    | tbd  | tbd         | tbd          |\n\n\u003e [!NOTE]  \n\u003e If you've implemented a new computer, please add it to the \"Contributed Computers\" section of the README.md file. Clearly indicate any auth / signup requirements. See the [Contributing](#contributing) section for more details.\n\n### Docker Setup\n\nIf you want to run the sample app with the `Docker` computer environment, you need to build and run a local Docker container.\n\nOpen a new shell to build and run the Docker image. The first time you do this, it may take a few minutes, but subsequent runs should be much faster. Once the logs stop, proceed to the next setup step. To stop the container, press CTRL+C on the terminal where you ran the command below.\n\n```shell\ndocker build -t cua-sample-app .\ndocker run --rm -it --name cua-sample-app -p 5900:5900 --dns=1.1.1.3 -e DISPLAY=:99 cua-sample-app\n```\n\n\u003e [!NOTE]  \n\u003e We use `--dns=1.1.1.3` to restrict accessible websites to a smaller, safer set. We highly recommend you take similar safety precautions.\n\n\u003e [!WARNING]  \n\u003e If you get the below error, then you need to kill that container.\n\u003e\n\u003e ```\n\u003e docker: Error response from daemon: Conflict. The container name \"/cua-sample-app\" is already in use by container \"e72fcb962b548e06a9dcdf6a99bc4b49642df2265440da7544330eb420b51d87\"\n\u003e ```\n\u003e\n\u003e Kill that container and try again.\n\u003e\n\u003e ```shell\n\u003e docker rm -f cua-sample-app\n\u003e ```\n\n### Hosted environment setup\n\nThis repository contains example implementations of third-party hosted environments.\nTo use these, you will need to set up an account with the service by following the links aboveand add your API key to the `.env` file.\n\n## Function Calling\n\nThe `Agent` class accepts regular function schemas in `tools` – it will return a hard-coded value for any invocations.\n\nHowever, if you pass in any `tools` that are also defined in your `Computer` methods, in addition to the required `Computer` methods, they will be routed to your `Computer` to be handled when called. **This is useful for cases where screenshots often don't capture the search bar or back arrow, so CUA may get stuck. So instead, you can provide a `back()` or `goto(url)` functions.** See `examples/playwright_with_custom_functions.py` for an example.\n\n## Risks \u0026 Safety considerations\n\nThis repository provides example implementations with basic safety measures in place.\n\nWe recommend reviewing the best practices outlined in our [guide](https://platform.openai.com/docs/guides/tools-computer-use#risks-and-safety), and making sure you understand the risks involved with using this tool.\n\n# Contributing\n\n## Computers\n\nTo contribute a new computer, you'll need to implement it, test it, and submit a PR. Please follow the steps below:\n\n### 1. Implement your computer\n\nYou will create or modify the following files (and only these files):\n\n| File                                        | Updates            |\n| ------------------------------------------- | ------------------ |\n| `computers/contrib/[your_computer_name].py` | Add computer file. |\n| `computers/contrib/__init__.py`             | Add to imports.    |\n| `computers/config.py`                       | Add to config.     |\n| `README.md`                                 | Add to README.     |\n\nCreate a new file in `computers/contrib/[your_computer_name].py` and define your computer class. Make sure to implement the methods defined in the `Computer` class – use the existing implementations as a reference.\n\n```python\nclass YourComputerName:\n    def __init__(self):\n        pass\n\n    def screenshot(self):\n        # TODO: implement\n        pass\n\n    def click(self, x, y):\n        # TODO: implement\n        pass\n\n    # ... add other methods as needed\n```\n\n\u003e [!NOTE]  \n\u003e For playwright-based computers, make sure to subclass `BasePlaywrightComputer` in `computers/shared/base_playwright.py` – see `computers/default/browserbase.py` for an example.\n\nImport your new computer in the `computers/contrib/__init__.py`:\n\n```python\n# ... existing computer imports\nfrom .your_computer_name import YourComputerName\n```\n\nAnd add your new computer to the `computers_config` dictionary in `computers/config.py`:\n\n```python\n# ... existing computers_config\n\"your_computer_name\": YourComputerName,\n```\n\nFeel free to add your new computer to the \"Contributed Computers\" section of the README.md file. Clearly indicate any auth / signup requirements.\n\n### 2. Test your computer\n\nTest your new computer (with the CLI). Make sure:\n\n- Basic search / navigation works.\n- Any setup / teardown is handled correctly.\n- Test e2e with a few different tasks.\n\nPotential gotchas (See `default` computers for reference):\n\n- Scrolling, dragging, and control/command keys.\n- Resource allocation and teardown.\n- Auth / signup requirements.\n\n### 3. Submit a PR\n\nYour PR should clearly define the following:\n\n- Title: `[contrib] Add computer: \u003cyour_computer_name\u003e`\n- Description:\n\n```\n# Add computer: \u003cyour_computer_name\u003e\n\n#### Affiliations\n\nWhat organization / company / institution are you affiliated with?\n\n#### Computer Description\n\n- Computer type (e.g. browser, linux)\n\n#### Testing Plan\n\n- Signup steps.\n- Auth steps.\n- Sample queries.\n```\n\nThank you for your contribution! Please follow all of the above guidelines. Failure to do so may result in your PR being rejected.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Fopenai-cua-sample-app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenai%2Fopenai-cua-sample-app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Fopenai-cua-sample-app/lists"}