{"id":19401018,"url":"https://github.com/google-research/android_world","last_synced_at":"2025-05-16T15:03:33.107Z","repository":{"id":241601030,"uuid":"800171422","full_name":"google-research/android_world","owner":"google-research","description":"AndroidWorld is an environment and benchmark for autonomous agents","archived":false,"fork":false,"pushed_at":"2025-04-11T20:23:31.000Z","size":21734,"stargazers_count":265,"open_issues_count":30,"forks_count":33,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-12T23:38:01.483Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-13T20:49:25.000Z","updated_at":"2025-04-11T20:23:34.000Z","dependencies_parsed_at":"2024-05-29T08:02:27.153Z","dependency_job_id":"09f293c3-5f3d-4c5c-aa6d-62c64f3d3a52","html_url":"https://github.com/google-research/android_world","commit_stats":null,"previous_names":["google-research/android_world"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fandroid_world","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fandroid_world/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fandroid_world/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fandroid_world/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/android_world/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248647255,"owners_count":21139081,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T11:16:43.385Z","updated_at":"2025-05-16T15:03:33.085Z","avatar_url":"https://github.com/google-research.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["Benchmark"],"readme":"# AndroidWorld\n\n[![Unittests](https://github.com/google-research/android_world/actions/workflows/pytest.yml/badge.svg)](https://github.com/google-research/android_world/actions/workflows/pytest.yml)\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://google-research.github.io/android_world/\"\u003eWebsite\u003c/a\u003e •\n\u003ca href=\"https://arxiv.org/pdf/2405.14573\"\u003ePaper\u003c/a\u003e •\n\u003ca href=\"https://google-research.github.io/android_world/task_list.html\"\u003eTasks\u003c/a\u003e •\n\u003ca href=\"https://docs.google.com/spreadsheets/d/1cchzP9dlTZ3WXQTfYNhh3avxoLipqHN75v1Tb86uhHo/edit?gid=0#gid=0\"\u003eLeaderboard\u003c/a\u003e\n\u003c/p\u003e\n\n![Overview](assets/overview.png)\n\n**AndroidWorld** is an environment for building and benchmarking autonomous computer control agents.\n\nIt runs on a live Android emulator and contains a highly reproducible benchmark of 116 hand-crafted tasks across 20 apps, which are dynamically instantiated with randomly-generated parameters to create millions of unique task variations.\n\nIn addition to the built-in tasks, AndroidWorld also supports the popular web benchmark, MiniWoB++ from [Liu et al.](http://arxiv.org/abs/1802.08802).\n\nKey features of AndroidWorld include:\n\n* 📝 **116 diverse tasks** across 20 real-world apps\n* 🎲 **Dynamic task instantiation** for millions of unique variations\n* 🏆 **Durable reward signals** for reliable evaluation\n* 🌐 **Open environment** with access to millions of Android apps and websites\n* 💾 **Lightweight footprint** (2 GB memory, 8 GB disk)\n* 🔧 **Extensible design** to easily add new tasks and benchmarks\n* 🖥️ **Integration with MiniWoB++** web-based tasks\n\nSee demo videos on our [website](https://google-research.github.io/android_world/).\n\n## Installation\n\n1. Set up the Android Emulator\n   1. Download Android Studio [here](https://developer.android.com/studio?gad_source=1\u0026gclid=Cj0KCQjw3ZayBhDRARIsAPWzx8oLcadBD0vAq8xmUutaunLGSzhgEtLz4xVZ_SpV4G0xJazS7LxQkDsaAuveEALw_wcB\u0026gclsrc=aw.ds)\n   2. Create an Android Virtual Device (AVD) by following these instructions. For hardware select **Pixel 6**, for System Image select **Tiramisu, API Level 33**, and choose AVD name as **AndroidWorldAvd**. [Watch the setup video.](https://github.com/google-research/android_world/assets/162379927/efc33980-8b36-44be-bb2b-a92d4c334a50)\n\n1. Launch the Android Emulator from the command line\n\n    Launch the emulator from the command line, not using the Android Studio UI, with the `-grpc 8554` flag which is needed communication with accessibility forwarding app.\n\n    ```bash\n    # Typically it's located in ~/Android/Sdk/emulator/emulator or\n    # ~/Library/Android/sdk/emulator/emulator\n    EMULATOR_NAME=AndroidWorldAvd # From previous step\n    ~/Library/Android/sdk/emulator/emulator -avd $EMULATOR_NAME -no-snapshot -grpc 8554\n    ```\n\n1. [Optional] It's recommended to use `conda`, which you can download [here](https://docs.anaconda.com/free/miniconda/miniconda-install/).\n\n    ```\n    conda create -n android_world python=3.11.8\n    conda activate android_world\n    ```\n\n1. Install AndroidWorld. *Note: Python 3.11 or above is required.*\n\n    ```python\n    git clone https://github.com/google-research/android_world.git\n    cd ./android_world\n    pip install -r requirements.txt\n    python setup.py install\n    ```\n\n1. Add model provider APIs as environment variables.\n\n    ```bash\n    # Add to .bashrc.\n    export OPENAI_API_KEY=your-key\n    export GCP_API_KEY=your-key\n    ```\n\n1. Install `ffmpeg`, if not already installed.\n\n    ```bash\n    # Linux (Ubuntu/Debian)\n    # sudo apt update \u0026\u0026 sudo apt install ffmpeg\n\n    # macOS\n    brew install ffmpeg\n    ```\n\n## Quickstart\n\nRun the `minimal_task_runner.py` script to see the basic mechanics of AndroidWorld components. It initializes the environment, sets up a task, and runs the default agent, M3A, on it.\n```bash\npython minimal_task_runner.py --task=ContactsAddContact\n```\n\nIf you don't specify a task, a random task will be selected. *NOTE: If you want to try open-source apps, i.e. not included with Android OS, please run `--perform_emulator_setup` in the script below.*\n\n## Run the benchmark\n\nNote: **Task Step Limits Update**\nAs of 11/18/2024, the max_steps/step_budget for each task in AndroidWorld have been updated to approximately **2x the human average completion time**. This adjustment ensures agents have sufficient time to complete tasks, while also reducing overhead of running thebenchmark. [Here](https://docs.google.com/spreadsheets/d/1KF-vY0Uy47o0mnursvs-HmS6hreU6U3rPrAjgEfjMK4/edit?usp=sharing) are the per-task updates.\n\n```bash\npython run.py \\\n  --suite_family=android_world \\\n  --agent_name=t3a_gpt4 \\\n  --perform_emulator_setup \\\n  --tasks=ContactsAddContact,ClockStopWatchRunning \\  # Optional: Just run on a subset.\n```\n\nThe first time you run this script, you must install the necessary apps and set permissions by specifying `--perform_emulator_setup`. This is a one-time setup. It may take several minutes depending on the connection speed.\n\nAbove we specify the optional `--tasks` flag to run on a subset of tasks. Leave it empty to run on the entire AndroidWorld suite.\n\nThe `n_task_combinations` argument specifies how many parameter permutations to use for each task. For example, for an SMS task, it would correspond to different phone number/message combinations for each run.\n\nIf a run fails part-way through, you can resume it by re-running the script with the `--checkpoint_dir` flag pointing to the output directory from the original run.\n\n## Running MiniWoB++ tasks\n\nTo run the MiniWoB++ web-based tasks in AndroidWorld, simply set\n`--suite_family=miniwob` and `--perform_emulator_setup` in the command above.\n\nA key advantage of running MiniWoB++ tasks is that common input elements are\nrendered as native, commonly used Android UI widgets, rather than as HTML. Thus agents must learn to use universal\nwidgets such as time- and date-pickers:\n\n\u003cp align=\"center\"\u003e\n   \u003cimg src=\"assets/miniwob.png\" style=\"width:30%\"\u003e\n\u003c/p\u003e\n\n## Create your own agent\n\nIn addition to the agents we provide [here](https://github.com/google-research/android_world/tree/main/android_world/agents), you can also easily create your own agent and run the benchmark with it as follows.\n\n1. Create an agent class that inherits from [EnvironmentInteractingAgent](https://github.com/google-research/android_world/blob/6e4feb00702735c9a7485f4ae714528a058cb2b7/android_world/agents/base_agent.py#L39C1-L39C44) and implement the [step](https://github.com/google-research/android_world/blob/6e4feb00702735c9a7485f4ae714528a058cb2b7/android_world/agents/base_agent.py#L116) method.\nIn the current workflow, the agent tries to complete a task in a for loop. In each round, the [step](https://github.com/google-research/android_world/blob/6e4feb00702735c9a7485f4ae714528a058cb2b7/android_world/agents/base_agent.py#L116) method will be called and this is where you implement your agent's logic. A typical approach involves first gathering information like the current screenshot, the UI elements (like buttons, icons) through the AndroidEnv instance within the agent, selecting one of the [supported actions](https://github.com/google-research/android_world/blob/main/android_world/env/json_action.py), executing it through the AndroidEnv and returning an [AgentInteractionResult](https://github.com/google-research/android_world/blob/6e4feb00702735c9a7485f4ae714528a058cb2b7/android_world/agents/base_agent.py#L26). The `done` property on AgentInteractionResult should be set to true to indicate that the task is finished.\n\n2. Import your agent in [run.py](https://github.com/google-research/android_world/blob/main/run.py) and also add it into the [_get_agent](https://github.com/google-research/android_world/blob/15471441ac306ff08bca87454b1b546ae81db7af/run.py#L147) method which takes in your agent's name and return an instance of it.\n\n3. Now you can run the benchmark with your new agent using the command above with the `agent_name` flag changed to your agent's name.\n\n## Adding new tasks\n\nPlease see [the guide](https://github.com/google-research/android_world/blob/main/docs/tasks_guide.md) on adding new tasks to AndroidWorld.\n\n## Citation\n\nIf you use our environment or data, please cite our paper:\n\n```\n@misc{rawles2024androidworlddynamicbenchmarkingenvironment,\n      title={AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents},\n      author={Christopher Rawles and Sarah Clinckemaillie and Yifan Chang and Jonathan Waltz and Gabrielle Lau and Marybeth Fair and Alice Li and William Bishop and Wei Li and Folawiyo Campbell-Ajala and Daniel Toyama and Robert Berry and Divya Tyamagundlu and Timothy Lillicrap and Oriana Riva},\n      year={2024},\n      eprint={2405.14573},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2405.14573},\n}\n```\n\n*This is not an officially supported Google product.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fandroid_world","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fandroid_world","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fandroid_world/lists"}