{"id":21644458,"url":"https://github.com/samsung/clickagent","last_synced_at":"2025-09-07T14:36:16.034Z","repository":{"id":258244453,"uuid":"872883832","full_name":"Samsung/ClickAgent","owner":"Samsung","description":"ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents","archived":false,"fork":false,"pushed_at":"2024-10-28T07:41:45.000Z","size":20489,"stargazers_count":14,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-31T12:57:14.201Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/pdf/2410.11872","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Samsung.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-15T08:37:14.000Z","updated_at":"2025-01-01T13:14:51.000Z","dependencies_parsed_at":"2024-10-18T05:45:52.355Z","dependency_job_id":null,"html_url":"https://github.com/Samsung/ClickAgent","commit_stats":null,"previous_names":["samsung/clickagent"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FClickAgent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FClickAgent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FClickAgent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FClickAgent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Samsung","download_url":"https://codeload.github.com/Samsung/ClickAgent/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931555,"owners_count":21827112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T05:40:19.006Z","updated_at":"2025-05-07T18:10:29.400Z","avatar_url":"https://github.com/Samsung.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./assets/title_image.png\" width=\"320\" height=\"320\"/\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003ch3\u003eEnhancing UI Location Capabilities of Autonomous Agents\u003c/h3\u003e\n\u003c/div\u003e\n  \n\u003cdiv align=\"center\"\u003e\n\u003chr\u003e\n\u003c/div\u003e\n\n  \n![](./assets/overview_architecture.png)\n## 📺 Demo\nhttps://github.com/user-attachments/assets/356ee6b2-86bb-417a-b24c-bbd206aafe2a\n\n## ☀️ Introduction\n\nWith the growing reliance on digital devices equipped with graphical user interfaces (GUIs), such as computers and smartphones, the need for effective automation tools has become increasingly important. Although multimodal large language models (MLLMs) like GPT-4V excel at tasks such as drafting emails, they struggle with GUI interactions, which limits their effectiveness in automating everyday tasks. In this paper, we introduce ClickAgent, a novel framework for building autonomous agents. In ClickAgent, the MLLM handles reasoning and action planning, while a separate UI location model (e.g., SeeClick) identifies the relevant UI elements on the screen. This approach addresses a key limitation of current-generation MLLMs: their difficulty in accurately locating UI elements.\n\nClickAgent significantly outperforms other prompt-based autonomous agents (such as CogAgent, AppAgent, and Auto-UI) on the AITW benchmark. Our evaluation was conducted on both an Android smartphone emulator and an actual Android smartphone, using the task success rate as the key metric for measuring agent performance.\n\n\n## 🚀 Getting Started\n\n### 🔧 Installation\n```bash\npip install -r requirements.txt\n```\n\n### 🤖 Android Environment Setup\n\n1. Download the [Android Debug Bridge](https://developer.android.com/tools/releases/platform-tools?hl=en).\n2. Turn on the ADB debugging switch on your Android phone, it needs to be turned on in the developer options first.If it is the HyperOS system, you need to turn on USB Debugging (Security Settings) at the same time.\n3. Connect your phone to the computer with a data cable and select \"Transfer files\".\n4. Test your ADB environment as follow: `/path/to/adb devices`. If the connected devices are displayed, the preparation is complete.\n5. If you are using a MAC or Linux system, make sure to turn on adb permissions as follow: `sudo chmod +x /path/to/adb`\n6. If you are using Windows system, the path will be `xx\\xx\\adb.exe`\n\nIn case you need additional information about ADB, you can find it here: [ADB Docs](https://developer.android.com/tools/adb)\n  \n#### 📋 Used apps\n\n\u003cdetails\u003e\n  \u003csummary\u003eShow list\u003c/summary\u003e\n\n    1. Clock\n    2. Calendar\n    3. Files\n    4. Messages\n    5. Contacts\n    6. Calculator\n    7. Settings\n    8. Gmail\n    9. Google Chrome\n    10. Google Maps\n    11. Google Play\n    12. Google Movies\n    13. Google Photos\n    14. YouTube \n    15. YouTube Music\n    16. Netflix\n    17. Spotify\n    18. Amazon Alexa\n    19. Amazon Music\n    20. Amazon Prime\n    21. X (Twitter)\n    22. Facebook\n    23. Instagram\n    24. Pandora\n    25. Yahoo\n    26. Yelp\n    27. eBay\n    28. Wikipedia\n\n\u003c/details\u003e\n\n#### 📱 Emulator\n\n1. Download and install [Android Studio](https://developer.android.com/studio)\n2. Tools -\u003e Device Manager\n3. Create Virtual Device\n4. Information about device, such as AVD ID and snapshot names you can find in details of created emulator\n\n### ⚙️ Configuration\n\nAll configurations are stored in `config.ini`. Every parameter can be overridden as a command line argument. \n\nList of arguments:\n```commandline\nrun.py \n--config-path path to configuration file\n\n--instruction instuction\n--action-file folder name in eval-save-folder where output will be saved\n\n--qwen IP:PORT Add port to host_api_worker.py default 21002\n--internvl IP:PORT\n--florence IP:PORT\n\n--use-eval True/False use reflection module\n--use-florence-only True/False use only UI Location Module\n\n--do-stop add to prompt STOP action\n--option \"1/2\" decision prompt choice\n\n--adb-path path to adb\n--aapt-path path to aapt\n--emu-path path ot emulator\n--device-type real or emu\n--device-id id of device\n--avd-name name of emulator AVD\n--snapshot run specified snapshot\n--run-apps-through-adb run applications directly with app id using adb if present\n\n--max-steps max number of steps per instruction \n--eval-save-folder folder name where output will be saved\n```\n\n### 🔬 Run Experiments\n\nAfter modifying the config to what you like, you can now run experiments with the following commands:\n\n```shell\ncd tests\npython run_test.py\n```\n\n### 📐 Main Results Reproduction\n\nTo reproduce the results shown in Table 1 of our paper, you must first host TinyClick, server and InternVL ([vLLM](https://docs.vllm.ai/en/latest/index.html) is recommended).\n\nAfter this use default values of [`config.ini`](config.ini).\n\n\n#### TinyClick server\n```shell\ncd api\npython host_florence.py\n```\n\n#### Host server\n```shell\ncd api\npython host_api_worker.py\n```\n\n#### InternVL2-Llama3-76B server \n```shell\npip install vllm\nvllm serve OpenGVLab/InternVL2-Llama3-76B --served-model-name internlm2 --tensor-parallel-size 4 \n```\n\n### 🏃 Run\n```shell\npython run.py\n```\n\n## 📝 To-Do List\n- [ ] Evaluate the ClickAgent on open-source models (e.g. GPT-4V)\n\n## 📖 Citation\n```bib\n@misc{hoscilowicz2024clickagent,\n    title={ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents},\n    author={Jakub Hoscilowicz and Bartosz Maj and Bartosz Kozakiewicz and Oleksii Tymoshchuk and Artur Janicki},\n    year={2024},\n    eprint={2410.11872},\n    archivePrefix={arXiv},\n    primaryClass={cs.HC}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamsung%2Fclickagent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsamsung%2Fclickagent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamsung%2Fclickagent/lists"}