{"id":32521547,"url":"https://github.com/opengvlab/scalecua","last_synced_at":"2025-10-28T06:29:43.451Z","repository":{"id":315517341,"uuid":"1053524611","full_name":"OpenGVLab/ScaleCUA","owner":"OpenGVLab","description":"ScaleCUA is the open-sourced computer use agents that can operate on corss-platform environments (Windows, macOS, Ubuntu, Android).","archived":false,"fork":false,"pushed_at":"2025-09-30T14:43:37.000Z","size":47799,"stargazers_count":396,"open_issues_count":1,"forks_count":21,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-09-30T15:29:34.463Z","etag":null,"topics":["computer-use-agents","data","gui-agents","models","online-evaluation-suite","scalecua"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenGVLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-09T15:08:00.000Z","updated_at":"2025-09-30T15:17:20.000Z","dependencies_parsed_at":"2025-09-19T04:38:20.703Z","dependency_job_id":null,"html_url":"https://github.com/OpenGVLab/ScaleCUA","commit_stats":null,"previous_names":["opengvlab/scalecua"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/OpenGVLab/ScaleCUA","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FScaleCUA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FScaleCUA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FScaleCUA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FScaleCUA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenGVLab","download_url":"https://codeload.github.com/OpenGVLab/ScaleCUA/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FScaleCUA/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281397339,"owners_count":26493908,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-28T02:00:06.022Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-use-agents","data","gui-agents","models","online-evaluation-suite","scalecua"],"created_at":"2025-10-28T06:29:41.521Z","updated_at":"2025-10-28T06:29:43.445Z","avatar_url":"https://github.com/OpenGVLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data\n\n\u003cp align=\"center\"\u003e\n\u0026nbsp\u0026nbsp📑 \u003ca href=\"https://arxiv.org/abs/2509.15221\"\u003ePaper\u003c/a\u003e\u0026nbsp\u0026nbsp | \u0026nbsp\u0026nbsp🤗 \u003ca href=\"https://huggingface.co/datasets/OpenGVLab/ScaleCUA-Data\"\u003eDataset\u003c/a\u003e\u0026nbsp\u0026nbsp | \u0026nbsp\u0026nbsp🤖 \u003ca href=\"https://huggingface.co/collections/OpenGVLab/scalecua-68c912cf56f7ff4c8e034003\"\u003eModel\u003c/a\u003e\u0026nbsp\u0026nbsp | \u0026nbsp\u0026nbsp🖥️  \u003ca href=\"https://github.com/OpenGVLab/OpenCUA\"\u003eModel Demo\u003c/a\u003e\u0026nbsp\u0026nbsp \n\u003c/p\u003e\n\n\u003cdiv style=\"max-width:900px;margin:0 auto;\"\u003e\n\nVision-Language Models (VLMs) have enabled computer use agents (**CUAs**) that operate GUIs autonomously with great potential. \nHowever, developing robust CUAs requires extensive in-domain knowledge about software interfaces and operations. \nUnlike image–text pairs that are widely available on the Internet, computer-use data, particularly operation trajectories, are rare, costly to collect. \nConsequently, progress in this field remains constrained by both data scale and the limited transferability of existing VLMs. \nIn this work, we introduce **ScaleCUA**, a step toward scaling open-source CUAs. It offers a large-scale dataset spanning 6 operating systems and 3 task domains, via a closed-loop pipeline uniting automated agents with human experts. Trained on this scaled-up data, ScaleCUA can operate seamlessly across platforms. \nSpecifically, it delivers strong gains over baselines (+**26.6** on WebArena-Lite-v2, +**10.7** on ScreenSpot-Pro) and sets new state-of-the-art results (**94.4**\\% on MMBench-GUI L1-Hard, **60.6**\\% on OSWorld-G, **47.4**\\% on WebArena-Lite-v2). These findings underscore the power of data-driven scaling for general-purpose cross-platform CUAs. \n\n\u003cimg width=\"4521\" height=\"3068\" alt=\"teaser\" src=\"https://github.com/user-attachments/assets/4ae7dd58-3747-427b-bf9c-73ebb1dd1f18\" /\u003e\n\n## 🤖 Video Demo\n**\u003ca id=\"draggan_demo\"\u003e\u003c/a\u003e**\n\nhttps://github.com/user-attachments/assets/73ee1048-5ef6-420d-8078-e73894dfa1db\n\n\n\n\n## 📋 Table of Contents\n- [ScaleCUA: An Open-Source Agent for Cross-Platform GUI Automation](#scalecua-an-open-source-agent-for-cross-platform-gui-automation)\n  - [📋 Table of Contents](#-table-of-contents)\n  - [🎉 News](#-news)\n  - [🚀 Key Features](#-key-features)\n  - [📂 Project Structure](#-project-structure)\n  - [⚙️ Setup](#️-setup)\n  - [🎮 Playground](#-playground)\n  - [📊 Evaluation](#-evaluation)\n  - [💻 Training](#-training)\n  - [💐 Acknowledgements](#-acknowledgements)\n  - [⚖️ License](#️-license)\n  - [📜 Citation](#-citation)\n\n## 🎉 News\n- `2025/09/30`: ScaleCUA-Demo on HuggingFace is under construction. Welcome to try our demo!\n- `2025/09/30`: ScaleCUA-Data is ready on HuggingFace.\n- `2025/09/19`: We have released models and code of ScaleCUA.\n\n## 🚀 Key Features\n\n  * **ScaleCUA-Data:** A large-Scale cross-platform dataset spanning 6 operating systems and 3 GUI-centric task domains.\n  * **ScaleCUA-Models:** An cross-platform general-purpose agent that excels at GUI-centric task completion on various environments.\n  * **SFT Codebase:** A comprehensive training framework that supports training computer use agent based on Qwen2.5-VL and InternVL.\n  * **Interactive Playground:** A series of realistic, interactive environments across Ubuntu, Android, and Web.\n  * **Online Evaluation Suite:** A set of online benchmarks to evaluate agents' capabilities of task completion on various platforms.\n\n\u003cimg width=\"2828\" height=\"1825\" alt=\"infer_mode\" src=\"https://github.com/user-attachments/assets/cccde5ba-641e-4e9f-8fdd-7628c0b5f4f3\" /\u003e\n\n\n## 📂 Project Structure\n\nThis repository is organized into three main components:\n\n  * **[`evaluation`](./evaluation)**: Contains all the code and benchmarks for the end-to-end evaluation of our agents.\n  * **[`playground`](./playground)**: Provides interactive environments (Android, Ubuntu, Web) and model implementations for users to experience the agent's capabilities firsthand.\n  * **[`agent-sft`](./agent-sft)**: Includes the training code, configurations, and instructions needed to reproduce ScaleCUA on the ScaleCUD dataset.\n\n## ⚙️ Setup\n\n1.  **Clone the repository:**\n    ```bash\n    git clone https://github.com/OpenGVLab/ScaleCUA.git\n    cd ScaleCUA\n    ```\n2.  **Install dependencies:**\n    ```bash\n    pip install -r requirements.txt\n    ```\n\n## 🎮 Playground\n\nThe Playground allows you to interactively experience the ScaleCUA agent's capabilities firsthand across Ubuntu, Android, and Web platforms. For a complete guide, please see the [[Playground]](playground/README.md).\n\nFollow these two steps to begin:\n1.  Deploy the ScaleCUA models with [vLLM](https://github.com/vllm-project/vllm) following our [[Model Deployment]](evaluation/README.md#-model-development). We support two modes of operation: **Native Agentic Model** using a single model for both UI grounding and planning, and **Agentic Workflow** supporting two different models for UI planning and grounding.\n\n2. Set up your environment following [[Playground Environment]](playground/README.md#environment-configuration). We provide pre-configured, interactive virtual machines for Ubuntu, Android, and Web to simplify this process.\n\nNow, you can try our agent in the interactive environment!\n\n\n## 📊 Evaluation\n\nWe provide a suite of benchmarks for end-to-end agent evaluation using a vision-only setup. ScaleCUA support using **[vLLM](https://github.com/vllm-project/vllm)** to deploy and evaluate it through an OpenAI-compatible API. To run the evaluation benchmarks, please refer to the specific instructions within the [[Evaluation]](evaluation/README.md).\n\nOur evaluation suite covers desktop, mobile, and web environments:\n\n  * **Android**: `AndroidWorld`, `AndroidLab`\n  * **Ubuntu**: `OSWorld`\n  * **macOS**: `MacOSArena`\n  * **Web**: `WebArenaLite-v2` (A refined version of WebArena-Lite suitable for visual-based agents)\n  * **Windows**: `WindowsAgentArena`\n\n## 🧠 Training\n\nThe directory `agent-sft/` contains all the necessary code and configuration files to train the ScaleCUA from scratch using our ScaleCUA-Data. We support training Computer Use Agents with both [InternVL](agent-sft/internvl_chat/README.md) and [Qwen-VL](agent-sft/qwen-vl-finetune/README.md).\n\n\n## 💐 Acknowledgements\nThanks to the following open-sourced projects:\n\n[OSWorld](https://github.com/xlang-ai/OSWorld) \u0026#8194; \n[WindowsAgentArena](https://github.com/microsoft/WindowsAgentArena) \u0026#8194; \n[WebArena](https://github.com/web-arena-x/webarena) \u0026#8194; \n[AndroidWorld](https://github.com/google-research/android_world)\u0026#8194; \n[ScienceBoard](https://github.com/OS-Copilot/ScienceBoard)\u0026#8194; \n[MMBench-GUI](https://github.com/open-compass/MMBench-GUI) \u0026#8194;\n[AGUVIS](https://github.com/xlang-ai/aguvis)\u0026#8194; \n[Qwen-VL](https://github.com/QwenLM/Qwen2.5-VL) \u0026#8194; \n[InternVL](https://github.com/OpenGVLab/InternVL) \u0026#8194;\n\n\n\n## ⚖️ License\n\nThis project is licensed under the [Apache 2.0 License](https://www.google.com/search?q=LICENSE).\n\n## 📜 Citation\n\nIf you find our work useful, please consider citing our paper:\n\n```bibtex\n@article{liu2025scalecua,\n  title        = {ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data},\n  author       = {Liu, Zhaoyang and Xie, Jingjing and Ding, Zichen and Li, Zehao and Yang, Bowen and Wu, Zhenyu and Wang, Xuehui and Sun, Qiushi and Liu, Shi and Wang, Weiyun and Ye, Shenglong and Li, Qingyun and Dong, Xuan and Yu, Yue and Lu, Chenyu and Mo, YunXiang and Yan, Yao and Tian, Zeyue and Zhang, Xiao and Huang, Yuan and Liu, Yiqian and Su, Weijie and Luo, Gen and Yue, Xiangyu and Qi, Biqing and Chen, Kai and Zhou, Bowen and Qiao, Yu and Chen, Qifeng and Wang, Wenhai},\n  journal      = {arXiv preprint arXiv:2509.15221},\n  year         = {2025},\n  note         = {Preprint},\n  url          = {https://github.com/OpenGVLab/ScaleCUA}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopengvlab%2Fscalecua","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopengvlab%2Fscalecua","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopengvlab%2Fscalecua/lists"}