acu
A curated list of resources about AI agents for Computer Use, including research papers, projects, frameworks, and tools.
https://github.com/trycua/acu
Last synced: 22 days ago
JSON representation
Articles
Contributing
Frameworks & Models
Papers
Benchmark
- A3: Android Agent Arena for Mobile GUI Agents
- Website
- Code
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- Code
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
- Code
- Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
- Code
- AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
- Code
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Code
- MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents
- Code
- VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
- Code
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
- Code
- Website
- Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction
- Code
Dataset
- OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
- Code
- AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
- ICAL: Continual Learning of Multimodal Agents by Transforming Trajectories into Actionable Insights
- Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale
- Multi-Turn Mind2Web: On the Multi-turn Instruction Following
- Code
- CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation
- Code
- AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks
- Mind2Web: Towards a Generalist Agent for the Web
- Code
- Android in the Wild: A Large-Scale Dataset for Android Device Control
- WebShop: Towards Scalable Real-World Web Interaction
- Code
- Rico: A Mobile App Dataset for Building Data-Driven Design Applications
- Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents
- UiPad: UI Parsing and Accessibility Dataset
- UiPad: UI Parsing and Accessibility Dataset
Frameworks & Models
- Reinforcement Learning for Long-Horizon Interactive LLM Agents
- Large Action Models: From Inception to Implementation
- Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigation
- SpiritSight Agent: Advanced GUI Agent with One Look
- AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs
- Simulate Before Act: Model-Based Planning for Web Agents
- Proposer-Agent-Evaluator (PAE): Autonomous Skill Discovery For Foundation Model Internet Agents
- Code
- Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents
- Digi-Q: Transforming VLMs to Device-Control Agents via Value-Based Offline RL
- Magentic-One
- Agent Workflow Memory
- Code
- The Impact of Element Ordering on LM Agent Performance
- Code
- Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
- OpenWebAgent: An Open Toolkit to Enable Web Agents on Large Language Models
- Code
- Agent-e: From autonomous web navigation to foundational design principles in agentic systems
- Apple Intelligence Foundation Language Models
- Tree search for language model agents
- DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
- Code
- Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
- Code
- Octopus Series: On-device Language Models for Computer Control
- Website
- AutoWebGLM: Bootstrap and reinforce a large language model-based web navigating agent
- Code
- Cradle: Empowering Foundation Agents towards General Computer Control
- Code
- Android in the Zoo: Chain-of-Action-Thought for GUI Agents
- Code
- ScreenAgent: A Computer Control Agent Driven by Visual Language Large Model
- Code
- OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
- Code
- UFO: A UI-Focused Agent for Windows OS Interaction
- Code
- CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation
- Intention-inInteraction (IN3): Tell Me More!
- Dual-view visual contextualization for web navigation
- ScreenAI: A Vision-Language Model for UI and Infographics Understanding
- GPT-4V(ision) is a Generalist Web Agent, if Grounded
- Code
- Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
- WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
- Code
- CogAgent: A Visual Language Model for GUI Agents
- Code
- AppAgent: Multimodal Agents as Smartphone Users
- LASER: LLM Agent with State-Space Exploration for Web Navigation
- Code
- AndroidEnv: A Reinforcement Learning Platform for Android
- Code
- Code
- Website
Safety
- Attacking Vision-Language Computer Agents via Pop-ups
- Code
- EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
- GuardAgent: Safeguard LLM Agent by a Guard Agent via Knowledge-Enabled Reasoning
Surveys
- AI Agents for Computer Use: A Review of Instruction-based Computer Control, GUI Automation, and Operator Assistants
- GUI Agents: A Survey
- Large Language Model-Brained GUI Agents: A Survey
- Website
- GUI Agents with Foundation Models: A Comprehensive Survey
UI Grounding
- OmniParser for Pure Vision Based GUI Agent
- Code
- Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
- Code
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
- Code
- Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
- Code
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents
- Code
- UI-Pro: A Hidden Recipe for Building Vision-Language Models for GUI Grounding
- Grounding Multimodal Large Language Model in GUI World
Projects
Automation
Environment & Sandbox
Frameworks & Models
- Anthropic Claude Computer Use
- AutoGen
- Browser Use
- Claude Computer Use Demo
- Claude Minecraft Use
- Computer Use OOTB
- Cybergod
- Grunty
- Inferable
- LaVague
- Mac Computer Use
- NatBot
- OpenAdapt
- OpenInterface
- OpenInterpreter
- Open Source Computer Use by E2B
- Self-Operating Computer
- Skyvern
- Surfkit
- WebMarker
- Runner H
- Upsonic
- Multion
- Notte Browser Using Agent
- Auto-GPT
- OpenInterface