awesome-gui-agent
💻 A curated list of papers and resources for multi-modal Graphical User Interface (GUI) agents.
https://github.com/showlab/awesome-gui-agent
Last synced: 24 days ago
JSON representation
Acknowledgements
- Awesome-Video-Diffusion - MLLM-Hallucination](https://github.com/showlab/Awesome-MLLM-Hallucination).
Datasets / Benchmarks
- World of Bits: An Open-Domain Platform for Web-Based Agents
- Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration
- ![Star
- Mapping Natural Language Instructions to Mobile UI Action Sequences
- ![Star - tensorflow)
- WebSRC: A Dataset for Web-Based Structural Reading Comprehension
- AndroidEnv: A Reinforcement Learning Platform for Android
- ![Star
- A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility
- META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
- ![Star - nlp/WebShop)
- Language Models can Solve Computer Tasks
- ![Star - agent)
- Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction
- ![GitHub - LANCE/Mobile-Env)
- Mind2Web: Towards a Generalist Agent for the Web
- ![Star - nlp-group/mind2web)
- Android in the Wild: A Large-Scale Dataset for Android Device Control
- WebArena: A Realistic Web Environment for Building Autonomous Agents
- ![Star - arena-x/webarena)
- Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large Language Models
- ![Star
- AssistGUI: Task-Oriented Desktop Graphical User Interface Automation
- VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
- OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
- WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
- ![Star - nlp/weblinx)
- On the Multi-turn Instruction Following for Conversational Web Agents
- ![Star - map)
- AgentStudio: A Toolkit for Building General Virtual Agents
- ![Star - studio)
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- ![Star - ai/OSWorld)
- Benchmarking Mobile Device Control Agents across Diverse Configurations
- ![Star - moca)
- MMInA: Benchmarking Multihop Multimodal Internet Agents
- ![Star
- Autonomous Evaluation and Refinement of Digital Agents
- ![Star - NLP/Agent-Eval-Refine)
- LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Automation Task Evaluation
- ![Star
- VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
- GUICourse: From General Vision Language Models to Versatile GUI Agents
- ![Star
- GUI-WORLD: A Dataset for GUI-oriented Multimodal LLM-based Agents
- ![Star - Chen/GUI-World)
- GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
- ![Star - Odyssey)
- VideoGUI: A Benchmark for GUI Automation from Instructional Videos
- ![Star
- Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
- ![Star - ai-lab/Screen-Point-and-Read)
- MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents
- ![Star - agent-bench)
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
- ![Star - research/android_world)
- Practical, Automated Scenario-based Mobile App Testing
- WebCanvas: Benchmarking Web Agents in Online Environments
- On the Effects of Data Scale on Computer Control Agents
- ![Star - research/google-research/tree/master/android_control)
- CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents
- ![Star - ai/crab)
- WebVLN: Vision-and-Language Navigation on Websites
- ![Star
- Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
- ![Star - ai/Spider2-V)
- AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
- ![Star - codebase)
- Windows Agent Arena
- ![Star
- Harnessing Webpage UIs for Text-Rich Visual Understanding
- ![Star
- A Unified Solution for Structured Web Data Extraction
- Rico: A Mobile App Dataset for Building Data-Driven Design Applications
- GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent
- ![Star - ACES-ISE/ChatUITest)
- OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
- ![Star - Copilot/OS-Genesis)
- A3: Android Agent Arena for Mobile GUI Agents
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
- ![Star - Pro-GUI-Grounding)
- ![Star - nlp/WebWalker)
- SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation
- WorldGUI: Dynamic Testing for Comprehensive Desktop GUI Automation
- ![Star - Thinker)
- LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark
- ![Star
- ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
- ![Star - Copilot/ScienceBoard)
- MONDAY: Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents
- ![Star
- Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System
- ![Star
Models / Agents
- ![arXiv
- ![Website - nlp-group.github.io/SeeAct/)
- Grounding Open-Domain Instructions to Automate Web Support Tasks
- Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
- A Data-Driven Approach for Learning to Control Computers
- Augmenting Autotelic Agents with Large Language Models
- Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
- ![Star
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
- LASER: LLM Agent with State-Space Exploration for Web Navigation
- CogAgent: A Visual Language Model for GUI Agents
- WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
- ![Star
- OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
- ![Star - Copilot/OS-Copilot)
- UFO: A UI-Focused Agent for Windows OS Interaction
- ![Star
- Comprehensive Cognitive LLM Agent for Smartphone GUI Automation
- Improving Language Understanding from Screenshots
- AutoWebGLM: Bootstrap And Reinforce A Large Language Model-based Web Navigating Agent
- ![Star
- SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models
- ![Star
- You Only Look at Screens: Multimodal Chain-of-Action Agents
- Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API
- OpenAgents: AN OPEN PLATFORM FOR LANGUAGE AGENTS IN THE WILD
- ![Star - ai/OpenAgents)
- GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
- AppAgent: Multimodal Agents as Smartphone Users
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
- ![Star
- GPT-4V(ision) is a Generalist Web Agent, if Grounded
- ![Star - NLP-Group/SeeAct)
- AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant
- ![Star - jia/AgentStore)
- Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
- Dual-View Visual Contextualization for Web Navigation
- DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
- ![Star - agent/digirl)
- Visual Grounding for User Interfaces
- ScreenAgent: A Computer Control Agent Driven by Visual Language Large Model
- ![Star
- ScreenAI: A Vision-Language Model for UI and Infographics Understanding
- Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
- Octopus: On-device language model for function calling of software APIs
- Octopus v2: On-device language model for super agent
- Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent
- Octopus v4: Graph of language models
- Search Beyond Queries: Training Smaller Language Models for Web Interactions via Reinforcement Learning
- Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Reranking
- AutoDroid: LLM-powered Task Automation in Android
- Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation
- Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as a Case Study
- Android in the Zoo: Chain-of-Action-Thought for GUI Agents
- ![Star
- Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning
- GUI Action Narrator: Where and When Did That Action Take Place?
- ![Star - Action-Narrator)
- Identifying User Goals from UI Trajectories
- VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning
- Octo-planner: On-device Language Model for Planner-Action Agents
- E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion
- Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
- ![Star - PLUG/MobileAgent)
- MobileFlow: A Multimodal LLM For Mobile GUI Agent
- Vision-driven Automated Mobile GUI Testing via Multimodal Large Language Model
- Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence
- ![Star
- MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices
- AUITestAgent: Automatic Requirements Oriented GUI Function Testing
- ![Star - lab/AUITestAgent)
- Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
- ![Star - E)
- OmniParser for Pure Vision Based GUI Agent
- VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents
- ![Star
- Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
- MindSearch: Mimicking Human Minds Elicits Deep AI Searcher
- ![Star
- AppAgent v2: Advanced Agent for Flexible Mobile Interactions
- Agent Workflow Memory
- ![Star - workflow-memory)
- MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandin
- Agent S: An Open Agentic Framework that Uses Computers Like a Human
- ![Star - ai/Agent-S)
- MobA: A Two-Level Agent System for Efficient Mobile Task Automation
- ![Star
- ![Dataset - MobBench)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
- ![Star - NLP-Group/UGround)
- OS-ATLAS: A Foundation Action Model For Generalist GUI Agents
- Attacking Vision-Language Computer Agents via Pop-ups
- ![Star - NLP/PopupAttack)
- AutoGLM: Autonomous Foundation Agents for GUIs
- ![Star
- AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
- ShowUI: One Vision-Language-Action Model for Generalist GUI Agent
- Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
- ![Star - ai/aguvis)
- Falcon-UI: Understanding GUI Before Following User Instructions
- PC Agent: While You Sleep, AI Works - A Cognitive Journey into Digital World
- ![Star - NLP/PC-Agent)
- Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining