{"id":65601,"url":"https://github.com/lectrician1/awesome-interface-agents","name":"awesome-interface-agents","description":"List of AI tools that can interact with user interfaces","projects_count":31,"last_synced_at":"2026-08-15T18:00:31.221Z","repository":{"id":244134664,"uuid":"814307389","full_name":"lectrician1/awesome-interface-agents","owner":"lectrician1","description":"List of AI tools that can interact with user interfaces","archived":false,"fork":false,"pushed_at":"2025-03-07T19:33:35.000Z","size":43,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-07-27T08:05:06.299Z","etag":null,"topics":["agent","agentic","agentic-ai","ai","ai-os","automation","awesome-list","interface","llava"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lectrician1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-12T18:44:17.000Z","updated_at":"2026-07-16T11:23:44.000Z","dependencies_parsed_at":"2024-06-13T03:42:28.236Z","dependency_job_id":"75b7d6d2-e697-4556-b979-21625198e6bb","html_url":"https://github.com/lectrician1/awesome-interface-agents","commit_stats":{"total_commits":10,"total_committers":1,"mean_commits":10.0,"dds":0.0,"last_synced_commit":"89622baa99e85199816c73f71f88118d28abb86a"},"previous_names":["lectrician1/awesome-interface-agents"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lectrician1/awesome-interface-agents","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lectrician1%2Fawesome-interface-agents","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lectrician1%2Fawesome-interface-agents/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lectrician1%2Fawesome-interface-agents/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lectrician1%2Fawesome-interface-agents/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lectrician1","download_url":"https://codeload.github.com/lectrician1/awesome-interface-agents/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lectrician1%2Fawesome-interface-agents/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36694198,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-08-06T04:43:03.162Z","status":"online","status_checked_at":"2026-08-15T02:00:05.847Z","response_time":94,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-09-05T00:00:21.314Z","updated_at":"2026-08-15T18:00:31.222Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Papers","Models","Complete solutions"],"sub_categories":["Web browser","VLMs","Segmenters","Operating system"],"readme":"# awesome-interface-agents\nList of AI tools that can interact with user interfaces. PRs welcome.\n\n## Models\n### VLMs\nThese are VLMs that support pointing / bounding boxes for user interaction. \n#### Open source\n* [Qwen 2.5-VL](https://github.com/QwenLM/Qwen2.5-VL) (Jan 2025)\n* [Moondream](https://moondream.ai/)\n* [Llama 3.2](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD_VISION.md) (Sep 2024): The two largest models of the Llama 3.2 collection, 11B and 90B, support image reasoning use cases, such as document-level understanding including charts and graphs, captioning of images, and visual grounding tasks such as directionally pinpointing objects in images based on natural language descriptions.\n* [Molmo](https://molmo.allenai.org/blog) (Sep 2024): VLM that matches GPT-4V performance with pointing ability.\n* [CogAgent](https://github.com/THUDM/CogVLM/tree/main?tab=readme-ov-file#introduction-to-cogagent) (Dec 2023): CogAgent is an open-source visual language model that can identify regions and points of UIs to interact with.\n* [Florence 2](https://arxiv.org/abs/2311.06242) (Nov 2023): Vision foundation model with a unified, prompt-based representation for a variety of computer vision and vision-language tasks including producing bounding boxes.\n\n#### Closed source\n* [OpenAI Operator](https://operator.chatgpt.com/) (Jan 2025): Backed by a Computer-Using Model.\n* [Claude 3.5 Computer Use](https://docs.anthropic.com/en/docs/build-with-claude/computer-use) (Oct 2024): Version of the Claude 3.5 model which supports computer use structured text and image tool inputs and actionable text outputs. \n\n### Segmenters\n* [Moondream](https://moondream.ai/)\n* [ScreenAI](https://research.google/blog/screenai-a-visual-language-model-for-ui-and-visually-situated-language-understanding/)\n* [Llava](https://llava-vl.github.io/)\n* [SegmentEverythingEverywhereAllAtOnce](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once)\n\n## Complete solutions\n### Operating system\n#### Open source\n* [Qwen 2.5-VL Cookbook](https://github.com/QwenLM/Qwen2.5-VL/blob/main/cookbooks/computer_use.ipynb)\n* [OpenAdapt.AI](https://openadapt.ai/): AI-First Process Automation with Large ([Language (LLMs) / Action (LAMs) / Multimodal (LMMs)] / Visual Language (VLMs)) Models\n* [ScreenAgent](https://github.com/niuzaisheng/ScreenAgent)\n* [Mobile-Agent](https://ar5iv.labs.arxiv.org/html/2401.16158v1)\n* [UI-ACT](https://github.com/TobiasNorlund/UI-Act): An AI agent for interacting with a computer using the graphical user interface\n* [OpenInterpreter](https://github.com/OpenInterpreter/open-interpreter): Uses code to interact with operating system.\n* [AIOS](https://github.com/agiresearch/AIOS): Can interact with operating system as backend.\n\n#### Closed source\n* [Manus AI](https://manus.im/) March 2025\n* [Claude 3.5 Computer Use Cookbook](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo)\n* [Adept](https://adept.ai): Company looking to automate user interface interaction through ML\n\n### Web browser\nThese are still mostly text-based\n#### Open source\n* [Skyvern](https://github.com/skyvern-ai/skyvern): Browser automation software\n* [AgentLLM](https://github.com/idosal/AgentLLM)\n* [LaVague](https://github.com/lavague-ai/LaVague)\n\n#### Closed source\n* [OpenAI Operator](https://operator.chatgpt.com/): A system using the Computer-Using Agent (CUA) model to interact with the user interface and ask for clarification from the user in your browser.\n* [Google Project Mariner](https://deepmind.google/technologies/project-mariner/): Browser extention to interact with pages.\n* [HyperWrite AI Agent](https://www.hyperwriteai.com/personal-assistant)\n\n## Papers\n* [Autonomous Interactive Agents](https://web.media.mit.edu/~lieber/Lieberary/Letizia/AIA/AIA.html): MIT\n* [Toolformer](https://arxiv.org/abs/2302.04761)\n* [Visual Programming: Compositional visual reasoning without training](https://openaccess.thecvf.com/content/CVPR2023/papers/Gupta_Visual_Programming_Compositional_Visual_Reasoning_Without_Training_CVPR_2023_paper.pdf)\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/lectrician1%2Fawesome-interface-agents/projects"}