{"id":51002841,"url":"https://github.com/black-yt/IrisGUI","last_synced_at":"2026-07-08T16:00:34.437Z","repository":{"id":333186779,"uuid":"1134729676","full_name":"black-yt/IrisGUI","owner":"black-yt","description":"A Lightweight Desktop GUI Agent via Dynamic Focus Vision and Hierarchical Memory. Your AI-powered hands and eyes for desktop automation.","archived":false,"fork":false,"pushed_at":"2026-04-28T11:06:01.000Z","size":1273,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-28T13:10:43.587Z","etag":null,"topics":["agent","ai","ai-agents","automomous-agents","computer-use","computer-use-agent","computer-use-agents","desktop","gui","guiagent","interaction","lightweight","linux","llm","macos","mllm","vllm","windows"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/black-yt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-15T05:49:35.000Z","updated_at":"2026-04-28T11:06:04.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/black-yt/IrisGUI","commit_stats":null,"previous_names":["black-yt/irisgui"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/black-yt/IrisGUI","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/black-yt%2FIrisGUI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/black-yt%2FIrisGUI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/black-yt%2FIrisGUI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/black-yt%2FIrisGUI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/black-yt","download_url":"https://codeload.github.com/black-yt/IrisGUI/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/black-yt%2FIrisGUI/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35270196,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","ai","ai-agents","automomous-agents","computer-use","computer-use-agent","computer-use-agents","desktop","gui","guiagent","interaction","lightweight","linux","llm","macos","mllm","vllm","windows"],"created_at":"2026-06-20T17:00:21.336Z","updated_at":"2026-07-08T16:00:34.413Z","avatar_url":"https://github.com/black-yt.png","language":"Python","funding_links":[],"categories":["Desktop Agents"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cpre\u003e\n██╗██████╗ ██╗███████╗\n██║██╔══██╗██║██╔════╝\n██║██████╔╝██║███████╗\n██║██╔══██╗██║╚════██║\n██║██║  ██║██║███████║\n╚═╝╚═╝  ╚═╝╚═╝╚══════╝\u003c/pre\u003e\n\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n[![Python](https://img.shields.io/badge/python-3.10+-yellow.svg)](https://www.python.org/)\n[![GitHub](https://img.shields.io/badge/GitHub-000000?logo=github\u0026logoColor=white)](https://github.com/black-yt/IrisGUI)\u0026#160;\n\u003c!-- \u003ca href=\"https://arxiv.org/pdf/xxxx.xxxx\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv\" height=\"21px\"\u003e\u003c/a\u003e --\u003e\n\n**A Lightweight Desktop GUI Agent via Dynamic Focus Vision and Hierarchical Memory**  \n\n*Lightweight • Minimal Code • Minimal Dependencies 🍃*\n\n*Visual Perception • Infinite Memory • Long Interaction 💪🏻*\n\u003c/div\u003e\n\n\n## 🚀 What is Iris?\n\n**Iris** is an intelligent agent designed to navigate your operating system just like a human does. It doesn't just blindly run scripts; it **sees** the screen, **thinks** about what to do, and **acts** with precision.\n\nIris is **lightweight**, with **minimal code** and **dependencies**, requiring only a **single API key**. Yet, it packs a punch with:\n*   **Visual Perception** 👁️\n*   **Infinite Memory** 🧠\n*   **Long Interaction** 🔄\n\nPowered by a robust **ReAct (Reasoning + Acting)** loop, Iris can handle complex workflows, recover from errors, and remember context over long periods thanks to its hierarchical memory system.\n\n---\n\n## 🆕 Latest News\n\n🚩 **Update** (2026-04-30) Iris now supports native user input requests during agent loops, with the model's question shown in the floating log and a reusable multi-line terminal input panel for immediate or delayed submission.\n\n🚩 **Update** (2026-04-29) Iris now supports OpenAI-compatible native tool calling, safe multi-tool GUI actions, a cross-platform terminal task launcher, structured loop output, traceable debug logs, and token-based memory compression.\n\n🚩 **Update** (2026-01-18) We release **Iris-v1.0**.\n\n---\n\n## 🧠 Architecture\n\nIris operates on a cycle of **Reasoning**, **Action**, **Observation** and **Reflection**. Here's how the magic happens:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/architecture.png\" alt=\"Iris Architecture\" width=\"850\"\u003e\n\u003c/p\u003e\n\nA **dynamic focusing view** strategy is adopted to improve positioning accuracy and action efficiency.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/dual_views.png\" alt=\"Dynamic Focusing View\" width=\"850\"\u003e\n\u003c/p\u003e\n\n**Hierarchical memory** can effectively avoid context explosion and prevent task forgetting.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/memory.png\" alt=\"Memory\" width=\"400\"\u003e\n\u003c/p\u003e\n\n---\n\n## ✨ Key Features\n\n| Feature | Description |\n| :--- | :--- |\n| 🍃 **Quick Installation**   | Only need to install **a few** dependencies and configure **an** API.                               |\n| 👁️ **Dynamic Focus Vision** | Uses **Global** (coarse) and **Local** (fine) views to locate elements with pixel-perfect accuracy. |\n| 🧠 **Hierarchical Memory**  | Smartly compresses history into **Short-term** and **Long-term** layers. No more token overflow!    |\n| 🔄 **Long Interaction**     | Complete super-long real-world tasks with **100 steps or more**.                                    |\n| 🛡️ **Self-Correction**      | Verifies cursor position before clicking. If it misses, it adjusts and tries again.                 |\n| 🎮 **Human-Like Control**   | Smooth mouse movements, typing, scrolling, and even drag-and-drop support.                          |\n| 📺 **Live Debug Mode**      | Watch Iris think and act in real-time with a dedicated GUI dashboard.                               |\n\n---\n\n## 🎞️ Demos\n\n* **Task**: `在浏览器中搜索“多模态大模型的基本原理与应用”等相关主题信息，对检索到的内容进行整理与归纳，并撰写一份结构清晰的中文Word文档（包含标题、分节与要点说明）。将该文档创建在桌面上，命名为“多模态报告.docx”。随后打开Google Translate网页，将该Word文档上传至文档翻译功能，选择中文到英文的翻译选项，等待系统生成翻译结果后下载英文版本文件，并使用Word打开检查翻译内容与格式是否正确。`\n\nhttps://github.com/user-attachments/assets/ab8ca64b-2878-452f-a6ac-32f58bbb6ae7\n\n\u003cbr\u003e\n\n* **Task**: `玩一局植物大战僵尸`\n\nhttps://github.com/user-attachments/assets/097baadc-e01f-441c-973d-f1260656d19a\n\n\u003cbr\u003e\n\n* **Task**: `Open Google Chrome and search for Shanghai's weather`\n\nhttps://github.com/user-attachments/assets/067b6fb4-d242-4b4b-b4e1-696eb170c3d2\n\n\u003cbr\u003e\n\n* **Task**: `Open Story.txt and write a short story of 100 words`\n\nhttps://github.com/user-attachments/assets/47c928c7-5226-4d9b-b5ff-dff41d17f874\n\n---\n\n## ⚡ Quick Start\n\nReady to let Iris take the wheel? Follow these steps to get started in minutes!\n\n### 1. Clone the Repository\n```bash\ngit clone https://github.com/black-yt/IrisGUI.git\ncd IrisGUI\n```\n\n### 2. Install Dependencies\nMake sure you have Python 3.10+ installed.\n```bash\npip install -r requirements.txt\n```\n\nOn Linux, Iris also needs a working desktop screenshot backend. Install `gnome-screenshot` for Wayland/X11 or `scrot` for X11 if screenshots fail.\n\nIf OpenAI/httpx fails during startup with `SSL_CERT_FILE` or `CURL_CA_BUNDLE` pointing to a missing file, unset or fix that environment variable. Iris ignores missing certificate bundle paths before creating the OpenAI client.\n\n### 3. Configure Environment\nCreate a `.env` file in the root directory (copy from [`.env.example`](.env.example)) and add your LLM credentials:\n```ini\nLLM_API_ENDPOINT=\"https://base-url/v1\"\nLLM_API_KEY=\"sk-your-api-key-here\"\nLLM_MODEL_NAME=\"gemini-3.1-pro-preview\"\nLLM_TIMEOUT_SECONDS=0\nLLM_MAX_RETRIES=1\nMEMORY_SHORT_TOKEN_BUDGET=128000\nMEMORY_LONG_TOKEN_BUDGET=128000\n```\n\n### 4. Run Iris\n```bash\npython main.py\n```\n\nThe terminal editor opens in a colored framed task input panel before the task starts. Type the task directly, use **Up/Down** to choose **New Line**, **Start Now**, **Start After 5s**, **Start After Custom Delay**, **Clear**, or **Exit**, then press **Enter** to confirm the selected action. **New Line** is selected by default, so pressing **Enter** normally inserts a line break. For custom delay, enter the number of seconds in the delay field, then confirm **Start After Custom Delay**.\n\nIris uses native tool calling for GUI actions. It can execute multiple tool calls in one model turn only when they are safe consecutive actions that do not depend on UI loading or a fresh screenshot.\n\nEach agent loop is shown as a color-coded structured box in both the terminal and the floating screen log, with separate sections for perception, model reasoning, tool calls, and feedback.\n\n\u003e **💡 Tip:** To stop Iris in an emergency, press **ESC** three times quickly! 🛑\n\n---\n\n## 📬 Contact\n\n- 💬 **GitHub Issues**: Please open an issue for bug reports or feature requests\n\n- 📧 **Email**: [xu_wanghan@sjtu.edu.cn](https://black-yt.github.io/)\n\n---\n\n\u003c!-- ## 📜 Citation\n\nIf you would like to cite our work, please use the following BibTeX.\n\n```bib\nComing soon...\n```\n\n--- --\u003e\n\n## 🌟 Star History\n\nIf you find this work helpful, please consider to **star⭐** this [repo](https://github.com/black-yt/IrisGUI). Thanks for your support! 🤩\n\n[![black-yt/IrisGUI Stargazers](https://reporoster.com/stars/black-yt/IrisGUI)](https://github.com/black-yt/IrisGUI/stargazers)\n\n\u003cp align=\"right\"\u003e\u003ca href=\"#top\"\u003e🔝Back to top\u003c/a\u003e\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblack-yt%2FIrisGUI","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblack-yt%2FIrisGUI","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblack-yt%2FIrisGUI/lists"}