{"id":16307185,"url":"https://github.com/francedot/Interface-Agent","last_synced_at":"2025-10-25T14:31:01.604Z","repository":{"id":220838190,"uuid":"751974079","full_name":"francedot/Interface-Agent","owner":"francedot","description":"InterfaceAgent: a versatile framework designed to create system and interface agents capable of managing mobile and desktop applications and features.","archived":false,"fork":false,"pushed_at":"2024-05-01T18:40:17.000Z","size":10501,"stargazers_count":110,"open_issues_count":4,"forks_count":3,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-02-06T00:03:26.018Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/francedot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-02T18:28:36.000Z","updated_at":"2025-02-04T15:29:08.000Z","dependencies_parsed_at":"2024-06-19T03:01:12.506Z","dependency_job_id":"39a1bed6-87c5-4483-9f5f-7f2521a31134","html_url":"https://github.com/francedot/Interface-Agent","commit_stats":null,"previous_names":["francedot/navaiguide","francedot/navaiguide-ts"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francedot%2FInterface-Agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francedot%2FInterface-Agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francedot%2FInterface-Agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francedot%2FInterface-Agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/francedot","download_url":"https://codeload.github.com/francedot/Interface-Agent/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238155552,"owners_count":19425734,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T21:13:43.393Z","updated_at":"2025-10-25T14:31:00.839Z","avatar_url":"https://github.com/francedot.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg align=\"center\" src=\"./img/logo.png\" width=\"27\"\u003e Agent\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"/\"\u003e\u003cimg src=\"https://img.shields.io/badge/typescript-gray?logo=typescript\" alt=\"TypeScript\"\u003e\u003c/a\u003e\n  \u003ca href=\"/\"\u003e\u003cimg src=\"https://img.shields.io/badge/node-20_LTS-blue\" alt=\"Node 20 LTS\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" alt=\"MIT License\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@interface-agent/core\"\u003e\u003cimg src=\"https://img.shields.io/badge/@interface--agent/core-0.0.1--preview-green\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@interface-agent/ios\"\u003e\u003cimg src=\"https://img.shields.io/badge/@interface--agent/ios-0.0.1--preview-green\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@interface-agent/windows\"\u003e\u003cimg src=\"https://img.shields.io/badge/@interface--agent/windows-0.0.1--preview-green\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg align=\"center\" width=\"320\" src=\"./img/logo.png\" alt=\"InterfaceAgent Screenshot\"\u003e\n\u003c/p\u003e\n\n# 🤔 What is InterfaceAgent?\n\nWelcome to **InterfaceAgent**, a versatile framework designed to create system and interface agents capable of managing mobile and desktop applications and features.\n\nHere are the key capabilities of **InterfaceAgent**:\n\n- **Planning \u0026 Goal Refinement**: The agent is capable of constructing multi-step plans across various applications to fulfill user requests. It can also adapt and refine these plans based on user feedback during the evaluation phase.\n\n- **Action Prediction (Pure Visual / Textual / Set-of-Mark Visual Prompting)**: InterfaceAgent employs a visual coordinate-based approach, pure DOM textual analysis, or set-of-marking to enhance the accuracy of predicting the next likely action.\n\n- **Mixture of Models**: InterfaceAgent is compatible with both GPT-4V and Claude models, excelling in determining the subsequent steps directly from page screenshots.\n\n- **Resilient Error Handling**: Recognizing that errors are an inherent part of AI Agents, InterfaceAgent incorporates a robust retry mechanism with exponential backoff. This allows it to skillfully navigate through temporary failures, ensuring the Agent's progress is uninterrupted.\n\n**InterfaceAgent** OS-specific agents extend the core toolkit with advanced automation for the target platform:\n\n- **Preview of iOS Agents:** Explore how your AI Agents can gain access to the ecosystem of apps and functionalities on your iOS device.\n- **Preview of Windows Agents:** Explore how your AI Agents can gain access to the ecosystem of apps and functionalities on your Windows 11 device.\n- **Preview of Appium Android Agents (Coming soon):** Explore how your AI Agents can gain access to the ecosystem of apps and functionalities on your Android device.\n- **Playwright-based Web Agents (Coming soon):** Learn how to build Web AI Agent Companions.\n\n## 💻 Getting Started\n\nYou can choose to either clone the repository or use npm, yarn, or pnpm to install InterfaceAgent.\n\n- For Core, see [installation steps](./packages/core/README.md).\n- For iOS, see [installation steps](./packages/ios/README.md).\n- For Windows, see [installation steps](./packages/windows/README.md).\n\n## 🎬 Demos\n\n### Windows\n\n```bash\n1) User Query: Help me download an app named EdgeTile\n```\n\u003cp align=\"center\"\u003e\n  \u003cimg align=\"center\" width=\"1280\" src=\"https://github.com/francedot/OS-Agent/assets/11706033/676218ad-c6db-4ada-8db2-72153131ac83\" alt=\"EdgeTile demo\"\u003e\n\u003c/p\u003e\n\n```bash\n2) User Query: Dropshipping products on Tiktok\n```\n\u003cp align=\"center\"\u003e\n  \u003cimg align=\"center\" width=\"1280\" src=\"https://github.com/francedot/OS-Agent/assets/11706033/c57db0e4-cc5c-42db-9f1b-fb9bd0a1fb4c\" alt=\"TikTok demo\"\u003e\n\u003c/p\u003e\n\n### iOS\n\n```bash\nUser Query: Help me prepare for a 30 days of fitness challenge\n```\n\u003cp align=\"center\"\u003e\n  \u003cimg align=\"center\" width=\"1280\" src=\"https://github.com/francedot/NavAIGuide-TS/assets/11706033/bdc79ec6-c05c-4fb4-a01a-0a4b3b802ce9\" alt=\"30 days of fitness demo\"\u003e\n\u003c/p\u003e\n\n## 🚀 Challenges and Focus\n\nInterfaceAgent continues to face challenges in long-horizon planning and selector inference accuracy. The current focus is on enhancing the stability of InterfaceAgent agents.\n\n## 🤓 Contributing\n\nWe welcome contributions. Please follow the standard fork-and-pull request workflow for your contributions.\n\n## 🛂 License\n\nInterfaceAgent is licensed under the [MIT License](LICENSE).\n\n## 🚑 Support\n\nFor support, questions, or feature requests, open an issue in the GitHub repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrancedot%2FInterface-Agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrancedot%2FInterface-Agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrancedot%2FInterface-Agent/lists"}