{"id":24850012,"url":"https://github.com/takahirom/arbigent","last_synced_at":"2025-12-29T11:55:48.244Z","repository":{"id":271073357,"uuid":"910426589","full_name":"takahirom/arbigent","owner":"takahirom","description":"AI Agent for testing Android, iOS, and Web apps. Get Started in 5 Minutes. Arbigent's intuitive UI and powerful code interface make it accessible to everyone, while its scenario breakdown feature ensures scalability for even the most complex tasks.","archived":false,"fork":false,"pushed_at":"2025-09-23T23:56:39.000Z","size":25302,"stargazers_count":395,"open_issues_count":23,"forks_count":29,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-09-24T01:22:14.765Z","etag":null,"topics":["agentic","ai","aiagent","android","gemini","ios","testing"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/takahirom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-31T08:38:58.000Z","updated_at":"2025-09-23T02:13:35.000Z","dependencies_parsed_at":"2025-01-20T02:26:09.541Z","dependency_job_id":"1caaf9fb-a699-4f24-90f2-d36029562ca4","html_url":"https://github.com/takahirom/arbigent","commit_stats":null,"previous_names":["takahirom/arbigent"],"tags_count":77,"template":false,"template_full_name":null,"purl":"pkg:github/takahirom/arbigent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takahirom%2Farbigent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takahirom%2Farbigent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takahirom%2Farbigent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takahirom%2Farbigent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/takahirom","download_url":"https://codeload.github.com/takahirom/arbigent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takahirom%2Farbigent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279021373,"owners_count":26087022,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic","ai","aiagent","android","gemini","ios","testing"],"created_at":"2025-01-31T13:02:05.541Z","updated_at":"2025-12-29T11:55:48.225Z","avatar_url":"https://github.com/takahirom.png","language":"Kotlin","funding_links":[],"categories":["HarmonyOS","Testing \u0026 Security"],"sub_categories":["Windows Manager","Other IDEs"],"readme":"# Arbigent(Arbiter-Agent): An AI Agent Testing Framework for Modern Applications\n\u003cimg width=\"2668\" height=\"1132\" alt=\"arbigent-banner-optimized\" src=\"https://github.com/user-attachments/assets/546c36ed-45fe-4ac2-a918-c7b0e7261f41\" /\u003e\n\n\n**Zero to AI agent testing in minutes. Arbigent's intuitive UI and powerful code interface make it accessible to everyone, while its scenario breakdown feature ensures scalability for even the most complex tasks.**\n\n\u003e [!WARNING]\n\u003e There seems to be a spam account posing as Arbigent, but the account is not related to me. The creator's accounts are [`https://x.com/_takahirom_`](https://x.com/_takahirom_) and [`https://x.com/new_runnable`](https://x.com/new_runnable) .\n\n## Screenshot\n\n\u003cimg width=\"650\" alt=\"arbigent-screenshot\" src=\"https://github.com/user-attachments/assets/77ebfcb1-3a44-4eaf-9775-3dff2597f9d1\" /\u003e\n\n## Demo movie\n\nhttps://github.com/user-attachments/assets/ec582760-5d6a-4ee3-8067-87cb2b673c8d\n\n## Motivation\n\n### Make AI Agent Testing Practical for Modern Applications\n\nTraditional UI testing often relies on brittle methods that are easily disrupted by even minor UI changes. A/B tests, updated tutorials, unexpected dialogs, dynamic advertising, or ever-changing user-generated content can cause tests to fail.  \nAI agents emerged as a solution, but testing with AI agents also presents challenges. AI agents often don't work as intended; for example, the agents might open other apps or click on the wrong button due to the complexity of the task.  \nTo address these challenges, I created Arbigent, an AI agent testing framework that can break down complex tasks into smaller, dependent scenarios. By decomposing tasks, Arbigent enables more predictable and scalable testing of AI agents in modern applications.\n\n### Customizable for Various AI Providers, OSes, Form Factors, etc.\n\nI believe many AI Agent testing frameworks will emerge in the future. However, widespread adoption might be delayed due to limitations in customization. For instance:\n\n*   **Limited AI Provider Support:** Frameworks might be locked to specific AI providers, excluding those used internally by companies.\n*   **Slow OS Adoption:** Support for different operating systems (like iOS and Android) could lag.\n*   **Delayed Form Factor Support:** Expanding to form factors beyond phones, such as Android TV, might take considerable time.\n\nTo address these issues, I aimed to create a framework that empowers users with extensive customization capabilities. Inspired by [OkHttp's interceptor](https://square.github.io/okhttp/features/interceptors/) pattern, Arbigent provides interfaces for flexible customization, allowing users to adapt the framework to their specific needs, such as those listed above.\n\n### Easy Integration into Development Workflows\n\nFurthermore, I wanted to make Arbigent accessible to QA engineers by offering a user-friendly UI. This allows for scenario creation within the UI and seamless test execution via the code interface.\n\n##  Key Feature Breakdown\n\n**I. Core Functionality \u0026 Design**\n\n*   **Complex Task Management:**\n    *   **Scenario Dependencies:** Breaks down complex goals into smaller, manageable scenarios that depend on each other (e.g., login -\u003e search).\n    *   **Orchestration:** Acts as a mediator, managing the execution flow of AI agents across multiple, interconnected scenarios.\n*   **Hybrid Development Workflow:**\n    *   **UI-Driven Scenario Creation:**  Allows non-programmers (e.g., QA engineers) to visually design test scenarios through a user-friendly interface.\n    *   **Code-Based Execution:** Enables software engineers to execute the saved scenarios programmatically (YAML files), allowing for integration with existing testing infrastructure.\n\n**II. Cross-Platform \u0026 Device Support**\n\n*   **Multi-Platform Compatibility:**\n    *   **Mobile \u0026 TV:** Supports testing on iOS, Android, Web, and TV interfaces.\n    *   **D-Pad Navigation:**  Handles TV interfaces that rely on D-pad navigation.\n\n**III. AI Optimization \u0026 Efficiency**\n\n*   **Enhanced AI Understanding:**\n    *   **UI Tree Optimization:** Simplifies and filters the UI tree to improve AI comprehension and performance.\n    *   **Accessibility-Independent:** Provides annotated screenshots to assist AI in understanding UIs that lack accessibility information.\n*   **Cost Savings:**\n    *   **Open Source:** Free to use, modify, and distribute, eliminating licensing costs.\n  *   **Efficient Model Usage:** Defaults to `gpt-4.1` for improved capabilities; also supports cost-effective models like `gpt-4o-mini`.\n\n**IV. Robustness \u0026 Reliability**\n\n*   **Double Check with AI-Powered Image Assertion:** Integrates [Roborazzi's feature](https://takahirom.github.io/roborazzi/ai-powered-image-assertion.html#behavior-of-ai-powered-image-assertion) to verify AI decisions using image-based prompts and allows the AI to re-evaluate if needed.\n*   **Stuck Screen Detection:** Identifies and recovers from situations where the AI agent gets stuck on the same screen, prompting it to reconsider its actions.\n\n**V. Advanced Features \u0026 Customization**\n\n*   **Flexible Code Interface:**\n    *   **Custom Hooks:** Offers a code interface for adding custom initialization and cleanup methods, providing greater control over scenario execution.\n*   **Maestro YAML Integration:**\n    *   **Pre-defined Test Flows:** Execute existing Maestro YAML test flows as initialization methods within Arbigent scenarios.\n    *   **Reusable Test Components:** Leverage your existing Maestro test automation scripts to set up complex application states before AI agent testing.\n    *   **Example Use Cases:**\n        * Run login flows using Maestro YAML before AI agent testing\n        * Set up specific application states (e.g., user onboarding completion)\n        * Execute complex setup sequences that require precise timing\n        * Integrate existing Maestro test assets into AI agent testing workflows\n* **Model Context Protocol (MCP) Support:**\n    * Introduced initial support for MCP, enabling Arbigent to leverage external tools and services defined via MCP servers. This significantly extends testing capabilities beyond direct UI interaction.\n    * You can configure MCP servers using a JSON string in the Project Settings.\n    * **Example MCP Use Cases:**\n        * Install and launch applications\n        * Check server logs (e.g., user behavior) using external tools\n        * Retrieve debug logs\n        * Interact with various other custom tools and services\n* **AI Hints - App-Provided Context:**\n    * Apps can provide domain-specific hints to help the AI understand the current screen better.\n    * Embed hints in the accessibility label using the `[[aihint:...]]` format:\n      ```kotlin\n      // Android (contentDescription)\n      view.contentDescription = \"Play button [[aihint:Video player, buffering]]\"\n      ```\n      ```swift\n      // iOS (accessibilityLabel)\n      button.accessibilityLabel = \"Play button [[aihint:Video player, buffering]]\"\n      ```\n      ```html\n      \u003c!-- Web (aria-label) --\u003e\n      \u003cbutton aria-label=\"Play button [[aihint:Video player, buffering]]\"\u003ePlay\u003c/button\u003e\n      ```\n    * The `[[...]]` format allows hints to coexist with accessibility labels and supports structured data like JSON.\n    * **Note:** Since this uses accessibility labels, it affects screen readers (TalkBack, VoiceOver, etc.). Enable hints only during testing to avoid accessibility issues in production.\n\n**VI. Community \u0026 Open Source**\n\n*   **Open Source Nature:**\n    *   **Free \u0026 Open:** Freely available for use, modification, and distribution.\n    *   **Community Driven:** Welcomes contributions from the community to enhance and expand the framework.\n\n## Arbigent's Strengths and Weaknesses Based on [SMURF](https://testing.googleblog.com/2024/10/smurf-beyond-test-pyramid.html)\n\nI categorized automated testing frameworks into five levels using the [SMURF](https://testing.googleblog.com/2024/10/smurf-beyond-test-pyramid.html) framework. Here's how Arbigent stacks up:\n\n*   **Speed (1/5):** Arbigent's speed is currently limited by the underlying AI technology and the need to interact with the application's UI in real-time. This makes it slower than traditional unit or integration tests.  \n    * We have introduced some mechanisms to address this:\n       *  Tests can be parallelized using the **`--shard` option** to speed up execution.  \n       *  **AI result caching** can be utilized when the UI tree and goal are identical, which is configurable in the project settings.  \n*   **Maintainability (4/5):** Arbigent excels in maintainability. The underlying AI model can adapt to minor UI changes, minimizing the need to rewrite tests for every small update, thus reducing maintenance effort. You can write tests in natural language (e.g., \"Complete the tutorial\"), making them resilient to UI changes. The task decomposition feature also reduces duplication, further enhancing maintainability. Maintenance can be done by non-engineers, thanks to the natural language interface.\n*   **Utilization (1/5):** Arbigent requires both device resources (emulators or physical devices) and AI resources, which can be costly. (AI cost can be around $0.005 per step and $0.02 per task when using GPT-4o.)\n*   **Reliability (3/5):** Arbigent has several features to improve reliability. It automatically waits during loading screens, handles unexpected dialogs, and even attempts self-correction. However, external factors like emulator flakiness can still impact reliability.\n    *   Recently I found Arbigent has retry feature and can execute the scenario from the beginning. But, **even without retry, Arbigent works fine without failures thanks to the flexibility of AI.**\n*   **Fidelity (5/5):** Arbigent provides high fidelity by testing on real or emulated devices with the actual application. It can even assess aspects that were previously difficult to test, such as verifying video playback by checking for visual changes on the screen.\n\nI believe that many of its current limitations, such as speed, maintainability, utilization, and reliability, will be addressed as AI technology continues to evolve. The need for extensive prompt engineering will likely diminish as AI models become more capable.\n\n## How to Use\n\n### Installation\n\nInstall the Arbigent UI binary from the [Release page](https://github.com/takahirom/arbigent/releases).\n\u003cimg width=\"632\" alt=\"image\" src=\"https://github.com/user-attachments/assets/499da604-4a43-4eb8-b27f-325a83caa013\" /\u003e\n\n### Installation for macOS Users\n\nIf you encounter security warnings when opening the app:\nRefer to [Apple's guide on opening apps from unidentified developers](https://support.apple.com/guide/mac-help/open-a-mac-app-from-an-unknown-developer-mh40616/mac).\n\n\u003cimg src=\"https://github.com/user-attachments/assets/44372fa0-1f58-4eea-9fff-a7cc672a0f86\" width=\"300\" /\u003e\n\nThis `Open Anyway` button is available for about an hour after you try to open the app.\n\u003cimg \n  src=\"https://github.com/user-attachments/assets/b8249cd6-3a40-4863-827c-78e63fdbfd87\" \n  width=\"400\" \n/\u003e\n\n### Device Connection and AI API Key Entry\n\n1. Connect your device to your PC.\n2. In the Arbigent UI, select your connected device from the list of available devices. This will establish a connection.\n3. Enter your AI provider's API key in the designated field within the Arbigent UI.\n\n\u003cimg src=\"https://github.com/user-attachments/assets/77a002f5-8ab3-4cb1-94f6-6c15a223900c\" width=\"450\" alt=\"Device Connection and AI API Key Entry\" /\u003e\n\n### Scenario Creation\n\nUse the intuitive UI to define scenarios. Simply specify the desired goal for the AI agent.\n\n\n### Generate scenarios\n\nYou can also generate scenarios from existing test cases. \n\u003cimg width=\"632\" alt=\"image\" src=\"https://github.com/user-attachments/assets/cbe078b0-8f2b-44a3-8980-1c5aff022a52\" /\u003e\n\n\n### Maestro YAML Integration\n\nArbigent allows you to integrate existing Maestro YAML test flows as initialization methods:\n\n1. **Add to Initialization Methods**: Select \"Maestro YAML\" as an initialization method type in your scenario configuration\n\n\u003cimg width=\"507\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/5b7a5b33-6d8d-4cbd-88a0-f4f922e5ae6d\" /\u003e\n\n2. **Add Maestro YAML**: Write a Maestro YAML file that defines the steps to be executed before the AI agent starts its task\n\n\u003cimg width=\"632\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/b348afa3-219e-4f3b-8275-ce0c45f882ee\" /\u003e\n\n3. **Choose Scenario**: Select from your predefined Maestro YAML scenarios to run before AI agent execution\n\n\u003cimg width=\"632\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/929170bb-9376-4fe4-8485-36a007ef07f6\" /\u003e\n\nThis enables you to:\n- Set up complex application states using deterministic Maestro flows\n- Reuse existing Maestro test automation assets\n- Combine precise setup sequences with AI-driven testing\n\n### Test Execution\n\nRun tests either directly through the UI or programmatically via the code interface or CLI.\n\n### CLI\n\nYou can install the CLI via Homebrew and run a saved YAML file.\n\n\u003cimg width=\"600\" alt=\"image\" src=\"https://github.com/user-attachments/assets/f20fdac8-ae46-4dc7-aa2b-6fc0b5bd4a9d\" /\u003e\n\n```bash\nbrew tap takahirom/homebrew-repo\nbrew install takahirom/repo/arbigent\n```\n\n```\nUsage: arbigent [\u003coptions\u003e] \u003ccommand\u003e [\u003cargs\u003e]...\n\nOptions:\n  -h, --help   Show this message and exit\n\nCommands:\n  run        Execute test scenarios\n  scenarios  List available scenarios\n  tags       Manage scenario tags\n```\n\n#### Configuration with Settings Files\n\nArbigent supports configuration via settings files in the `.arbigent` directory, which simplifies CLI usage by eliminating the need to specify common parameters repeatedly. Create a `.arbigent` directory in your project root and place your configuration files inside.\n\nArbigent supports multiple configuration files with automatic priority ordering - local configuration files have higher priority than global ones, allowing you to maintain separate settings for different environments while providing appropriate fallbacks.\n\n**Prioritized Configuration Support:**\n\nArbigent supports both global and command-specific configurations with prioritization:\n\n1. **High Priority**: Command-specific settings (e.g., `run.ai-type`)\n2. **Low Priority**: Global settings (e.g., `ai-type`) - used as fallback\n\n**Example hybrid configuration:**\n```yaml\n# Global settings (shared across all commands)\nproject-file: tests/arbigent-project.yml\nworking-directory: /path/to/your/project\nlog-level: info\n\n# Default AI configuration (fallback)\nai-type: openai\nopenai-api-key: sk-xxxxxxxxxxxxxxxxxx\nopenai-model-name: gpt-4.1\n\n# Run command specific settings (overrides global settings)\nrun:\n  ai-type: azureopenai  # Overrides global ai-type\n  azure-openai-endpoint: https://xxxxxxxxx.openai.azure.com/openai/deployments/xxxxx/\n  azure-openai-api-version: 2025-xx-xx\n  azure-openai-model-name: gpt-4o\n  azure-openai-api-key: xxxxxxxxxxxxxxxxxx\n  os: android\n```\n\n**Simple flat configuration (still supported):**\n```yaml\nai-type: azureopenai\nazure-openai-endpoint: https://xxxxxxxxx.openai.azure.com/openai/deployments/xxxxx/\nazure-openai-api-version: 2025-xx-xx\nazure-openai-model-name: gpt-4o\nazure-openai-api-key: xxxxxxxxxxxxxxxxxx\nproject-file: tests/arbigent-project.yml\nos: android\nlog-level: info\nworking-directory: /path/to/your/project\n```\n\nWhen using a settings file, you can run tests with the simplified command:\n\n```bash\narbigent run\n```\n\nAll parameters configured in the settings files will be shown as `(source: already provided by property file)` in the help output, indicating which settings are already configured.\n\n#### arbigent run command\n\n```\nUsage: arbigent run [\u003coptions\u003e]\n\nOptions for OpenAI API AI:\n  --openai-endpoint=\u003ctext\u003e    Endpoint URL (default: https://api.openai.com/v1/)\n  --openai-model-name=\u003ctext\u003e  Model name (default: gpt-4.1)  \n  --openai-api-key, --openai-key=\u003ctext\u003e API key\n\nOptions for Gemini API AI:\n  --gemini-endpoint=\u003ctext\u003e    Endpoint URL (default: https://generativelanguage.googleapis.com/v1beta/openai/)\n  --gemini-model-name=\u003ctext\u003e  Model name (default: gemini-1.5-flash)\n  --gemini-api-key=\u003ctext\u003e     API key\n\nOptions for Azure OpenAI:\n  --azure-openai-endpoint=\u003ctext\u003e         Endpoint URL\n  --azure-openai-api-version=\u003ctext\u003e      API version\n  --azure-openai-model-name=\u003ctext\u003e       Model name (default: gpt-4.1)  \n  --azure-openai-api-key, --azure-openai-key=\u003ctext\u003e API key\n\nOptions:\n  --ai-type=(openai|gemini|azureopenai)  Type of AI to use\n  --ai-api-logging                       Enable AI API debug logging\n  --os=(android|ios|web)                 Target operating system\n  --project-file=\u003ctext\u003e                  Path to the project YAML file\n  --log-level=(debug|info|warn|error)    Log level\n  --log-file=\u003ctext\u003e                      Log file path\n  --working-directory=\u003ctext\u003e             Working directory for the project\n  --path=\u003ctext\u003e                          Path to a file\n  --scenario-ids=\u003ctext\u003e                  Scenario IDs to execute (comma-separated)\n  --tags=\u003ctext\u003e                          Tags to filter scenarios (comma-separated)\n  --dry-run                              Dry run mode\n  --shard=\u003cvalue\u003e                        Shard specification (e.g., 1/5)\n  -h, --help                             Show this message and exit\n```\n\nWhen parameters are provided via the settings files, the help output will indicate `(source: already provided by property file)` for those options, so you know which parameters are already configured.\n\n#### Other commands\n\n**List scenarios:**\n```bash\narbigent scenarios\n```\n\n**Manage tags:**\n```bash\narbigent tags\n```\n\n\n### Shard Option to Enable Parallel Tests\n\nYou can run tests separately with the `--shard` option. This allows you to split your test suite and run tests in parallel, reducing overall test execution time.\n\n**Example:**\n\n```bash\narbigent run --shard=1/4\n```\n\nThis command will run the first quarter of your test suite.\n\n#### Running Specific Scenarios\n\nYou can run specific scenarios by their IDs:\n\n```bash\narbigent run --scenario-ids=\"scenario-1,scenario-2\"\n```\n\nOr filter scenarios by tags:\n\n```bash\narbigent run --tags=\"smoke,regression\"\n```\n\n**Integrating with GitHub Actions:**\n\nHere's an example of how to integrate the `--shard` option with GitHub Actions to run parallel tests on multiple Android emulators:\n\n```yaml\n  cli-e2e-android:\n    runs-on: ubuntu-latest\n    strategy:\n      fail-fast: false\n      matrix:\n        shardIndex: [ 1, 2, 3, 4 ]\n        shardTotal: [ 4 ]\n    steps:\n...\n      - name: CLI E2E test\n        uses: reactivecircus/android-emulator-runner@v2\n...\n          script: |\n            arbigent run --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --os=android --project-file=sample-test/src/main/resources/projects/e2e-test-android.yaml --ai-type=gemini --gemini-model-name=gemini-2.0-flash-exp\n...\n\n      - uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4\n        if: ${{ always() }}\n        with:\n          name: cli-report-android-${{ matrix.shardIndex }}-${{ matrix.shardTotal }}\n          path: |\n            arbigent-result/*\n          retention-days: 90\n```\n\n### Minimal GitHub Actions CI sample\n\nYou can use the CLI in GitHub Actions like in this sample. There are only two files: `.github/workflows/arbigent-test.yaml` and `arbigent-project.yaml`. This example demonstrates GitHub Actions and an `arbigent-project.yaml` file created by the Arbigent UI.\n\nhttps://github.com/takahirom/arbigent-sample\n\n## Supported AI Providers\n\n| AI Provider | Supported |\n|-------------|-----------|\n| OpenAI      | Yes       |\n| Gemini      | Yes       |\n| OpenAI based APIs like Ollama | Yes |\n\nYou can add AI providers by implementing the `ArbigentAi` interface.\n\n## Supported OSes / Form Factors\n\n| OS          | Supported | Test Status in the Arbigent repository            |\n|-------------|-----------|---------------------------------------------------|\n| Android     | Yes       | End-to-End including Android emulator and real AI |\n| iOS         | Yes       | End-to-End including iOS simulator and real AI    |\n| Web(Chrome) | Yes       | Currently, Testing not yet conducted              |\n\nYou can add OSes by implementing the `ArbigentDevice` interface. Thanks to the excellent [Maestro](https://github.com/mobile-dev-inc/maestro) library, we are able to support multiple OSes.\n\n| Form Factor    | Supported |\n|----------------|-----------|\n| Phone / Tablet | Yes       |\n| TV(D-Pad)      | Yes       |\n\n# Learn More\n\n## Basic Structure\n\n### Execution Flow\n\nThe execution flow involves the UI, Arbigent, ArbigentDevice, and ArbigentAi. The UI sends a project creation request to Arbigent, which fetches the UI tree from ArbigentDevice. ArbigentAi then decides on an action based on the goal and UI tree. The action is performed by ArbigentDevice, and the results are returned to the UI for display.\n\n```mermaid\nsequenceDiagram\n  participant UI(or Tests)\n  participant ArbigentAgent\n  participant ArbigentDevice\n  participant ArbigentAi\n  UI(or Tests)-\u003e\u003eArbigentAgent: Execute\n  loop\n    ArbigentAgent-\u003e\u003eArbigentDevice: Fetch UI tree\n    ArbigentDevice-\u003e\u003eArbigentAgent: Return UI tree\n    ArbigentAgent-\u003e\u003eArbigentAi: Decide Action by goal and UI tree and histories\n    ArbigentAi-\u003e\u003eArbigentAgent: Return Action\n    ArbigentAgent-\u003e\u003eArbigentDevice: Perform actions\n    ArbigentDevice-\u003e\u003eArbigentAgent: Return results\n  end\n  ArbigentAgent-\u003e\u003eUI(or Tests): Display results\n```\n\n###  Class Diagram\n\nThe class diagram illustrates the relationships between ArbigentProject, ArbigentScenario, ArbigentTask, ArbigentAgent, ArbigentScenarioExecutor, ArbigentAi, ArbigentDevice, and ArbigentInterceptor.\n\n```mermaid\nclassDiagram\n  direction TB\n  class ArbigentProject {\n    +List~ArbigentScenario~ scenarios\n    +execute()\n  }\n  class ArbigentAgentTask {\n    +String goal\n  }\n  class ArbigentAgent {\n    +ArbigentAi ai\n    +ArbigentDevice device\n    +List~ArbigentInterceptor~ interceptors\n    +execute(arbigentAgentTask)\n  }\n  class ArbigentScenarioExecutor {\n    +execute(arbigentScenario)\n  }\n  class ArbigentScenario {\n    +List~ArbigentAgentTask~ agentTasks\n  }\n  ArbigentProject o--\"*\" ArbigentScenarioExecutor\nArbigentScenarioExecutor o--\"*\" ArbigentAgent\nArbigentScenario o--\"*\" ArbigentAgentTask\nArbigentProject o--\"*\" ArbigentScenario\n```\n\n### Saved project file\n\n\u003e [!WARNING]\n\u003e The yaml format is still under development and may change in the future.\n\nThe project file is saved in YAML format and contains scenarios with goals, initialization methods, and cleanup data. Dependencies between scenarios are also defined.\nYou can write a project file in YAML format by hand or create it using the Arbigent UI.\n\nThe id is auto-generated UUID by Arbigent UI but you can change it to any string.\n\n#### Available Initialization Methods\n\n- **Back**: Press the back button\n- **Wait**: Wait for a specified duration\n- **LaunchApp**: Launch an application\n- **CleanupData**: Clear application data\n- **OpenLink**: Open a URL/link\n- **MaestroYaml**: Execute a predefined Maestro YAML scenario\n- **Reconnect**: Disconnect and reconnect the device\n\n```yaml\nscenarios:\n  - id: \"7788d7f4-7276-4cb3-8e98-7d3ad1d1cd47\"\n    goal: \"Open the Now in Android app from the app list. The goal is to view the list\\\n    \\ of topics.  Do not interact with the app beyond this.\"\n    initializationMethods:\n      - type: \"CleanupData\"\n        packageName: \"com.google.samples.apps.nowinandroid\"\n      - type: \"LaunchApp\"\n        packageName: \"com.google.samples.apps.nowinandroid\"\n      - type: \"MaestroYaml\"\n        scenarioId: \"login-flow-maestro\"\n  - id: \"f0ef0129-c764-443f-897d-fc4408e5952b\"\n    goal: \"In the Now in Android app, select an tech topic and complete the form in\\\n    \\ the \\\"For you\\\" tab. The goal is reached when articles are displayed.  Do not\\\n    \\ click on any articles. If the browser opens, return to the app.\"\n    dependency: \"7788d7f4-7276-4cb3-8e98-7d3ad1d1cd47\"\n    imageAssertions:\n      - assertionPrompt: \"Articles are visible on the screen\"\n  - id: \"73c785f7-0f45-4709-97b5-601b6803eb0d\"\n    goal: \"Save an article using the Bookmark button.\"\n    dependency: \"f0ef0129-c764-443f-897d-fc4408e5952b\"\n  - id: \"797514d2-fb04-4b92-9c07-09d46cd8f931\"\n    goal: \"Check if a saved article appears in the Saved tab.\"\n    dependency: \"73c785f7-0f45-4709-97b5-601b6803eb0d\"\n    imageAssertions:\n      - assertionPrompt: \"The screen is showing Saved tab\"\n      - assertionPrompt: \"There is an article in the screen\"\n\n# Maestro YAML scenarios for initialization\nfixedScenarios:\n  - id: \"login-flow-maestro\"\n    title: \"Login Flow\"\n    description: \"Performs user login using predefined credentials\"\n    yamlText: |\n      appId: com.google.samples.apps.nowinandroid\n      ---\n      - tapOn: \"Sign In\"\n      - inputText: \"test@example.com\"\n      - tapOn: \"Password\"\n      - inputText: \"password123\"\n      - tapOn: \"Login\"\n      - assertVisible: \"Welcome\"\n```\n\n## Code Interface\n\n\u003e [!WARNING]\n\u003e The code interface is still under development and may change in the future.\n\nArbigent provides a code interface for executing tests programmatically. Here's an example of how to run a test:\n\n\n### Dependency\n\nStay tuned for the release of Arbigent on Maven Central.\n\n### Running saved project yaml file\n\nYou can load a project yaml file and execute it using the following code:\n\n```kotlin\nclass ArbigentTest {\n  private val scenarioFile = File(this::class.java.getResource(\"/projects/nowinandroidsample.yaml\").toURI())\n\n  @Test\n  fun tests() = runTest(\n    timeout = 10.minutes\n  ) {\n    val arbigentProject = ArbigentProject(\n      file = scenarioFile,\n      aiFactory = {\n        OpenAIAi(\n          apiKey = System.getenv(\"OPENAI_API_KEY\")\n        )\n      },\n      deviceFactory = {\n        AvailableDevice.Android(\n          dadb = Dadb.discover()!!\n        ).connectToDevice()\n      }\n    )\n    arbigentProject.execute()\n  }\n}\n```\n\n### Run a scenario directly\n\n```kotlin\nval agentConfig = AgentConfig {\n  deviceFactory { FakeDevice() }\n  ai(FakeAi())\n}\nval arbigentScenarioExecutor = ArbigentScenarioExecutor {\n}\nval arbigentScenario = ArbigentScenario(\n  id = \"id2\",\n  agentTasks = listOf(\n    ArbigentAgentTask(\"id1\", \"Login in the app and see the home tab.\", agentConfig),\n    ArbigentAgentTask(\"id2\", \"Search an episode and open detail\", agentConfig)\n  ),\n  maxStepCount = 10,\n)\narbigentScenarioExecutor.execute(\n  arbigentScenario\n)\n```\n\n### Run a goal directly\n\n```kotlin\nval agentConfig = AgentConfig {\n  deviceFactory { FakeDevice() }\n  ai(FakeAi())\n}\n\nval task = ArbigentAgentTask(\"id1\", \"Login in the app and see the home tab.\", agentConfig)\nArbigentAgent(agentConfig)\n  .execute(task)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftakahirom%2Farbigent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftakahirom%2Farbigent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftakahirom%2Farbigent/lists"}