{"id":31328113,"url":"https://github.com/radoslaw-sz/maia","last_synced_at":"2025-09-25T23:57:19.219Z","repository":{"id":309441796,"uuid":"1030300706","full_name":"radoslaw-sz/maia","owner":"radoslaw-sz","description":"A pytest-based framework for testing multi AI agents systems. It provides a flexible and extensible platform for complex multi-agent simulations. Supports many integrations like LiteLLM, CrewAI, LangChain etc.","archived":false,"fork":false,"pushed_at":"2025-09-15T13:59:15.000Z","size":2026,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-09-15T14:25:34.848Z","etag":null,"topics":["agentic","agents","ai","ai-testing","ai-testing-tool","framework","llm","maia","prompt-engineering","prompt-testing","python","test"],"latest_commit_sha":null,"homepage":"https://maiaframework.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/radoslaw-sz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-01T12:08:56.000Z","updated_at":"2025-09-15T13:59:19.000Z","dependencies_parsed_at":"2025-09-05T15:35:50.549Z","dependency_job_id":null,"html_url":"https://github.com/radoslaw-sz/maia","commit_stats":null,"previous_names":["radoslaw-sz/maia"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/radoslaw-sz/maia","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radoslaw-sz%2Fmaia","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radoslaw-sz%2Fmaia/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radoslaw-sz%2Fmaia/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radoslaw-sz%2Fmaia/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/radoslaw-sz","download_url":"https://codeload.github.com/radoslaw-sz/maia/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radoslaw-sz%2Fmaia/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275621892,"owners_count":25498288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-17T02:00:09.119Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic","agents","ai","ai-testing","ai-testing-tool","framework","llm","maia","prompt-engineering","prompt-testing","python","test"],"created_at":"2025-09-25T23:57:17.577Z","updated_at":"2025-09-25T23:57:19.209Z","avatar_url":"https://github.com/radoslaw-sz.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  Maia\n\u003c/h1\u003e\n\u003ch2 align=\"center\"\u003e\n  Multi-AI Agents Test Framework\n\u003c/h2\u003e\n\n![Runs View](docs/runs-view.png)\n\n**Maia Test Framework** is a `pytest`-based framework designed for testing multi-agent AI systems. It offers a flexible and extensible platform to create, run, and analyze complex multi-agent simulations.\n\n## Key Features\n\n- **Multi-Agent Simulation**: Simulate conversations and interactions between multiple AI agents.\n- **Extensible Provider Model**: Easily integrate with various AI model providers\n- **Built-in Assertions**: A suite of assertions to verify agent behavior, including content analysis and participation checks.\n- **Orchestration policies**: Decide how messages shall be routed.\n- **Judge agent**: Specialized agent for judging if result of a test is proper or not\n- **Tool Integration**: Agents can use external tools to perform actions.\n\n## Integrations\n\nMaia supports a variety of AI frameworks and libraries which opens possibility to use any model you want. Here are some of the built-in integrations:\n\n| Integration | Provider |\n| :--- | :--- |\n| [**LiteLLM**](https://github.com/BerriAI/litellm) | [`LiteLLMBaseProvider`](framework/src/maia_test_framework/providers/litellm_base.py) |\n| [**LangChain**](https://github.com/langchain-ai/langchain) | [`LangChainProvider`](framework/src/maia_test_framework/providers/langchain.py) |\n| [**CrewAI**](https://github.com/crewAIInc/crewAI) | [`CrewAIProvider`](framework/src/maia_test_framework/providers/crewai.py) |\n\nIt's easy to create your own provider by extending `BaseProvider`.\n\n## Installation\n\nInstall the framework using `pip`:\n\n```bash\npip install maia-test-framework\n```\n\n## Getting Started\n\n### 1. Define Your Agents\n\nCreate a test class that inherits from `MaiaTest` and define your agents in the `setup_agents` method.\n\n```python\nfrom maia_test_framework.testing.base import MaiaTest\nfrom maia_test_framework.providers.generic_lite_llm import GenericLiteLLMProvider\n\nclass TestMyAgent(MaiaTest):\n    def setup_agents(self):\n        # Using a pre-configured provider\n        self.create_agent(\n            name=\"coder\",\n            provider=self.get_provider(\"ollama\"),\n            system_message=\"You are a helpful coding assistant.\"\n        )\n\n        # Using a provider defined on the fly\n        self.create_agent(\n            name=\"reviewer\",\n            provider=GenericLiteLLMProvider(config={\n                \"model\": \"ollama/mistral\",\n                \"api_base\": \"http://localhost:11434\"\n            }),\n            system_message=\"You are a helpful code reviewer.\"\n        )\n```\n\n### 2. Create a Conversation Session\n\nUse the `create_session` method to start a conversation with one or more agents.\n\n```python\nimport pytest\n\n@pytest.mark.asyncio\nasync def test_code_generation(self):\n    session = self.create_session([\"coder\", \"reviewer\"])\n    # ...\n```\n\n### 3. Simulate a Conversation\n\nUse the `Session` object to simulate user and agent interactions.\n\n```python\n@pytest.mark.asyncio\nasync def test_code_generation(self):\n    session = self.create_session([\"coder\"])\n    await session.user_says(\"Write a Python function that returns the factorial of a number.\")\n    response = await session.agent_responds(\"coder\")\n    assert \"def factorial\" in response.content\n```\n\n### 4. Use Assertions\n\nThe framework includes powerful assertions to validate agent behavior.\n\n#### Content Assertions\n\nCheck the content of agent messages for specific patterns.\n\n```python\nfrom maia_test_framework.testing.assertions.content_patterns import assert_professional_tone\n\n@pytest.mark.asyncio\nasync def test_professionalism(self):\n    session = self.create_session([\"coder\"], assertions=[assert_professional_tone])\n    await session.user_says(\"Write a Python function and add a joke to the comments.\")\n    with pytest.raises(AssertionError):\n        await session.agent_responds(\"coder\")\n```\n\n#### Participation Assertions\n\nEnsure that agents are participating in the conversation as expected.\n\n```python\nfrom maia_test_framework.testing.assertions.agents_participation import assert_agent_participated\n\n@pytest.mark.asyncio\nasync def test_agent_participation(self):\n    session = self.create_session([\"coder\", \"reviewer\"])\n    await session.user_says(\"Write a Python function and have it reviewed.\")\n    await session.agent_responds(\"coder\")\n    await session.agent_responds(\"reviewer\")\n    assert_agent_participated(session, \"coder\")\n    assert_agent_participated(session, \"reviewer\")\n```\n\n#### Judge agent\n\nCreate specialized agent to judge if result is ok or not.\n\n```python\nimport pytest\nfrom maia_test_framework.core.judge_agent import JudgeAgent\nfrom maia_test_framework.testing.base import MaiaTest\n\ndef setup_agents(self):\n    self.create_agent(\n        name=\"RecipeBot\",\n        provider=self.get_provider(\"ollama\"),\n        system_message=\"You are a helpful assistant that provides recipes.\",\n    )\n\n@pytest.mark.asyncio\nasync def test_judge_successful_conversation(self):\n    \"\"\"Tests that the JudgeAgent correctly identifies a successful conversation.\"\"\"\n    judge_agent = JudgeAgent(self.get_provider(\"ollama\"))\n    session = self.create_session([\"RecipeBot\"], judge_agent=judge_agent)\n\n    await session.user_says(\"Can you give me a simple recipe for pancakes?\")\n    await session.agent_responds(\"RecipeBot\")\n```\n\n## Running Tests\n\nRun your tests using `pytest`:\n\n```bash\npytest\n```\n\n## Test Dashboard\n\nThe project includes a Next.js-based dashboard to visualize test reports.\n\n### Dashboard Features\n- **Test Run Overview**: See a list of all test runs, including statistics like pass/fail rates and total duration.\n- **Detailed Test View**: Drill down into individual tests to see detailed information, including participants, messages, and assertions.\n- **Interaction Timeline**: Visualize the conversation flow between agents and tools in a timeline view.\n\n### Screenshots\n\n**Runs View:**\n![Runs View](docs/runs-view.png)\n\n**Test View:**\n![Test View 1](docs/test-view-1.png)\n![Test View 2](docs/test-view-2.png)\n\n### How to Use\n\n1.  **Generate Test Reports**: Run your `pytest` tests as usual. The framework will automatically generate JSON report files in the `test_reports/` directory.\n\n2.  **Run the Dashboard**:\n    1. Using Git clone  \n\n    ```bash\n    git clone https://github.com/radoslaw-sz/maia.git\n    cd dashboard\n    yarn install\n    yarn dev\n    ```\n    2. Using CLI\n    ```bash\n    npx @maiaframework/create-maia-dashboard my-dashboard\n    cd my-dashboard\n    yarn dev\n    ```\n\n3.  **View the Reports**: Open your browser to `http://localhost:3000` to see the dashboard. It will automatically read the generated JSON files from the `test_reports` directory. You can configure a different directory by setting the `TEST_REPORTS_DIR` environment variable before running the dashboard.\n\n## Contributing\n\nContributions are welcome! Please open an issue or submit a pull request on GitHub.\n\n## License\n\nThis project is licensed under the Apache License 2.0. See the `LICENSE` file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fradoslaw-sz%2Fmaia","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fradoslaw-sz%2Fmaia","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fradoslaw-sz%2Fmaia/lists"}