{"id":20137081,"url":"https://github.com/ctrf-io/ai-test-reporter","last_synced_at":"2025-09-21T06:32:24.852Z","repository":{"id":256279470,"uuid":"851936275","full_name":"ctrf-io/ai-test-reporter","owner":"ctrf-io","description":"Generate a test report with AI summaries from various models including OpenAI, Azure and Claude","archived":false,"fork":false,"pushed_at":"2025-08-10T19:06:50.000Z","size":1756,"stargazers_count":44,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-16T04:52:47.764Z","etag":null,"topics":["ai","ctrf","test-automation","test-reporting"],"latest_commit_sha":null,"homepage":"https://ctrf.io","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ctrf-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-04T00:05:56.000Z","updated_at":"2025-08-12T06:15:38.000Z","dependencies_parsed_at":"2024-09-10T01:17:00.338Z","dependency_job_id":"b7f30c79-bc2a-4b61-b444-eb583a7e6909","html_url":"https://github.com/ctrf-io/ai-test-reporter","commit_stats":null,"previous_names":["ctrf-io/ai-test-reporter"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ctrf-io/ai-test-reporter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctrf-io%2Fai-test-reporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctrf-io%2Fai-test-reporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctrf-io%2Fai-test-reporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctrf-io%2Fai-test-reporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ctrf-io","download_url":"https://codeload.github.com/ctrf-io/ai-test-reporter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctrf-io%2Fai-test-reporter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275405943,"owners_count":25459312,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-16T02:00:10.229Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ctrf","test-automation","test-reporting"],"created_at":"2024-11-13T21:25:06.190Z","updated_at":"2025-09-21T06:32:24.846Z","avatar_url":"https://github.com/ctrf-io.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI Test Reporter\n\nAI Test Reporter is a powerful tool that generates intelligent summaries of test results using a wide range of AI models. With access to over 300 models through various providers (OpenAI, Anthropic Claude, Google Gemini, Mistral, Perplexity, OpenRouter, and more), it analyzes failing tests and provides actionable insights about what went wrong and how to fix it.\n\n\u003cdiv align=\"center\"\u003e\n\u003cdiv style=\"padding: 1.5rem; border-radius: 8px; margin: 1rem 0; border: 1px solid #30363d;\"\u003e\n\u003cspan style=\"font-size: 23px;\"\u003e💚\u003c/span\u003e\n\u003ch3 style=\"margin: 1rem 0;\"\u003eCTRF tooling is open source and free to use\u003c/h3\u003e\n\u003cp style=\"font-size: 16px;\"\u003eYou can support the project with a follow and a star\u003c/p\u003e\n\n\u003cdiv style=\"margin-top: 1.5rem;\"\u003e\n\u003ca href=\"https://github.com/ctrf-io/ai-test-reporter\"\u003e\n\u003cimg src=\"https://img.shields.io/github/stars/ctrf-io/ai-test-reporter?style=for-the-badge\u0026color=2ea043\" alt=\"GitHub stars\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://github.com/ctrf-io\"\u003e\n\u003cimg src=\"https://img.shields.io/github/followers/ctrf-io?style=for-the-badge\u0026color=2ea043\" alt=\"GitHub followers\"\u003e\n\u003c/a\u003e\n\u003c/div\u003e\n\u003c/div\u003e\n\n\u003cp style=\"font-size: 14px; margin: 1rem 0;\"\u003e\nContributions are very welcome! \u003cbr/\u003e\nExplore more \u003ca href=\"https://www.ctrf.io/integrations\"\u003eintegrations\u003c/a\u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n## Why Use AI Test Reporter?\n\n- 🤖 **Smart Analysis**: Get AI-powered explanations of why tests failed and suggestions for fixes\n- 🔌 **Multiple Providers**: Choose from 300+ AI models across different providers\n- 💰 **Cost-Effective Options**: Start with providers offering free credits like Mistral and Google Gemini\n- 🔄 **Developer Tool Integration**: Seamlessly integrate AI summaries into your workflow:\n  - GitHub Actions and Pull Requests\n  - Slack / Teams notifications\n  - And more through standardized CTRF reports\n- 🎯 **Consolidated Insights**: Get high-level summaries of test suite failures to identify patterns and root causes\n- ⚡ **Framework Agnostic**: Works with any testing framework through CTRF reports\n- 📊 **Flexible Reporting**: AI summaries are embedded in your CTRF JSON report, allowing you to:\n  - Include AI analysis in your existing reporting workflows\n  - Programmatically customize how and where summaries appear in your Developer Tools\n\n## Models\n\n\u003e [!TIP]\n\u003e The following integrations are available with free tiers so are great to try out AI test reporter:\n\u003e\n\u003e - **Google Gemini**: Offers a free tier with lower rate limits. [Pricing details](https://ai.google.dev/pricing#2_0flash)\n\u003e - **Mistral AI**: Provides a free API tier to explore the service. [Tier details](https://docs.mistral.ai/deployment/laplateforme/tier/#usage-tiers)\n\nYou can use any of the models supported by the following providers:\n\n- OpenAI\n- Anthropic Claude\n- Google Gemini\n- Mistral\n- Grok\n- DeepSeek\n- Azure OpenAI\n- Perplexity\n- OpenRouter\n\nYou use your own API keys for the models you select.\n\n## Usage\n\nGenerate a CTRF report using your testing framework. [CTRF reporters](https://github.com/orgs/ctrf-io/repositories) are available for most testing frameworks and easy to install.\n\n**No CTRF reporter? No problem!**\n\nUse [junit-to-ctrf](https://github.com/ctrf-io/junit-to-ctrf) to convert a JUnit report to CTRF\n\n## OpenAI\n\nRun the following command:\n\n```bash\nnpx ai-ctrf openai \u003cpath-to-ctrf-report\u003e\n```\n\nAn AI summary for each failed test will be added to your test report.\n\nThe package interacts with the OpenAI API, you must set `OPENAI_API_KEY` environment variable.\n\nYou will be responsible for any charges incurred from using your selected OpenAI model. Make sure you are aware of the associated cost.\n\nA message is sent to OpenAI for each failed test.\n\n### Options\n\n`--model`: OpenAI model to use (default: gpt-3.5-turbo).\n\n`--systemPrompt`: Custom system prompt to guide the AI response.\n\n`--frequencyPenalty`: OpenAI frequency penalty parameter (default: 0).\n\n`--maxTokens`: Maximum number of tokens for the response.\n\n`--presencePenalty`: OpenAI presence penalty parameter (default: 0).\n\n`--temperature`: Sampling temperature (conflicts with topP).\n\n`--topP`: Top-p sampling parameter (conflicts with temperature).\n\n`--log`: Whether to log the AI responses to the console (default: true).\n\n`--maxMessages`: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)\n\n`consolidate`: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)\n\n## Azure OpenAI\n\nRun the following command:\n\n```bash\nnpx ai-ctrf azure-openai \u003cpath-to-ctrf-report\u003e\n```\n\nAn AI summary for each failed test will be added to your test report.\n\nThe package interacts with the Azure OpenAI API, you must set `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, and `AZURE_OPENAI_DEPLOYMENT_NAME` environment variable or provide them as arguments.\n\nYou will be responsible for any charges incurred from using your selected Azure OpenAI model. Make sure you are aware of the associated cost.\n\nA message is sent to Azure OpenAI for each failed test.\n\n### Options\n\n`--model`: OpenAI model to use (default: gpt-3.5-turbo).\n\n`--systemPrompt`: Custom system prompt to guide the AI response.\n\n`--frequencyPenalty`: OpenAI frequency penalty parameter (default: 0).\n\n`--maxTokens`: Maximum number of tokens for the response.\n\n`--presencePenalty`: OpenAI presence penalty parameter (default: 0).\n\n`--temperature`: Sampling temperature (conflicts with topP).\n\n`--topP`: Top-p sampling parameter (conflicts with temperature).\n\n`--log`: Whether to log the AI responses to the console (default: true).\n\n`--maxMessages`: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)\n\n`consolidate`: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)\n\n## Claude\n\nRun the following command:\n\n```bash\nnpx ai-ctrf claude \u003cpath-to-ctrf-report\u003e\n```\n\nAn AI summary for each failed test will be added to your test report.\n\nThe package interacts with the Anthropic API, you must set `ANTHROPIC_API_KEY` environment variable.\n\nYou will be responsible for any charges incurred from using your selected Claude model. Make sure you are aware of the associated costs.\n\nA message is sent to Claude for each failed test.\n\n### Claude Options\n\n`--model`: Claude model to use (default: claude-3-5-sonnet-20240620).\n\n`--systemPrompt`: Custom system prompt to guide the AI response.\n\n`--maxTokens`: Maximum number of tokens for the response.\n\n`--temperature`: Sampling temperature.\n\n`--log`: Whether to log the AI responses to the console (default: true).\n\n`--maxMessages`: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)\n\n`consolidate`: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)\n\n## Grok\n\nRun the following command:\n\n```bash\nnpx ai-ctrf grok \u003cpath-to-ctrf-report\u003e\n```\n\nAn AI summary for each failed test will be added to your test report.\n\nThe package interacts with the Grok API, you must set `GROK_API_KEY` environment variable.\n\nYou will be responsible for any charges incurred from using your selected Grok model. Make sure you are aware of the associated cost.\n\nA message is sent to Grok for each failed test.\n\n### Grok Options\n\n`--model`: Grok model to use (default: grok-1).\n\n`--systemPrompt`: Custom system prompt to guide the AI response.\n\n`--maxTokens`: Maximum number of tokens for the response.\n\n`--temperature`: Sampling temperature.\n\n`--log`: Whether to log the AI responses to the console (default: true).\n\n`--maxMessages`: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)\n\n`consolidate`: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)\n\n## DeepSeek\n\nRun the following command:\n\n```bash\nnpx ai-ctrf deepseek \u003cpath-to-ctrf-report\u003e\n```\n\nAn AI summary for each failed test will be added to your test report.\n\nThe package interacts with the DeepSeek API, you must set `DEEPSEEK_API_KEY` environment variable.\n\nYou will be responsible for any charges incurred from using your selected DeepSeek model. Make sure you are aware of the associated cost.\n\nA message is sent to DeepSeek for each failed test.\n\n### DeepSeek Options\n\n`--model`: DeepSeek model to use (default: deepseek-coder).\n\n`--systemPrompt`: Custom system prompt to guide the AI response.\n\n`--maxTokens`: Maximum number of tokens for the response.\n\n`--temperature`: Sampling temperature.\n\n`--log`: Whether to log the AI responses to the console (default: true).\n\n`--maxMessages`: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)\n\n`consolidate`: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)\n\n## Mistral\n\nRun the following command:\n\n```bash\nnpx ai-ctrf mistral \u003cpath-to-ctrf-report\u003e\n```\n\nAn AI summary for each failed test will be added to your test report.\n\nThe package interacts with the Mistral API, you must set `MISTRAL_API_KEY` environment variable.\n\nYou will be responsible for any charges incurred from using your selected Mistral model. Make sure you are aware of the associated cost.\n\nA message is sent to Mistral for each failed test.\n\nMistral offers free API credits upon signup, making it a great option to test the tool without immediate costs.\n\n### Mistral Options\n\n`--model`: Mistral model to use (default: mistral-medium).\n\n`--systemPrompt`: Custom system prompt to guide the AI response.\n\n`--maxTokens`: Maximum number of tokens for the response.\n\n`--temperature`: Sampling temperature.\n\n`--topP`: Top-p sampling parameter.\n\n`--log`: Whether to log the AI responses to the console (default: true).\n\n`--maxMessages`: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)\n\n`consolidate`: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)\n\n## Google Gemini\n\nRun the following command:\n\n```bash\nnpx ai-ctrf gemini \u003cpath-to-ctrf-report\u003e\n```\n\nAn AI summary for each failed test will be added to your test report.\n\nThe package interacts with the Google Gemini API, you must set `GOOGLE_API_KEY` environment variable.\n\nYou will be responsible for any charges incurred from using your selected Gemini model. Make sure you are aware of the associated cost.\n\nA message is sent to Gemini for each failed test.\n\nGoogle offers free API credits for Gemini, providing a cost-effective way to try out the tool.\n\n### Gemini Options\n\n`--model`: Gemini model to use (default: gemini-pro).\n\n`--systemPrompt`: Custom system prompt to guide the AI response.\n\n`--maxTokens`: Maximum number of tokens for the response.\n\n`--temperature`: Sampling temperature.\n\n`--topP`: Top-p sampling parameter.\n\n`--log`: Whether to log the AI responses to the console (default: true).\n\n`--maxMessages`: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)\n\n`consolidate`: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)\n\n## Perplexity\n\nRun the following command:\n\n```bash\nnpx ai-ctrf perplexity \u003cpath-to-ctrf-report\u003e\n```\n\nAn AI summary for each failed test will be added to your test report.\n\nThe package interacts with the Perplexity API, you must set `PERPLEXITY_API_KEY` environment variable.\n\nYou will be responsible for any charges incurred from using your selected Perplexity model. Make sure you are aware of the associated cost.\n\nA message is sent to Perplexity for each failed test.\n\n### Perplexity Options\n\n`--model`: Perplexity model to use (default: pplx-7b-online).\n\n`--systemPrompt`: Custom system prompt to guide the AI response.\n\n`--maxTokens`: Maximum number of tokens for the response.\n\n`--temperature`: Sampling temperature.\n\n`--topP`: Top-p sampling parameter.\n\n`--log`: Whether to log the AI responses to the console (default: true).\n\n`--maxMessages`: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)\n\n`consolidate`: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)\n\n## OpenRouter\n\nRun the following command:\n\n```bash\nnpx ai-ctrf openrouter \u003cpath-to-ctrf-report\u003e\n```\n\nAn AI summary for each failed test will be added to your test report.\n\nThe package interacts with the OpenRouter API, you must set `OPENROUTER_API_KEY` environment variable.\n\nOpenRouter provides access to multiple AI models through a single API, including models from Anthropic, OpenAI, Google, Meta, and more.\n\nYou will be responsible for any charges incurred from using your selected OpenRouter model. Make sure you are aware of the associated cost.\n\nA message is sent to OpenRouter for each failed test.\n\n### OpenRouter Options\n\n`--model`: OpenRouter model to use (default: anthropic/claude-3-opus). Available models include:\n- anthropic/claude-3-opus\n- google/gemini-pro\n- meta-llama/llama-2-70b-chat\n- mistral/mixtral-8x7b\nAnd many more from the OpenRouter catalog.\n\n`--systemPrompt`: Custom system prompt to guide the AI response.\n\n`--maxTokens`: Maximum number of tokens for the response.\n\n`--temperature`: Sampling temperature.\n\n`--topP`: Top-p sampling parameter.\n\n`--frequencyPenalty`: Frequency penalty parameter.\n\n`--presencePenalty`: Presence penalty parameter.\n\n`--log`: Whether to log the AI responses to the console (default: true).\n\n`--maxMessages`: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)\n\n`consolidate`: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)\n\n## Test Information Analyzed by AI Model\n\nThe AI model analyzes information to:\n\n1. Understand the complete context of the failure\n2. Identify potential root causes\n3. Suggest specific fixes\n4. Highlight patterns across multiple failures\n\nWhen consolidation is enabled (`--consolidate`), the AI analyzes all test failures AI summaries together to provide a high-level summary of issues.\n\n### Test Object\n\nFor each failing test, the AI receives the complete test object.\n\n### Environment Context\n\nThe complete environment details from the `environment` object in the CTRF report is provided to the AI model.\n\n### Tool Context\n\nThe complete tool details from the `tool` object in the CTRF report is provided to the AI model.\n\n## Charges\n\nYou are responsible for any charges incurred from using the AI models. Make sure you are aware of the associated costs.\n\n## CTRF Report Example\n\n``` json\n{\n  \"results\": {\n    \"tool\": {\n      \"name\": \"AnyFramework\"\n    },\n    \"summary\": {\n      \"tests\": 1,\n      \"passed\": 0,\n      \"failed\": 1,\n      \"pending\": 0,\n      \"skipped\": 0,\n      \"other\": 1,\n      \"start\": 1722511783500,\n      \"stop\": 1722511804528\n    },\n    \"tests\": [\n        {\n            \"name\": \"should display profile information\",\n            \"status\": \"failed\",\n            \"duration\": 800,\n            \"message\": \"Assertion Failure: profile mismatch\",\n            \"trace\": \"ProfileTest.js:45...\",\n            \"ai\": \"The test failed because there was a profile mismatch at line 45 of the ProfileTest.js file. To resolve this issue,   review the code at line 45 to ensure that the expected profile information matches the actual data being displayed. Check for any discrepancies and make necessary adjustments to align the expected and actual profile information.\"\n        },\n    ]\n  }\n}\n```\n\n## Standard Output\n\n![stdout](assets/stdout.png)\n\n## GitHub Actions Integration\n\nView AI summaries in directly in the Github Actions workflow:\n\n![Github](assets/github.png)\n\nAdd a Pull Request comment with your AI summary:\n\n![Github](assets/github-pr.png)\n\n## Slack Integration\n\nSend a Slack message with your AI test summary:\n\n![Slack](assets/slack.png)\n\n## Support Us\n\nIf you find this project useful, consider giving it a GitHub star ⭐ It means a lot to us.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fctrf-io%2Fai-test-reporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fctrf-io%2Fai-test-reporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fctrf-io%2Fai-test-reporter/lists"}