{"id":45820752,"url":"https://github.com/skyramp/testbot","last_synced_at":"2026-05-06T01:04:45.586Z","repository":{"id":340832319,"uuid":"1165807592","full_name":"skyramp/testbot","owner":"skyramp","description":"AI Powered Deterministic Testing for your Web Applications.","archived":false,"fork":false,"pushed_at":"2026-04-09T21:15:28.000Z","size":11369,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-09T23:20:15.368Z","etag":null,"topics":["agentic-workflow","ai-agent","qa","qa-automation","rest-api","testing"],"latest_commit_sha":null,"homepage":"https://www.skyramp.dev/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skyramp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-24T15:09:52.000Z","updated_at":"2026-04-09T21:15:33.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/skyramp/testbot","commit_stats":null,"previous_names":["skyramp/testbot"],"tags_count":35,"template":false,"template_full_name":null,"purl":"pkg:github/skyramp/testbot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skyramp%2Ftestbot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skyramp%2Ftestbot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skyramp%2Ftestbot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skyramp%2Ftestbot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skyramp","download_url":"https://codeload.github.com/skyramp/testbot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skyramp%2Ftestbot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31826902,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"online","status_checked_at":"2026-04-15T02:00:06.175Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic-workflow","ai-agent","qa","qa-automation","rest-api","testing"],"created_at":"2026-02-26T20:13:14.186Z","updated_at":"2026-05-06T01:04:45.577Z","avatar_url":"https://github.com/skyramp.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Skyramp Testbot\n\n\u003e Automated test maintenance for your REST APIs using Skyramp's AI-powered Testbot\n\n[![GitHub Marketplace](https://img.shields.io/badge/Marketplace-Skyramp%20Testbot-blue?logo=github)](https://github.com/marketplace/actions/skyramp-testbot)\n[![License](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)\n\n## Features\n\n- 🤖 **Skyramp Powered Test Maintenance** - Automatically updates tests when code changes\n- 🔍 **Smart Change Detection** - Analyzes git diffs to identify impacted tests\n- ✨ **Test Generation** - Creates new tests for new code additions\n- ✅ **Automated Test Execution** - Runs tests and validates results\n- 💬 **PR Integration** - Posts detailed summaries as PR comments\n- 🔄 **Auto-commit** - Optionally commits test changes automatically\n\n## Quick Start\n\nThe fastest way to get started is with the **Skyramp Testbot Installer** — a guided wizard that installs the GitHub App, configures secrets, and opens a ready-to-merge setup PR in your repository.\n\n1. Go to [testbot.skyramp.dev](https://testbot.skyramp.dev) and sign in with GitHub.\n2. Install the Skyramp Testbot GitHub App on your organization or personal account.\n3. Select a repository, configure your Skyramp license and AI agent key, and review the generated workflow.\n4. Click **Deploy** — the installer creates a PR with the workflow file and configures your secrets automatically.\n5. Merge the PR, and Testbot will run on every pull request.\n\n### Manual Setup\n\nIf you prefer to set things up manually:\n\n1. Add 2 secrets to your repository:\n    1. Obtain a [Skyramp](https://skyramp.dev) license key and store it as `SKYRAMP_LICENSE`.\n    2. Add an API key for your chosen AI agent (`ANTHROPIC_API_KEY`, `CURSOR_API_KEY`, or `COPILOT_PAT`).\n2. Add this workflow to your repository (`.github/workflows/skyramp-testbot.yml`):\n\n    ```yaml\n    name: Skyramp Testbot\n    on: [pull_request]\n\n    jobs:\n      testbot:\n        runs-on: ubuntu-latest\n        permissions:\n          contents: write\n          pull-requests: write\n        steps:\n          - uses: actions/checkout@v4\n            with:\n              fetch-depth: 0\n\n          - uses: skyramp/testbot@v0.9\n            with:\n              skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n    ```\n\n## Agent Customization\n\nShould you want to use your own AI agent subscription, provision a key and use the appropriate input.\n\n- Claude Code - Coding Agent by Anthropic\n- **Cursor CLI** - Powerful AI agent from Cursor\n- **GitHub Copilot CLI** - GitHub's AI coding assistant\n\nChoose the agent that best fits your needs and existing subscriptions.\n\n## Prerequisites\n\nBefore using this action, ensure you have:\n\n- [ ] Skyramp license file content stored in GitHub Secrets as `SKYRAMP_LICENSE`\n- [ ] **For Claude Code**: Claude Code API key stored in GitHub Secrets as `ANTHROPIC_API_KEY`\n- [ ] **For Cursor**: Cursor API key stored in GitHub Secrets as `CURSOR_API_KEY`\n- [ ] **For Copilot**: GitHub token with Copilot access stored in GitHub Secrets as `GITHUB_TOKEN` or `COPILOT_PAT`\n- [ ] Docker available in your runner (for Skyramp Executor)\n- [ ] Node.js compatible project (action installs Node.js automatically)\n- [ ] Existing Skyramp tests or a test directory structure\n\n\u003e **Note:** The agent type is automatically detected based on which API key you provide. Provide only one key (not both).\n\n## Inputs\n\n### Required\n\n| Input | Description |\n|-------|-------------|\n| `skyrampLicenseFile` | Skyramp license file content (store in GitHub Secrets) |\n| `anthropicApiKey` | Anthropic API key (provide this to use Claude Code) |\n| `cursorApiKey` | Cursor API key (provide this to use Cursor agent) |\n| `copilotApiKey` | GitHub token with Copilot access (provide this to use Copilot agent) |\n\n### Optional - Service Lifecycle\n\n| Input | Description | Default |\n|-------|-------------|---------|\n| `targetSetupCommand` | Command to start services before test maintenance | `docker compose up -d` |\n| `skipTargetSetup` | Skip running service startup command | `false` |\n| `targetReadyCheckCommand` | Command to verify services are ready (retried until success or timeout) | `sleep 5` |\n| `targetReadyCheckTimeout` | Max seconds to wait for ready check to succeed | `30` |\n| `targetReadyCheckDiagnosticsCommand` | Command to collect diagnostics on ready check timeout | Docker container status/logs |\n| `targetTeardownCommand` | Command to tear down services after tests (runs in post step, guaranteed even on failure/cancellation) | `''` |\n| `skipTargetTeardown` | Skip running service teardown command | `false` |\n| `authTokenCommand` | Shell command to generate an authentication token (stdout is captured and set as `SKYRAMP_TEST_TOKEN`) | `''` |\n\n### Optional - Other\n\n| Input | Description | Default |\n|-------|-------------|---------|\n| `testDirectory` | Directory containing Skyramp tests | `tests` |\n| `skyrampExecutorVersion` | Skyramp Executor Docker image version | `v1.3.23` |\n| `skyrampMcpVersion` | Skyramp MCP package version | `latest` |\n| `nodeVersion` | Node.js version to use | `lts/*` |\n| `workingDirectory` | Working directory for the action | `.` |\n| `autoCommit` | Automatically commit test changes | `true` |\n| `commitMessage` | Commit message for test changes | `Skyramp Testbot: test maintenance suggestions` |\n| `postPrComment` | Post summary as PR comment | `true` |\n| `testbotMaxRetries` | Maximum number of retries for transient agent CLI errors | `3` |\n| `testbotRetryDelay` | Delay in seconds between agent retry attempts | `10` |\n| `testExecutionTimeout` | Timeout in seconds for individual MCP tool calls (e.g., test execution) | `300` |\n| `testbotTimeout` | Timeout in minutes for the agent execution | `60` |\n| `reportCollapsed` | Wrap report sections in collapsible `\u003cdetails\u003e` blocks | `true` |\n| `enableDebug` | Enable debug logging | `true` |\n\n## Outputs\n\n| Output | Description |\n|--------|-------------|\n| `test_summary` | Full summary of test maintenance actions |\n| `tests_modified` | Number of tests modified |\n| `tests_created` | Number of tests created |\n| `tests_executed` | Number of tests executed |\n| `skipped_self_trigger` | Whether execution was skipped due to detecting own commit |\n| `commit_sha` | SHA of the commit made by testbot (empty if no commit) |\n\n## Usage Examples\n\n### Basic Usage with Claude Code\n\n```yaml\n- uses: skyramp/testbot@v0.9.0\n  with:\n    skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n    anthropicApiKey: ${{ secrets.ANTHROPIC_API_KEY }}\n\n```\n\n### Basic Usage with Cursor\n\n```yaml\n- uses: skyramp/testbot@v0.9.0\n  with:\n    skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n    cursorApiKey: ${{ secrets.CURSOR_API_KEY }}\n```\n\n### Using GitHub Copilot CLI\n\n```yaml\n- uses: skyramp/testbot@v0.9.0\n  with:\n    skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n    copilotApiKey: ${{ secrets.COPILOT_PAT }}\n```\n\n### Custom Service Startup Command\n\n```yaml\n- uses: skyramp/testbot@v0.9.0\n  with:\n    skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n    cursorApiKey: ${{ secrets.CURSOR_API_KEY }}\n    targetSetupCommand: 'npm run start:services'\n```\n\n### Without Auto-commit (Manual Review)\n\n```yaml\n- uses: skyramp/testbot@v0.9.0\n  with:\n    skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n    cursorApiKey: ${{ secrets.CURSOR_API_KEY }}\n    autoCommit: false\n```\n\n### Custom Test Directory Location\n\n```yaml\n- uses: skyramp/testbot@v0.9.0\n  with:\n    skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n    cursorApiKey: ${{ secrets.CURSOR_API_KEY }}\n    testDirectory: 'api/tests'\n```\n\n### Authentication\n\nIf your API under test requires authentication, there are two ways to provide a token for test execution.\n\n#### Static Token\n\nIf your token is fixed (e.g. a test API key), set `SKYRAMP_TEST_TOKEN` as a workflow environment variable:\n\n```yaml\nenv:\n  SKYRAMP_TEST_TOKEN: ${{ secrets.SKYRAMP_TEST_TOKEN }}\n\njobs:\n  test-maintenance:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n        with:\n          fetch-depth: 0\n\n      - uses: skyramp/testbot@v0.9.0\n        with:\n          skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n          cursorApiKey: ${{ secrets.CURSOR_API_KEY }}\n```\n\n#### Dynamic Token\n\nIf your token must be generated at runtime (e.g. by calling a login endpoint or running a CLI), use the `authTokenCommand` input. The command runs after services start, and its stdout is captured as the token:\n\n```yaml\n- uses: skyramp/testbot@v0.9.0\n  with:\n    skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n    cursorApiKey: ${{ secrets.CURSOR_API_KEY }}\n    authTokenCommand: 'curl -s https://my-api.com/auth/token'\n```\n\nThe token is automatically masked in GitHub Actions logs via `::add-mask::`. If the command fails, the action stops before running any tests.\n\n### Using Outputs\n\n```yaml\n- uses: skyramp/testbot@v0.9.0\n  id: skyramp\n  with:\n    skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n    cursorApiKey: ${{ secrets.CURSOR_API_KEY }}\n\n- name: Check Results\n  run: |\n    echo \"Tests Modified: ${{ steps.skyramp.outputs.tests_modified }}\"\n    echo \"Tests Created: ${{ steps.skyramp.outputs.tests_created }}\"\n    echo \"Tests Executed: ${{ steps.skyramp.outputs.tests_executed }}\"\n```\n\n## Triggering Other Workflows\n\nBy default, commits made by GitHub Actions using `GITHUB_TOKEN` don't trigger other workflows (this is GitHub's built-in recursion prevention). If you want Testbot's commits to trigger your CI/CD pipelines, linters, or other workflows, you need to use a Personal Access Token (PAT).\n\n### Setup Steps\n\n1. **Create a fine-grained PAT** scoped to your repository with `Contents: Read and Write` permission at [github.com/settings/personal-access-tokens](https://github.com/settings/personal-access-tokens). Set an expiration date and rotate regularly.\n2. **Add it as a secret** named `PAT_TOKEN` in your repository settings\n3. **Update your workflow** to use the PAT at checkout and add recursion prevention:\n\n```yaml\njobs:\n  test-maintenance:\n    runs-on: ubuntu-latest\n    # Prevent infinite loops - on push events, skip if triggered by Testbot's own commits\n    # head_commit is only available on push events; pull_request events are handled by the action-level check\n    if: github.event_name != 'push' || github.event.head_commit.author.name != 'Skyramp Testbot'\n\n    steps:\n      - uses: actions/checkout@v4\n        with:\n          fetch-depth: 0\n          token: ${{ secrets.PAT_TOKEN }}  # Use PAT instead of GITHUB_TOKEN\n\n      - uses: skyramp/testbot@v0.9.0\n        with:\n          skyrampLicenseFile: ${{ secrets.SKYRAMP_LICENSE }}\n          cursorApiKey: ${{ secrets.CURSOR_API_KEY }}\n```\n\nSee [examples/trigger-workflows.yml](examples/trigger-workflows.yml) for a complete example.\n\n### How It Works\n\nThe recursion prevention has two layers:\n\n1. **Job-level condition**: On `push` events, skips the entire job if the commit was made by Testbot\n2. **Action-level detection**: The action detects self-triggers (using `git log` for `pull_request` events where `head_commit` is unavailable) and exits gracefully\n\nThis ensures Testbot never runs on its own commits while allowing other workflows to run normally.\n\n## How It Works\n\n1. **Change Detection** - Generates a git diff between the base branch and current PR\n2. **License Setup** - Configures Skyramp license from secrets\n3. **Environment Setup** - Installs Node.js, Skyramp MCP, and selected AI agent CLI\n4. **MCP Configuration** - Configures the Skyramp MCP server for agent access\n5. **Service Startup** - Starts your services using the configured command\n6. **AI Analysis** - AI agent analyzes changes and identifies test impacts\n7. **Test Maintenance** - Updates existing tests or generates new ones using Skyramp MCP\n8. **Test Execution** - Runs tests and validates results\n9. **Summary Generation** - Creates detailed summary of actions taken\n10. **PR Comment** - Posts summary to PR (if enabled)\n11. **Auto-commit** - Commits test changes (if enabled)\n\n## Troubleshooting\n\n### Common Issues\n\n**CLI installation fails**\n\n- Check runner network connectivity\n- Verify the installation endpoint is accessible\n- Try enabling debug mode: `enableDebug: true`\n- For Copilot: Ensure npm is working correctly\n\n**License validation errors**\n\n- Ensure license content is properly stored in GitHub Secrets\n- Check that license file is not expired\n- Verify secret name matches input parameter\n\n**Service startup issues**\n\n- Verify docker-compose.yml exists in repository\n- Check that Docker is available in runner\n- Use `skipTargetSetup: true` if services not needed\n- Customize with `targetSetupCommand` for different startup methods\n\n**Agent timeout or failures**\n\n- **Cursor**: Check API key is valid and has quota remaining\n- **Copilot**: Verify Copilot subscription is active and token is valid\n- Review agent logs for specific errors\n- Enable debug mode for more detailed output\n\nFor more detailed troubleshooting, see [docs/troubleshooting.md](docs/troubleshooting.md).\n\n## Configuration Guide\n\nFor advanced configuration options and patterns, see [docs/configuration.md](docs/configuration.md).\n\n## Security Best Practices\n\n1. **Never commit secrets** - Always use GitHub Secrets for sensitive values\n2. **Limit permissions** - Only grant necessary permissions in workflow\n3. **Pin versions** - Use specific versions (`@v1.0.0`) for production workflows\n4. **Review auto-commits** - Consider disabling auto-commit for sensitive repositories\n5. **Audit logs** - Enable debug mode periodically to review action behavior\n\n## Support\n\n- **Documentation**: [docs/](docs/)\n- **Website**: [skyramp.dev](https://skyramp.dev)\n\n## License\n\nThis project is licensed under the ISC License - see the [LICENSE](LICENSE) file for details.\n\n---\n\nMade with ⚡ by [Skyramp](https://skyramp.dev)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskyramp%2Ftestbot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskyramp%2Ftestbot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskyramp%2Ftestbot/lists"}