{"id":26644986,"url":"https://github.com/moinsen-dev/testronaut","last_synced_at":"2025-07-25T18:42:36.788Z","repository":{"id":284121908,"uuid":"953883730","full_name":"moinsen-dev/testronaut","owner":"moinsen-dev","description":"Testronaut is an AI-powered, containerized framework for end-to-end testing of CLI tools. It analyzes commands, generates test plans, verifies outputs semantically with LLMs, and runs everything safely in Docker.","archived":false,"fork":false,"pushed_at":"2025-03-24T09:28:01.000Z","size":394,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-24T09:40:18.964Z","etag":null,"topics":["ai","cli","docker","llm","python3","test-automation","testing-tools"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moinsen-dev.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-24T08:29:42.000Z","updated_at":"2025-03-24T09:28:05.000Z","dependencies_parsed_at":"2025-03-24T09:40:22.101Z","dependency_job_id":"5728a7fd-563e-474c-996c-26dccc7a52ef","html_url":"https://github.com/moinsen-dev/testronaut","commit_stats":null,"previous_names":["moinsen-dev/testronaut"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moinsen-dev%2Ftestronaut","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moinsen-dev%2Ftestronaut/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moinsen-dev%2Ftestronaut/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moinsen-dev%2Ftestronaut/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moinsen-dev","download_url":"https://codeload.github.com/moinsen-dev/testronaut/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245352287,"owners_count":20601122,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","cli","docker","llm","python3","test-automation","testing-tools"],"created_at":"2025-03-24T21:21:16.291Z","updated_at":"2025-07-25T18:42:36.776Z","avatar_url":"https://github.com/moinsen-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Version](https://img.shields.io/badge/version-0.5.0-blue.svg)](https://github.com/your-username/testronaut)\n[![Code Coverage](https://img.shields.io/badge/coverage-72%25-yellow.svg)](https://github.com/your-username/testronaut)\n[![CI Status](https://img.shields.io/github/actions/workflow/status/your-username/testronaut/main.yml?label=CI)](https://github.com/your-username/testronaut/actions/workflows/main.yml)\n[![Documentation Status](https://img.shields.io/github/actions/workflow/status/your-username/testronaut/docs.yml?label=docs)](https://github.com/your-username/testronaut/actions/workflows/docs.yml)\n[![Security Status](https://img.shields.io/github/actions/workflow/status/your-username/testronaut/security.yml?label=security)](https://github.com/your-username/testronaut/actions/workflows/security.yml)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](http://makeapullrequest.com)\n[![Contributions Welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg)](https://github.com/your-username/testronaut/issues)\n[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/your-username/testronaut/graphs/commit-activity)\n[![Ask Me Anything !](https://img.shields.io/badge/Ask%20me-anything-1abc9c.svg)](https://github.com/your-username/testronaut/discussions)\n\n# Testronaut 🚀\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/testronaut-logo.png\" alt=\"Testronaut Logo\" width=\"300\"/\u003e\n  \u003cp\u003e\u003cem\u003eAI-powered testing for CLI tools - from analysis to execution\u003c/em\u003e\u003c/p\u003e\n\u003c/div\u003e\n\nTestronaut is an AI-powered framework that revolutionizes CLI testing by automatically analyzing command-line tools, generating comprehensive test plans, and executing them with intelligent result verification.\n\n## 🌟 What's New in v0.5.0\n\n- ✨ **Test Plan Generation**: Automatically generate test plans from analyzed CLI tools\n- 🚀 **Test Execution**: Execute test plans locally with detailed result reporting\n- 📊 **Smart Analysis**: Enhanced CLI analysis with better command discovery\n- 🎯 **End-to-End Workflow**: Complete pipeline from analysis to test execution\n\n## 🎬 Quick Demo\n\n```bash\n# Analyze a CLI tool\n$ testronaut analyze git\n✓ Found 27 commands in git\n\n# Generate a test plan\n$ testronaut generate plan git --output git_tests.json\n✓ Generated 54 test cases\n\n# Execute the tests\n$ testronaut execute plan git_tests.json\n✓ Executing 54 tests...\n✓ 21 passed, 33 failed (commands need proper setup)\n```\n\n## 🚀 Key Features\n\n### 🔍 Intelligent CLI Analysis\n- Automatically discovers commands and subcommands\n- Extracts options, arguments, and help text\n- Handles complex command hierarchies\n- LLM-enhanced parsing for challenging formats\n\n### 🧪 Smart Test Generation\n- Creates comprehensive test plans automatically\n- Generates both positive and negative test cases\n- Produces executable pytest files\n- Covers help commands, basic usage, and edge cases\n\n### ⚡ Automated Test Execution\n- Runs tests in isolated environments\n- Captures stdout, stderr, and exit codes\n- Provides detailed failure reports\n- Supports timeout configuration\n\n### 🤖 AI-Powered Verification (Coming Soon)\n- Semantic comparison of outputs\n- Handles dynamic content gracefully\n- Reduces false positives\n- Supports multiple LLM providers\n\n## 📦 Installation\n\n### Development Setup\n\nWe use [uv](https://github.com/astral-sh/uv) for fast, reliable Python packaging:\n\n```bash\n# Install uv (if not already installed)\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n\n# Clone and setup\ngit clone https://github.com/yourusername/testronaut.git\ncd testronaut\n./setup.sh\n\n# Activate environment\nsource .venv/bin/activate\n```\n\n### Quick Start\n\n```bash\n# Configure your LLM provider (optional for basic features)\ntestronaut config set-key openai YOUR_API_KEY\n\n# Analyze a CLI tool\ntestronaut analyze npm\n\n# Generate a test plan\ntestronaut generate plan npm --output npm_tests.json\n\n# Execute the tests\ntestronaut execute plan npm_tests.json\n\n# View analyzed tools\ntestronaut generate list\n```\n\n## 🛠️ Usage Examples\n\n### Analyze Popular CLI Tools\n\n```bash\n# Analyze git\ntestronaut analyze git\n\n# Analyze with enhanced LLM parsing\ntestronaut analyze kubectl --enhanced\n\n# Interactive browser for results\ntestronaut analyze browser\n```\n\n### Generate Test Plans\n\n```bash\n# Generate JSON test plan\ntestronaut generate plan docker --output docker_tests.json\n\n# Generate pytest file too\ntestronaut generate plan npm --pytest tests/test_npm.py\n\n# List all analyzed tools\ntestronaut generate list\n```\n\n### Execute Tests\n\n```bash\n# Execute test plan\ntestronaut execute plan my_tests.json\n\n# Custom timeout (seconds)\ntestronaut execute plan my_tests.json --timeout 10\n\n# Save results to specific directory\ntestronaut execute plan my_tests.json --output results/\n```\n\n## 🏗️ Architecture\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/testronaut-architecture.png\" alt=\"Testronaut Architecture\" width=\"700\"/\u003e\n\u003c/div\u003e\n\nTestronaut consists of five main components:\n\n1. **CLI Analyzer**: Parses and understands command structure\n2. **Test Generator**: Creates comprehensive test plans\n3. **Test Executor**: Runs tests in isolated environments\n4. **Result Verifier**: Compares expected vs actual results\n5. **LLM Integration**: Enhances analysis and verification\n\n## 🧪 Project Status\n\n| Phase | Component | Status | Progress |\n|-------|-----------|--------|----------|\n| 004 | CLI Analysis Engine | ✅ Completed | 100% |\n| 005 | Test Plan Generator | ✅ Completed | 100% |\n| 006 | Test Executor (Local) | ✅ Completed | 100% |\n| 007 | Docker Execution | 🚧 In Progress | 20% |\n| 008 | AI Result Verification | 📋 Planned | 0% |\n| 009 | Reporting System | 📋 Planned | 0% |\n\n## 🤝 Contributing\n\nWe welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n### Development Workflow\n\n```bash\n# Run tests\nuv run pytest\n\n# Check code quality\nruff check .\nruff format .\n\n# Type checking\nmypy src/testronaut\n\n# Run all checks\npre-commit run --all-files\n```\n\n## 📊 Roadmap\n\n### Near Term (v0.6.0)\n- [ ] Docker-based test execution\n- [ ] Basic semantic verification\n- [ ] HTML test reports\n- [ ] Test retry mechanisms\n\n### Medium Term (v0.7.0)\n- [ ] Advanced LLM verification\n- [ ] Parallel test execution\n- [ ] CI/CD integration helpers\n- [ ] Test coverage analysis\n\n### Long Term (v1.0.0)\n- [ ] Multi-language CLI support\n- [ ] Distributed test execution\n- [ ] Custom test generators\n- [ ] Enterprise features\n\n## 📚 Documentation\n\nFull documentation is available at our [documentation site](https://your-username.github.io/testronaut/).\n\n- [Installation Guide](docs/guides/installation.md)\n- [Quick Start Tutorial](docs/guides/quickstart.md)\n- [Architecture Overview](docs/api/overview.md)\n- [CLI Reference](docs/reference/cli.md)\n\n## 🙏 Acknowledgments\n\n- Built with [Typer](https://typer.tiangolo.com/) for the CLI\n- Uses [Rich](https://rich.readthedocs.io/) for beautiful terminal output\n- Powered by [SQLModel](https://sqlmodel.tiangolo.com/) for data persistence\n- Package management by [uv](https://github.com/astral-sh/uv)\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\n\u003cdiv align=\"center\"\u003e\n  \u003cp\u003eMade with ❤️ by the Testronaut team\u003c/p\u003e\n  \u003cp\u003e\n    \u003ca href=\"https://github.com/yourusername/testronaut/issues\"\u003eReport Bug\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/yourusername/testronaut/issues\"\u003eRequest Feature\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/yourusername/testronaut/discussions\"\u003eJoin Discussion\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/div\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoinsen-dev%2Ftestronaut","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoinsen-dev%2Ftestronaut","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoinsen-dev%2Ftestronaut/lists"}