https://github.com/moinsen-dev/testronaut
Testronaut is an AI-powered, containerized framework for end-to-end testing of CLI tools. It analyzes commands, generates test plans, verifies outputs semantically with LLMs, and runs everything safely in Docker.
https://github.com/moinsen-dev/testronaut
ai cli docker llm python3 test-automation testing-tools
Last synced: 10 months ago
JSON representation
Testronaut is an AI-powered, containerized framework for end-to-end testing of CLI tools. It analyzes commands, generates test plans, verifies outputs semantically with LLMs, and runs everything safely in Docker.
- Host: GitHub
- URL: https://github.com/moinsen-dev/testronaut
- Owner: moinsen-dev
- License: mit
- Created: 2025-03-24T08:29:42.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-24T09:28:01.000Z (about 1 year ago)
- Last Synced: 2025-03-24T09:40:18.964Z (about 1 year ago)
- Topics: ai, cli, docker, llm, python3, test-automation, testing-tools
- Language: Python
- Homepage:
- Size: 385 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://opensource.org/licenses/MIT)
[](https://github.com/your-username/testronaut)
[](https://github.com/your-username/testronaut)
[](https://github.com/your-username/testronaut/actions/workflows/main.yml)
[](https://github.com/your-username/testronaut/actions/workflows/docs.yml)
[](https://github.com/your-username/testronaut/actions/workflows/security.yml)
[](http://makeapullrequest.com)
[](https://github.com/your-username/testronaut/issues)
[](https://github.com/your-username/testronaut/graphs/commit-activity)
[](https://github.com/your-username/testronaut/discussions)
# Testronaut 🚀
AI-powered testing for CLI tools - from analysis to execution
Testronaut is an AI-powered framework that revolutionizes CLI testing by automatically analyzing command-line tools, generating comprehensive test plans, and executing them with intelligent result verification.
## 🌟 What's New in v0.5.0
- ✨ **Test Plan Generation**: Automatically generate test plans from analyzed CLI tools
- 🚀 **Test Execution**: Execute test plans locally with detailed result reporting
- 📊 **Smart Analysis**: Enhanced CLI analysis with better command discovery
- 🎯 **End-to-End Workflow**: Complete pipeline from analysis to test execution
## 🎬 Quick Demo
```bash
# Analyze a CLI tool
$ testronaut analyze git
✓ Found 27 commands in git
# Generate a test plan
$ testronaut generate plan git --output git_tests.json
✓ Generated 54 test cases
# Execute the tests
$ testronaut execute plan git_tests.json
✓ Executing 54 tests...
✓ 21 passed, 33 failed (commands need proper setup)
```
## 🚀 Key Features
### 🔍 Intelligent CLI Analysis
- Automatically discovers commands and subcommands
- Extracts options, arguments, and help text
- Handles complex command hierarchies
- LLM-enhanced parsing for challenging formats
### 🧪 Smart Test Generation
- Creates comprehensive test plans automatically
- Generates both positive and negative test cases
- Produces executable pytest files
- Covers help commands, basic usage, and edge cases
### ⚡ Automated Test Execution
- Runs tests in isolated environments
- Captures stdout, stderr, and exit codes
- Provides detailed failure reports
- Supports timeout configuration
### 🤖 AI-Powered Verification (Coming Soon)
- Semantic comparison of outputs
- Handles dynamic content gracefully
- Reduces false positives
- Supports multiple LLM providers
## 📦 Installation
### Development Setup
We use [uv](https://github.com/astral-sh/uv) for fast, reliable Python packaging:
```bash
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Clone and setup
git clone https://github.com/yourusername/testronaut.git
cd testronaut
./setup.sh
# Activate environment
source .venv/bin/activate
```
### Quick Start
```bash
# Configure your LLM provider (optional for basic features)
testronaut config set-key openai YOUR_API_KEY
# Analyze a CLI tool
testronaut analyze npm
# Generate a test plan
testronaut generate plan npm --output npm_tests.json
# Execute the tests
testronaut execute plan npm_tests.json
# View analyzed tools
testronaut generate list
```
## 🛠️ Usage Examples
### Analyze Popular CLI Tools
```bash
# Analyze git
testronaut analyze git
# Analyze with enhanced LLM parsing
testronaut analyze kubectl --enhanced
# Interactive browser for results
testronaut analyze browser
```
### Generate Test Plans
```bash
# Generate JSON test plan
testronaut generate plan docker --output docker_tests.json
# Generate pytest file too
testronaut generate plan npm --pytest tests/test_npm.py
# List all analyzed tools
testronaut generate list
```
### Execute Tests
```bash
# Execute test plan
testronaut execute plan my_tests.json
# Custom timeout (seconds)
testronaut execute plan my_tests.json --timeout 10
# Save results to specific directory
testronaut execute plan my_tests.json --output results/
```
## 🏗️ Architecture
Testronaut consists of five main components:
1. **CLI Analyzer**: Parses and understands command structure
2. **Test Generator**: Creates comprehensive test plans
3. **Test Executor**: Runs tests in isolated environments
4. **Result Verifier**: Compares expected vs actual results
5. **LLM Integration**: Enhances analysis and verification
## 🧪 Project Status
| Phase | Component | Status | Progress |
|-------|-----------|--------|----------|
| 004 | CLI Analysis Engine | ✅ Completed | 100% |
| 005 | Test Plan Generator | ✅ Completed | 100% |
| 006 | Test Executor (Local) | ✅ Completed | 100% |
| 007 | Docker Execution | 🚧 In Progress | 20% |
| 008 | AI Result Verification | 📋 Planned | 0% |
| 009 | Reporting System | 📋 Planned | 0% |
## 🤝 Contributing
We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
### Development Workflow
```bash
# Run tests
uv run pytest
# Check code quality
ruff check .
ruff format .
# Type checking
mypy src/testronaut
# Run all checks
pre-commit run --all-files
```
## 📊 Roadmap
### Near Term (v0.6.0)
- [ ] Docker-based test execution
- [ ] Basic semantic verification
- [ ] HTML test reports
- [ ] Test retry mechanisms
### Medium Term (v0.7.0)
- [ ] Advanced LLM verification
- [ ] Parallel test execution
- [ ] CI/CD integration helpers
- [ ] Test coverage analysis
### Long Term (v1.0.0)
- [ ] Multi-language CLI support
- [ ] Distributed test execution
- [ ] Custom test generators
- [ ] Enterprise features
## 📚 Documentation
Full documentation is available at our [documentation site](https://your-username.github.io/testronaut/).
- [Installation Guide](docs/guides/installation.md)
- [Quick Start Tutorial](docs/guides/quickstart.md)
- [Architecture Overview](docs/api/overview.md)
- [CLI Reference](docs/reference/cli.md)
## 🙏 Acknowledgments
- Built with [Typer](https://typer.tiangolo.com/) for the CLI
- Uses [Rich](https://rich.readthedocs.io/) for beautiful terminal output
- Powered by [SQLModel](https://sqlmodel.tiangolo.com/) for data persistence
- Package management by [uv](https://github.com/astral-sh/uv)
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
---