{"id":31419403,"url":"https://github.com/compiler-explorer/ce-stresstest","last_synced_at":"2025-09-29T22:56:53.535Z","repository":{"id":307008477,"uuid":"1028008548","full_name":"compiler-explorer/ce-stresstest","owner":"compiler-explorer","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-29T11:02:00.000Z","size":96,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-19T09:10:07.043Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/compiler-explorer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-28T22:07:27.000Z","updated_at":"2025-09-14T19:07:52.000Z","dependencies_parsed_at":"2025-07-29T00:26:26.586Z","dependency_job_id":"3bcdaef6-2998-4ad1-b80b-7c10d800edcc","html_url":"https://github.com/compiler-explorer/ce-stresstest","commit_stats":null,"previous_names":["compiler-explorer/ce-stresstest"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/compiler-explorer/ce-stresstest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compiler-explorer%2Fce-stresstest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compiler-explorer%2Fce-stresstest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compiler-explorer%2Fce-stresstest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compiler-explorer%2Fce-stresstest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/compiler-explorer","download_url":"https://codeload.github.com/compiler-explorer/ce-stresstest/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compiler-explorer%2Fce-stresstest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277601097,"owners_count":25845629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-29T02:00:09.175Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-29T22:56:50.698Z","updated_at":"2025-09-29T22:56:53.527Z","avatar_url":"https://github.com/compiler-explorer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Compiler Explorer Stress Testing Framework\n\nA comprehensive stress testing framework for Compiler Explorer's compilation workers. This framework tests the performance and scalability of SQS-based compilation worker infrastructure using realistic workloads and various load patterns.\n\n## Features\n\n- **Multiple Load Patterns**: Steady, burst, ramp, wave, and sustained high load patterns\n- **Realistic Workloads**: CPU-intensive, memory-heavy, IO-intensive, quick, and error scenarios\n- **Scaling Analysis**: Test performance across different worker instance counts\n- **Real-time Monitoring**: Live dashboard with metrics and alerts\n- **Comprehensive Reporting**: HTML reports with visualizations and analysis\n- **Baseline Validation**: Automatic detection of performance regressions\n- **CLI and Interactive Modes**: Flexible usage options\n\n## Quick Start\n\n### Installation\n\n1. Install dependencies using Poetry:\n```bash\ncd stress-test\npoetry install\n```\n\n2. Or using pip:\n```bash\npip install -r requirements.txt  # If you had created this instead of pyproject.toml\n```\n\n### Basic Usage\n\n#### Command Line Interface\n\nRun a simple steady load test:\n```bash\npoetry run python run_stress_test.py steady --rps 10 --duration 300\n```\n\nRun a scaling test:\n```bash\npoetry run python run_stress_test.py scaling --instances 2,4,6,8 --rps-per-instance 5 --duration 180\n```\n\nRun a burst load test:\n```bash\npoetry run python run_stress_test.py burst --baseline-rps 5 --burst-rps 20 --duration 600\n```\n\n#### Interactive Mode\n\nFor guided configuration:\n```bash\npoetry run python run_stress_test.py interactive\n```\n\n#### Using the Framework Directly\n\n```python\nimport asyncio\nfrom src.framework import CompilerStressTest, TestConfiguration\n\nasync def main():\n    config = TestConfiguration(\n        endpoint=\"https://beta.compiler-explorer.com\",\n        compiler=\"g122\",\n        scenarios=[\"cpu_intensive\", \"simple\", \"memory_heavy\"]\n    )\n    \n    async with CompilerStressTest(config) as tester:\n        # Run steady load test\n        summary = await tester.steady_load_test(rps=10, duration_seconds=300)\n        \n        # Run scaling test\n        results = await tester.scaling_test(\n            instance_counts=[2, 4, 6, 8],\n            rps_per_instance=5,\n            duration_per_test=180\n        )\n\nasyncio.run(main())\n```\n\n## Available Test Patterns\n\n### Steady Load\nMaintains constant requests per second throughout the test duration.\n```bash\npoetry run python run_stress_test.py steady --rps 15 --duration 300\n```\n\n### Burst Load\nBaseline traffic with periodic bursts of higher load.\n```bash\npoetry run python run_stress_test.py burst \\\n    --baseline-rps 5 \\\n    --burst-rps 25 \\\n    --duration 600 \\\n    --burst-duration 30 \\\n    --burst-interval 120\n```\n\n### Ramp Tests\nGradually increase (ramp up) or decrease (ramp down) load over time.\n```bash\n# Ramp up\npoetry run python run_stress_test.py ramp --min-rps 1 --max-rps 20 --duration 300\n\n# Ramp down\npoetry run python run_stress_test.py ramp --min-rps 1 --max-rps 20 --duration 300 --ramp-down\n```\n\n### Wave Pattern\nSinusoidal load variation for testing response to cyclical traffic.\n```bash\npoetry run python run_stress_test.py wave \\\n    --base-rps 10 \\\n    --amplitude-rps 5 \\\n    --duration 600 \\\n    --period 300\n```\n\n### Scaling Tests\nTest performance across different worker instance counts.\n```bash\npoetry run python run_stress_test.py scaling \\\n    --instances 2,4,6,8,10 \\\n    --rps-per-instance 5 \\\n    --duration 180\n```\n\n## Workload Scenarios\n\nThe framework includes several built-in workload scenarios:\n\n### CPU-intensive Scenarios\n- **cpu_intensive**: Heavy template metaprogramming with recursive instantiation\n- Complex optimization with loop unrolling and LTO\n- Recursive constexpr computation\n\n### Memory-intensive Scenarios  \n- **memory_heavy**: Large static array allocations and deep template instantiation\n- Heavy macro expansion\n- Deep template instantiation trees\n\n### IO-intensive Scenarios\n- **io_intensive**: Many standard library includes and large generated output\n- Large debug symbol generation\n\n### Quick Scenarios\n- **simple**: Basic \"Hello World\" program\n- Simple mathematical operations\n- Basic class definitions\n\n### Error Scenarios\n- **error_syntax**: Syntax error compilation failures\n- **error_template**: Template instantiation errors\n- Missing header file errors\n\n## Configuration\n\n### Environment Variables\n- `DEBUG_API=1`: Enable detailed API response logging\n- `VERBOSE=1`: Enable verbose output\n\n### Custom Scenarios\nAdd your own C++ files to `examples/workloads/` with metadata comments:\n\n```cpp\n// compile: -O3 -std=c++20\n// baseline_min_ms: 500\n// baseline_max_ms: 2000\n// weight: 0.8\n// description: Custom CPU-intensive workload\n\n#include \u003ciostream\u003e\n// Your code here...\n```\n\n### Configuration Files\nUse YAML configuration files for complex test setups:\n\n```yaml\n# examples/configs/my_test.yaml\ntest_name: \"custom_test\"\nendpoint: \"https://beta.compiler-explorer.com\"\ncompiler: \"g122\"\nscenarios: [\"cpu_intensive\", \"simple\"]\ntest_patterns:\n  - type: \"steady\"\n    rps: 10.0\n    duration_seconds: 300\n```\n\n## CLI Commands\n\n### Available Commands\n\n```bash\n# List all available scenarios\npoetry run python run_stress_test.py list-scenarios\n\n# Generate report from results\npoetry run python run_stress_test.py report results/test_results.json\n\n# Interactive configuration mode\npoetry run python run_stress_test.py interactive\n```\n\n### Global Options\n\nMost commands support these options:\n- `--endpoint`: API endpoint (default: https://beta.compiler-explorer.com)\n- `--compiler`: Compiler ID (default: g122)\n- `--concurrent`: Max concurrent requests (default: 50)\n- `--scenarios`: Comma-separated scenario names\n- `--workload-dir`: Custom workload directory\n- `--results-dir`: Results output directory (default: results)\n- `--no-dashboard`: Disable live dashboard\n- `--test-name`: Custom test name\n\n## Output and Reporting\n\n### Results Structure\n```\nresults/\n├── test_name_timestamp_summary.json    # Test summary\n├── test_name_timestamp_raw.json        # Raw request/response data\n└── reports/\n    ├── test_name_report.html           # HTML report\n    └── test_name_charts/               # Generated charts\n        ├── latency_distribution.png\n        ├── throughput_timeline.png\n        ├── success_rate_comparison.png\n        ├── error_breakdown.png\n        ├── scaling_analysis.png\n        └── performance_heatmap.png\n```\n\n### Metrics Collected\n- **Latency**: Mean, median, P95, P99, min, max response times\n- **Throughput**: Requests per second, successful RPS\n- **Success Rate**: Percentage of successful compilations\n- **Error Breakdown**: Categorized error types and frequencies\n- **Baseline Violations**: Responses outside expected time ranges\n- **Scaling Efficiency**: Performance across instance counts\n\n### HTML Reports\nComprehensive reports include:\n- Executive summary with key metrics\n- Interactive charts and visualizations\n- Detailed test results breakdown\n- Alerts and performance issues\n- Scaling analysis (when applicable)\n\n## Advanced Usage\n\n### Custom Load Patterns\n```python\nfrom src.load_patterns import LoadPatternFactory\n\n# Create custom pattern\npattern = LoadPatternFactory.create_pattern_from_config({\n    \"type\": \"burst\",\n    \"baseline_rps\": 5,\n    \"burst_rps\": 30,\n    \"duration_seconds\": 600,\n    \"burst_duration_seconds\": 45,\n    \"burst_interval_seconds\": 150\n}, scenarios)\n```\n\n### Performance Analysis\n```python\nfrom src.metrics import PerformanceAnalyzer\n\n# Detect regressions\nregressions = PerformanceAnalyzer.detect_performance_regression(\n    current_metrics, baseline_metrics, regression_threshold=0.15\n)\n\n# Analyze scaling efficiency\nscaling_analysis = PerformanceAnalyzer.analyze_scaling_efficiency(\n    scaling_results\n)\n```\n\n### Custom Scenarios\n```python\nfrom src.scenarios import create_custom_scenario\nfrom pathlib import Path\n\nscenario = create_custom_scenario(\n    name=\"my_custom_test\",\n    source_file=Path(\"my_code.cpp\"),\n    compiler_options=\"-O3 -std=c++20\",\n    baseline_min_ms=200,\n    baseline_max_ms=1000,\n    description=\"Custom optimization test\"\n)\n```\n\n## Best Practices\n\n### Test Design\n1. **Start Small**: Begin with low RPS and short durations\n2. **Baseline First**: Establish performance baselines before stress testing\n3. **Gradual Scaling**: Increase load gradually to find breaking points\n4. **Monitor Resources**: Watch for API rate limits and server constraints\n5. **Realistic Workloads**: Use scenarios that match your actual usage patterns\n\n### Performance Monitoring\n1. **Live Dashboard**: Use the live dashboard for real-time monitoring\n2. **Baseline Validation**: Set appropriate baseline thresholds for your scenarios\n3. **Alert Thresholds**: Configure alerts for critical performance metrics\n4. **Regular Testing**: Run tests regularly to detect performance regressions\n\n### Troubleshooting\n1. **API Limits**: Reduce `--rate-limit-rps` if hitting API limits\n2. **Timeouts**: Increase `--timeout` for slow compilations\n3. **Memory Issues**: Reduce `--concurrent` for large workloads\n4. **Network Issues**: Use `--debug-api` to diagnose API problems\n\n## Examples\n\n### Comprehensive Baseline Test\n```bash\npoetry run python run_stress_test.py steady \\\n    --rps 8 \\\n    --duration 600 \\\n    --scenarios \"simple,cpu_intensive,memory_heavy,io_intensive\" \\\n    --test-name \"nightly_baseline\" \\\n    --results-dir \"results/baseline\"\n```\n\n### Performance Regression Testing\n```bash\n# Run current build\npoetry run python run_stress_test.py steady --rps 10 --duration 300 --test-name \"current_build\"\n\n# Compare with baseline (manual process using reports)\npoetry run python run_stress_test.py report results/current_build_*_summary.json\n```\n\n### Scaling Analysis\n```bash\npoetry run python run_stress_test.py scaling \\\n    --instances 2,4,6,8,10,12 \\\n    --rps-per-instance 4 \\\n    --duration 240 \\\n    --scenarios \"simple,cpu_intensive\" \\\n    --test-name \"scaling_analysis_v2\"\n```\n\n### High Load Stress Test\n```bash\npoetry run python run_stress_test.py burst \\\n    --baseline-rps 15 \\\n    --burst-rps 75 \\\n    --duration 900 \\\n    --burst-duration 60 \\\n    --burst-interval 180 \\\n    --concurrent 100 \\\n    --test-name \"high_load_stress\"\n```\n\n## Contributing\n\n1. Add new workload scenarios to `examples/workloads/`\n2. Extend load patterns in `src/load_patterns.py`\n3. Improve metrics collection in `src/metrics.py`\n4. Enhance reporting in `src/reporting.py`\n\n## License\n\nThis framework is provided as-is for testing Compiler Explorer infrastructure. Use responsibly and respect API rate limits.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompiler-explorer%2Fce-stresstest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcompiler-explorer%2Fce-stresstest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompiler-explorer%2Fce-stresstest/lists"}