{"id":31738202,"url":"https://github.com/kilo-org/alex-trebench","last_synced_at":"2025-10-09T09:55:49.751Z","repository":{"id":312580722,"uuid":"1047871237","full_name":"Kilo-Org/alex-treBENCH","owner":"Kilo-Org","description":"🎮 Benchmark LLMs with Jeopardy! questions. Tournament-style testing for large language models. What is... your model's true performance?","archived":false,"fork":false,"pushed_at":"2025-09-15T01:42:31.000Z","size":1072,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-15T03:36:27.244Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kilo-Org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-31T12:28:52.000Z","updated_at":"2025-09-15T01:42:35.000Z","dependencies_parsed_at":"2025-08-31T18:24:06.962Z","dependency_job_id":"ffe6892c-6a0a-4785-968b-e49142384b48","html_url":"https://github.com/Kilo-Org/alex-treBENCH","commit_stats":null,"previous_names":["kilo-org/alex-trebench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Kilo-Org/alex-treBENCH","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kilo-Org%2Falex-treBENCH","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kilo-Org%2Falex-treBENCH/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kilo-Org%2Falex-treBENCH/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kilo-Org%2Falex-treBENCH/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kilo-Org","download_url":"https://codeload.github.com/Kilo-Org/alex-treBENCH/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kilo-Org%2Falex-treBENCH/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001065,"owners_count":26083022,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-09T09:55:48.668Z","updated_at":"2025-10-09T09:55:49.744Z","avatar_url":"https://github.com/Kilo-Org.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# alex-treBENCH!\n\nJeopardy Language Model Benchmarking System.\n\n🎮 Benchmark LLMs with Jeopardy! questions. Tournament-style testing for large language models. What is... your model's true performance?\n\n[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Tests](https://img.shields.io/badge/tests-passing-green.svg)](tests/)\n\nA comprehensive benchmarking application that evaluates language models using Jeopardy questions from Kaggle, providing statistically significant and repeatable performance analysis through OpenRouter's API.\n\n## 🎯 Project Overview\n\nThis system is designed to:\n\n- ✅ Test multiple language models simultaneously using authentic Jeopardy questions\n- ✅ Provide statistically significant benchmarking with proper sampling methodologies\n- ✅ Measure key performance metrics: accuracy, response speed, cost efficiency, and consistency\n- ✅ Generate comprehensive reports with category and difficulty-level analysis\n- ✅ Support both CLI interface and future web interface expansion\n\n## 📋 Key Features\n\n### Core Capabilities\n\n- **Statistical Sampling**: Scientifically valid question selection ensuring 95% confidence level\n- **Fuzzy Answer Matching**: Intelligent answer evaluation handling variations and formats\n- **Multi-Model Support**: Concurrent testing of 5-10 language models via OpenRouter API\n- **Comprehensive Metrics**: Accuracy, latency, tokens/second, cost analysis, and consistency tracking\n- **Category Analysis**: Performance breakdown by Jeopardy categories and difficulty levels\n- **Reproducible Results**: Deterministic benchmarking with configurable parameters\n\n### Performance Metrics Tracked\n\n- Response accuracy (correct/incorrect with confidence scoring)\n- Response speed (latency and tokens per second)\n- Cost per query and cost-effectiveness ratios\n- Model consistency across similar question types\n- Category-specific performance analysis\n- Difficulty-level performance based on Jeopardy dollar values\n\n## 🏗️ Architecture Overview\n\n### System Components\n\n```mermaid\ngraph TB\n    A[Data Ingestion Layer] --\u003e B[Question Selection Engine]\n    B --\u003e C[Model Testing Engine]\n    C --\u003e D[Answer Evaluation Engine]\n    D --\u003e E[Metrics Calculation Engine]\n    E --\u003e F[Results Storage Layer]\n    F --\u003e G[Reporting \u0026 Analytics]\n\n    H[OpenRouter API] --\u003e C\n    I[Kaggle Dataset] --\u003e A\n    J[SQLite Database] --\u003e F\n```\n\n### Technology Stack\n\n- **Backend**: Python 3.8+ with async/await support\n- **Database**: SQLite with SQLAlchemy ORM\n- **API Integration**: OpenRouter via aiohttp\n- **Data Processing**: Pandas, NumPy for statistical analysis\n- **Text Matching**: FuzzyWuzzy with Levenshtein distance\n- **CLI Interface**: Click with Rich for enhanced output\n- **Testing**: Pytest with async support\n\n## 📁 Project Structure\n\n```\nalex-trebench/\n├── config/                    # Configuration files (YAML)\n│   ├── default.yaml           # Main configuration\n│   └── models/                # Model-specific settings\n├── src/\n│   ├── main.py                # CLI entry point (alex command)\n│   ├── core/                  # Foundation components\n│   ├── data/                  # Data ingestion and preprocessing\n│   ├── models/                # LLM API clients and adapters\n│   ├── evaluation/            # Answer matching and grading\n│   ├── benchmark/             # Execution engine and reporting\n│   ├── storage/               # Database models and repositories\n│   ├── cli/                   # Command-line interface\n│   ├── commands/              # Command implementations\n│   └── utils/                 # Shared utilities\n├── tests/                     # Comprehensive test suite\n│   ├── unit/                  # Unit tests\n│   ├── integration/           # Integration tests\n│   └── e2e/                   # End-to-end tests\n├── docs/                      # Documentation\n│   ├── USER_GUIDE.md          # Complete user guide\n│   └── API_REFERENCE.md       # API documentation\n├── scripts/                   # Utility scripts\n├── examples/                  # Usage examples\n└── data/                      # Local data storage and cache\n```\n\n## 📖 Documentation\n\n### User Documentation\n\n- **[User Guide](docs/USER_GUIDE.md)**: Complete user guide with installation, configuration, and usage examples\n- **[API Reference](docs/API_REFERENCE.md)**: Comprehensive API documentation with code examples\n\n### Technical Documentation\n\n- **[Technical Specification](TECHNICAL_SPEC.md)**: Complete system architecture, database schema, algorithms, and API integration patterns\n- **[Project Structure](PROJECT_STRUCTURE.md)**: Detailed directory organization, module responsibilities, and technology stack\n- **[Implementation Roadmap](ROADMAP.md)**: Development phases, priorities, and delivery timeline\n\n### Key Specifications\n\n#### Statistical Sampling\n\n- **Sample Size**: 1000 questions for statistical significance (95% confidence, 5% margin of error)\n- **Stratified Sampling**: Proportional representation across categories and difficulty levels\n- **Reproducibility**: Configurable random seed for consistent benchmark runs\n\n#### Answer Evaluation Methodology\n\n- **Multi-level Matching**: Exact match, normalized comparison, and semantic similarity\n- **Fuzzy Scoring**: Weighted combination of similarity metrics with confidence thresholds\n- **Format Flexibility**: Handles Jeopardy answer format variations and common response patterns\n\n#### Performance Metrics\n\n```python\n# Core metrics calculated\naccuracy_rate = correct_answers / total_questions\navg_response_time = mean(response_times_ms)\ntokens_per_second = mean(tokens_generated / response_time_seconds)\ncost_per_correct = total_cost / correct_answers\nconsistency_score = 1 - std_deviation(response_times) / mean(response_times)\n```\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Python 3.8 or higher\n- uv (recommended) or pip for package management\n- OpenRouter API key (get one at [openrouter.ai](https://openrouter.ai))\n- Internet connection for API access\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone \u003crepository-url\u003e\ncd alex-trebench\n\n# Install using uv (recommended)\nuv pip install -e .\n\n# Or using pip\npip install -e .\n```\n\n### Configuration\n\n```bash\n# Set up environment variables\nexport OPENROUTER_API_KEY=\"your_api_key_here\"\n\n# Or create .env file\necho \"OPENROUTER_API_KEY=your_api_key_here\" \u003e .env\n\n# Initialize the database\nalex init\n```\n\n### Basic Usage\n\n```bash\n# Run a quick benchmark (50 questions)\nalex benchmark run --model openai/gpt-3.5-turbo --size quick\n\n# Run a standard benchmark (200 questions)\nalex benchmark run --model openai/gpt-4 --size standard\n\n# Compare multiple models\nalex benchmark compare --models \"openai/gpt-3.5-turbo,openai/gpt-4\" --size quick\n\n# View benchmark history\nalex benchmark history --model openai/gpt-4\n\n# Generate a report\nalex benchmark report --run-id 1 --format markdown\n```\n\n## 📊 Sample Output\n\n### Benchmark Results Summary\n\n```\n┌─────────────────┬──────────┬─────────────┬──────────────┬─────────────┐\n│ Model           │ Accuracy │ Avg Time    │ Cost/Query   │ Consistency │\n├─────────────────┼──────────┼─────────────┼──────────────┼─────────────┤\n│ gpt-4-turbo     │ 73.2%    │ 1,240ms     │ $0.003       │ 0.89        │\n│ claude-3-sonnet │ 71.8%    │ 980ms       │ $0.002       │ 0.92        │\n│ gpt-3.5-turbo   │ 64.5%    │ 650ms       │ $0.001       │ 0.85        │\n└─────────────────┴──────────┴─────────────┴──────────────┴─────────────┘\n\nCategory Performance:\n• Science \u0026 Technology: GPT-4 (78%) \u003e Claude-3 (75%) \u003e GPT-3.5 (68%)\n• History: Claude-3 (74%) \u003e GPT-4 (72%) \u003e GPT-3.5 (63%)\n• Literature: GPT-4 (69%) \u003e Claude-3 (67%) \u003e GPT-3.5 (59%)\n```\n\n## 📈 Implementation Status\n\n### ✅ Completed Features\n\n- **Core Infrastructure**: Complete project setup with modular architecture\n- **Data Pipeline**: Kaggle integration, preprocessing, and statistical sampling\n- **Model Integration**: OpenRouter API client with support for 20+ models\n- **Benchmark Engine**: Complete benchmarking workflow with async processing\n- **Evaluation System**: Fuzzy answer matching, grading, and metrics calculation\n- **Database Layer**: SQLite with SQLAlchemy ORM and migration support\n- **CLI Interface**: Comprehensive command-line interface with Rich formatting\n- **Reporting System**: Multiple output formats (terminal, markdown, JSON)\n- **Testing Suite**: Unit, integration, and end-to-end tests with 80%+ coverage\n- **Documentation**: Complete user guide and API reference\n\n### 🚧 Current Development\n\n- **Performance Optimization**: Memory usage and concurrent processing improvements\n- **Web Interface**: Optional FastAPI-based REST API (future enhancement)\n- **Advanced Analytics**: Trend analysis and model comparison tools\n\n### 📋 Usage Examples\n\n#### Single Model Benchmark\n\n```bash\n# Quick test with GPT-3.5-turbo\nalex benchmark run --model openai/gpt-3.5-turbo --size quick\n\n# Comprehensive evaluation with GPT-4\nalex benchmark run \\\n  --model openai/gpt-4 \\\n  --size comprehensive \\\n  --name \"GPT-4 Comprehensive Test\" \\\n  --report-format markdown \\\n  --output gpt4_report.md\n```\n\n#### Model Comparison\n\n```bash\n# Compare popular models\nalex benchmark compare \\\n  --models \"openai/gpt-3.5-turbo,openai/gpt-4,anthropic/claude-3-haiku\" \\\n  --size standard \\\n  --concurrent-limit 3\n\n# Generate comparison report\nalex benchmark compare \\\n  --models \"openai/gpt-4,anthropic/claude-3-sonnet\" \\\n  --size quick \\\n  --report-format json \\\n  --output model_comparison.json\n```\n\n#### Advanced Configuration\n\n```bash\n# Custom benchmark with specific settings\nalex benchmark run \\\n  --model openai/gpt-4 \\\n  --size custom \\\n  --sample-size 500 \\\n  --timeout 120 \\\n  --grading-mode lenient \\\n  --name \"Custom Benchmark\" \\\n  --description \"Testing with custom parameters\"\n```\n\n#### Data Management\n\n```bash\n# Initialize dataset\nalex data init\n\n# Sample questions by category\nalex data sample \\\n  --category \"SCIENCE\" \\\n  --size 100 \\\n  --output science_questions.json\n\n# View dataset statistics\nalex data stats\n```\n\n#### Model Management\n\n```bash\n# List all available models\nalex models list\n\n# Test model connectivity\nalex models test --model openai/gpt-3.5-turbo\n\n# Estimate costs\nalex models costs --model openai/gpt-4 --questions 1000\n```\n\n## 🧪 Testing \u0026 Verification\n\n### Quick System Verification\n\nVerify your alex-treBENCH installation is working correctly:\n\n```bash\n# Quick verification script\n./scripts/quick_test.sh\n\n# Or run the smoke test directly\npython scripts/smoke_test.py\n\n# Using Make\nmake smoke-test\n```\n\n### Smoke Test\n\nThe smoke test provides complete end-to-end verification of the alex-treBENCH system:\n\n- ✅ **Database initialization** - Creates and verifies database schema\n- ✅ **Sample data loading** - Loads test questions into database\n- ✅ **API connectivity** - Tests OpenRouter integration (real or simulated)\n- ✅ **Benchmark execution** - Runs minimal benchmark with 3 questions\n- ✅ **Report generation** - Creates and validates performance reports\n- ✅ **System health** - Verifies all critical components\n\n**Cost**: ~$0.001-0.005 per run with API key, $0.00 in simulation mode\n\n### Test Categories\n\n```bash\n# Comprehensive test suite\nmake test              # All tests\nmake test-coverage     # With coverage report\nmake test-unit         # Unit tests only\nmake test-integration  # Integration tests\nmake test-e2e          # End-to-end tests\n\n# Component-specific testing\nmake test-agents       # Individual component tests\npython scripts/test_agents.py\n```\n\n### Expected Output (Smoke Test Success)\n\n```\n🔥 alex-treBENCH Smoke Test\nRunning complete end-to-end system verification\n\n✅ Setting up test environment...\n✅ Initializing database...\n✅ Loading sample data...\n✅ Running minimal benchmark...\n✅ Generating report...\n✅ Verifying system health...\n\n🎉 Smoke Test PASSED\nalex-treBENCH system is working correctly!\n```\n\n### Continuous Integration\n\nTests automatically run on:\n\n- Pull requests to main/develop branches\n- Pushes to main/develop branches\n- Manual workflow triggers with optional real API testing\n\nSee [`.github/workflows/smoke-test.yml`](.github/workflows/smoke-test.yml) for CI configuration.\n\n### Full Testing Documentation\n\nFor comprehensive testing information, troubleshooting, and advanced test scenarios:\n\n📖 **[Complete Testing Guide](docs/TESTING.md)**\n\nCovers:\n\n- Detailed test agent documentation\n- Troubleshooting common issues\n- Cost management strategies\n- Performance testing\n- Writing new tests\n- CI/CD integration\n\n## 🙏 Acknowledgments\n\n- **Kaggle**: For providing the Jeopardy dataset (aravindram11/jeopardy-dataset-updated)\n- **OpenRouter**: For unified language model API access\n- **Jeopardy!**: For creating the foundational question format that makes this benchmarking meaningful\n\n## 📞 Support\n\nFor questions, issues, or contributions:\n\n- 📖 Read the [User Guide](docs/USER_GUIDE.md) for detailed usage instructions\n- 🔧 Check the [API Reference](docs/API_REFERENCE.md) for technical details\n- 🐛 Create an issue in the GitHub repository\n- 💬 Review the technical documentation in [`TECHNICAL_SPEC.md`](TECHNICAL_SPEC.md)\n\n---\n\n**🎉 Implementation Complete**: This system is now fully implemented and production-ready. All core features are functional with comprehensive testing and documentation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkilo-org%2Falex-trebench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkilo-org%2Falex-trebench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkilo-org%2Falex-trebench/lists"}