{"id":31767891,"url":"https://github.com/webdevtodayjason/llm-wars","last_synced_at":"2025-10-10T01:48:46.667Z","repository":{"id":281073301,"uuid":"944116913","full_name":"webdevtodayjason/LLM-Wars","owner":"webdevtodayjason","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-06T20:24:40.000Z","size":12,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-06T01:30:31.773Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/webdevtodayjason.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-06T20:14:21.000Z","updated_at":"2025-03-07T00:22:14.000Z","dependencies_parsed_at":"2025-03-06T21:36:15.046Z","dependency_job_id":null,"html_url":"https://github.com/webdevtodayjason/LLM-Wars","commit_stats":null,"previous_names":["webdevtodayjason/llm-wars"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/webdevtodayjason/LLM-Wars","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webdevtodayjason%2FLLM-Wars","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webdevtodayjason%2FLLM-Wars/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webdevtodayjason%2FLLM-Wars/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webdevtodayjason%2FLLM-Wars/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/webdevtodayjason","download_url":"https://codeload.github.com/webdevtodayjason/LLM-Wars/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webdevtodayjason%2FLLM-Wars/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002400,"owners_count":26083375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-10T01:48:36.461Z","updated_at":"2025-10-10T01:48:46.662Z","avatar_url":"https://github.com/webdevtodayjason.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM Wars\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\nA powerful CLI tool for comparing LLM performance across multiple providers simultaneously. Test prompts against OpenAI, Anthropic, Google Gemini, and more in a single interface.\n\n![LLM Wars Screenshot](https://via.placeholder.com/800x400?text=LLM+Wars+Screenshot)\n\n## Features\n\n- 🚀 **Multi-Provider Support**: Test prompts across OpenAI, Anthropic, Google Gemini, and CodeGPT\n- 🧠 **Model Selection**: Choose specific models from each provider\n- ⚡ **Parallel Querying**: Send prompts to all selected models simultaneously\n- 📊 **Performance Metrics**: Track token usage and response times\n- 📦 **Clean Interface**: Rich text UI with side-by-side comparison\n- 🔄 **Multi-View Responses**: View full responses individually or sequentially\n- 📝 **Question Library**: Comprehensive benchmark suite with questions from industry-standard tests:\n  - Instruction Following (IFEval)\n  - Big-Bench Hard (BBH)\n  - Mathematical Reasoning (MATH)\n  - Grade School Questions (GPQA)\n  - Massive Multitask Language Understanding (MMLU)\n  - Multi-Turn User Simulation (MUSR)\n\n## Installation\n\n### Option 1: Standard Python Environment\n\n```bash\n# Clone the repository\ngit clone https://github.com/webdevtodayjason/LLM-Wars.git\ncd LLM-Wars\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### Option 2: Conda Environment (Recommended)\n\n```bash\n# Clone the repository\ngit clone https://github.com/webdevtodayjason/LLM-Wars.git\ncd LLM-Wars\n\n# Create and activate conda environment\nconda env create -f environment.yml\nconda activate llm-wars\n```\n\n### Configure API Keys\n\nThe repository includes an `.env.example` file with the structure for all required API keys.\n\n1. Copy the example file to create your own `.env` file:\n\n```bash\n# Copy example env file\ncp .env.example .env\n```\n\n2. Edit the `.env` file and add your API keys:\n\n```bash\n# Open the file in your editor\nnano .env  # or use any text editor you prefer\n```\n\n3. Replace the placeholder values with your actual API keys:\n\n```\nOPENAI_API_KEY=sk-yourOpenAIKeyHere        # From OpenAI\nANTHROPIC_API_KEY=sk-ant-yourAnthropicKey  # From Anthropic\nGOOGLE_API_KEY=AIza-yourGoogleKey          # From Google AI Studio\nCODEGPT_API_KEY=yourCodeGPTKeyHere         # If applicable\n```\n\nWhere to get API keys:\n- OpenAI: https://platform.openai.com/account/api-keys\n- Anthropic: https://console.anthropic.com/settings/keys\n- Google: https://makersuite.google.com/app/apikey\n\nNote: The application will work even if you don't have all API keys. You'll be able to select only the providers for which you have keys configured.\n\n## Usage\n\n```bash\n# If using standard Python environment\npython app.py\n\n# If using conda environment\nconda activate llm-wars\npython app.py\n```\n\n### Step-by-Step Guide\n\n1. **Select Providers**: Choose up to 4 LLM providers from the available options\n2. **Choose Models**: Select specific models for each provider \n3. **Query Models**: Select from predefined benchmark questions or enter your own prompt\n4. **View Responses**: Compare side-by-side responses with performance metrics\n5. **Full Responses**: View complete responses for each model:\n   - View individual model responses\n   - View all responses sequentially\n   - Return to the comparison view at any time\n\n### Advanced Features\n\n#### Predefined Question Categories\n\nLLM Wars includes a comprehensive question library with categories designed to test different aspects of LLM capabilities:\n\n- **General Knowledge**: Basic factual knowledge\n- **Programming Tasks**: Code generation and explanation\n- **Creative Writing**: Creativity and text generation\n- **Logic \u0026 Reasoning**: Problem-solving capabilities\n- **Instruction Following (IFEval)**: Following specific instructions\n- **Big-Bench Hard (BBH)**: Complex reasoning tasks\n- **Mathematical Reasoning (MATH)**: Mathematical problem-solving\n- **Grade School Questions (GPQA)**: Simple educational questions\n- **Massive Multitask Language Understanding (MMLU)**: Specialized knowledge\n- **Multi-Turn User Simulation (MUSR)**: Conversational capabilities\n\n#### Performance Metrics\n\nFor each model response, LLM Wars provides:\n\n- **Completion Time**: How long the model took to generate the response\n- **Token Count (Input)**: Number of tokens in the prompt\n- **Token Count (Output)**: Number of tokens in the response\n\n## Roadmap\n\n- [x] Add a file with predefined questions to test\n- [x] Create side-by-side comparison interface\n- [x] Add multi-view response capability\n- [ ] Export results to CSV/JSON for analysis\n- [ ] Add scoring and benchmarking capabilities\n- [ ] Support for custom provider integrations\n- [ ] Web interface with visualization tools\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add some amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## About the Author\n\nCreated by [Jason Brashear](https://github.com/webdevtodayjason/)\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n## Acknowledgements\n\n- [Rich](https://github.com/Textualize/rich) for the beautiful terminal UI\n- [OpenAI](https://openai.com), [Anthropic](https://www.anthropic.com), [Google](https://deepmind.google/technologies/gemini/), and other LLM providers","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwebdevtodayjason%2Fllm-wars","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwebdevtodayjason%2Fllm-wars","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwebdevtodayjason%2Fllm-wars/lists"}