{"id":30190781,"url":"https://github.com/monarch1108/llm_labs","last_synced_at":"2025-08-12T20:08:29.039Z","repository":{"id":308859015,"uuid":"1034348370","full_name":"MONARCH1108/LLM_Labs","owner":"MONARCH1108","description":"Advanced AI model comparison platform supporting Groq, Gemini, and Ollama with multiple prompt strategies, performance analytics, and comprehensive reporting capabilities.","archived":false,"fork":false,"pushed_at":"2025-08-08T08:49:43.000Z","size":50,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-08T10:28:26.910Z","etag":null,"topics":["comparison-tool","flask-application","gemini-api","groq-api","llm-evaluation","llm-evaluation-toolkit","ollama","prompt-engineering"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MONARCH1108.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-08T08:45:02.000Z","updated_at":"2025-08-08T08:51:11.000Z","dependencies_parsed_at":"2025-08-08T10:28:29.071Z","dependency_job_id":"6fe2a2ec-32ef-470e-950d-b30f8ae12165","html_url":"https://github.com/MONARCH1108/LLM_Labs","commit_stats":null,"previous_names":["monarch1108/llm_labs"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/MONARCH1108/LLM_Labs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2FLLM_Labs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2FLLM_Labs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2FLLM_Labs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2FLLM_Labs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MONARCH1108","download_url":"https://codeload.github.com/MONARCH1108/LLM_Labs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2FLLM_Labs/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270127392,"owners_count":24531793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["comparison-tool","flask-application","gemini-api","groq-api","llm-evaluation","llm-evaluation-toolkit","ollama","prompt-engineering"],"created_at":"2025-08-12T20:08:25.277Z","updated_at":"2025-08-12T20:08:29.005Z","avatar_url":"https://github.com/MONARCH1108.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🤖 LLM Comparison \u0026 Evaluation Tool\n\nA comprehensive Python tool for comparing Large Language Models (LLMs) across multiple providers using various prompting strategies. Supports both CLI and web interfaces for interactive evaluation and performance analysis.\n\n## Documentaion:\nhttps://medium.com/@abhayemani8/llm-labs-149b8174bd33\n\n## 🌟 Features\n\n### Multi-Provider Support\n- **Groq**: High-performance LLM inference\n- **Google Gemini**: Advanced AI capabilities\n- **Ollama**: Local LLM deployment\n\n### Advanced Prompting Strategies\n- **Zero-shot**: Direct prompting without examples\n- **One-shot**: Single example-based prompting\n- **Few-shot**: Multiple example-based prompting\n- **Chain-of-Thought (CoT)**: Step-by-step reasoning\n- **ReAct**: Reasoning + Action prompting\n- **Self-Ask**: Self-questioning approach\n- **Tree-of-Thought**: Multi-path exploration\n- **Instruction + Constraints**: Structured output control\n- **Persona-based**: Role-playing with custom personas\n\n### Comprehensive Metrics\n- **Performance**: Response time, token usage\n- **Quality**: Readability scores (Flesch Reading Ease)\n- **Analysis**: Prompt classification and evaluation\n- **Reporting**: Automated report generation with insights\n\n### Dual Interface\n- **CLI Mode**: Terminal-based interaction\n- **Web Interface**: Modern Flask-based UI\n\n## 📋 Requirements\n\n### System Requirements\n- Python 3.8+\n- For Ollama: Local Ollama installation\n- For Groq/Gemini: Valid API keys\n\n## 🚀 Installation\n\n### 1. Clone the Repository\n```bash\ngit clone https://github.com/MONARCH1108/LLM_Labs.git\ncd llm-comparison-tool\n```\n\n### 2. Install Dependencies\n```bash\npip install -r requirements.txt\n```\n\n### 3. Install spaCy Language Model\n```bash\npython -m spacy download en_core_web_sm\n```\n\n### 4. Environment Setup\nCreate a `.env` file in the project root:\n```env\nGROQ_API_KEY=your_groq_api_key_here\nGEMINI_API_KEY=your_gemini_api_key_here\n```\n\n### 5. Install Ollama (Optional)\nFor local LLM support:\n```bash\n# Visit https://ollama.ai/download for installation instructions\n# Then pull models:\nollama pull llama3:latest\nollama pull mistral:latest\n```\n\n## 📁 Project Structure\n\n```\nllm-comparison-tool/\n├── app.py                 # CLI interface\n├── flask_app.py                  # Flask web interface\n├── requirements.txt        # Python dependencies\n├── .env                    # Environment variables\n├── utils/\n│   ├── __init__.py\n│   ├── comparison.py       # Core comparison logic\n│   ├── llms.py            # LLM provider interfaces\n│   ├── prompts.py         # Prompting strategies\n│   └── report_generator.py # Report generation\n├── comparison_tools/\n│   ├── __init__.py\n│   ├── tokenizer.py       # Token counting utilities\n│   └── prompt_classifier.py # Prompt classification\n├── templates/\n│   └── index.html         # Web UI template\n├── reports/               # Generated reports\n└── logs/                  # Application logs\n```\n\n## 🖥️ Usage\n\n### CLI Mode\n\n#### Single LLM Chat\n```bash\npython main.py\n# Choose option 1 for single LLM chat\n# Select provider, model, and prompting strategy\n# Start interactive conversation\n```\n\n#### Comparative Evaluation\n```bash\npython main.py\n# Choose option 2 for comparative evaluation\n# Select multiple providers and models\n# Choose prompting strategies\n# Enter your question for comparison\n```\n\n### Web Interface\n\n#### Start the Web Server\n```bash\npython app.py\n```\nAccess the web interface at `http://localhost:5000`\n\n#### Features Available in Web UI\n- **Interactive Model Selection**: Browse available models\n- **Real-time Ollama Status**: Check local model availability\n- **Live Comparison**: See results as they generate\n- **Report Download**: Export detailed performance reports\n- **Visual Metrics**: Performance charts and statistics\n\n## 🔧 Configuration\n\n### API Keys Setup\n\n#### Groq API\n1. Visit [Groq Console](https://console.groq.com/)\n2. Create an account and generate API key\n3. Add to `.env` file\n\n#### Google Gemini API\n1. Visit [Google AI Studio](https://makersuite.google.com/)\n2. Create project and generate API key\n3. Add to `.env` file\n\n### Model Configuration\n\n#### Groq Models\n- `llama3-8b-8192`\n- `llama3-70b-8192`\n- `mixtral-8x7b-32768`\n- `gemma-7b-it`\n\n#### Gemini Models\n- `gemini-1.5-pro`\n- `gemini-1.5-flash`\n- `gemini-2.0-flash`\n\n#### Ollama Models\n```bash\nollama pull llama3:latest\nollama pull mistral:latest\nollama pull codellama:latest\nollama pull vicuna:latest\n```\n\n## 📊 Evaluation Metrics\n\n### Performance Metrics\n- **Response Time**: Latency in seconds\n- **Token Usage**: Input/Output/Total tokens\n- **Throughput**: Tokens per second\n\n### Quality Metrics\n- **Readability**: Flesch Reading Ease score\n- **Structure**: Sentence and syllable counts\n- **Prompt Classification**: Automatic categorization\n\n### Comparative Analysis\n- **Speed Comparison**: Response time rankings\n- **Efficiency Analysis**: Token usage patterns\n- **Quality Assessment**: Readability scores\n- **Best Performer**: Overall performance winner\n\n## 📈 Sample Output\n\n```\n📊 LLM Comparison Results:\n\n🔹 GROQ (llama3-8b-8192) | Prompt: Chain of Thought\n🧠 Prompt Type: Reasoning | Prompt Length: 45 words\n🔢 Tokens - Input: 123 | Output: 256 | Total: 379\n⏱️ Response Time: 2.3 seconds\n📝 Response Length: 187 words\n📚 Readability - Sentences: 12 | Syllables: 298 | Flesch Score: 65.4\n```\n\n## 🔍 Advanced Features\n\n### Custom Persona Prompting\n```python\n# Example: Detective persona\nrole = \"detective\"\ntone = \"analytical\"\nstyle = \"detailed\"\n# Automatically generates specialized prompts\n```\n\n### Automated Report Generation\n- **Performance Summary**: Statistical analysis\n- **Model Comparison**: Side-by-side metrics\n- **Best Practices**: Recommendations based on results\n- **Export Options**: TXT, JSON formats\n\n### Error Handling \u0026 Logging\n- **Comprehensive Logging**: All interactions logged\n- **Error Recovery**: Graceful handling of API failures\n- **Status Monitoring**: Real-time provider status\n\n## 🛠️ Development\n\n### Running Tests\n```bash\npython -m pytest tests/\n```\n\n### Contributing\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests for new functionality\n5. Submit a pull request\n\n### Code Style\n- Follow PEP 8 guidelines\n- Use type hints where applicable\n- Add docstrings for functions\n- Keep functions focused and modular\n\n## 📝 API Reference\n\n### Core Functions\n\n#### `run_comparative_evaluation(providers_models, prompts, user_input)`\nRuns comparative evaluation across multiple LLMs.\n\n**Parameters:**\n- `providers_models`: Dict of providers and their models\n- `prompts`: Dict of prompt templates\n- `user_input`: User question for evaluation\n\n#### `query_groq_llm(user_input, model, prompt)`\nQueries Groq LLM with specified parameters.\n\n#### `query_gemini_llm(user_input, model, prompt)`\nQueries Google Gemini with specified parameters.\n\n#### `query_ollama_llm(user_input, model, prompt)`\nQueries local Ollama model with specified parameters.\n\n## 🤝 Contributing\n\nWe welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for details.\n\n### Areas for Contribution\n- Additional LLM providers\n- New prompting strategies\n- Enhanced metrics and analysis\n- UI/UX improvements\n- Documentation and examples\n\n# Frontend (index.html)\n## Key Features \u0026 Important Information\n\n### 1. **Multi-Provider AI Support**\n- **Groq**: Ultra-fast inference with specialized hardware for real-time applications\n- **Gemini**: Google's advanced AI models with strong reasoning capabilities\n- **Ollama**: Run models locally for complete privacy and control (requires local installation)\n\n### 2. **Advanced Prompt Engineering Strategies**\n- Zero Shot: Direct questions without examples\n- Chain of Thought: Step-by-step reasoning for complex problems\n- Few Shot: Learning from multiple examples\n- Persona Based: Role-playing with customizable personas\n- Self-Questioning, Tree of Thoughts, and more advanced techniques\n\n### 3. **Dual Interface Modes**\n- **Single LLM Chat**: Test individual models with real-time conversations\n- **Comparative Evaluation**: Run the same question across multiple models and prompt strategies simultaneously for comprehensive analysis\n\n### 4. **Real-Time Performance Analytics**\n- Response time tracking and analysis\n- Token usage estimation\n- Chat statistics (message count, average response time)\n- Model performance comparisons\n\n### 5. **Local Model Support (Ollama Integration)**\n- Complete privacy - models run entirely on your machine\n- No internet required after initial setup\n- Supports popular models like Llama 3, CodeLlama, Mistral\n- API endpoint to check locally installed models: `/api/ollama-models`\n\n### 6. **Export \u0026 Sharing Capabilities**\n- Export chat histories as JSON files\n- Generate comprehensive performance reports\n- Share comparison results via URL\n- Download detailed logs for analysis\n- Save and load configuration presets\n\n### 7. **Interactive UI/UX Features**\n- Dark theme with glassmorphism effects\n- Animated progress indicators during model testing\n- Toast notifications for user feedback\n- Auto-scroll chat interface with toggle\n- Copy message functionality with one-click\n\n### 8. **API Endpoints Required**\n```\nPOST /api/single-chat          # Single model chat\nPOST /api/comparative-evaluation # Multi-model comparison\nPOST /api/generate-report      # Performance report generation\nGET  /api/ollama-models        # List local Ollama models\nGET  /api/get-log             # Retrieve comparison logs\nGET  /api/prompt-templates    # Load available prompt strategies\n```\n\n### 9. **Setup Requirements**\n- Modern web browser with ES6+ support\n- Backend API server (endpoints listed above)\n- For Ollama: Local Ollama installation and running service\n- For Groq: API key and model access\n- For Gemini: Google AI API credentials\n\n### 10. **Built-in Help \u0026 User Guidance**\n- Interactive welcome tour for first-time users\n- Contextual help tooltips and information panels\n- Sample questions categorized by use case (coding, reasoning, creative, analysis)\n- Model validation and setup verification\n- Comprehensive error handling with user-friendly messages\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonarch1108%2Fllm_labs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonarch1108%2Fllm_labs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonarch1108%2Fllm_labs/lists"}