{"id":30665263,"url":"https://github.com/monarch1108/llm-labs","last_synced_at":"2025-08-31T20:17:18.373Z","repository":{"id":310445655,"uuid":"1039867058","full_name":"MONARCH1108/llm-labs","owner":"MONARCH1108","description":"LLM Labs is an experimental project to evaluate, compare, and improve Large Language Models (LLMs).   The repo evolves daily with new metrics, features, and experiments.","archived":false,"fork":false,"pushed_at":"2025-08-18T05:35:08.000Z","size":24,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-18T07:20:54.527Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MONARCH1108.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-18T05:28:34.000Z","updated_at":"2025-08-18T05:35:11.000Z","dependencies_parsed_at":"2025-08-18T07:20:56.809Z","dependency_job_id":"cf2bc981-886f-4a88-9122-2d1cb30b0221","html_url":"https://github.com/MONARCH1108/llm-labs","commit_stats":null,"previous_names":["monarch1108/llm-labs"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/MONARCH1108/llm-labs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2Fllm-labs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2Fllm-labs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2Fllm-labs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2Fllm-labs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MONARCH1108","download_url":"https://codeload.github.com/MONARCH1108/llm-labs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2Fllm-labs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273032934,"owners_count":25034067,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-31T02:00:09.071Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-31T20:17:12.694Z","updated_at":"2025-08-31T20:17:18.366Z","avatar_url":"https://github.com/MONARCH1108.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📁 LLM Labs - Directory Structure\n\n\n```\nLLM_Labs_V1/\n│\n├─ comparision_tools/\n│   ├─ metrics.py\n│   │   ├─ get_text_stats(text: str) -\u003e dict\n│   │   ├─ compute_flesch_reading_ease(sentence_count: int, word_count: int, syllable_count: int) -\u003e float|str\n│   │   ├─ compute_smog_index(sentence_count: int, polysyllable_count: int) -\u003e float|str\n│   │   ├─ compute_coleman_liau_index(char_count: int, word_count: int, sentence_count: int) -\u003e float|str\n│   │   ├─ compute_gunning_fog_index(sentence_count: int, word_count: int, polysyllable_count: int) -\u003e float|str\n│   │   ├─ compute_automated_readability_index(char_count: int, word_count: int, sentence_count: int) -\u003e float|str\n│   │   ├─ compute_dale_chall_index(text: str) -\u003e float\n│   │   ├─ compute_forcast_index(text: str) -\u003e float\n│   │   ├─ compute_linsear_write_index(text: str) -\u003e float\n│   │   ├─ compute_lix(text: str) -\u003e float\n│   │   ├─ compute_rix(text: str) -\u003e float\n│   │   └─ get_readability_metrics(text: str) -\u003e dict\n│   │\n│   ├─ prompt_classifier.py\n│   │   └─ classify_prompt(prompt: str) -\u003e str\n│   │\n│   └─ tokenizer.py\n│       └─ count_tokens(text: str) -\u003e int\n│\n├─ utils/\n│   ├─ comparison.py\n│   │   └─ run_comparative_evaluation(providers_models: dict, prompts: dict, user_input: str)\n│   │\n│   ├─ llms.py\n│   │   ├─ query_groq_llm(user_input: str, model: str, prompt: str) -\u003e str\n│   │   ├─ query_gemini_llm(user_input: str, model: str, prompt: str) -\u003e str\n│   │   └─ query_ollama_llm(user_input: str, model: str, prompt: str) -\u003e str\n│   │\n│   ├─ prompts.py\n│   │   ├─ zero_shot_prompt() -\u003e str\n│   │   ├─ one_shot_prompt() -\u003e str\n│   │   ├─ few_shot_prompt() -\u003e str\n│   │   ├─ chain_of_thought_prompt() -\u003e str\n│   │   ├─ react_prompt() -\u003e str\n│   │   ├─ self_ask_prompt() -\u003e str\n│   │   ├─ tree_of_thought_prompt() -\u003e str\n│   │   ├─ instruction_constraints_prompt() -\u003e str\n│   │   └─ persona_based_prompt_template(role: str, tone: str = \"professional\", style: str = \"clear and concise\") -\u003e str\n│   │\n│   └─ report_generator.py\n│       ├─ safe_float(value: str)\n│       ├─ extract_log_metrics() -\u003e list\n│       └─ generate_report(summary: list, user_question: str)\n│\n└─ app.py\n    ├─ select_prompt_templates() -\u003e dict\n    ├─ run_comparative()\n    ├─ run_single_llm_chat()\n    └─ main()\n```\n\n## Key Corrections from Your Original Format:\n\n### 1. **File Organization**:\n\n- `metrics.py` contains all readability functions (not in `tokenizer.py`)\n- `llms.py` contains all LLM query functions (separate from `comparison.py`)\n- `prompts.py` contains all prompt templates (separate from `comparison.py`)\n\n### 2. **Function Distribution**:\n\n- **`metrics.py`**: 12 functions for text analysis and readability\n- **`tokenizer.py`**: Only 1 function for token counting\n- **`llms.py`**: 3 LLM provider functions\n- **`prompts.py`**: 9 prompting strategy functions\n- **`comparison.py`**: 1 main orchestration function\n- **`report_generator.py`**: 3 functions for report generation\n\n### 3. **Missing Functions Added**:\n\n- `safe_float()` in `report_generator.py`\n- All individual readability metric functions in `metrics.py`\n- Proper parameter types and return types specified\n\n### 4. **Accurate File Count**:\n\n- **Total Files**: 7 Python files\n- **Total Functions**: 30 functions\n- **Modules**: 2 main directories (`comparision_tools/`, `utils/`)\n\n\n\n## 📄 File Details \u0026 Functions\n\n### `app.py` - Main Application Entry Point\n\n**Purpose**: Central application controller and user interface\n\n#### Functions:\n\n- **`select_prompt_templates()`**\n    - Interactive prompt strategy selection\n    - Handles persona-based prompts with custom parameters\n    - Returns dictionary of selected prompt templates\n- **`run_comparative()`**\n    - Orchestrates multi-provider LLM comparison\n    - Manages provider/model selection interface\n    - Executes comparative evaluation and report generation\n- **`run_single_llm_chat()`**\n    - Single LLM interactive chat interface\n    - Provider and model selection\n    - Continuous conversation loop with exit functionality\n- **`main()`**\n    - Application entry point\n    - Mode selection (Single Chat vs Comparative)\n    - Route to appropriate functionality\n\n---\n\n### 📁 `comparision_tools/` - Core Analysis Tools\n\n#### `metrics.py` - Text Analysis \u0026 Readability Metrics\n\n**Purpose**: Comprehensive text analysis and readability scoring\n\n##### Functions:\n\n- **`get_text_stats(text: str) -\u003e dict`**\n    - Extract basic text statistics using spaCy NLP\n    - Returns sentence count, word count, syllable count, etc.\n- **`compute_flesch_reading_ease(sentence_count, word_count, syllable_count) -\u003e float | str`**\n    - Flesch Reading Ease score calculation\n    - Score range: 0-100 (higher = easier to read)\n- **`compute_smog_index(sentence_count, polysyllable_count) -\u003e float | str`**\n    - SMOG (Simple Measure of Gobbledygook) readability\n    - Estimates years of education needed to understand text\n- **`compute_coleman_liau_index(char_count, word_count, sentence_count) -\u003e float | str`**\n    - Coleman-Liau Index calculation\n    - Grade level estimation based on character/word ratios\n- **`compute_gunning_fog_index(sentence_count, word_count, polysyllable_count) -\u003e float | str`**\n    - Gunning Fog Index for readability assessment\n    - Estimates years of formal education required\n- **`compute_automated_readability_index(char_count, word_count, sentence_count) -\u003e float | str`**\n    - Automated Readability Index (ARI)\n    - Character-based readability measurement\n- **`compute_dale_chall_index(text: str) -\u003e float`**\n    - Dale-Chall Readability Score using textstat\n    - Vocabulary difficulty assessment\n- **`compute_forcast_index(text: str) -\u003e float`**\n    - FORCAST (FORmula for CASTing readability)\n    - Technical text readability measurement\n- **`compute_linsear_write_index(text: str) -\u003e float`**\n    - Linsear Write Formula\n    - Military readability standard\n- **`compute_lix(text: str) -\u003e float`**\n    - LIX readability index\n    - Swedish-origin text density metric\n- **`compute_rix(text: str) -\u003e float`**\n    - RIX readability index\n    - Long words per sentence measurement\n- **`get_readability_metrics(text: str) -\u003e dict`**\n    - Master function combining all readability metrics\n    - Returns comprehensive analysis dictionary\n\n#### `prompt_classifier.py` - Automatic Prompt Categorization\n\n**Purpose**: Intelligent prompt classification system\n\n##### Functions:\n\n- **`classify_prompt(prompt: str) -\u003e str`**\n    - Keyword-based prompt classification\n    - Categories: Math, Creative, Reasoning, Knowledge, General\n    - Returns classification string\n\n#### `tokenizer.py` - Token Counting Utilities\n\n**Purpose**: Token usage tracking and analysis\n\n##### Functions:\n\n- **`count_tokens(text: str) -\u003e int`**\n    - GPT-2 tokenizer-based token counting\n    - Supports truncation for long texts\n    - Returns integer token count\n\n---\n\n### 📁 `utils/` - Utility Modules\n\n#### `comparison.py` - Comparative Evaluation Engine\n\n**Purpose**: Multi-provider LLM comparison orchestration\n\n##### Functions:\n\n- **`run_comparative_evaluation(providers_models: dict, prompts: dict, user_input: str)`**\n    - Main comparison execution function\n    - Iterates through all provider/model/prompt combinations\n    - Captures performance metrics, readability scores, and responses\n    - Logs detailed results and displays summary\n    - Handles errors and edge cases gracefully\n\n#### `llms.py` - LLM Provider API Integrations\n\n**Purpose**: Unified interface for multiple LLM providers\n\n##### Functions:\n\n- **`query_groq_llm(user_input: str, model: str, prompt: str) -\u003e str`**\n    - Groq API integration with chat completions\n    - Handles API key authentication\n    - Returns cleaned response text\n- **`query_gemini_llm(user_input: str, model: str, prompt: str) -\u003e str`**\n    - Google Gemini API integration\n    - GenerativeModel interface usage\n    - Returns cleaned response text\n- **`query_ollama_llm(user_input: str, model: str, prompt: str) -\u003e str`**\n    - Local Ollama client integration\n    - Chat interface for local models\n    - Returns cleaned response text\n\n#### `prompts.py` - Prompting Strategy Templates\n\n**Purpose**: Collection of advanced prompting techniques\n\n##### Functions:\n\n- **`zero_shot_prompt() -\u003e str`**\n    - Basic zero-shot prompting template\n    - Direct task completion without examples\n- **`one_shot_prompt() -\u003e str`**\n    - Single example-guided prompting\n    - Pattern-following instruction template\n- **`few_shot_prompt() -\u003e str`**\n    - Multiple example-based learning template\n    - Consistent style guidance\n- **`chain_of_thought_prompt() -\u003e str`**\n    - Step-by-step reasoning template\n    - Logical problem decomposition\n- **`react_prompt() -\u003e str`**\n    - Reasoning + Action framework template\n    - Thought-Action-Observation cycle\n- **`self_ask_prompt() -\u003e str`**\n    - Self-questioning methodology template\n    - Sub-question decomposition approach\n- **`tree_of_thought_prompt() -\u003e str`**\n    - Multiple reasoning path exploration\n    - Path evaluation and selection\n- **`instruction_constraints_prompt() -\u003e str`**\n    - Precise output control template\n    - Clear instructions with strict constraints\n- **`persona_based_prompt_template(role: str, tone: str, style: str) -\u003e str`**\n    - Dynamic persona-based prompting\n    - Customizable role, tone, and style parameters\n\n#### `report_generator.py` - Performance Report Generation\n\n**Purpose**: Automated analysis and reporting system\n\n##### Functions:\n\n- **`safe_float(value: str)`**\n    - Safe string-to-float conversion\n    - Handles \"N/A\" and invalid values gracefully\n- **`extract_log_metrics()`**\n    - Parse execution logs for performance data\n    - Regex-based metric extraction\n    - Returns structured summary list\n- **`generate_report(summary: list, user_question: str)`**\n    - Generate comprehensive performance report\n    - Statistical analysis and interpretation guides\n    - Gemini AI-powered insights generation\n    - Markdown format output with timestamps\n\n---\n\n## 🔧 Configuration Files\n\n### `requirements.txt` - Python Dependencies\n\n```\ngroq\u003e=0.4.1\ngoogle-generativeai\u003e=0.3.0\nollama\u003e=0.1.7\npython-dotenv\u003e=0.19.0\nspacy\u003e=3.4.0\ntextstat\u003e=0.7.3\ntransformers\u003e=4.21.0\n```\n\n### `.env` - Environment Variables (User-Created)\n\nenv\n\n```env\nGROQ_API_KEY=your_groq_api_key_here\nGEMINI_API_KEY=your_gemini_api_key_here\n```\n\n---\n\n## 📊 Auto-Generated Files\n\n### `llm_comparison.log` - Execution Logs\n\n- Detailed execution traces\n- Performance metrics logging\n- Error tracking and debugging info\n- Timestamp-based entries\n\n### `llm_report_YYYYMMDD_HHMMSS.txt` - Performance Reports\n\n- Comprehensive analysis reports\n- Statistical summaries\n- AI-generated insights\n- Export-ready format\n\n---\n\n## 🚀 Key Features by Module\n\n### 📈 Analytics \u0026 Metrics\n\n- **10+ Readability Indices**: Comprehensive text analysis\n- **Token Usage Tracking**: Input/output/total consumption\n- **Response Time Metrics**: Performance benchmarking\n- **Prompt Classification**: Automatic categorization\n\n### 🤖 LLM Integration\n\n- **Multi-Provider Support**: Groq, Gemini, Ollama\n- **Unified API**: Consistent interface across providers\n- **Error Handling**: Graceful failure management\n- **Model Flexibility**: Support for various model architectures\n\n### 💡 Prompting Strategies\n\n- **9 Advanced Techniques**: From zero-shot to persona-based\n- **Dynamic Templates**: Customizable parameters\n- **Best Practices**: Industry-standard implementations\n- **Scalable Framework**: Easy addition of new strategies\n\n### 📊 Reporting System\n\n- **Automated Analysis**: AI-powered insights\n- **Statistical Summaries**: Comprehensive metrics\n- **Export Formats**: Multiple output options\n- **Visual Interpretations**: Clear metric explanations","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonarch1108%2Fllm-labs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonarch1108%2Fllm-labs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonarch1108%2Fllm-labs/lists"}