{"id":28946685,"url":"https://github.com/dmitriz/llm-selection","last_synced_at":"2026-02-27T12:05:27.689Z","repository":{"id":294870496,"uuid":"988202708","full_name":"dmitriz/llm-selection","owner":"dmitriz","description":"Strategic selection, evaluation, and routing of large language models","archived":false,"fork":false,"pushed_at":"2025-06-04T09:27:50.000Z","size":35,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-04T16:54:27.897Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmitriz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-22T07:51:34.000Z","updated_at":"2025-05-22T12:38:46.000Z","dependencies_parsed_at":"2025-05-22T14:06:10.575Z","dependency_job_id":null,"html_url":"https://github.com/dmitriz/llm-selection","commit_stats":null,"previous_names":["dmitriz/llm-selection"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dmitriz/llm-selection","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitriz%2Fllm-selection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitriz%2Fllm-selection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitriz%2Fllm-selection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitriz%2Fllm-selection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmitriz","download_url":"https://codeload.github.com/dmitriz/llm-selection/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitriz%2Fllm-selection/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261446937,"owners_count":23159513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-23T08:42:06.972Z","updated_at":"2026-02-27T12:05:27.648Z","avatar_url":"https://github.com/dmitriz.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM Selection Project\n\nA system for selecting, routing, and evaluating large language models (LLMs) based on task type, reasoning ability, access methods, costs, and reliability.\n\n## 🚀 Getting Started\n\nThis project is designed to help you make strategic model selection decisions for your LLM tasks. The repository will grow to include:\n\n- Model catalogs with capability tracking\n- Routing logic for task-to-model mapping\n- Evaluation tools and reviewer prompts\n- Integration patterns for various LLM access methods\n\n## 📂 Repository Structure\n\n- `models.md` - Catalog of available LLMs and their capabilities (planned)\n- `routing-strategy.md` - Logic for matching tasks to models (planned)\n- `reviewer-prompts.md` - Evaluation criteria and prompts (planned)\n- `PROJECT_BACKLOG.md` - Complete project backlog with detailed goals and plans\n\n## 🏁 Initial Tasks\n\n- [ ] Create `models.md` for cataloging available LLMs\n- [ ] Add `routing-strategy.md` to outline logic by task type\n- [ ] Prepare reviewer prompt set in `reviewer-prompts.md`\n- [ ] Design and test a selection prompt (e.g., \"Which model should I use for X?\")\n\n## 📊 Roadmap \u0026 Prioritization\n\n### Phase 1: Model Cataloging (June 2025)\n\n- **Focus**: Comprehensive model inventory and capability assessment\n- **Deliverable**: `models.md` with detailed capability matrix\n- **Key milestones**: \n  - Complete API documentation for all major providers\n  - Document token limits and rate restrictions\n  - Create standardized capability scoring system\n\n### Phase 2: Routing Strategy (July 2025)\n\n- **Focus**: Decision framework for model selection\n- **Deliverable**: `routing-strategy.md` with decision trees\n- **Key milestones**:\n  - Map task types to optimal model profiles\n  - Create fallback pathways for rate limits and failures\n  - Design cost optimization strategies\n\n### Phase 3: Evaluation Framework (August 2025)\n\n- **Focus**: Output quality assessment\n- **Deliverable**: `reviewer-prompts.md` with scoring metrics\n- **Key milestones**:\n  - Develop standardized evaluation criteria\n  - Create reviewer prompts for different output types\n  - Build comparison methodology for cross-model evaluation\n\n### Success Metrics\n\n- **Latency targets**: \u003c500ms for routing decisions, \u003c5s end-to-end for simple tasks\n- **Hallucination thresholds**: \u003c2% factual errors in final outputs\n- **Cost efficiency**: 30% reduction in token usage vs. single-model approach\n- **Selection accuracy**: \u003e95% appropriate model selection for given task types\n\n## 📖 Detailed Documentation\n\nFor complete project details including:\n\n- Core components\n- Key insights and strategy notes\n- Planned experiments\n- Roles and agents\n\nPlease refer to the [PROJECT_BACKLOG.md](PROJECT_BACKLOG.md) file.\n\n## 🤝 Contributing\n\nContributions are welcome! Check the PROJECT_BACKLOG.md for areas where you can help, or suggest new features and improvements.\n\n---\n\n\u003e Note: This project is in active development. Features and documentation will evolve over time.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmitriz%2Fllm-selection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmitriz%2Fllm-selection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmitriz%2Fllm-selection/lists"}