{"id":43973788,"url":"https://github.com/richard-gyiko/which-llm","last_synced_at":"2026-06-28T09:00:53.405Z","repository":{"id":333565089,"uuid":"1137400553","full_name":"richard-gyiko/which-llm","owner":"richard-gyiko","description":"CLI + agent skill for selecting the right LLM based on benchmarks, capabilities, and cost. Works with Cursor, Claude Code, Copilot, and more.","archived":false,"fork":false,"pushed_at":"2026-05-22T07:28:23.000Z","size":321,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-22T15:30:15.996Z","etag":null,"topics":["agent-skill","ai","artificial-analysis","benchmarks","cli","llm","model-selection","rust"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/richard-gyiko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-01-19T10:17:38.000Z","updated_at":"2026-02-11T08:09:43.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/richard-gyiko/which-llm","commit_stats":null,"previous_names":["richard-gyiko/artificial-analysis-cli","richard-gyiko/which-llm-cli","richard-gyiko/which-llm"],"tags_count":116,"template":false,"template_full_name":null,"purl":"pkg:github/richard-gyiko/which-llm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richard-gyiko%2Fwhich-llm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richard-gyiko%2Fwhich-llm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richard-gyiko%2Fwhich-llm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richard-gyiko%2Fwhich-llm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/richard-gyiko","download_url":"https://codeload.github.com/richard-gyiko/which-llm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richard-gyiko%2Fwhich-llm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34882751,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-skill","ai","artificial-analysis","benchmarks","cli","llm","model-selection","rust"],"created_at":"2026-02-07T08:04:06.127Z","updated_at":"2026-06-28T09:00:53.368Z","avatar_url":"https://github.com/richard-gyiko.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# which-llm\n\n**Stop guessing which LLM to use.** Get data-driven model recommendations based on your task requirements, budget, and performance needs.\n\nWith 100+ LLMs available—each with different strengths, pricing, and capabilities—choosing the right one is overwhelming. `which-llm` queries real benchmark data and gives you actionable recommendations.\n\n\u003e **Note:** This tool provides best-effort suggestions based on benchmark scores and capability metadata. It does not substitute proper evaluation on your specific use case. Benchmarks have known limitations and may not reflect real-world performance for your domain.\n\n## Quick Start\n\nThe easiest way to use `which-llm` is through the **agent skill**—your AI coding assistant (Cursor, Claude Code, Copilot, etc.) learns how to recommend models for you automatically.\n\n### 1. Install the CLI\n\n```bash\n# macOS / Linux\nbrew tap richard-gyiko/tap\nbrew install which-llm\n\n# Windows\nscoop bucket add richard-gyiko https://github.com/richard-gyiko/scoop-bucket\nscoop install which-llm\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eOther installation methods\u003c/summary\u003e\n\n**Manual download** from [GitHub Releases](https://github.com/richard-gyiko/which-llm/releases):\n\n```bash\n# macOS (Apple Silicon)\ncurl -LO https://github.com/richard-gyiko/which-llm/releases/latest/download/which-llm-aarch64-apple-darwin.tar.gz\ntar -xzf which-llm-aarch64-apple-darwin.tar.gz\nsudo mv which-llm /usr/local/bin/\n\n# macOS (Intel)\ncurl -LO https://github.com/richard-gyiko/which-llm/releases/latest/download/which-llm-x86_64-apple-darwin.tar.gz\ntar -xzf which-llm-x86_64-apple-darwin.tar.gz\nsudo mv which-llm /usr/local/bin/\n\n# Linux\ncurl -LO https://github.com/richard-gyiko/which-llm/releases/latest/download/which-llm-x86_64-unknown-linux-gnu.tar.gz\ntar -xzf which-llm-x86_64-unknown-linux-gnu.tar.gz\nsudo mv which-llm /usr/local/bin/\n```\n\n**From source** (requires Rust):\n\n```bash\ncargo install --path .\n```\n\n\u003c/details\u003e\n\n### 2. Start Using It\n\n**No API key required!** The CLI fetches pre-built benchmark data from GitHub Releases, updated daily.\n\n```bash\n# Refresh data (run once to populate cache)\nwhich-llm refresh\n\n# Query models using SQL\nwhich-llm query \"SELECT name, intelligence, coding, price FROM benchmarks LIMIT 10\"\n\n# List available tables\nwhich-llm tables\n\n# Check data source info\nwhich-llm info\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eOptional: Configure API access for real-time data\u003c/summary\u003e\n\nFor the freshest data (instead of daily snapshots), you can configure direct API access to [Artificial Analysis](https://artificialanalysis.ai):\n\n1. Create an account at [artificialanalysis.ai/login](https://artificialanalysis.ai/login)\n2. Generate an API key\n3. Configure the CLI:\n\n```bash\nwhich-llm profile create default --api-key YOUR_API_KEY\n```\n\nOr set the `ARTIFICIAL_ANALYSIS_API_KEY` environment variable.\n\nThen use the `--use-api` flag to fetch directly from the API:\n\n```bash\nwhich-llm refresh --use-api\n```\n\n\u003c/details\u003e\n\n### 3. Install the Skill\n\n```bash\n# Pick your AI coding tool\nwhich-llm skill install cursor      # Cursor\nwhich-llm skill install claude      # Claude Code\nwhich-llm skill install opencode    # OpenCode\nwhich-llm skill install codex       # Codex CLI\nwhich-llm skill install windsurf    # Windsurf\nwhich-llm skill install copilot     # GitHub Copilot\nwhich-llm skill install antigravity # Antigravity\n\n# Or install globally (available in all projects)\nwhich-llm skill install cursor --global\n```\n\nNow just ask your AI assistant: *\"Which LLM should I use for [your task]?\"*\n\n\u003e **Tip:** Most AI assistants will automatically load the skill when you mention \"which llm\" in your question—no need to explicitly tell it to load the skill.\n\n\u003e **Experimental:** The skill is under active development. Recommendations and output format may change as we refine the task classification and benchmark interpretation.\n\nThe skill follows the [Agent Skills](https://agentskills.io) open standard. See the [full skill documentation](skills/which-llm/SKILL.md) for details on how it classifies tasks and selects models.\n\n## How It Works\n\nThe skill teaches your AI assistant to:\n\n1. **Classify your task** — Is it transformational (summarize, extract), analytical (compare, justify), tool-using (API calls), or agentic (planning, orchestration)?\n2. **Derive requirements** — Map task complexity to minimum benchmark thresholds\n3. **Query real data** — Filter models by capability (tool calling, context window, structured output)\n4. **Recommend Primary + Fallback** — Suggest a cost-optimized cascade (cheap model first, escalate when needed)\n\nData sources:\n- **[Artificial Analysis](https://artificialanalysis.ai)** — Benchmark scores (intelligence, coding, math) and performance metrics (price, latency, throughput)\n- **[models.dev](https://models.dev)** — Capability metadata (context window, tool calling, structured output, reasoning)\n\n## Example\n\n**You:** *\"I'm building a feature that summarizes long Slack threads so users can catch up quickly. Threads can be pretty long, like 200+ messages. Which LLM should I use?\"*\n\n**Assistant (with skill loaded):**\n\n\u003e ## Task Classification\n\u003e - **Skill Type:** Transformational (summarization)\n\u003e - **Key Constraints:** Large context window (200+ messages)\n\u003e - **Priority:** Balanced (quality + cost + speed matter equally)\n\u003e \n\u003e ## Recommendations\n\u003e \n\u003e ### Primary: gpt-oss-20B ($0.10/M)\n\u003e - Intelligence: 24.7 | Context: 131k tokens | Throughput: 305 tps\n\u003e - Why: Excellent cost-capability ratio. 131k context easily handles 200+ messages.\n\u003e \n\u003e ### Fallback: MiMo-V2-Flash ($0.15/M)\n\u003e - Intelligence: 30-39 | Context: 256k tokens\n\u003e - Use if: Primary struggles with nuanced summaries or threads exceed 300+ messages\n\u003e \n\u003e ## Cost Estimate\n\u003e - **Cascade (70/30 split):** $0.115/M tokens\n\u003e - **Savings vs always using fallback:** 23%\n\u003e\n\u003e **Validation step:** Before deploying, test both models on 5-10 representative Slack threads from your workspace.\n\n[View full transcript](examples/opencode_haiku_45_transcript.md) — shows the complete flow including CLI queries and scoring.\n\n## CLI Reference\n\nFor power users, scripting, or debugging, you can query the data directly.\n\n### SQL Queries (Primary Interface)\n\nUse full SQL expressiveness on the cached benchmark data:\n\n```bash\n# Best coding models under $5/M (benchmarks table)\nwhich-llm query \"SELECT name, creator, coding, output_price FROM benchmarks WHERE coding \u003e 40 AND output_price \u003c 5 ORDER BY coding DESC\"\n\n# Models with tool calling and large context (models table)\nwhich-llm query \"SELECT model_name, provider_name, context_window, tool_call FROM models WHERE tool_call = true AND context_window \u003e 100000\"\n\n# List available tables\nwhich-llm tables\n\n# Show schema for a specific table\nwhich-llm tables benchmarks\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eAvailable tables and columns\u003c/summary\u003e\n\n#### Tables\n\n| Table | Description | Source |\n|-------|-------------|--------|\n| `benchmarks` | LLM benchmark scores and pricing | Artificial Analysis |\n| `models` | Capability metadata and provider info | models.dev |\n| `text_to_image` | Text-to-image models | Artificial Analysis |\n| `image_editing` | Image editing models | Artificial Analysis |\n| `text_to_speech` | Text-to-speech models | Artificial Analysis |\n| `text_to_video` | Text-to-video models | Artificial Analysis |\n| `image_to_video` | Image-to-video models | Artificial Analysis |\n\n#### Benchmarks Table (Artificial Analysis)\n\n| Column | Type | Description |\n|--------|------|-------------|\n| `name` | VARCHAR | Model name |\n| `creator` | VARCHAR | Creator (OpenAI, Anthropic, etc.) |\n| `intelligence` | DOUBLE | Intelligence index |\n| `coding` | DOUBLE | Coding index |\n| `math` | DOUBLE | Math index |\n| `input_price` | DOUBLE | Price per 1M input tokens |\n| `output_price` | DOUBLE | Price per 1M output tokens |\n| `tps` | DOUBLE | Tokens per second |\n| `latency` | DOUBLE | Time to first token (seconds) |\n\n#### Models Table (models.dev)\n\n| Column | Type | Description |\n|--------|------|-------------|\n| `model_name` | VARCHAR | Model name |\n| `provider_name` | VARCHAR | Provider (OpenAI, Anthropic, etc.) |\n| `context_window` | BIGINT | Maximum context window |\n| `tool_call` | BOOLEAN | Supports function calling |\n| `structured_output` | BOOLEAN | Supports JSON mode |\n| `reasoning` | BOOLEAN | Chain-of-thought model |\n| `open_weights` | BOOLEAN | Weights publicly available |\n\n\u003e **Note:** The `benchmarks` and `models` tables are independent. Use SQL to join or correlate data between them based on model/provider names.\n\n\u003c/details\u003e\n\n### Compare Models\n\nCompare models side-by-side with highlighted winners:\n\n```bash\n# Compare two or more models\nwhich-llm compare \"gpt-5 (high)\" \"claude 4.5 sonnet\" \"gemini 2.5 pro\"\n\n# Show additional fields\nwhich-llm compare \"gpt-5\" \"claude-4.5\" --verbose\n\n# Output formats: --json, --csv, --table, --plain\nwhich-llm compare \"gpt-5\" \"claude-4.5\" --json\n```\n\nThe compare command uses fuzzy matching on model names and displays a transposed table with models as columns and metrics as rows. Winners for each metric are marked with `*`.\n\n### Calculate Token Costs\n\nEstimate token costs with projections:\n\n```bash\n# Single model cost calculation\nwhich-llm cost \"gpt-5 (high)\" --input 10k --output 5k\n\n# Compare costs across models\nwhich-llm cost \"gpt-5\" \"claude 4.5\" --input 1M --output 500k\n\n# Daily/monthly projections with request volume\nwhich-llm cost \"gpt-5 (high)\" --input 2k --output 1k --requests 1000 --period daily\n\n# Supports token units: k (thousands), M (millions), B (billions)\nwhich-llm cost \"claude-4.5\" --input 1.5M --output 750k\n```\n\n### Other Commands\n\n```bash\n# Refresh data from sources\nwhich-llm refresh\n\n# View data source and attribution info\nwhich-llm info\n\n# Manage cache\nwhich-llm cache status\nwhich-llm cache clear\n\n# Manage profiles (for API access)\nwhich-llm profile list\nwhich-llm profile create work --api-key KEY\nwhich-llm profile default work\n\n# Skill management\nwhich-llm skill list\nwhich-llm skill uninstall cursor\n```\n\n## Attribution\n\n- Benchmark data provided by [Artificial Analysis](https://artificialanalysis.ai)\n- Capability metadata provided by [models.dev](https://models.dev)\n\nThis tool uses data from the [Artificial Analysis API](https://artificialanalysis.ai/documentation). Per the API terms, attribution is required for all use of the data.\n\n## Data Freshness\n\nThe CLI uses pre-built benchmark data hosted on GitHub Releases, updated daily via automated workflows. This means:\n\n- **No API key required** for basic usage\n- Data is typically **less than 24 hours old**\n- Use `which-llm info` to see when data was last updated\n- Use `which-llm refresh` to fetch fresh data from sources\n- Use `which-llm refresh --use-api` with an API key for real-time data\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frichard-gyiko%2Fwhich-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frichard-gyiko%2Fwhich-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frichard-gyiko%2Fwhich-llm/lists"}