{"id":126362,"url":"https://github.com/sam-blackfly/awesome-llm-tools","name":"awesome-llm-tools","description":"A curated directory of essential LLM tools: Local inference engines, agent frameworks, RAG pipelines, vector databases, and fine-tuning kits.","projects_count":350,"last_synced_at":"2026-09-28T17:00:33.706Z","repository":{"id":341876291,"uuid":"1171834146","full_name":"sam-blackfly/awesome-llm-tools","owner":"sam-blackfly","description":"A curated directory of essential LLM tools: Local inference engines, agent frameworks, RAG pipelines, vector databases, and fine-tuning kits.","archived":false,"fork":false,"pushed_at":"2026-03-03T19:18:41.000Z","size":31,"stargazers_count":2,"open_issues_count":6,"forks_count":6,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-08-20T01:06:17.576Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sam-blackfly.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-03T16:55:35.000Z","updated_at":"2026-08-14T16:59:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/sam-blackfly/awesome-llm-tools","commit_stats":null,"previous_names":["sam-blackfly/awesome-llm-tools"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sam-blackfly/awesome-llm-tools","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sam-blackfly%2Fawesome-llm-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sam-blackfly%2Fawesome-llm-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sam-blackfly%2Fawesome-llm-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sam-blackfly%2Fawesome-llm-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sam-blackfly","download_url":"https://codeload.github.com/sam-blackfly/awesome-llm-tools/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sam-blackfly%2Fawesome-llm-tools/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":341189360,"owners_count":37167410,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-08-22T15:14:58.755Z","status":"online","status_checked_at":"2026-09-08T02:00:05.531Z","response_time":108,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2026-04-03T22:19:13.747Z","updated_at":"2026-09-28T17:00:33.707Z","primary_language":"Python","list_of_lists":false,"displayable":true,"categories":["License","1. Local Agents","2. Libraries \u0026 Frameworks","12. Miscellaneous","11. Benchmarks \u0026 Leaderboards","6. RAG Pipelines","9. Fine-Tuning","8. Inference Engines","3. Prompt Optimization","7. Vector Databases","5. Agent Memory","4. Context Optimization","10. Model Preparation \u0026 Quantization"],"sub_categories":["Data \u0026 Alignment Tools","Coding Agents \u0026 IDE Extensions","Python","Research \u0026 Knowledge Agents","Rust","Training Frameworks","Server / Production","Desktop / Local","Autonomous Agents","Go","Audio / Voice Agents","General-Purpose Local Agents","JavaScript / TypeScript","Low-code \u0026 Visual Flow","Java","Computer Use \u0026 OS Agents","AI Browsers","Kotlin"],"readme":"# awesome-llm-tools\n\n\u003e A curated directory of essential LLM tools spanning the full stack: from model preparation and quantization to inference engines, local agents, agent frameworks, RAG pipelines, vector databases, fine-tuning, and more.\n\n[![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n\n---\n\n## Table of Contents\n\n1. [Local Agents](#1-local-agents)\n   1. [Coding Agents \u0026 IDE Extensions](#coding-agents--ide-extensions)\n   2. [Autonomous Agents](#autonomous-agents)\n   3. [AI Browsers](#ai-browsers)\n   4. [Low-code \u0026 Visual Flow](#low-code--visual-flow)\n   5. [Research \u0026 Knowledge Agents](#research--knowledge-agents)\n   6. [Computer Use \u0026 OS Agents](#computer-use--os-agents)\n   7. [Audio / Voice Agents](#audio--voice-agents)\n   8. [General-Purpose Local Agents](#general-purpose-local-agents)\n2. [Libraries \u0026 Frameworks](#2-libraries--frameworks)\n   1. [Python](#python)\n   2. [JavaScript / TypeScript](#javascript--typescript)\n   3. [Java](#java)\n   4. [Kotlin](#kotlin)\n   5. [Go](#go)\n   6. [Rust](#rust)\n3. [Prompt Optimization](#3-prompt-optimization)\n4. [Context Optimization](#4-context-optimization)\n5. [Agent Memory](#5-agent-memory)\n6. [RAG Pipelines](#6-rag-pipelines)\n7. [Vector Databases](#7-vector-databases)\n8. [Inference Engines](#8-inference-engines)\n   1. [Desktop / Local](#desktop--local)\n   2. [Server / Production](#server--production)\n9. [Fine-Tuning](#9-fine-tuning)\n   1. [Training Frameworks](#training-frameworks)\n   2. [PEFT Methods](#peft-methods-techniques)\n   3. [Data \u0026 Alignment Tools](#data--alignment-tools)\n10. [Model Preparation \u0026 Quantization](#10-model-preparation--quantization)\n11. [Benchmarks \u0026 Leaderboards](#11-benchmarks--leaderboards)\n12. [Miscellaneous](#12-miscellaneous)\n13. [Related Resources](#13-related-resources)\n\n---\n\n## 1. Local Agents\n\n### Coding Agents \u0026 IDE Extensions\n\n| Tool | Type | Editor | Local LLM? | Notes |\n|------|------|--------|------------|-------|\n| [Continue](https://github.com/continuedev/continue) | Extension | VS Code / JetBrains | ✅ Native | Best open-source local AI coding assistant |\n| [Cursor](https://cursor.com) | IDE Fork | VS Code | ⚠️ Via proxy | AI-first IDE; local LLMs via ngrok workaround |\n| [Windsurf](https://codeium.com/windsurf) | IDE Fork | VS Code | ⚠️ Partial | Codeium's agentic IDE |\n| [Aider](https://github.com/Aider-AI/aider) | CLI Agent | Terminal | ✅ Ollama/Llama.cpp | AI pair programmer in the terminal |\n| [Cline](https://github.com/cline/cline) | Extension | VS Code | ✅ OpenAI-compat | Autonomous coding agent in VS Code |\n| [KiloCode](https://github.com/Kilo-Org/kilocode) | Extension | VS Code / JetBrains / CLI | ✅ Local | Open-source; supports many local \u0026 cloud LLMs |\n| [OpenCode](https://github.com/opencode-ai/opencode) | CLI Agent | Terminal | ✅ 75+ providers | Go-based TUI agent with LSP, Git-context; 26k+ stars |\n| [Kiro](https://kiro.dev) | IDE | Standalone (Code OSS) | ⚠️ Cloud (Bedrock) | AWS's spec-driven agentic IDE; agent hooks \u0026 steering |\n| [Antigravity](https://antigravity.dev) | In-editor Agent | VS Code / JetBrains | ✅ Via API | Google DeepMind AI coding assistant |\n| [Qoder](https://qoder.ai) | IDE | Standalone | ⚠️ Cloud (multi-model) | Alibaba's agentic IDE; repo-scale context, Repo Wiki, Quest mode |\n| [Copilot (GitHub)](https://github.com/features/copilot) | Extension | VS Code / JetBrains / Neovim | ❌ Cloud only | Industry standard; cloud-only |\n| [Cody (Sourcegraph)](https://sourcegraph.com/cody) | Extension | VS Code / JetBrains | ✅ Via API | Enterprise code search + AI |\n| [Tabby](https://github.com/TabbyML/tabby) | Self-hosted | VS Code / JetBrains / Vim | ✅ Self-hosted | Self-hosted coding assistant server |\n| [Fauxpilot](https://github.com/fauxpilot/fauxpilot) | Self-hosted | VS Code (Copilot compat) | ✅ Self-hosted | Open-source GitHub Copilot alternative |\n| [Void](https://github.com/voideditor/void) | IDE Fork | VS Code | ✅ Native | Open-source Cursor alternative |\n| [Ghostwriter](https://replit.com/ai) | IDE Extension | Replit | ❌ Cloud | Replit's AI-powered pair programmer |\n| [Amazon Q](https://aws.amazon.com/q/) | Extension / AWS | VS Code / JetBrains | ❌ Cloud | AWS's generative AI assistant for coding and automation |\n| [Tabnine](https://www.tabnine.com/) | Extension | VS Code / JetBrains | ✅ Local | Enterprise-grade local/cloud AI code completions |\n| [TurboPilot](https://github.com/ravenscroftj/turbopilot) | Self-hosted | VS Code (Copilot compat) | ✅ Local | Self-hosted Copilot clone using llama.cpp to run Salesforce Codegen |\n| [v0](https://v0.dev) | Web Tool | Browser | ❌ Cloud | Prompt-driven UI generation for React and Next.js by Vercel |\n| [Lovable](https://lovable.dev) | Web Tool | Browser | ❌ Cloud | Conversational full-stack app generation from natural language |\n| [Warp](https://www.warp.dev/) | AI Terminal | Terminal | ✅ Local + Cloud | AI-native terminal with natural language commands, agent mode, and team sharing |\n| [Warp Oz](https://www.warp.dev/oz) | Autonomous Agent | Terminal | ✅ Cloud | Warp's autonomous terminal agent; runs multi-step tasks end-to-end with full shell context |\n| [goose](https://github.com/block/goose) | CLI Agent | Terminal | ✅ Local | Open-source, extensible AI agent that goes beyond code suggestions |\n| [Roo-Code](https://github.com/RooCodeInc/Roo-Code) | In-editor Agent | VS Code | ✅ OpenAI-compat | Autonomous coding agent formerly known as Roo-Cline |\n| [crush](https://github.com/charmbracelet/crush) | CLI Agent | Terminal | ✅ Local | Glamorous AI coding agent for the terminal by Charm |\n| [Plandex](https://github.com/plandex-ai/plandex) | CLI Agent | Terminal | ✅ OpenAI-compat | Long-running terminal agent for complex tasks |\n| [SWE-agent](https://github.com/princeton-nlp/SWE-agent) | CLI Agent | Terminal | ✅ Via API | Autonomous GitHub issue solver |\n| [OpenHands (OpenDevin)](https://github.com/All-Hands-AI/OpenHands) | Web + CLI | Browser / Terminal | ✅ OpenAI-compat | Full autonomous software engineering agent |\n| [kilocode](https://github.com/Kilo-Org/kilocode) | In-editor Agent | VS Code | ✅ Mixed | AI coding assistant for planning, building, and fixing code |\n| [humanlayer](https://github.com/humanlayer/humanlayer) | Middleware | API | ✅ Layer | Human-in-the-loop layer for AI agents to solve hard problems in complex codebases |\n| [ProxyAI](https://github.com/carlrobertoh/ProxyAI) | IDE Plugin | JetBrains | ✅ Native | Open-source AI copilot for JetBrains IDEs |\n| [Claude Code](https://www.anthropic.com/claude-code) | CLI Agent | Terminal | ❌ Cloud (Claude) | Anthropic's terminal agent; code onboarding, PR generation, powerful edits |\n| [Jules](https://jules.google/) | Async Agent | GitHub | ❌ Cloud (Gemini) | Google's async coding agent; works in the background on GitHub issues |\n| [Browser-Use](https://github.com/browser-use/browser-use) | Browser Agent | Browser | ✅ OpenAI-compat | Enable AI to control your browser; web automation agent |\n| [Neovim + llm.nvim](https://github.com/huggingface/llm.nvim) | Plugin | Neovim | ✅ Native | HuggingFace's Neovim LLM plugin |\n\n### Autonomous Agents\n\nFull autonomous agents that can plan, act, use tools, and run 24/7 without constant human supervision.\n\n| Tool | Description | Language | Notes |\n|------|-------------|----------|-------|\n| [OpenClaw](https://github.com/openclaw-ai/openclaw) | Self-hosted autonomous agent; multi-channel gateway, self-installing tools, persistent memory | TypeScript | 100k+ GitHub stars |\n| [TinyClaw](https://github.com/tinyclaw-ai/tinyclaw) | Lightweight personal AI companion; multi-agent team (coder, writer, researcher) with live dashboard \u0026 file-based task queue | Python | Personal, multi-channel |\n| [OpenFang](https://www.openfang.sh/) | Agent OS: 7 autonomous capability packages, 16 security layers, 40 channel adapters, 30 pre-built agents, MCP \u0026 A2A | Rust | Self-hosted |\n| [NullClaw](https://github.com/nullclaw/nullclaw) | Fastest, smallest autonomous AI assistant infra; hybrid host + WASM, edge-optimised | Zig | Self-hosted |\n| [ZeroClaw](https://github.com/zeroclaw-labs/zeroclaw) | Fast, small autonomous AI assistant infra; deploy anywhere, hot-swap any component | Rust | Self-hosted |\n| [SuperAgent](https://github.com/homanp/superagent) | Deploy LLM Agents to production; provides an API for agent creation and management | Node.js | Agent Platform |\n| [ix](https://github.com/kreneskyp/ix) | Autonomous GPT-4 agent platform; provides a full-featured UI for agent orchestration | Python | Agent OS |\n| [waggledance.ai](https://github.com/agi-merge/waggle-dance) | Concurrent system of AI Agents implementing Plan-Validate-Solve approach | Python | Goal Solving |\n| [GPT Pilot](https://github.com/Pythagora-io/gpt-pilot) | Dev tool that writes scalable apps from scratch with developer oversight | Python | Agentic Dev |\n| [Davika](https://github.com/stitionai/devika) | Agentic AI software engineer capable of complex coding tasks | Python | Open-source |\n| [n8n](https://github.com/n8n-io/n8n) | Multi-agent workflow automation; combines AI nodes with 400+ business process integrations | Node.js | Low-code |\n| [Sauna](https://www.sauna.ai) | Context-compounding assistant that learns taste and detects patterns proactively | Python | Personal assistant |\n| [AgentMail](https://www.agentmail.to) | Dedicated email inboxes for AI agents to communicate and act on emails | API / Service | Agent Tool |\n| [MetaGPT](https://github.com/geekan/MetaGPT) | Multi-agent framework that assigns roles (Product Manager, Architect, Coder) to agents | Python | Software engineering |\n| [ChatDev](https://github.com/OpenBMB/ChatDev) | Virtual software company powered by multiple agents; simulates a full dev cycle | Python | Agentic Dev |\n| [GPT Prompt Engineer](https://github.com/mshumer/gpt-prompt-engineer) | Automated prompt engineering; generates, tests, and ranks prompts to find the best | Python | Automation |\n\n### AI Browsers\n\n| Tool | Description | Platform |\n|------|-------------|----------|\n| [Perplexity Comet](https://www.perplexity.ai/comet) | Perplexity's AI-native browser with built-in search intelligence and agentic web interaction | Web / Desktop |\n| [Dia](https://www.diabrowser.com/) | AI browser from The Browser Company (makers of Arc); chat with tabs, write, learn, shop with privacy controls | macOS |\n| [Browserbase](https://www.browserbase.com/) | Serverless cloud browser infrastructure for AI agents; Playwright, Puppeteer, Selenium at scale with stealth mode | Cloud / API |\n| [ChatGPT Atlas](https://chatgpt.com/atlas/) | OpenAI's agentic interface for web-aware tasks and autonomous browsing within ChatGPT | Web |\n| [Page Assist](https://github.com/n4ze3m/page-assist) | Browser Extension | Chrome/Firefox | ✅ Local | Use locally running AI models to assist in web browsing via sidebar |\n| [nanobrowser](https://github.com/nanobrowser/nanobrowser) | Browser Extension | Chrome | ✅ Local | Open-source Chrome extension for AI-powered web automation |\n| [MultiOn](https://www.multion.ai/) | AI browser agent that can book flights, shop, and navigate complex sites autonomously | Cloud / API | Agentic Web |\n| [Skyvern](https://github.com/Skyvern-ai/Skyvern) | Browser automation agent using LLMs to automate complex browser-based workflows | Python / Playwright | Open-source |\n\n### Low-code \u0026 Visual Flow\n\n| Tool | Type | Description |\n| :--- | :--- | :--- |\n| [Flowise](https://github.com/FlowiseAI/Flowise) | Visual Flow | Drag \u0026 drop UI to build customized LLM flows using LangChainJS |\n| [Langflow](https://github.com/logspace-ai/langflow) | Visual Flow | Low-code RAG and agentic workflow designer for Python developers |\n| [Dify](https://github.com/langgenius/dify) | App Platform | LLM application development platform with visual orchestration and backend features |\n| [Chainlit](https://github.com/Chainlit/chainlit) | UI Framework | Build Python LLM apps with a professional look and feel in minutes |\n| [Flock](https://github.com/Onelevenvy/flock) | Low-code | Workflow-based platform for building chatbots, RAG, and coordinating agent teams |\n| [DemoGPT](https://github.com/melih-unsal/DemoGPT) | Visual Flow | Create quick demos by just using prompts; applies ToT approach on LangChain docs |\n\n### Research \u0026 Knowledge Agents\n\n| Tool | Description | Platform |\n|------|-------------|----------|\n| [Perplexica](https://github.com/ItzCrazyKns/Perplexica) | AI-powered search engine. An OpenSource alternative to Perplexity AI | Self-hosted |\n| [DocsGPT](https://github.com/arc53/docsgpt) | GPT-powered chat for documentation search \u0026 assistance | Web |\n| [Chaindesk](https://github.com/gmpetrov/databerry) | The no-code platform for semantic search and documents retrieval | Web |\n| [Paper QA](https://github.com/whitead/paper-qa) | LLM Chain for answering questions from documents with citations | Python |\n| [DB-GPT](https://github.com/csunny/DB-GPT) | Interact with your data and environment privately with local LLMs | Web |\n| [gpt-researcher](https://github.com/assafelovic/gpt-researcher) | Autonomous agent designed for comprehensive online research | Python |\n| [SurfSense](https://github.com/MODSetter/SurfSense) | Personal knowledge assistant; alternative to NotebookLM / Perplexity / Glean | Self-hosted |\n| [open-notebook](https://github.com/lfnovo/open-notebook) | Open-source implementation of NotebookLM with local model support | Web |\n| [RD-Agent](https://github.com/microsoft/RD-Agent) | Autonomous agent for automating industrial R\u0026D processes | Python |\n| [local-deep-researcher](https://github.com/langchain-ai/local-deep-researcher) | Fully local web research and report writing assistant | Python / LangGraph |\n| [maestro](https://github.com/murtaza-nasir/maestro) | AI-powered research application designed to streamline complex tasks | Web |\n| [Phind](https://phind.com/) | AI-powered search engine for developers; answers complex technical questions with sources | Web |\n| [Metaphor](https://metaphor.systems/) | Language model powered search engine that understands internet-scale context | Web |\n| [You.com](https://you.com/) | AI search assistant with customized experiences and strict data privacy | Web |\n| [Komo](https://komo.ai/) | AI-powered search engine focused on quick, relevant results | Web |\n\n### Computer Use \u0026 OS Agents\n\n| Tool | Description | Platform |\n|------|-------------|----------|\n| [OmniParser](https://github.com/microsoft/OmniParser) | Pure vision-based GUI agent screen parsing tool | Python / Model |\n| [cua](https://github.com/trycua/cua) | Docker container environment specialized for Computer-Use AI agents | Docker |\n| [self-operating-computer](https://github.com/OthersideAI/self-operating-computer) | Framework enabling multimodal models to operate a computer keyboard/mouse | Python |\n| [Agent-S](https://github.com/simular-ai/Agent-S) | Open agentic framework that uses computers like a human | Python |\n| [openwork](https://github.com/different-ai/openwork) | Open-source alternative to Claude Cowork powered by OpenCode | Desktop |\n| [mobile-use](https://github.com/minitap-ai/mobile-use) | AI agent that controls Android or iOS devices via natural language | Python |\n\n### Audio / Voice Agents\n\n\u003e For speech/audio **models**, see the companion models repo.\n\n| Tool | Description | Local? | Task |\n|------|-------------|--------|------|\n| [faster-whisper](https://github.com/SYSTRAN/faster-whisper) | CTranslate2-based Whisper reimplementation; 4× faster with less memory | ✅ | Speech-to-Text |\n| [whisper.cpp](https://github.com/ggerganov/whisper.cpp) | C/C++ port of Whisper; runs on CPU with no dependencies | ✅ | Speech-to-Text |\n| [Piper](https://github.com/rhasspy/piper) | Fast, local neural text-to-speech engine | ✅ | Text-to-Speech |\n| [Kokoro-FastAPI](https://github.com/remsky/Kokoro-FastAPI) | FastAPI wrapper for Kokoro TTS with OpenAI-compatible API | ✅ | Text-to-Speech |\n| [Coqui TTS](https://github.com/idiap/coqui-ai-TTS) | Community fork of Coqui TTS toolkit; multi-voice \u0026 multilingual | ✅ | Text-to-Speech |\n| [SpeechBrain](https://github.com/speechbrain/speechbrain) | All-in-one speech toolkit (ASR, TTS, speaker ID, emotion recognition) | ✅ | Multi-task |\n| [Home Assistant Wyoming](https://github.com/rhasspy/wyoming) | Protocol + server for local voice pipelines (STT/TTS/wake-word) | ✅ | Voice Pipeline |\n| [Speecht5 (HuggingFace)](https://github.com/microsoft/SpeechT5) | Microsoft's unified speech-text pre-training framework | ✅ | ASR + TTS |\n| [Vapi](https://vapi.ai/) | Voice AI platform for building low-latency conversational agents with natural speech | ❌ Cloud | API / Voice |\n| [Retell AI](https://www.retellai.com/) | Real-time conversational AI for developers to build natural-sounding voice agents | ❌ Cloud | API / Voice |\n\n### General-Purpose Local Agents\n\n| Tool | Description | Notes |\n|------|-------------|-------|\n| [Open WebUI](https://github.com/open-webui/open-webui) | Feature-rich self-hosted web UI for Ollama / OpenAI APIs | Tools, RAG, image gen built-in |\n| [AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) | All-in-one local AI app with agents, RAG, tools | Desktop + Docker |\n| [LibreChat](https://github.com/danny-avila/LibreChat) | Multi-provider AI chat with agents, plugins, RAG | Self-hosted |\n| [SillyTavern](https://github.com/SillyTavern/SillyTavern) | Advanced frontend for character-based AI interactions | Self-hosted |\n| [Lobe Chat](https://github.com/lobehub/lobe-chat) | Modern design AI chat framework; support for multiple backends, plugins, and RAG | Docker / Vercel |\n| [ChatBox](https://github.com/ChatBoxAI/ChatBox) | User-friendly desktop client app for AI models (Ollama, OpenAI, Claude, etc.) | Windows/macOS/Linux |\n| [PrivateGPT](https://github.com/zylon-ai/private-gpt) | Production-ready private document Q\u0026A agent | 100% local |\n| [Dash (Agno)](https://github.com/agno-agi/dash) | Self-learning data / text-to-SQL agent grounded in 6 layers of context; inspired by OpenAI's internal impl | Built on Agno framework |\n| [visual-explainer](https://github.com/nicobailon/visual-explainer) | Agent skill that generates rich HTML pages and slide decks for diagrams, diff reviews, plan audits, and project recaps | Agent skill / tool |\n\n---\n\n## 2. Libraries \u0026 Frameworks\n\n### Python\n\n| Framework | Description | Best For |\n|-----------|-------------|----------|\n| [LangChain](https://github.com/langchain-ai/langchain) | The most widely adopted LLM framework; chains, tools, agents, RAG | Rapid prototyping, complex chains |\n| [LangGraph](https://github.com/langchain-ai/langgraph) | Graph-based agent orchestration (state machines) from LangChain team | Deterministic multi-agent workflows |\n| [LlamaIndex](https://github.com/run-llama/llama_index) | Data framework for LLM applications; advanced RAG \u0026 agents | Data-heavy, enterprise knowledge systems |\n| [AutoGen](https://github.com/microsoft/autogen) | Microsoft's async multi-agent conversation framework | Collaborative multi-agent problem solving |\n| [CrewAI](https://github.com/crewAIInc/crewAI) | Role-based multi-agent orchestration; fast \u0026 standalone | Specialized crew workflows |\n| [Pydantic AI](https://github.com/pydantic/pydantic-ai) | Type-safe agent framework by Pydantic team | Production-grade, type-safe pipelines |\n| [Agno (Phidata)](https://github.com/agno-agi/agno) | Lightweight, fast multi-modal agent framework | Multi-modal agents with memory \u0026 storage |\n| [Haystack](https://github.com/deepset-ai/haystack) | Pipeline-based NLP / RAG framework for production | Production NLP \u0026 search pipelines |\n| [Smolagents](https://github.com/huggingface/smolagents) | Minimal, powerful agents library by Hugging Face | Code agents, tool calling |\n| [Atomic Agents](https://github.com/BrainBlend-AI/atomic-agents) | Modular, dependency-injection-based agent framework | Composable, testable agents |\n| [CAMEL-AI](https://github.com/camel-ai/camel) | Role-playing communicative multi-agent framework; research and multi-agent systems | Multi-agent Communication |\n| [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) | Official OpenAI Python SDK for building agents | OpenAI-native agent workflows |\n| [Google ADK](https://google.github.io/adk-docs/) | Google's Agent Development Kit — modular, model-agnostic framework with multi-agent orchestration, rich tool ecosystem, built-in eval | Google Cloud / Gemini agents |\n| [aisuite](https://github.com/andrewyng/aisuite) | Simple unified interface to multiple Generative AI providers, by Andrew Ng | Multi-provider prototyping |\n| [swarm](https://github.com/openai/swarm) | OpenAI's educational framework for lightweight ergonomic multi-agent orchestration | Learning multi-agent patterns |\n| [guidance](https://github.com/guidance-ai/guidance) | Interleave generation, prompting, and logic control in a continuous flow matching LLM processing | Constrained, structured generation |\n| [outlines](https://github.com/outlines-dev/outlines) | Library for reliable structured generation (JSON, regex, grammar) from LLMs and diffusers, by dottxt.ai | Structured outputs |\n| [Multi-Agent Orchestrator](https://github.com/awslabs/multi-agent-orchestrator) | AWS framework for managing multiple AI agents and complex multi-turn conversations | Enterprise multi-agent systems |\n| [rowboat](https://github.com/rowboatlabs/rowboat) | AI-powered multi-agent builder with no-code UI, powered by OpenAI Agents SDK | Rapid agent prototyping |\n| [Medusa](https://github.com/FasterDecoding/Medusa) | Framework accelerating LLM generation with multiple decoding heads; no specialized hardware | Inference acceleration |\n| [LMQL](https://lmql.ai/) | Programming language for LLMs based on logic programming; enables robust prompting | Advanced Prompting |\n| [Griptape](https://github.com/griptape-ai/griptape) | Python framework for AI workflows and pipelines with chain of thought reasoning | Workflows |\n| [Embedchain](https://github.com/embedchain/embedchain) | Framework to easily create LLM powered bots over any dataset | RAG Builder |\n| [SymbolicAI](https://github.com/Xpitfire/symbolicai) | Neuro-symbolic framework for building applications with LLMs at the core | Hybrid AI |\n| [Ludwig](https://github.com/ludwig-ai/ludwig) | Low-code framework for building custom AI models like LLMs and other deep neural networks | Low-code ML |\n| **Core Libraries** |\n| [transformers](https://github.com/huggingface/transformers) | HuggingFace's flagship model library; 300k+ models, inference \u0026 fine-tuning for every major architecture | Foundation library |\n| [accelerate](https://github.com/huggingface/accelerate) | HuggingFace library to run PyTorch training/inference on any hardware with minimal code changes | Multi-GPU / mixed-precision |\n| [sentence-transformers](https://github.com/UKPLab/sentence-transformers) | Easy-to-use library for state-of-the-art sentence, text, and image embeddings | Embeddings / semantic search |\n| [tiktoken](https://github.com/openai/tiktoken) | OpenAI's fast BPE tokenizer; used to count tokens before sending to APIs | Tokenization |\n| [openai](https://github.com/openai/openai-python) | Official OpenAI Python SDK | OpenAI API client |\n| [anthropic](https://github.com/anthropics/anthropic-sdk-python) | Official Anthropic Python SDK | Claude API client |\n| [google-genai](https://github.com/googleapis/python-genai) | Official Google Gen AI Python SDK for Gemini models | Gemini API client |\n| [google-adk](https://pypi.org/project/google-adk/) | Google Agent Development Kit Python package | Google ADK agents |\n| [llm](https://github.com/simonw/llm) | CLI tool + Python library for running prompts against any LLM; extensible via plugins (Ollama, Claude, Gemini, etc.) | CLI / scripting |\n\n### JavaScript / TypeScript\n\n| Framework | Description | Best For |\n|-----------|-------------|----------|\n| [LangChain.js](https://github.com/langchain-ai/langchainjs) | Official JS/TS port of LangChain | Full-stack JS LLM apps |\n| [LlamaIndex.TS](https://github.com/run-llama/LlamaIndexTS) | TypeScript version of LlamaIndex | RAG in Node/Deno/edge |\n| [VoltAgent](https://github.com/voltagent/voltagent) | TypeScript-native open-source agentic framework | TypeScript-first developers |\n| [Mastra](https://github.com/mastra-ai/mastra) | TypeScript agent framework with workflows, memory, tools | Modern Node.js AI apps |\n| [Vercel AI SDK](https://github.com/vercel/ai) | Full-stack AI toolkit for Next.js / React | AI-powered web UIs |\n| [GenKit (Firebase)](https://github.com/firebase/genkit) | Google's framework for production AI flows in JS/TS | Firebase / Google Cloud apps |\n| [Bee Agent Framework](https://github.com/i-am-bee/bee-agent-framework) | IBM's open-source TypeScript agent framework | Enterprise TypeScript agents |\n| **Core Libraries** |\n| [openai](https://github.com/openai/openai-node) | Official OpenAI Node.js / TypeScript SDK | OpenAI API client |\n| [anthropic](https://github.com/anthropics/anthropic-sdk-typescript) | Official Anthropic TypeScript SDK | Claude API client |\n| [@google/genai](https://github.com/googleapis/js-genai) | Official Google Gen AI JS/TS SDK for Gemini models | Gemini API client |\n| [ollama-js](https://github.com/ollama/ollama-js) | Official Ollama JavaScript library | Local model client |\n| [ai](https://www.npmjs.com/package/ai) | Vercel AI SDK — stream-first unified API for LLMs, RSC support, tool calling | Universal LLM SDK for JS/TS |\n| [genkit](https://www.npmjs.com/package/genkit) | Google's GenKit — production AI flows, plugins, tracing, evaluation | Google Cloud / Firebase AI apps |\n| [@google/adk](https://www.npmjs.com/package/@google/adk) | Google Agent Development Kit for Node.js | Google ADK agents |\n\n### Java\n\n| Framework | Description | Best For |\n|-----------|-------------|----------|\n| [Semantic Kernel](https://github.com/microsoft/semantic-kernel) | Microsoft's cross-language AI SDK | Enterprise, .NET / Java ecosystem |\n| [Spring AI](https://github.com/spring-projects/spring-ai) | Spring Framework for AI / LLM integration; prompt chaining, routing, RAG | Spring Boot apps |\n| [LangChain4j](https://github.com/langchain4j/langchain4j) | Java port of LangChain; chains, tools, agents, RAG | Java LLM apps |\n| [Quarkus AI](https://quarkus.io/ai/) | Official Quarkiverse extension for seamless LangChain4j integration and native compilation | Quarkus / Native AI |\n| [Apache Camel AI](https://camel.apache.org/components/next/ai-summary.html) | Enterprise integration components (LangChain4j, DJL) for intelligent flows | Integration / AI Flows |\n| [Google ADK Java](https://google.github.io/adk-docs/) | Google's Agent Development Kit for Java — same multi-agent, tool-use capabilities as the Python SDK | Google Cloud / Gemini Java agents |\n| **Core Libraries** |\n| [DJL (Deep Java Library)](https://github.com/deepjavalibrary/djl) | Amazon's engine-agnostic deep learning library for Java; run HuggingFace \u0026 PyTorch models | Java ML inference |\n| [openai-java](https://github.com/openai/openai-java) | Official OpenAI Java SDK | OpenAI API client |\n\n### Kotlin\n\n| Framework | Description | Best For |\n|-----------|-------------|----------|\n| [Koog](https://github.com/JetBrains/koog) | JetBrains' pure-Kotlin agent framework; multiplatform (JVM, Android, iOS, WASM), fault-tolerant with retries \u0026 state restore | Enterprise Kotlin / multiplatform agents |\n| **Core Libraries** |\n| [DJL (Kotlin)](https://github.com/deepjavalibrary/djl) | Amazon's DJL works seamlessly in Kotlin; run PyTorch, ONNX, HuggingFace models on JVM | Kotlin ML inference |\n| [Kotlin OpenAI client](https://github.com/aallam/openai-kotlin) | Multiplatform OpenAI API client for Kotlin (JVM, Android, JS, Native) | OpenAI API client |\n\n### Go\n\n| Framework | Description | Best For |\n|-----------|-------------|----------|\n| [LangChainGo](https://github.com/tmc/langchaingo) | Go port of LangChain; chains, tools, memory, RAG | Go LLM apps |\n| [Eino](https://github.com/cloudwego/eino) | ByteDance's LLM framework inspired by LangChain \u0026 ADK; reusable components, multi-agent coordination | High-performance Go services |\n| [adk-go](https://github.com/google/adk-go) | Google's Agent Development Kit for Go — multi-agent orchestration, tool use, Gemini/Vertex AI integration | Google ADK Go agents |\n| **Core Libraries** |\n| [go-openai](https://github.com/sashabaranov/go-openai) | Unofficial but most widely used OpenAI Go client; OpenAI, Azure OpenAI support | OpenAI API client |\n| [ollama-go](https://github.com/ollama/ollama/tree/main/api) | Official Ollama Go API client | Local model client |\n\n### Rust\n\n| Framework | Description | Best For |\n|-----------|-------------|----------|\n| [Rig](https://github.com/0xPlaygrounds/rig) | Modular, production-ready Rust library; 20+ providers, 10+ vector stores, WASM support | Performance-critical Rust agents |\n| [LLM-Chain](https://github.com/sobelio/llm-chain) | Structured multi-step LLM workflows in Rust; prompt templates, chains, tool calling | Type-safe Rust pipelines |\n| **Core Libraries** |\n| [candle](https://github.com/huggingface/candle) | HuggingFace's minimalist Rust ML framework; GPU support, WASM-ready, no Python dependency | Rust-native ML inference |\n| [async-openai](https://github.com/64bit/async-openai) | Async Rust client for OpenAI API; idiomatic, type-safe | OpenAI API client |\n| [ollama-rs](https://github.com/pepperoni21/ollama-rs) | Async Rust client for the Ollama API | Local model client |\n| [aichat](https://github.com/sigoden/aichat) | All-in-one Rust CLI with shell integration, RAG, agents, and 20+ LLM providers | CLI / scripting |\n\n---\n\n## 3. Prompt Optimization\n\nTools for systematic prompt engineering, evaluation, and automated optimization.\n\n| Tool | Description | Approach |\n|------|-------------|----------|\n| [DSPy](https://github.com/stanfordnlp/dspy) | Stanford's framework for programming (not prompting) LLMs — declarative, auto-optimized | Automatic |\n| [PromptFoo](https://github.com/promptfoo/promptfoo) | CLI/CI tool for testing, evaluating, and red-teaming prompts | Evaluation |\n| [Langfuse](https://github.com/langfuse/langfuse) | Open-source LLM observability with prompt management, tracing, evals | Monitoring |\n| [PromptLayer](https://promptlayer.com) | Prompt versioning, logging, and A/B testing SaaS | Management |\n| [LangSmith](https://smith.langchain.com) | LangChain's debugging, testing, and evaluation platform | Evaluation |\n| [Promptflow (Azure)](https://github.com/microsoft/promptflow) | Microsoft's end-to-end LLM app development and evaluation | Azure |\n| [PromptPerfect](https://promptperfect.jina.ai) | AI-driven auto-optimization of prompts for any model | Automatic |\n| [Agenta](https://github.com/agenta-ai/agenta) | Open-source LLM dev platform: prompt playground, evaluation | Evaluation |\n| [EvalPlus](https://github.com/evalplus/evalplus) | Rigorous code evaluation framework for LLMs | Code eval |\n| [Opik](https://github.com/comet-ml/opik) | Open-source LLM evaluation and tracing by Comet | Evaluation |\n| [OPRO](https://github.com/google-deepmind/opro) | Google DeepMind's optimization by prompting — LLM as optimizer | Automatic |\n| [TextGrad](https://github.com/zou-group/textgrad) | Automatic differentiation through text for prompt/system optimization | Gradient-based |\n| [DeepEval](https://github.com/confident-ai/deepeval) | Unit testing framework for LLMs; test-driven development for RAG \u0026 agents | Evaluation |\n| [Giskard](https://github.com/Giskard-AI/giskard) | Open-source testing framework for LLMs and ML models; automated red-teaming | Evaluation |\n\n---\n\n## 4. Context Optimization\n\nTools and techniques for managing, compressing, and optimizing the LLM context window.\n\n| Tool | Description | Technique |\n|------|-------------|-----------|\n| [LLMLingua](https://github.com/microsoft/LLMLingua) | Microsoft's prompt compression library; removes redundant tokens | Compression |\n| [LongLLMLingua](https://github.com/microsoft/LLMLingua) | Extension for long-context scenarios with key information extraction | Compression |\n| [Rerankers](https://github.com/AnswerDotAI/rerankers) | Unified interface for cross-encoder rerankers to prioritize context | Reranking |\n| [FlashAttention](https://github.com/Dao-AILab/flash-attention) | IO-aware exact attention algorithm; enables longer context efficiently | Kernel optimization |\n| [RoPE scaling / YaRN](https://github.com/jquesnelle/yarn) | Context length extension techniques (RoPE NTK, YaRN) | Positional Encoding |\n| [LongRoPE](https://github.com/microsoft/LongRoPE) | Microsoft's long-context RoPE extension method | Positional Encoding |\n| [Sliding Window Attention](https://github.com/mistralai/mistral-src) | Mistral's local attention mechanism for efficient long contexts | Architecture |\n| [Semantic Chunking](https://github.com/aurelio-labs/semantic-chunker) | Context-aware document chunking based on semantic similarity | Chunking |\n| [LlamaIndex Context Management](https://docs.llamaindex.ai/en/stable/module_guides/querying/response_synthesizers/) | Built-in context window management in LlamaIndex pipelines | Pipeline |\n| [Context Caching (Gemini)](https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview) | Reuse prompt prefixes across requests to reduce latency + cost | Caching |\n| [claude-context-mode](https://github.com/mksglu/claude-context-mode) | Stops context loss from large outputs; smart snippets, fuzzy search, progressive throttling, subagent routing for Claude | Context Management |\n| [LMCache](https://github.com/LMCache/LMCache) | KV Cache sharing layer; share and reuse KV caches across requests for faster inference and higher throughput | Caching |\n| [GPTCache](https://github.com/zilliztech/GPTCache) | A Library for Creating Semantic Cache for LLM Queries to reduce costs | Semantic Caching |\n\n---\n\n## 5. Agent Memory\n\nPersistent, long-term, and working memory systems for AI agents.\n\n| Tool | Description | Memory Type | Self-hosted |\n|------|-------------|-------------|-------------|\n| [Mem0](https://github.com/mem0ai/mem0) | Intelligent memory layer for AI; multi-level (user, session, agent) memory with LLM-based extraction | Semantic + Graph | ✅ |\n| [Letta (MemGPT)](https://github.com/letta-ai/letta) | OS-inspired agent memory; tiered main/external context (formerly MemGPT) | Hierarchical + Persistent | ✅ |\n| [Zep](https://github.com/getzep/zep) | Temporal knowledge graph memory for assistants; extracts entities \u0026 relations | Knowledge Graph | ✅ |\n| [Cognee](https://github.com/topoteretes/cognee) | Structured memory with knowledge graphs built from conversation history | Knowledge Graph | ✅ |\n| [Memary](https://github.com/kingjulio8238/memary) | Open-source memory system for agents with routing and self-healing | Semantic | ✅ |\n| [LangMem](https://github.com/langchain-ai/langmem) | LangChain's long-term memory SDK for LangGraph agents | Semantic | ✅ |\n| [Graphiti (Zep)](https://github.com/getzep/graphiti) | Real-time knowledge graph for AI agents from the Zep team | Knowledge Graph | ✅ |\n| [R2R (Retrieval to Reasoning)](https://github.com/SciPhi-AI/R2R) | Production-ready RAG + memory system with knowledge graphs | Hybrid | ✅ |\n| [ReMemory](https://github.com/agentscope-ai/reme) | Memory Management Kit for agents; compacts history, writes summaries to file, semantic recall via MemorySearch | File + Semantic | ✅ |\n| [supermemory](https://github.com/supermemoryai/supermemory) | Personal AI second brain; fast, scalable memory engine and application | Second Brain | ✅ |\n| [memU](https://github.com/NevaMind-AI/memU) | Open-source memory framework for AI companions and persona-driven agents | Persona Memory | ✅ |\n| [quivr](https://github.com/StanGirard/quivr) | Open-source RAG framework that acts as your generative AI second brain | Second Brain | ✅ |\n\n---\n\n## 6. RAG Pipelines\n\nFrameworks and tools for building Retrieval-Augmented Generation systems.\n\n| Tool | Description | Best For |\n|------|-------------|----------|\n| [LlamaIndex](https://github.com/run-llama/llama_index) | Data-first framework; 150+ connectors, advanced indexing, sub-question agents | Document-heavy RAG |\n| [Haystack](https://github.com/deepset-ai/haystack) | Modular pipeline framework for enterprise RAG \u0026 search; 50+ integrations | Production pipelines |\n| [LangChain](https://github.com/langchain-ai/langchain) | End-to-end RAG chains with rich integrations and agent-based retrieval | Flexible RAG prototyping |\n| [Ragas](https://github.com/explodinggradients/ragas) | Reference-free RAG evaluation (faithfulness, relevance, recall) | RAG Evaluation |\n| [TRULENS](https://github.com/truera/trulens) | LLM app evaluation and RAG feedback functions | Evaluation + Observability |\n| [FlashRAG](https://github.com/RUC-NLPIR/FlashRAG) | Modular unified framework for reproducible RAG research | Research |\n| [Pathway](https://github.com/pathwaycom/pathway) | Real-time streaming RAG pipeline framework | Live data RAG |\n| [R2R](https://github.com/SciPhi-AI/R2R) | Production RAG system with hybrid search, knowledge graphs, eval | Production |\n| [Cognita](https://github.com/truefoundry/cognita) | Modular, production-ready RAG framework by TrueFoundry | Organized, modular RAG |\n| [Verba](https://github.com/weaviate/verba) | Weaviate-powered open-source RAG chatbot | Weaviate + RAG |\n| [DSPy](https://github.com/stanfordnlp/dspy) | Declarative RAG pipelines with auto-optimization of retrieval | Optimized RAG |\n| [Camel RAG](https://github.com/camel-ai/camel) | RAG tools within the CAMEL multi-agent framework | Multi-agent + RAG |\n| [PageIndex](https://github.com/VectifyAI/PageIndex) | Vectorless, reasoning-based RAG; builds a hierarchical document index and uses LLM reasoning over it instead of embeddings | No-vector RAG |\n| [GraphRAG](https://github.com/microsoft/graphrag) | Microsoft's graph-based RAG; builds knowledge graph communities and uses hierarchical summarisation for global question answering | Knowledge Graph RAG |\n| [HashIndex](https://github.com/JasonHonKL/HashIndex) | Ultra-fast LLM-optimised document indexing without vector search; hash-based retrieval | No-vector RAG |\n| [LightRAG](https://github.com/HKUDS/LightRAG) | Dual-level retrieval system combining graph and vector search for deep context awareness | Graph + Vector RAG |\n| [vanna](https://github.com/vanna-ai/vanna) | Open-source Python RAG framework for SQL generation and related functionality | Text-to-SQL RAG |\n| [graphiti](https://github.com/getzep/graphiti) | Real-time knowledge graph for AI agents with temporal awareness | Knowledge Graph |\n| [onyx](https://github.com/onyx-dot-app/onyx) | Enterprise AI platform connected to company docs, apps, and people | Enterprise RAG |\n| [crawl4ai](https://github.com/unclecode/crawl4ai) | Open-source web crawler and scraper designed for LLMs and AI agents | Web Crawling |\n| [Firecrawl](https://github.com/mendableai/firecrawl) | API to scrape and crawl websites into clean, LLM-ready markdown or structured data | Web Scraper / API |\n| [Docling](https://github.com/DS4SD/docling) | IBM's open-source toolkit to transform unstructured documents (PDF, etc.) into structured data for AI | Document Parser |\n| [claude-context](https://github.com/zilliztech/claude-context) | Make entire codebase the context for any coding agent | Codebase RAG |\n| [pipeshub-ai](https://github.com/pipeshub-ai/pipeshub-ai) | Extensible workplace AI platform for enterprise search and workflow automation | Enterprise RAG |\n\n---\n\n## 7. Vector Databases\n\nPurpose-built stores for high-dimensional vector embeddings.\n\n| Database | Type | Best For | License |\n|----------|------|----------|---------|\n| [Chroma](https://github.com/chroma-core/chroma) | Open-source, Python-native | Local dev, prototyping, RAG | Apache-2.0 |\n| [Qdrant](https://github.com/qdrant/qdrant) | Open-source, Rust-built, cloud option | Production, fast filtering, hybrid search | Apache-2.0 |\n| [Milvus](https://github.com/milvus-io/milvus) | Open-source, Kubernetes-scale | Billion-vector scale, enterprise | Apache-2.0 |\n| [Weaviate](https://github.com/weaviate/weaviate) | Open-source, GraphQL API | Semantic + GraphQL queries, hybrid search | BSD-3 |\n| [pgvector](https://github.com/pgvector/pgvector) | PostgreSQL extension | Existing PostgreSQL users, \u003c1M vectors | PostgreSQL |\n| [LanceDB](https://github.com/lancedb/lancedb) | Embedded, serverless, columnar | Serverless RAG, edge apps | Apache-2.0 |\n| [zvec](https://github.com/alibaba/zvec) | In-process embedded vector database; lightweight and extremely fast (AliBaba) | Embedded RAG, local search | Apache-2.0 |\n| [Faiss](https://github.com/facebookresearch/faiss) | Library, not a DB | Raw ANN search in Python/C++ | MIT |\n| [Annoy](https://github.com/spotify/annoy) | Library, memory-mapped | Static indexes, read-heavy | Apache-2.0 |\n| [Vespa](https://github.com/vespa-engine/vespa) | Full search engine with vector support | Hybrid text + vector at scale | Apache-2.0 |\n| [Marqo](https://github.com/marqo-ai/marqo) | End-to-end multimodal tensor search | Image + text search | SSPL |\n| [Turbopuffer](https://turbopuffer.com) | Serverless vector DB on object storage | Low-cost, high-scale cloud | Proprietary |\n| **Managed / Cloud** | | | |\n| [Pinecone](https://pinecone.io) | Fully managed cloud vector DB | Managed, auto-scaling, \u003c50ms p95 | Proprietary |\n| [Zilliz Cloud](https://zilliz.com) | Managed Milvus; enterprise-grade | Managed Milvus | Proprietary |\n| [Weaviate Cloud](https://weaviate.io/pricing) | Managed Weaviate | Managed hybrid search | Proprietary |\n\n---\n\n## 8. Inference Engines\n\n### Desktop / Local\n\nOptimized for personal hardware — laptops, workstations, or edge devices.\n\n| Tool | Description | Backends | GUI |\n|------|-------------|----------|-----|\n| [llama.cpp](https://github.com/ggerganov/llama.cpp) | The foundational C/C++ LLM inference engine; cross-platform CPU+GPU, GGUF format, powers most local tools | CUDA, Metal, Vulkan, ROCm | ✅ Web (built-in server) |\n| [Ollama](https://github.com/ollama/ollama) | \"Docker for LLMs\" — one-command model pull \u0026 run, OpenAI-compatible API | llama.cpp, MLX | Web UI via Open WebUI |\n| [LM Studio](https://lmstudio.ai) | Polished cross-platform desktop app; model discovery, local server, chat | llama.cpp, MLX | ✅ Native |\n| [GPT4All](https://github.com/nomic-ai/gpt4all) | Privacy-first desktop app; curated model library, no GPU required | llama.cpp | ✅ Native |\n| [Jan](https://github.com/janhq/jan) | ChatGPT-like open-source desktop app; local-first, extensions | llama.cpp, TensorRT | ✅ Native |\n| [koboldcpp](https://github.com/LostRuins/koboldcpp) | Single-binary llama.cpp with a web UI for creative writing \u0026 roleplay | llama.cpp | ✅ Web |\n| [text-generation-webui](https://github.com/oobabooga/text-generation-webui) | Feature-rich Gradio web UI with multi-backend support (Oobabooga) | llama.cpp, ExLlamaV2, GPTQ, AWQ | ✅ Web |\n| [llamafile](https://github.com/Mozilla-Ocho/llamafile) | Single-file executable LLMs that run on any OS without install | llama.cpp | ✅ Web |\n| [MLC LLM](https://github.com/mlc-ai/mlc-llm) | Universal LLM deployment framework; targets CPU, GPU, Mobile, WebGPU | TVM | ✅ Web |\n| [LocalAI](https://github.com/mudler/LocalAI) | Self-hosted OpenAI-compatible REST API for LLMs, images, audio | llama.cpp, GPU backends | ✅ Web |\n| [OpenLLM](https://github.com/bentoml/OpenLLM) | Open-source platform for deploying and operating LLMs in production via BentoML | Multiple | ✅ Web |\n| [msty](https://msty.app/) | Simple, clean desktop app for running local and online AI models side-by-side | llama.cpp | ✅ Native |\n| [Open Interpreter](https://github.com/KillianLucas/open-interpreter) | Locally running implementation of OpenAI's Code Interpreter; executes code in the terminal | llama.cpp, Ollama | ✅ Terminal |\n| [bitnet.cpp](https://github.com/microsoft/BitNet) | Official inference framework for 1-bit LLMs (BitNet b1.58); fast lossless CPU inference | CPU (NPU/GPU coming) | ❌ |\n| [PowerInfer](https://github.com/SJTU-IPADS/PowerInfer) | CPU/GPU inference engine exploiting neuron activation locality for faster inference | CUDA + CPU | ❌ |\n| [exo](https://github.com/exo-explore/exo) | Distributed inference engine that clusters multiple devices (macOS, Linux) into a single GPU for running frontier models | Metal, CUDA | ✅ Dashboard |\n| [distributed-llama](https://github.com/b4rtaz/distributed-llama) | Connect home devices into a powerful cluster to accelerate LLM inference | C++ | ❌ |\n| [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) | llama.cpp fork with additional SOTA quants and improved performance | llama.cpp fork | ❌ |\n| [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM) | Optimized inference engine for running LLMs on AMD Ryzen™ AI NPUs | AMD NPU | ❌ |\n| [lemonade](https://github.com/lemonade-sdk/lemonade) | Local LLM server with GPU and NPU Acceleration | Multiple | ❌ |\n\n### Server / Production\n\nDesigned for high-throughput, multi-user, and production API serving.\n\n| Tool | Description | Key Feature | GPU Support |\n|------|-------------|-------------|-------------|\n| [vLLM](https://github.com/vllm-project/vllm) | State-of-the-art serving engine with PagedAttention \u0026 continuous batching | Highest throughput | NVIDIA, AMD, TPU |\n| [TGI (Text Generation Inference)](https://github.com/huggingface/text-generation-inference) | Hugging Face's production-grade Rust/Python server | Flash Attention, tensor parallelism | NVIDIA, AMD, Gaudi |\n| [flashinfer](https://github.com/flashinfer-ai/flashinfer) | High-performance kernel library for LLM serving | Kernel optimization | NVIDIA |\n| [SGLang](https://github.com/sgl-project/sglang) | Fast structured generation runtime with RadixAttention | Structured outputs | NVIDIA |\n| [LiteLLM](https://github.com/BerriAI/litellm) | Unified OpenAI-compatible proxy for 100+ LLM providers | Provider routing | Cloud + Local |\n| [Triton Inference Server](https://github.com/triton-inference-server/server) | NVIDIA's enterprise multi-framework model serving | Multi-model, multi-GPU | NVIDIA |\n| [Ray Serve](https://github.com/ray-project/ray) | Scalable model serving via Ray distributed framework | Autoscaling | Any |\n| [FastChat](https://github.com/lm-sys/FastChat) | Multi-model API server and chat UI (LLM Arena backend) | Multi-model | NVIDIA |\n| [Aphrodite Engine](https://github.com/PygmalionAI/aphrodite-engine) | vLLM fork with extra quantization methods and creative writing focus | GPTQ, AWQ, EXL2 | NVIDIA |\n| [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) | NVIDIA's peak-performance compiled inference for Blackwell/Hopper | FP8, NVFP4 | NVIDIA |\n| [Llama.cpp server](https://github.com/ggerganov/llama.cpp/tree/master/examples/server) | Built-in OpenAI-compatible HTTP server inside llama.cpp | Cross-platform CPU+GPU | Any |\n| [ClawRouter](https://github.com/BlockRunAI/ClawRouter) | Agent-native LLM cost router; 15-dimension classifier routes to cheapest capable model in \u003c1ms, 41+ providers, x402 micropayments | Any | Any |\n| [Portkey Gateway](https://github.com/Portkey-AI/gateway) | Unified API for 100+ open \u0026 closed-source models with caching, fallbacks, retries, load-balancing, and edge deployment | Any | Any |\n| [Nano-vLLM](https://github.com/GeeeekExplorer/nano-vllm) | Lightweight vLLM implementation built from scratch for minimal serving overhead | Least overhead | NVIDIA |\n| [gpustack](https://github.com/gpustack/gpustack) | Simple, scalable AI model deployment on GPU clusters; managing multiple nodes easily | Multi-node | NVIDIA/AMD |\n| [mini-sglang](https://github.com/sgl-project/mini-sglang) | Lightweight yet high-performance inference framework for LLMs based on SGLang | Fast serving | NVIDIA |\n| [vllm-gfx906](https://github.com/nlzy/vllm-gfx906) | vLLM optimized for AMD gfx906 GPUs (Radeon VII, MI50, MI60) | AMD specialized | AMD |\n| [llm-scaler](https://github.com/intel/llm-scaler) | Run and scale LLMs on Intel Arc™ Pro B60 GPUs | Intel specialized | Intel |\n\n---\n\n\n## 9. Fine-Tuning\n\nFrameworks and tools for fine-tuning, alignment, and PEFT methods.\n\n### Training Frameworks\n\n| Tool | Description | Methods | Memory Efficient |\n|------|-------------|---------|-----------------|\n| [Unsloth](https://github.com/unslothai/unsloth) | 2–5× faster fine-tuning; 70–90% less VRAM via custom CUDA/Triton kernels | QLoRA, LoRA, GRPO | ✅✅ |\n| [Axolotl](https://github.com/axolotl-ai-cloud/axolotl) | YAML-configured fine-tuning for any model; multi-GPU with DeepSpeed/FSDP | LoRA, QLoRA, Full, DPO, ORPO, GRPO | ✅ |\n| [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) | User-friendly fine-tuning with Web UI + CLI; 100+ model support | LoRA, QLoRA, Full, SFT, DPO, PPO, ORPO | ✅ |\n| [TRL (HuggingFace)](https://github.com/huggingface/trl) | HuggingFace library for RL-based LLM training | SFT, DPO, GRPO, PPO, Reward Modeling | ✅ |\n| [PEFT (HuggingFace)](https://github.com/huggingface/peft) | Parameter-Efficient Fine-Tuning methods library | LoRA, QLoRA, Prefix Tuning, Adapters, IA³ | ✅ |\n| [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) | High-performance RLHF framework built on Ray, vLLM, ZeRO-3, and DeepSpeed | RLHF, PPO, DPO, GRPO, SFT | ✅ |\n| [Kiln](https://github.com/kiln-ai/kiln) | Fine-tuning tool with synthetic data generation and dataset collaboration | SFT / Synthetic Data | ✅ |\n| [augmentoolkit](https://github.com/e-p-armstrong/augmentoolkit) | Tool for training open-source LLMs on new factual datasets | Fact Training | ✅ |\n| [torchtune](https://github.com/pytorch/torchtune) | PyTorch's native fine-tuning library; no frills, pure PyTorch | LoRA, QLoRA, Full | ✅ |\n| [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) | NVIDIA's framework for training \u0026 fine-tuning at massive scale | Full FT, tensor/pipeline parallelism | ✅ (multi-node) |\n| [DeepSpeed](https://github.com/microsoft/DeepSpeed) | Microsoft's deep learning optimization library; ZeRO memory optimization | Full FT, ZeRO-1/2/3 | ✅✅ |\n\n### PEFT Methods (Techniques)\n\n| Method | Description | Implementation |\n|--------|-------------|----------------|\n| LoRA | Low-Rank Adaptation; trains small rank-decomposition matrices | PEFT, Unsloth, Axolotl |\n| QLoRA | Quantized LoRA; 4-bit base + LoRA adapters | PEFT + bitsandbytes |\n| DoRA | Weight-decomposed LoRA; often outperforms LoRA | Unsloth, PEFT |\n| GaLore | Gradient Low-Rank Projection for full-parameter training efficiency | PEFT |\n| Adapters | Small bottleneck modules inserted between layers | PEFT, AdapterHub |\n| Prefix Tuning | Learned prefix tokens prepended to each Transformer layer | PEFT |\n\n### Data \u0026 Alignment Tools\n\n| Tool | Description |\n|------|-------------|\n| [Alpaca Eval](https://github.com/tatsu-lab/alpaca_eval) | Automated instruction-following model evaluation |\n| [Open Instruct](https://github.com/allenai/open-instruct) | AllenAI's framework for instruction tuning with SFT/DPO/RL |\n| [DPO Trainer (TRL)](https://huggingface.co/docs/trl/dpo_trainer) | Direct Preference Optimization trainer in TRL |\n| [DataTrove](https://github.com/huggingface/datatrove) | Large-scale data processing pipeline for pre-training data |\n| [Distilabel](https://github.com/argilla-io/distilabel) | Synthetic data generation and AI feedback pipeline |\n| [Argilla](https://github.com/argilla-io/argilla) | Open-source data annotation platform for LLM datasets |\n\n---\n \n ## 10. Model Preparation \u0026 Quantization\n \n Tools for merging, quantizing, and optimizing models before deployment.\n \n | Tool | Description | Best For |\n |------|-------------|----------|\n | [mergekit](https://github.com/arcee-ai/mergekit) | Powerful toolkit for merging multiple LLMs (SLERP, TIES, DARE, Passthrough) | Model Merging |\n | [bitsandbytes](https://github.com/TimDettmers/bitsandbytes) | Lightweight wrapper for CUDA custom functions, particularly 8-bit optimizers and 4-bit quantization | Quantization Base |\n | [AutoGPTQ](https://github.com/PanQiQi/AutoGPTQ) | Easy-to-use LLM quantization package based on the GPTQ algorithm | GPTQ Quantization |\n | [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) | Efficient LLM quantization package based on the AWQ algorithm | AWQ Quantization |\n | [GGUF (llama.cpp)](https://github.com/ggerganov/llama.cpp/blob/master/examples/quantize/README.md) | The standard quantization format for CPU/local inference | Local Quantization |\n | [ExLlamaV2](https://github.com/turboderp/exllamav2) | Fast inference for EXL2 quantized models on NVIDIA GPUs | GPU Optimization |\n \n ---\n \n | [outlines](https://github.com/dottxt-ai/outlines) | Structured outputs and generation control for LLMs | ![Stars](https://img.shields.io/github/stars/dottxt-ai/outlines?style=flat-square) | Apache-2.0 |\n \n ---\n \n ## 11. Benchmarks \u0026 Leaderboards\n \n | Resource | Description |\n |----------|-------------|\n | [LMSYS Chatbot Arena](https://chat.lmsys.org/?leaderboard) | Crowdsourced open platform for LLM evaluation; ELO-based leaderboard |\n | [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) | HuggingFace's tracking of open-source LLMs across multiple benchmarks |\n | [MMLU](https://github.com/hendrycks/test) | Massive Multitask Language Understanding benchmark |\n | [HumanEval](https://github.com/openai/human-eval) | OpenAI's benchmark for evaluating code generation |\n | [SWE-bench](https://github.com/princeton-nlp/SWE-bench) | Evaluation for software engineering agents on real GitHub issues |\n | [LiveCodeBench](https://livecodebench.github.io/leaderboard.html) | Holistic and contamination-free evaluation of LLMs for code |\n \n ---\n \n ## 12. Miscellaneous\n\nTools that are useful across multiple categories or don't fit neatly into a single section.\n\n| Tool | Description | Stars | License |\n|------|-------------|-------|---------|\n| [llmfit](https://github.com/AlexsJones/llmfit) | Rust TUI/CLI that scans your hardware and shows which models from 100+ providers will fit and run on it | ![Stars](https://img.shields.io/github/stars/AlexsJones/llmfit?style=flat-square) | MIT |\n| [llm-checker](https://github.com/Sunwood-ai-labs/llm-checker) | Python CLI that scans hardware (Apple Silicon, NVIDIA, AMD, Intel Arc), scores compatible models on quality/speed/fit, Ollama integration + built-in MCP server | ![Stars](https://img.shields.io/github/stars/Sunwood-ai-labs/llm-checker?style=flat-square) | MIT |\n| [selfhostllm](https://github.com/erans/selfhostllm) | Web calculator estimating GPU memory requirements and max concurrent requests for self-hosted LLM inference | ![Stars](https://img.shields.io/github/stars/erans/selfhostllm?style=flat-square) | MIT |\n| [gitingest](https://github.com/coderamp-labs/gitingest) | Converts any GitHub repo into a single prompt-friendly text dump; replace \"hub\" with \"ingest\" in any GitHub URL | ![Stars](https://img.shields.io/github/stars/coderamp-labs/gitingest?style=flat-square) | MIT |\n| [Repomix](https://github.com/yamadashy/repomix) | Pack your codebase into AI-friendly formats; similar to gitingest but focused on Node ecosystem | ![Stars](https://img.shields.io/github/stars/yamadashy/repomix?style=flat-square) | MIT |\n| [Stenography](https://stenography.dev/) | Automatic code documentation using AI to explain complex logic in natural language | ![Stars](https://img.shields.io/github/stars/stenography-ai/stenography?style=flat-square) | Proprietary |\n| [Mintlify](https://mintlify.com/) | AI-powered documentation writer that monitors code changes to keep docs in sync | ![Stars](https://img.shields.io/github/stars/mintlify/mintlify?style=flat-square) | Proprietary |\n| [Phoenix](https://github.com/Arize-ai/phoenix) | Open-source AI observability platform for tracing, evaluating, and monitoring LLM apps | ![Stars](https://img.shields.io/github/stars/Arize-ai/phoenix?style=flat-square) | Apache-2.0 |\n| [OpenLIT](https://github.com/openlit/openlit) | Open-source GenAI and LLM observability platform native to OpenTelemetry | ![Stars](https://img.shields.io/github/stars/openlit/openlit?style=flat-square) | Apache-2.0 |\n| [Openllmetry](https://github.com/traceloop/openllmetry) | Open-source observability for LLM applications based on OpenTelemetry | ![Stars](https://img.shields.io/github/stars/traceloop/openllmetry?style=flat-square) | Apache-2.0 |\n| [MLflow](https://github.com/mlflow/mlflow) | Open-source platform for the machine learning lifecycle, including LLM tracking | ![Stars](https://img.shields.io/github/stars/mlflow/mlflow?style=flat-square) | Apache-2.0 |\n| [Trigger.dev](https://github.com/triggerdotdev/trigger.dev) | Background jobs platform for long-running AI workflows and dependable agents | ![Stars](https://img.shields.io/github/stars/triggerdotdev/trigger.dev?style=flat-square) | Apache-2.0 |\n| [context7](https://github.com/upstash/context7) | Up-to-date code documentation for LLMs and AI code editors | ![Stars](https://img.shields.io/github/stars/upstash/context7?style=flat-square) | MIT |\n| [deepwiki-open](https://github.com/AsyncFuncAI/deepwiki-open) | AI-powered wiki generator for GitHub/Gitlab/Bitbucket repositories | ![Stars](https://img.shields.io/github/stars/AsyncFuncAI/deepwiki-open?style=flat-square) | MIT |\n| [cai](https://github.com/aliasrobotics/cai) | Cybersecurity AI framework (CAI) for AI security testing | ![Stars](https://img.shields.io/github/stars/aliasrobotics/cai?style=flat-square) | MIT |\n| [speakr](https://github.com/murtaza-nasir/speakr) | Personal, self-hosted web application for transcribing audio recordings | ![Stars](https://img.shields.io/github/stars/murtaza-nasir/speakr?style=flat-square) | MIT |\n| [presenton](https://github.com/presenton/presenton) | Open-source AI presentation generator and API | ![Stars](https://img.shields.io/github/stars/presenton/presenton?style=flat-square) | MIT |\n| [Gorilla](https://github.com/ShishirPatil/gorilla) | Large Language Model-Connected APIs; enables LLMs to use 1600+ APIs | ![Stars](https://img.shields.io/github/stars/ShishirPatil/gorilla?style=flat-square) | Apache-2.0 |\n| [LlamaHub](https://github.com/emptycrown/llama-hub) | A library of data loaders for LLMs made by the community | ![Stars](https://img.shields.io/github/stars/emptycrown/llama-hub?style=flat-square) | MIT |\n| [OmniGen2](https://github.com/VectorSpaceLab/OmniGen2) | Exploration into advanced multimodal generation | ![Stars](https://img.shields.io/github/stars/VectorSpaceLab/OmniGen2?style=flat-square) | MIT |\n| [4o-ghibli-at-home](https://github.com/TheAhmadOsman/4o-ghibli-at-home) | Self-hosted AI photo stylizer for performance and privacy | ![Stars](https://img.shields.io/github/stars/TheAhmadOsman/4o-ghibli-at-home?style=flat-square) | MIT |\n| [Observer](https://github.com/Roy3838/Observer) | Local micro-agents that observe, log and react to system events | ![Stars](https://img.shields.io/github/stars/Roy3838/Observer?style=flat-square) | MIT |\n| [gabber](https://github.com/gabber-dev/gabber) | Tool to build AI apps that see, hear, and speak using screen/mic/cam | ![Stars](https://img.shields.io/github/stars/gabber-dev/gabber?style=flat-square) | MIT |\n| [promptcat](https://github.com/sevenreasons/promptcat) | Zero-dependency prompt manager/catalog in a single HTML file | ![Stars](https://img.shields.io/github/stars/sevenreasons/promptcat?style=flat-square) | MIT |\n| [heretic](https://github.com/p-e-w/heretic) | Fully automatic censorship removal for language models | ![Stars](https://img.shields.io/github/stars/p-e-w/heretic?style=flat-square) | MIT |\n| [llama-swap](https://github.com/mostlygeek/llama-swap) | Reliable model swapping for local OpenAI compatible servers | ![Stars](https://img.shields.io/github/stars/mostlygeek/llama-swap?style=flat-square) | MIT |\n| [outlines](https://github.com/dottxt-ai/outlines) | Structured outputs and generation control for LLMs | ![Stars](https://img.shields.io/github/stars/dottxt-ai/outlines?style=flat-square) | Apache-2.0 |\n\n\n---\n\n## Contributing\n\nContributions welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) before submitting a PR.\n\n**Format for new entries:**\n- Use GitHub Stars badges where applicable\n- Include a concise, accurate description\n- Link to the primary GitHub repo or official site\n\n---\n\n## License\n\n[![CC0](https://mirrors.creativecommons.org/presskit/buttons/88x31/svg/cc-zero.svg)](https://creativecommons.org/publicdomain/zero/1.0)\n\nThis list is released into the public domain under CC0.\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/sam-blackfly%2Fawesome-llm-tools/projects"}