Management

• NVIDIA GPU Operator - operator.svg?style=flat&color=green)  

Scheduling

• KAI Scheduler - Scheduler.svg?style=flat&color=green)  
• Project-HAMi - HAMi/HAMi.svg?style=flat&color=green)  

AI Gateway

• APISIX - Native API Gateway and AI Gateway with extensive plugin system and AI capabilities.   
• Envoy AI Gateway - gateway.svg?style=flat&color=green)  
• kgateway - Native API Gateway and AI Gateway.   
• Kong - Native API Gateway and AI Gateway.   
• gateway-api-inference-extension - sigs/gateway-api-inference-extension.svg?style=flat&color=green)  
• agentgateway - commit/agentgateway/agentgateway?color=green)
• LLM Semantic Router - of-Models (MoM) System with Privacy Preservation and Prompt Guard. The semantic router intelligently directs OpenAI compliant API requests to the most suitable backend models based on semantic understanding of request content.   
• TrustGate - hosted Go Agent Gateway (Apache-2.0) — OpenAI-compatible LLM proxy + MCP aggregation with per-consumer auth/audit. 

Benchmark

• Inference Perf - sigs/inference-perf.svg?style=flat&color=green)  
• Inference Benchmark - Hypercomputer/inference-benchmark.svg?style=flat&color=green)  
• genai-bench - bench is a powerful benchmark tool designed for comprehensive token-level performance evaluation of large language model (LLM) serving systems.   

Inference Engine

• LMDeploy - commit/internlm/lmdeploy?color=green)
• MInference - context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.    
• MLC LLM - ai/mlc-llm.svg?style=flat&color=green)  
• MLServer - model serving and more.   
• Ollama - R1, Phi-4, Gemma 3, and other large language models.   
• OpenLLM - source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.   
• OpenVINO - commit/openvinotoolkit/openvino?color=green)
• Ratchet - platform browser ML framework.    
• SGLang - project/sglang.svg?style=flat&color=green)  
• Triton Inference Server - inference-server/server.svg?style=flat&color=green)  
• Text Generation Inference - generation-inference.svg?style=flat&color=green)  
• vLLM - throughput and memory-efficient inference and serving engine for LLMs.   
• web-llm - performance In-browser LLM Inference Engine.    
• zml - commit/zml/zml?color=green)
• LoRAX - LoRA inference server that scales to 1000s of fine-tuned LLMs.    
• Cortex.cpp - commit/janhq/cortex.cpp?color=green)
• Nvidia Dynamo - dynamo/dynamo.svg?style=flat&color=green)  
• Llumnix - instance LLM serving.   
• transformers.js - of-the-art Machine Learning for the web. Run 🤗 Transformers directly in your browser, with no need for a server!    
• DeepSpeed-MII - latency and high-throughput inference possible, powered by DeepSpeed.   
• TinyGrad - commit/tinygrad/tinygrad?color=green)
• Xinference - source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.   
• llama-box - box.svg?style=flat&color=green)  
• Petals - style. Fine-tuning and inference up to 10x faster than offloading   
• llama.cpp - commit/ggerganov/llama.cpp?color=green)
• ipex-llm - VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.   

Inference Platform

• BentoML - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!   
• llmaz - commit/inftyai/llmaz?color=green)
• Kserve - commit/kserve/kserve?color=green)
• AIBrix - efficient and pluggable Infrastructure components for GenAI inference.   
• KubeAI - to-text.   
• Mooncake - ai/mooncake.svg?style=flat&color=green)  
• Kaito - model inference and fine-tuning, with GPU auto-provisioning, container-based hosting, and CRD-based orchestration.   
• llm-d - d is a Kubernetes-native high-performance distributed LLM inference framework   
• OME - grade management and serving of Large Language Models (LLMs)   
• Modular - commit/modular/modular?color=green)
• beta9 - cloud/beta9.svg?style=flat&color=green)  
• Paralleliq - aware GPU optimization layer for AI inference clusters. Understands which model runs on which GPU to identify waste and risk — tier misplacement, dark capacity, OOM risk, and CPU:GPU imbalance — with human-in-the-loop approval workflows and a full audit trail.

LLM Router

• LiteLLM - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq].   
• RouteLLM - save LLM costs without compromising quality.   
• AI Gateway - ai/gateway.svg?style=flat&color=green)  
• vLLM Semantic Router - of-Models Router for Efficient LLM Inference   
• bifrost - commit/maximhq/bifrost?color=green)
• LLMRouter - Source Library for LLM Routing   
• Ferro Labs AI Gateway - native LiteLLM & Kong AI Gateway alternative.   
• Hebo AI Gateway - compatible /chat/completions, /embeddings & /models endpoints   
• Doubleword Control Layer - hosted, etc.) behind a single authentication layer - with API key generation, user management, request logging, and more   
• Otari - source, OpenAI-compatible LLM gateway you run yourself. One endpoint for 40+ providers, with virtual keys, budgets, and usage tracking.   
• llm-cache-router - first Python library combining semantic caching, multi-provider routing (OpenAI, Anthropic, Gemini, Ollama, MiniMax, Qwen) and cost tracking. Cuts LLM spend 30-70% via vector-similarity cache with in-memory/Redis/Qdrant backends.   
• llm-swarm-router - hosted mesh router for local LLM backends — OpenAI `/v1` + Anthropic Messages on `:11400`, mDNS LAN peers. Complements cloud gateways with **local LAN mesh** for Cursor, Claude Code, Codex, and Honcho.   
• Ollama Herd - to-text and embedding requests across a fleet of machines you own. Auto-discovers nodes over mDNS and scores each on memory fit, thermal state, queue depth and already-loaded models. OpenAI-, Ollama- and Anthropic-compatible.   
• OpenPaths - source, OpenAI-compatible model router/AI gateway routing chat, image, video, music, speech, transcription, embeddings and reasoning models to the lowest-latency provider via a unified API.   
• Swobu - commit/swobuforge/swobu?color=green)

Middleware

• LMCache - Context LLM By Smart KV Cache Optimizations.    
• Checkpoint Engine - engine is a simple middleware to update model weights in LLM inference engines   
• kvcached - project/kvcached.svg?style=flat&color=green)  
• KV Cache Store - cache artifact registry plus an open-source Rust CLI. Precompute, verify, quantize, and share attention states across RAG and long-context prompts to cut prefill cost and latency.  ([CLI source](https://github.com/kvcachestore/kvcdn))

Output

• Outlines - ai/outlines.svg?style=flat&color=green)  
• XGrammar - ai/xgrammar.svg?style=flat&color=green)  
• BAML - commit/boundaryml/baml?color=green)

Simulator

• Vidur - scale simulation framework for LLM inference   

• Nanoflow - oriented high-performance serving framework for LLMs |
• TensorRT-LLM - LLM) |  |  | TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.||

MCP Client

• awesome-mcp-clients - mcp-clients.svg?style=flat&color=green)  

MCP Server

• awesome-mcp-servers - mcp-servers.svg?style=flat&color=green)  
• mcp-directory - directory.svg?style=flat&color=green)  
• BaiLian MCP
• Cline MCP Marketplace
• Docker MCP Catalog - quality MCP servers as Docker Images, spanning database solutions, developer tools, productivity platforms, and API integrations.
• Higress MCP Marketplace
• MCPMarket
• ModelScope MCP
• MCPMarket

Agent

• OpenManus - commit/mannaandpoem/openmanus?color=green)
• Claudia - Create custom agents, manage interactive Claude Code sessions, run secure background agents, and more.   

Agent Framework

• Flowise - commit/flowiseai/flowise?color=green)
• AutoGPT - gravitas/autogpt.svg?style=flat&color=green)  
• MetaGPT - Agent Framework: First AI Software Company, Towards Natural Language Programming.   
• PydanticAI - ai.svg?style=flat&color=green)  
• Swarm - agent orchestration. Managed by OpenAI Solution team.    
• Agno - source framework for building multi-agent systems with memory, knowledge and reasoning.   
• LangGraph - ai/langgraph.svg?style=flat&color=green)  
• OpenAI Agents SDK - agent workflows.   
• Semantic Kernel - edge LLM technology quickly and easily into your apps.   
• CAMEL - agent framework. Finding the Scaling Law of Agents.   
• kagent - dev/kagent.svg?style=flat&color=green)   
• Agent Development Kit (ADK) - source, code-first Python toolkit for building, evaluating, and deploying sophisticated AI agents with flexibility and control.   
• Suna - Open Source Generalist AI Agent   
• Qwen-Agent - Agent.svg?style=flat&color=green)  
• autogen - commit/microsoft/autogen?color=green)
• fast-agent - agent.svg?style=flat&color=green)  
• crewAI - playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.   
• AgentField - scalable, observable, and identity-aware from day one.   
• Agent Shadow Brain - evolving AI coding intelligence with infinite memory (TurboQuant), genetic algorithm evolution, predictive bug detection, PageRank knowledge graphs, and swarm intelligence. The world's first autonomous coding brain.   

Application Framework

• phoenix - ai/phoenix.svg?style=flat&color=green)  
• Helicone - commit/helicone/helicone?color=green)
• Langfuse - commit/langfuse/langfuse?color=green)
• OpenLIT - native LLM Observability, GPU Monitoring, Guardrails, Evaluations, Prompt Management, Vault, Playground. 🚀💻 Integrates with 50+ LLM Providers, VectorDBs, Agent Frameworks and GPUs.   
• ragas - commit/explodinggradients/ragas?color=green)
• Evidently - source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.   
• lunaary - ai/lunary.svg?style=flat&color=green)  
• PostHog - source web & product analytics, session recording, feature flagging and A/B testing that you can self-host. Get started - free.   
• Weave - powered applications, built by Weights & Biases.   
• DeepEval - ai/deepeval.svg?style=flat&color=green)  
• Neurolink - provider support, and professional CLI. Build, test, and deploy AI applications with multiple ai providers.   
• Cordum.io - only control plane for autonomous AI Agents and external workers. It uses NATS for the bus, Redis for state and payload pointers, and CAP v2 wire contracts for jobs, results, and heartbeats. Workers and product packs live outside this repo.Core cordum   
• AgentMark - source, Git-native platform for building and observing reliable AI agents. Prompts and datasets in your repo, evals in CI, and OpenTelemetry tracing.   
• Future AGI - source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.   

Evolutionary Framework

• AI-Researcher - Researcher: Autonomous Scientific Innovation" -- A production-ready version: https://novix.science/chat   
• AIDE ML - Driven Exploration in the Space of Code. The machine Learning engineering agent that automates AI R&D.   
• OpenEvolve - source implementation of AlphaEvolve   
• ShinkaEvolve - Ended and Sample-Efficient Program Evolution   
• LoongFlow - Grade AI Agents.   
• SkyDiscover - Driven Scientific and Algorithmic Discovery   
• Kapso - improving AI software factory (for measurable objectives). \#1 open-source on MLE-Bench; ALE-Bench; RelBench.   

RAG

• LightRAG - Augmented Generation"   
• GraphRAG - based Retrieval-Augmented Generation (RAG) system.   
• RAGFlow - source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.   
• quivr - commit/QuivrHQ/quivr?color=green)
• RAG-Anything - Anything: All-in-One RAG Framework"   
• EmbedGuard - layer detection and provenance attestation for adversarial embedding attacks in RAG systems; published in IJCESEN 2026 (DOI 10.22399/ijcesen.4869).   

Workflow

• Dify - ready platform for agentic workflow development.   
• Haystack - ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.   
• LangChain - aware reasoning applications.   
• LlamaIndex - powered agents over your data.   
• Inference - commit/roboflow/inference?color=green) 
• FastGPT - based platform built on the LLMs, offers a comprehensive suite of out-of-the-box capabilities such as data processing, RAG retrieval, and visual AI workflow orchestration, letting you easily develop and deploy complex question-answering systems without the need for extensive setup or configuration.   
• Heym - available, self-hosted visual platform for building, running, evaluating, and observing AI workflows with agents, RAG, and MCP.   
• Nika - as-code AI workflow engine in a single Rust binary — reviewable YAML DAGs statically checked (schema, permits, honest cost floor) before any token is spent, tamper-evident traces after.   

AI Agent

• Magentic-UI - centered web agent   
• OpenManus - commit/FoundationAgents/openmanus?color=green)
• Tongyi Deep Research - source DeepResearch Agent   
• Agent QA - source self-improving QA agent for software teams. A test harness with memory. Write tests in natural language for web and mobile. agent-qa learns from every run, adapts to UI changes, and catches regressions before you ship.   

AI Terminal

• Codex - commit/openai/codex?color=green) 
• kubectl-ai - ai.svg?style=flat&color=green)  
• Gemini CLI - source AI agent that brings the power of Gemini directly into your terminal.   
• aider - AI/aider.svg?style=flat&color=green)  
• Crush - commit/charmbracelet/crush?color=green)
• Stakpak - commit/stakpak/agent?color=green)
• OpenCode - commit/sst/opencode?color=green)
• Atomic Agent - first CLI and TUI coding agent that runs open-weight models entirely on your machine through a llama.cpp fork, with no account or API key required. 56 built-in tools, MCP support, and a five-layer local memory system. Currently a developer preview.    

Chatbot

• FastChat - sys/fastchat.svg?style=flat&color=green)  
• Open WebUI - friendly AI Interface (Supports Ollama, OpenAI API, ...).   
• PrivateGPT - ai/private-gpt.svg?style=flat&color=green)  
• Jan - commit/janhq/jan?color=green)
• Lobe Chat - an open-source, modern-design AI chat framework. Supports Multi AI Providers( OpenAI / Claude 3 / Gemini / Ollama / DeepSeek / Qwen), Knowledge Base (file upload / knowledge management / RAG ), Multi-Modals (Plugins/Artifacts) and Thinking. One-click FREE deployment of your private ChatGPT/ Claude / DeepSeek application.   
• NextChat - commit/chatgptnextweb/nextchat?color=green)
• 5ire - platform desktop AI assistant, MCP client. It compatible with major service providers, supports local knowledge base and tools via model context protocol servers.   
• Chatbot UI - ui.svg?style=flat&color=green)  
• Cherry Studio - r1.   
• Gradio - app/gradio.svg?style=flat&color=green)  
• Chat SDK - featured, hackable Next.js AI chatbot built by Vercel   
• AnythingLLM - in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more.   
• LLM - line   
• opcode - Create custom agents, manage interactive Claude Code sessions, run secure background agents, and more.   

Code Agent

• Continue - source IDE extensions and hub of models, rules, prompts, docs, and other building blocks.   
• Tabby - hosted AI coding assistant.   
• SWE-agent - agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]   
• Open SWE - Source Asynchronous Coding Agent   
• Kolega Code - agent workflows (Gigacode); provider-agnostic with MCP support and journaled resume.   

Code Assistant

• Auto-dev - powered coding wizard（AI 驱动编程助手）with multilingual support 🌐, auto code generation 🏗️, and a helpful bug-slaying assistant 🐞! Customizable prompts 🎨 and a magic Auto Dev/Testing/Document/Agent feature 🧪 included! 🚀.   
• Codefuse-chatbot - Agent Framework, working with DevOps Toolkits, Code&Doc Repo RAG, etc.   
• Cody - commit/sourcegraph/cody?color=green)
• Sweep - commit/sweepai/sweep?color=green)

Database

• chroma - native open-source embedding database.   
• deeplake - time to PyTorch/TensorFlow.   
• Faiss - commit/facebookresearch/faiss?color=green)
• milvus - performance, cloud-native vector database built for scalable vector ANN search.   
• weaviate - source vector database that stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native database​.   
• Hindsight - io/hindsight.svg?style=flat&color=green)  

Evolve Agent

• AgentEvolver - Evolving Agent System   
• EvoAgentX - Evolving Ecosystem of AI Agents   

Observation

• OpenLLMetry - source observability for your LLM application, based on OpenTelemetry.   
• wandb - tune models, and manage models from experimentation to production.   
• Latitude - source ai monitoring platform.   
• ClawMetry - hosted observability for coding agents. Reads the session logs runtimes already write on disk, so there is no SDK and nothing in the request path.   
• Gait - gated high-risk tool calls for agent workflows.   
• LLMeter - source LLM cost monitoring with per-customer cost attribution. Polls provider billing APIs (OpenAI, Anthropic, DeepSeek, OpenRouter) directly — no proxy or code changes — with budget alerts before surprise bills.   
• Observatory - source TypeScript and Python SDKs for instrumenting AI agents and analyzing production runs, traces, sessions, feedback, and recurring patterns.   

Sandbox

• Daytona - Generated Code.   
• E2B - dev/E2B.svg?style=flat&color=green)  

Tool

• Mem0 - commit/mem0ai/mem0?color=green)
• Browser Use - use/browser-use.svg?style=flat&color=green)  
• OpenAI CUA - cua-sample-app.svg?style=flat&color=green)