Management

• NVIDIA GPU Operator - operator.svg?style=flat&color=green)  

Scheduling

• KAI Scheduler - Scheduler.svg?style=flat&color=green)  
• Project-HAMi - HAMi/HAMi.svg?style=flat&color=green)  

AI Gateway

• APISIX - Native API Gateway and AI Gateway with extensive plugin system and AI capabilities.   
• Envoy AI Gateway - gateway.svg?style=flat&color=green)  
• Higress - commit/alibaba/higress?color=green)
• kgateway - Native API Gateway and AI Gateway.   
• Kong - Native API Gateway and AI Gateway.   
• gateway-api-inference-extension - sigs/gateway-api-inference-extension.svg?style=flat&color=green)  
• agentgateway - commit/agentgateway/agentgateway?color=green)
• LLM Semantic Router - of-Models (MoM) System with Privacy Preservation and Prompt Guard. The semantic router intelligently directs OpenAI compliant API requests to the most suitable backend models based on semantic understanding of request content.   

Benchmark

• Inference Perf - sigs/inference-perf.svg?style=flat&color=green)  
• Inference Benchmark - Hypercomputer/inference-benchmark.svg?style=flat&color=green)  
• genai-bench - bench is a powerful benchmark tool designed for comprehensive token-level performance evaluation of large language model (LLM) serving systems.   

Inference Engine

• LMDeploy - commit/internlm/lmdeploy?color=green)
• MInference - context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.    
• MLC LLM - ai/mlc-llm.svg?style=flat&color=green)  
• MLServer - model serving and more.   
• Ollama - R1, Phi-4, Gemma 3, and other large language models.   
• OpenLLM - source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.   
• OpenVINO - commit/openvinotoolkit/openvino?color=green)
• Ratchet - platform browser ML framework.    
• SGLang - project/sglang.svg?style=flat&color=green)  
• Triton Inference Server - inference-server/server.svg?style=flat&color=green)  
• Text Generation Inference - generation-inference.svg?style=flat&color=green)  
• vLLM - throughput and memory-efficient inference and serving engine for LLMs.   
• web-llm - performance In-browser LLM Inference Engine.    
• zml - commit/zml/zml?color=green)
• LoRAX - LoRA inference server that scales to 1000s of fine-tuned LLMs.    
• Cortex.cpp - commit/janhq/cortex.cpp?color=green)
• Nvidia Dynamo - dynamo/dynamo.svg?style=flat&color=green)  
• Llumnix - instance LLM serving.   
• transformers.js - of-the-art Machine Learning for the web. Run 🤗 Transformers directly in your browser, with no need for a server!    
• DeepSpeed-MII - latency and high-throughput inference possible, powered by DeepSpeed.   
• TinyGrad - commit/tinygrad/tinygrad?color=green)
• Xinference - source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.   
• llama-box - box.svg?style=flat&color=green)  
• Petals - style. Fine-tuning and inference up to 10x faster than offloading   
• llama.cpp - commit/ggerganov/llama.cpp?color=green)
• ipex-llm - VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.   

Inference Platform

• BentoML - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!   
• llmaz - commit/inftyai/llmaz?color=green)
• Kserve - commit/kserve/kserve?color=green)
• AIBrix - efficient and pluggable Infrastructure components for GenAI inference.   
• KubeAI - to-text.   
• Mooncake - ai/mooncake.svg?style=flat&color=green)  
• Kaito - model inference and fine-tuning, with GPU auto-provisioning, container-based hosting, and CRD-based orchestration.   
• llm-d - d is a Kubernetes-native high-performance distributed LLM inference framework   
• OME - grade management and serving of Large Language Models (LLMs)   
• Modular - commit/modular/modular?color=green)
• beta9 - cloud/beta9.svg?style=flat&color=green)  

LLM Router

• LiteLLM - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq].   
• RouteLLM - save LLM costs without compromising quality.   
• AI Gateway - ai/gateway.svg?style=flat&color=green)  
• vLLM Semantic Router - of-Models Router for Efficient LLM Inference   
• bifrost - commit/maximhq/bifrost?color=green)
• LLMRouter - Source Library for LLM Routing   
• Ferro Labs AI Gateway - native LiteLLM & Kong AI Gateway alternative.   
• Hebo AI Gateway - compatible /chat/completions, /embeddings & /models endpoints   
• Doubleword Control Layer - hosted, etc.) behind a single authentication layer - with API key generation, user management, request logging, and more   
• Otari - source, OpenAI-compatible LLM gateway you run yourself. One endpoint for 40+ providers, with virtual keys, budgets, and usage tracking.   

Middleware

• LMCache - Context LLM By Smart KV Cache Optimizations.    
• Checkpoint Engine - engine is a simple middleware to update model weights in LLM inference engines   
• kvcached - project/kvcached.svg?style=flat&color=green)  

Output

• Instructor - ai/instructor.svg?style=flat&color=green)  
• Outlines - ai/outlines.svg?style=flat&color=green)  
• XGrammar - ai/xgrammar.svg?style=flat&color=green)  
• BAML - commit/boundaryml/baml?color=green)

Simulator

• Vidur - scale simulation framework for LLM inference   

• Nanoflow - oriented high-performance serving framework for LLMs |
• TensorRT-LLM - LLM) |  |  | TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.||

MCP Client

• awesome-mcp-clients - mcp-clients.svg?style=flat&color=green)  

MCP Server

• awesome-mcp-servers - mcp-servers.svg?style=flat&color=green)  
• mcp-directory - directory.svg?style=flat&color=green)  
• BaiLian MCP
• Cline MCP Marketplace
• Docker MCP Catalog - quality MCP servers as Docker Images, spanning database solutions, developer tools, productivity platforms, and API integrations.
• Higress MCP Marketplace
• MCPMarket
• ModelScope MCP
• MCPMarket

Agent

• OpenManus - commit/mannaandpoem/openmanus?color=green)
• Claudia - Create custom agents, manage interactive Claude Code sessions, run secure background agents, and more.   

Agent Framework

• Flowise - commit/flowiseai/flowise?color=green)
• AutoGPT - gravitas/autogpt.svg?style=flat&color=green)  
• MetaGPT - Agent Framework: First AI Software Company, Towards Natural Language Programming.   
• PydanticAI - ai.svg?style=flat&color=green)  
• Swarm - agent orchestration. Managed by OpenAI Solution team.    
• Agno - source framework for building multi-agent systems with memory, knowledge and reasoning.   
• LangGraph - ai/langgraph.svg?style=flat&color=green)  
• OpenAI Agents SDK - agent workflows.   
• Semantic Kernel - edge LLM technology quickly and easily into your apps.   
• CAMEL - agent framework. Finding the Scaling Law of Agents.   
• kagent - dev/kagent.svg?style=flat&color=green)   
• Agent Development Kit (ADK) - source, code-first Python toolkit for building, evaluating, and deploying sophisticated AI agents with flexibility and control.   
• Suna - Open Source Generalist AI Agent   
• Qwen-Agent - Agent.svg?style=flat&color=green)  
• autogen - commit/microsoft/autogen?color=green)
• fast-agent - agent.svg?style=flat&color=green)  
• crewAI - playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.   
• AgentField - scalable, observable, and identity-aware from day one.   
• Agent Shadow Brain - evolving AI coding intelligence with infinite memory (TurboQuant), genetic algorithm evolution, predictive bug detection, PageRank knowledge graphs, and swarm intelligence. The world's first autonomous coding brain.   

Application Framework

• phoenix - ai/phoenix.svg?style=flat&color=green)  
• Helicone - commit/helicone/helicone?color=green)
• Langfuse - commit/langfuse/langfuse?color=green)
• OpenLIT - native LLM Observability, GPU Monitoring, Guardrails, Evaluations, Prompt Management, Vault, Playground. 🚀💻 Integrates with 50+ LLM Providers, VectorDBs, Agent Frameworks and GPUs.   
• ragas - commit/explodinggradients/ragas?color=green)
• Evidently - source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.   
• lunaary - ai/lunary.svg?style=flat&color=green)  
• PostHog - source web & product analytics, session recording, feature flagging and A/B testing that you can self-host. Get started - free.   
• Weave - powered applications, built by Weights & Biases.   
• DeepEval - ai/deepeval.svg?style=flat&color=green)  
• Neurolink - provider support, and professional CLI. Build, test, and deploy AI applications with multiple ai providers.   
• Cordum.io - only control plane for autonomous AI Agents and external workers. It uses NATS for the bus, Redis for state and payload pointers, and CAP v2 wire contracts for jobs, results, and heartbeats. Workers and product packs live outside this repo.Core cordum   
• AgentMark - source, Git-native platform for building and observing reliable AI agents. Prompts and datasets in your repo, evals in CI, and OpenTelemetry tracing.   
• Future AGI - source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.   

Evolutionary Framework

• AI-Researcher - Researcher: Autonomous Scientific Innovation" -- A production-ready version: https://novix.science/chat   
• AIDE ML - Driven Exploration in the Space of Code. The machine Learning engineering agent that automates AI R&D.   
• OpenEvolve - source implementation of AlphaEvolve   
• ShinkaEvolve - Ended and Sample-Efficient Program Evolution   
• LoongFlow - Grade AI Agents.   
• SkyDiscover - Driven Scientific and Algorithmic Discovery   

RAG

• LightRAG - Augmented Generation"   
• GraphRAG - based Retrieval-Augmented Generation (RAG) system.   
• RAGFlow - source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.   
• quivr - commit/QuivrHQ/quivr?color=green)
• RAG-Anything - Anything: All-in-One RAG Framework"   

Workflow

• Dify - ready platform for agentic workflow development.   
• Haystack - ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.   
• LangChain - aware reasoning applications.   
• LlamaIndex - powered agents over your data.   
• Inference - commit/roboflow/inference?color=green) 
• FastGPT - based platform built on the LLMs, offers a comprehensive suite of out-of-the-box capabilities such as data processing, RAG retrieval, and visual AI workflow orchestration, letting you easily develop and deploy complex question-answering systems without the need for extensive setup or configuration.   
• Heym - available, self-hosted visual platform for building, running, evaluating, and observing AI workflows with agents, RAG, and MCP.   

AI Agent

• Magentic-UI - centered web agent   
• goose - install, execute, edit, and test with any LLM   
• OpenManus - commit/FoundationAgents/openmanus?color=green)
• Tongyi Deep Research - source DeepResearch Agent   

AI Terminal

• Codex - commit/openai/codex?color=green) 
• kubectl-ai - ai.svg?style=flat&color=green)  
• Gemini CLI - source AI agent that brings the power of Gemini directly into your terminal.   
• aider - AI/aider.svg?style=flat&color=green)  
• Crush - commit/charmbracelet/crush?color=green)
• Stakpak - commit/stakpak/agent?color=green)
• OpenCode - commit/sst/opencode?color=green)

Chatbot

• FastChat - sys/fastchat.svg?style=flat&color=green)  
• Open WebUI - friendly AI Interface (Supports Ollama, OpenAI API, ...).   
• PrivateGPT - ai/private-gpt.svg?style=flat&color=green)  
• Jan - commit/janhq/jan?color=green)
• Lobe Chat - an open-source, modern-design AI chat framework. Supports Multi AI Providers( OpenAI / Claude 3 / Gemini / Ollama / DeepSeek / Qwen), Knowledge Base (file upload / knowledge management / RAG ), Multi-Modals (Plugins/Artifacts) and Thinking. One-click FREE deployment of your private ChatGPT/ Claude / DeepSeek application.   
• NextChat - commit/chatgptnextweb/nextchat?color=green)
• 5ire - platform desktop AI assistant, MCP client. It compatible with major service providers, supports local knowledge base and tools via model context protocol servers.   
• Chatbot UI - ui.svg?style=flat&color=green)  
• Cherry Studio - r1.   
• Gradio - app/gradio.svg?style=flat&color=green)  
• Chat SDK - featured, hackable Next.js AI chatbot built by Vercel   
• AnythingLLM - in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more.   
• LLM - line   
• opcode - Create custom agents, manage interactive Claude Code sessions, run secure background agents, and more.   

Code Agent

• Continue - source IDE extensions and hub of models, rules, prompts, docs, and other building blocks.   
• Tabby - hosted AI coding assistant.   
• SWE-agent - agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]   
• Open SWE - Source Asynchronous Coding Agent   

Code Assistant

• Auto-dev - powered coding wizard（AI 驱动编程助手）with multilingual support 🌐, auto code generation 🏗️, and a helpful bug-slaying assistant 🐞! Customizable prompts 🎨 and a magic Auto Dev/Testing/Document/Agent feature 🧪 included! 🚀.   
• Codefuse-chatbot - Agent Framework, working with DevOps Toolkits, Code&Doc Repo RAG, etc.   
• Cody - commit/sourcegraph/cody?color=green)
• Sweep - commit/sweepai/sweep?color=green)

Database

• chroma - native open-source embedding database.   
• deeplake - time to PyTorch/TensorFlow.   
• Faiss - commit/facebookresearch/faiss?color=green)
• milvus - performance, cloud-native vector database built for scalable vector ANN search.   
• weaviate - source vector database that stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native database​.   
• Hindsight - io/hindsight.svg?style=flat&color=green)  

Evolve Agent

• AgentEvolver - Evolving Agent System   
• EvoAgentX - Evolving Ecosystem of AI Agents   

Observation

• OpenLLMetry - source observability for your LLM application, based on OpenTelemetry.   
• wandb - tune models, and manage models from experimentation to production.   
• Latitude - source ai monitoring platform.   

Sandbox

• Daytona - Generated Code.   
• E2B - dev/E2B.svg?style=flat&color=green)  

Tool

• Mem0 - commit/mem0ai/mem0?color=green)
• Browser Use - use/browser-use.svg?style=flat&color=green)  
• OpenAI CUA - cua-sample-app.svg?style=flat&color=green)  
• Graphiti - Time Knowledge Graphs for AI Agents.   
• Beads - A memory upgrade for your coding agent   
• WritBase - native task management for AI agent fleets   
• SidClaw - commit/sidclawhq/platform?color=green)

Agentic RL

• AReaL - Fast RL for LLM Reasoning and Agents. Made Simple & Flexible.   
• rLLM - org/rllm.svg?style=flat&color=green)  
• slime - training framework for RL Scaling.   
• verl - commit/volcengine/verl?color=green)

Benchmark

• AgentBench - commit/thudm/agentbench?color=green)
• lm-evaluation-harness - shot evaluation of language models.   
• LongBench - commit/thudm/longbench?color=green)
• OpenCompass - 4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.   
• LiveBench - Free LLM Benchmark   
• opik - ready dashboards.   
• terminal-bench - institute/terminal-bench.svg?style=flat&color=green)