awesome-llmops
An awesome & curated list of best LLMOps tools for developers
https://github.com/tensorchord/awesome-llmops
Last synced: 9 days ago
JSON representation
-
AutoML
-
Profiling
- TPOT - source software packages. |  |
- Archai - square) |
- autoai - square) |
- AutoGL - square) |
- automl-gs - gs.svg?style=flat-square) |
- autokeras - team/autokeras.svg?style=flat-square) |
- Auto-PyTorch - PyTorch.svg?style=flat-square) |
- auto-sklearn - in replacement for a scikit-learn estimator. |  |
- Dragonfly - square) |
- Determined - ai/determined.svg?style=flat-square) |
- DEvol (DeepEvolution) - square) |
- EvalML - square) |
- FEDOT - itmo/FEDOT.svg?style=flat-square) |
- FLAML - us/research/publication/flaml-a-fast-and-lightweight-automl-library/)). |  |
- Goptuna - bata/goptuna.svg?style=flat-square) |
- HpBandSter - square) |
- Hyperband - square) |
- Hypernets - square) |
- Hyperopt - square) |
- hyperunity - box hyperparameter optimisation. |  |
- Intelli - square) |
- Katib - native project for automated machine learning (AutoML). |  |
- Keras Tuner - team/keras-tuner.svg?style=flat-square) |
- learn2learn - learning Framework for Researchers. |  |
- MOE - square) |
- Model Search - square) |
- NASGym - of-concept OpenAI Gym environment for Neural Architecture Search (NAS). |  |
- NNI - parameter tuning. |  |
- Optuna - square) |
- Pycaret - source, low-code machine learning library in Python that automates machine learning workflows. |  |
- REMBO - dimensions via random embedding. |  |
- RoBO - square) |
- scikit-optimize(skopt) - based optimization with a `scipy.optimize` interface. |  |
- Spearmint - square) |
- Torchmeta - Learning library for PyTorch. |  |
- Vegas - noah/vega.svg?style=flat-square) |
- AutoRAG - Boost your LLM app performance with your own data |  |
- AutoGluon - square) |
- FEDOT - itmo/FEDOT.svg?style=flat-square) |
- Ludwig - square) |
-
-
Awesome Lists
-
Profiling
- Awesome Federated Learning Systems - paper.svg?style=flat-square) |
- Awesome AutoDL - depth analysis) |  |
- Awesome AutoML - related research, tools, projects and other resources |  |
- Awesome AutoML Papers - automl-papers.svg?style=flat-square) |
- Awesome-Code-LLM - LLM for research. |  |
- Awesome Federated Learning - organized from Arxiv (mostly) |  |
- awesome-federated-learning - federated-learning.svg?style=flat-square) |
- Awesome Open MLOps - open-mlops.svg?style=flat-square) |
- Awesome Production Machine Learning - production-machine-learning.svg?style=flat-square) |
- Awesome Tensor Compilers - tensor-compilers.svg?style=flat-square) |
- kelvins/awesome-mlops - mlops.svg?style=flat-square) |
- visenger/awesome-mlops - An awesome list of references for MLOps |  |
- currentslab/awesome-vector-search - vector-search.svg?style=flat-square) |
- pleisto/flappy - Ready LLM Agent SDK for Every Developer |  |
-
-
Code AI
-
Vector search
- CodeGeeX - square) |
- CodeGen - source model for program synthesis. Trained on TPU-v4. Competitive with OpenAI Codex. |  |
- CodeT5 - square) |
- Continue - source autopilot for software development—bring the power of ChatGPT to VS Code |  |
- fauxpilot - source alternative to GitHub Copilot server |  |
- tabby - hosted AI coding assistant. An opensource / on-prem alternative to GitHub Copilot. |  |
- promptext - square) |
- AgentsMesh - hostable AI Agent Workforce Platform. Multi-agent orchestration with remote AI workstations (AgentPods), PTY sandbox + git worktree isolation, built-in Kanban, and per-pod MCP server. Supports Claude Code, Codex CLI, Gemini CLI, Aider, OpenCode. |  |
- Bernstein - class MCP server, quality gates, cost tracking with budgets. |  |
- AIDE - source ML engineering agent that uses tree search to explore solution spaces. Automates machine learning experimentation from data analysis to model training. [Paper](https://arxiv.org/abs/2502.13138). |  |
-
-
Data
-
Data/Feature enrichment
- Upgini - to-use features from public and community shared data sources and enriches your training dataset with only the accuracy improving features |  |
- Feast - dev/feast.svg?style=flat-square) |
- distilabel - quality outputs, full data ownership, and overall efficiency. |  |
- FastDatasets - quality training datasets for Large Language Models. |  |
-
Data Management
- ArtiVC - square) |
- Dolt - square) |
- Delta-Lake - io/delta.svg?style=flat-square) |
- Pachyderm - square) |
- Quilt - organizing data hub for S3. |  |
- DVC - Git for Data & Models - ML Experiments Management. |  |
-
Data Storage
-
Data Tracking
-
Feature Engineering
- Featureform - square) |
- FeatureTools - square) |
-
-
Federated ML
-
Profiling
- EasyFL - to-use Federated Learning Platform |  |
- FATE - square) |
- FedML - scale cross-silo federated learning, cross-device federated learning on smartphones/IoTs, and research simulation. |  |
- Flower - square) |
- Harmonia - source project aiming at developing systems/infrastructures and libraries to ease the adoption of federated learning (abbreviated to FL) for researches and production usage. |  |
- TensorFlow Federated - square) |
-
-
Large Scale Deployment
-
ML Platforms
- MLRun - square) |
- Weights & Biases - powered applications, featuring W&B Prompts for LLM execution flow visualization, input and output monitoring, and secure management of prompts and LLM chain configurations. |  |
- Hopsworks - tuning and serving LLMs. Hopsworks includes both a feature store and vector database for RAG. |  |
- OpenLLM - tune, serve, deploy, and monitor any LLMs with ease. |  |
- MLflow - square) |
- ModelFox - square) |
- Kserve - square) |
- Kubeflow - square) |
- Polyaxon - square) |
- Primehub - square) |
- OpenModelZ - click machine learning deployment (LLM, text-to-image and so on) at scale on any cluster (GCP, AWS, Lambda labs, your home lab, or even a single machine). |  |
- Seldon-core - core.svg?style=flat-square) |
- Starwhale - tuning. |  |
- TrueFoundry - tune and serve LLM Models on a company’s own Infrastructure with Data Security and Optimal GPU and Cost Management. Launch your LLM Application at Production scale with best DevSecOps practices. | |
-
Model Management
-
Scheduling
- PAI - sourced by Microsoft). |  |
- Kueue - native Job Queueing. |  |
- Slurm - square) |
- Volcano - sh/volcano.svg?style=flat-square) |
- Yunikorn - weight, universal resource scheduler for container orchestrator systems. |  |
-
Workflow
- Airflow - square) |
- ZenML - io/zenml.svg?style=flat-square) |
- aqueduct - Source Platform for Production Data Science |  |
- Argo Workflows - workflows.svg?style=flat-square) |
- Flyte - native workflow automation platform for complex, mission-critical data and ML processes at scale. |  |
- Kubeflow Pipelines - square) |
- Metaflow - life data science projects with ease! |  |
- Ploomber - square) |
- Prefect - square) |
- simulate-sdk - grade Voice AI simulation SDK for scenario-driven stress testing of multimodal and agentic systems. |  |
- VDP - source unstructured data ETL tool to streamline the end-to-end unstructured data processing pipeline. |  |
- Hamilton - inc/hamilton.svg?style=flat-square) |
- Kitaru - io/kitaru.svg?style=flat-square) |
-
-
LLMOps
-
Observability
- Portkey - efficient apps. | |
- Fiddler AI - production to production. | |
- TrueFoundry - prem) Infra including deploying, Fine-tuning, tracking Prompts and serving Open Source LLM Models with full Data Security and Optimal GPU Management. Train and Launch your LLM Application at Production scale with best Software Engineering practices. | |
- Parea AI - controlled enhanced prompt playground. |  |
- Vellum
- Izlo
- Keywords AI
- Literal AI - modal LLM observability and evaluation platform. Create prompt templates, deploy prompts versions, debug LLM runs, create datasets, run evaluations, monitor LLM metrics and collect human feedback. | |
- Evidently - source framework to evaluate, test and monitor ML and LLM-powered systems. |  |
- agenta - AI/agenta.svg?style=flat-square) |
- AI studio - square) |
- Arize-Phoenix - ai/phoenix.svg?style=flat-square) |
- BudgetML - square) |
- deeplake - square) |
- Dify - source framework aims to enable developers (and even non-developers) to quickly build useful applications based on large language models, ensuring they are visual, operable, and improvable. |  |
- Dstack - effective LLM development in any cloud (AWS, GCP, Azure, Lambda, etc). |  |
- Glide - Native LLM Routing Engine. Improve LLM app resilience and speed. |  |
- GPTCache - square) |
- Haystack - answering and more. |  |
- Langfuse - square) |
- LangKit - of-the-box LLM telemetry collection library that extracts features and profiles prompts, responses and metadata about how your LLM is performing over time to find problems at scale. |  |
- LLMApp - time LLM-enabled data pipelines with few lines of code. |  |
- LLMFlows - answering systems, and agents. |  |
- magentic - powered functionality. |  |
- Mirascope - fast, efficient development and ensuring quality in LLM-based applications |  |
- OpenLIT - native GenAI and LLM Application Observability tool and provides OpenTelmetry Auto-instrumentation for monitoring LLMs, VectorDBs and Frameworks. It provides valuable insights into token & cost usage, user interaction, and performance related metrics. |  |
- Pezzo 🕹️ - source LLMOps platform built for developers and teams. In just two lines of code, you can seamlessly troubleshoot your AI operations, collaborate and manage your prompts in one place, and instantly deploy changes to any environment. |  |
- PromptMage - source tool to simplify the process of creating and managing LLM workflows and prompts as a self-hosted solution. |  |
- prompttools - source tools for testing and experimenting with prompts. The core idea is to enable developers to evaluate prompts using familiar interfaces like code and notebooks. In just a few lines of codes, you can test your prompts and parameters across different models (whether you are using OpenAI, Anthropic, or LLaMA models). You can even evaluate the retrieval accuracy of vector databases. |  |
- xTuring - tuning. |  |
- Helicone - source LLM observability platform for logging, monitoring, and debugging AI applications. Simple 1-line integration to get started. |  |
- PromptFoundry - foundry/python-sdk.svg?style=flat-square) |
- gotoHuman - based and agentic workflows. Prompt users to approve actions, select next steps, or review and validate generated results. |
- GPUStack - source GPU cluster manager for running and managing LLMs |  |
- PromptLayer 🍰 - layer-library.svg?style=flat-square) |
- Opik - ml/opik.svg?style=flat-square) |
- Laminar - source all-in-one platform for engineering AI products. Traces, Evals, Datasets, Labels. |  |
- Dataoorts
- PromptDX - ai/promptdx.svg?style=flat-square) |
- systemprompt.io
- MLflow - source framework for the end-to-end machine learning lifecycle, helping developers track experiments, evaluate models/prompts, deploy models, and add observability with tracing. |  |
- Epsilla - in-one platform to create vertical AI agents powered by your private data and knowledge. | |
- PromptSite - works directly with your local filesystem, ideal for data scientists and engineers to easily integrate into existing LLM workflows | |
- AgentMark - Safe Markdown-based Agents |  |
- Cheshire Cat AI - cat-ai/core.svg?style=flat-square) |
- Lunary - and-play integration into LangChain. |  |
- AI studio - square) |
- LiteLLM 🚅 - square) |
- LlamaIndex - square) |
- langchain - square) |
- Manag.ai - in-one prompt management and observability platform. Craft, track, and perfect your LLM prompts with ease. | |
- TensorZero - source framework for building production-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluations, and experimentation. |  |
- Manag.ai - in-one prompt management and observability platform. Craft, track, and perfect your LLM prompts with ease. | |
- Dataoorts
- Hypersigil - source prompt lifecycle management and gateway with a Web UI. |  |
- Neurolink - provider AI agent framework that unifies 12+ LLM providers (OpenAI, Google, Anthropic, AWS, Azure, Groq, etc.) with workflow orchestration. Production-grade platform for building LLM applications with streaming, tool calling, caching, and enterprise features. Battle-tested at 15M+ requests/month. |  |
- Weights & Biases (Prompts) - first W&B MLOps platform. Utilize W&B Prompts for visualizing and inspecting LLM execution flow, tracking inputs and outputs, viewing intermediate results, securely managing prompts and LLM chain configurations. | |
- Embedchain - square) |
- Roundtable - configuration unified AI assistant management built on the FastMCP framework. Provides seamless integration with Claude, ChatGPT, and other AI assistants through a single MCP interface with session management, logging, and production-ready operations. |  |
- Future AGI - agi/ai-evaluation?style=flat-square) |
- PraisonAI - ready Multi-AI Agents framework with self-reflection. Fastest agent instantiation (3.77μs), 100+ LLM support via LiteLLM, MCP integration, agentic workflows (route/parallel/loop/repeat), built-in memory, Python & JS SDKs. |  |
- Puzzlet AI - Based LLM Engineering Platform. Achieve more from GenAI: Manage, evaluate, and improve your full-stack LLM application - with version control, type-safety, and local development built-in. | |
- LRM - powered translation via Ollama, validation, and code scanning for unused/missing keys. |  |
- TeamoRouter - 4o, Gemini, DeepSeek, Kimi, MiniMax. Smart routing modes (teamo-best, teamo-balanced, teamo-eco) auto-pick the optimal model. Up to 50% off official prices. 2-second install via skill.md. | |
- Semantic Cache Router - semantic-cache-and-stateful-routing-system.svg?style=flat-square) |
- AgentField - source control plane for building and operating AI agents like APIs at scale, with routing, memory, observability, identity, auth, and policy controls. |  |
- Contexto - hosted context engine for AI agents with persistent conversation memory and recall. Works as a drop-in OpenAI-compatible proxy, OpenClaw plugin, or memory SDK — no code changes required. |  |
- Hive - source AI agent framework for building goal-driven, self-improving autonomous agents with auto-generated graphs, evolution loops, and MCP integration. |  |
- Mengram - source memory infrastructure for AI agents. Provides semantic (entities/facts), episodic (conversations), and procedural (learned behaviors) memory with auto-reflection. Python SDK, JS SDK, MCP server, and REST API. |  |
- TeamoRouter - 4o, Gemini, DeepSeek, Kimi, MiniMax. Smart routing modes (teamo-best, teamo-balanced, teamo-eco) auto-pick the optimal model. Up to 50% off official prices. 2-second install via skill.md. | |
- Rhesis - source testing infrastructure for LLM and agentic applications. Collaborative platform enabling teams to define quality metrics, run evaluations, and ship confidently with version control and peer review workflows built for AI engineering. |  |
- Statewave - source memory runtime for AI agents. Compiles events into deterministic, provenance-tagged context bundles instead of query-time retrieval. Apache-2.0, self-hostable on Postgres + pgvector. |  |
- SwarmClaw - hosted multi-agent AI runtime with 23+ LLM providers, persistent memory, skills, schedules, sub-agent spawning, and MCP client + server support. Ships as desktop app, CLI, or Docker. |  |
- future-agi - source self-hostable end-to-end agent engineering and optimization platform unifying tracing, evals, simulations, datasets, gateway, and guardrails for LLM and AI agent applications. |  |
- PromptDX - ai/promptdx.svg?style=flat-square) |
- Registry Broker - online/registry-broker.svg?style=flat-square) |
-
-
ML Platforms
- TrueFoundry - A PaaS to deploy, Fine-tune and serve LLM Models on a company’s own Infrastructure with Data Security and Optimal GPU and Cost Management. Launch your LLM Application at Production scale with best DevSecOps practices.
Programming Languages
Categories
Sub Categories
Observability
91
Profiling
73
Vector search
32
Large Model Serving
23
Frameworks for Training
22
Frameworks/Servers for Serving
17
Large Language Model
14
ML Platforms
14
Workflow
13
Visualization
10
Foundation Model Fine Tuning
9
IDEs and Workspaces
6
Experiment Tracking
6
Data Management
6
CV Foundation Model
5
Scheduling
5
Robotics Foundation Model
5
Model Management
4
Frameworks for LLM security
4
Data/Feature enrichment
4
ML Compiler
3
Data Storage
3
Data Tracking
2
Feature Engineering
2
Audio Foundation Model
2
Hybrid search
1
Model Editing
1
Keywords
machine-learning
103
llm
62
python
60
deep-learning
54
ai
46
mlops
45
pytorch
39
data-science
39
llmops
29
tensorflow
25
kubernetes
25
automl
24
ml
23
inference
18
openai
18
large-language-models
17
hyperparameter-optimization
17
llms
17
rag
16
prompt-engineering
15
developer-tools
15
langchain
15
chatgpt
14
keras
14
vector-database
14
gpu
13
observability
12
gpt
12
vector-search
12
neural-architecture-search
12
evaluation
12
docker
11
neural-network
11
llama
10
golang
10
artificial-intelligence
10
agents
10
generative-ai
10
chatbot
9
analytics
9
ai-agents
9
hyperparameter-tuning
9
scikit-learn
9
open-source
9
go
8
transformers
8
workflow
8
nearest-neighbor-search
8
model-serving
8
language-model
8