https://github.com/pmady/llmops
🚀 The Ultimate Curated List of LLMOps Tools, Frameworks, and Resources - A comprehensive collection of the best tools for Large Language Model Operations
https://github.com/pmady/llmops
List: llmops
ai-tools artificial-intelligence awesome-list deep-learning generative-ai huggingface inference langchain large-language-models llm llmops machine-learning ml-tools mlops model-serving model-training openai prompt-engineering rag vector-database
Last synced: 5 months ago
JSON representation
🚀 The Ultimate Curated List of LLMOps Tools, Frameworks, and Resources - A comprehensive collection of the best tools for Large Language Model Operations
- Host: GitHub
- URL: https://github.com/pmady/llmops
- Owner: pmady
- License: cc0-1.0
- Created: 2026-01-11T03:41:01.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2026-01-11T20:06:13.000Z (6 months ago)
- Last Synced: 2026-01-11T23:52:45.468Z (6 months ago)
- Topics: ai-tools, artificial-intelligence, awesome-list, deep-learning, generative-ai, huggingface, inference, langchain, large-language-models, llm, llmops, machine-learning, ml-tools, mlops, model-serving, model-training, openai, prompt-engineering, rag, vector-database
- Language: Shell
- Size: 33.2 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
- ultimate-awesome - llmops - 🚀 The Ultimate Curated List of LLMOps Tools, Frameworks, and Resources - A comprehensive collection of the best tools for Large Language Model Operations. (Other Lists / TeX Lists)
README
# Awesome LLMOps [](https://awesome.re)
🚀 The Ultimate Curated List of LLMOps Tools, Frameworks, and Resources
A comprehensive collection of the best tools, frameworks, models, and resources for Large Language Model Operations (LLMOps)
---
## 📋 Table of Contents
- [What's New](#whats-new)
- [What is LLMOps?](#what-is-llmops)
- [LLMOps vs MLOps](#llmops-vs-mlops)
- [Models](#models)
- [Large Language Models](#large-language-models)
- [Multimodal Models](#multimodal-models)
- [Audio Foundation Models](#audio-foundation-models)
- [Inference & Serving](#inference--serving)
- [Inference Engines](#inference-engines)
- [Inference Platforms](#inference-platforms)
- [Model Serving Frameworks](#model-serving-frameworks)
- [Orchestration](#orchestration)
- [Application Frameworks](#application-frameworks)
- [Agent Frameworks](#agent-frameworks)
- [Workflow Management](#workflow-management)
- [Training & Fine-Tuning](#training--fine-tuning)
- [Training Frameworks](#training-frameworks)
- [Fine-Tuning Tools](#fine-tuning-tools)
- [Experiment Tracking](#experiment-tracking)
- [Prompt Engineering](#prompt-engineering)
- [Vector Search & RAG](#vector-search--rag)
- [Observability & Monitoring](#observability--monitoring)
- [Security & Safety](#security--safety)
- [Data Management](#data-management)
- [Optimization & Performance](#optimization--performance)
- [Development Tools](#development-tools)
- [LLMOps Platforms](#llmops-platforms)
- [Resources & Learning](#resources--learning)
- [Contributing](#contributing)
---
## What's New
### 🆕 Recently Added (January 2026)
**Infrastructure & Deployment:**
- [Skypilot](https://github.com/skypilot-org/skypilot) - Run LLMs on any cloud with one command
- [Modal](https://modal.com/) - Serverless platform for AI/ML workloads
**Evaluation & Testing:**
- [Ragas](https://github.com/explodinggradients/ragas) - Evaluation framework for RAG pipelines
- [PromptFoo](https://github.com/promptfoo/promptfoo) - Test and evaluate LLM outputs
**Agent Frameworks:**
- [Phidata](https://github.com/phidatahq/phidata) - Build AI assistants with memory and knowledge
- [Composio](https://github.com/ComposioHQ/composio) - Integration platform for AI agents
**Monitoring & Observability:**
- [Traceloop](https://github.com/traceloop/openllmetry) - OpenTelemetry for LLMs
- [LangWatch](https://github.com/langwatch/langwatch) - LLM monitoring and analytics
### 📈 Trending This Month
- **vLLM** continues to dominate high-throughput inference
- **LangGraph** gaining traction for stateful agent workflows
- **Ollama** becoming the go-to for local LLM deployment
- **DeepSeek** models showing impressive cost-performance ratios
---
## What is LLMOps?
**LLMOps (Large Language Model Operations)** is a set of practices, tools, and workflows designed to deploy, monitor, and maintain large language models in production environments. It encompasses the entire lifecycle of LLM applications, from development and training to deployment, monitoring, and continuous improvement.
### Key Components of LLMOps:
- **Model Development**: Training, fine-tuning, and optimizing LLMs
- **Deployment**: Serving models efficiently at scale
- **Monitoring**: Tracking performance, costs, and quality
- **Prompt Management**: Version control and optimization of prompts
- **Security**: Ensuring safe and responsible AI usage
- **Evaluation**: Testing and validating model outputs
- **Data Management**: Handling training data and embeddings
---
## LLMOps vs MLOps
| Aspect | MLOps | LLMOps |
|--------|-------|--------|
| **Model Size** | Typically smaller models | Very large models (billions of parameters) |
| **Training** | Full model training common | Fine-tuning and prompt engineering preferred |
| **Deployment** | Standard serving infrastructure | Specialized inference optimization required |
| **Monitoring** | Metrics-focused | Quality, safety, and cost-focused |
| **Versioning** | Model versions | Model + prompt + configuration versions |
| **Cost** | Moderate compute costs | High compute and inference costs |
| **Latency** | Milliseconds | Seconds (streaming helps) |
| **Data** | Structured/tabular data | Unstructured text, multimodal data |
---
## Models
### Large Language Models
| Model | Description | Stars | License |
|-------|-------------|-------|---------|
| [LLaMA](https://github.com/facebookresearch/llama) | Meta's foundational large language models |  | Research |
| [Mistral](https://github.com/mistralai/mistral-src) | High-performance open models from Mistral AI |  | Apache 2.0 |
| [Gemma](https://www.kaggle.com/models/google/gemma) | Google's lightweight open models | N/A | Gemma License |
| [Qwen](https://github.com/QwenLM/Qwen) | Alibaba's multilingual LLM series |  | Apache 2.0 |
| [DeepSeek](https://github.com/deepseek-ai/DeepSeek-LLM) | Cost-effective open-source LLMs |  | MIT |
| [Phi](https://huggingface.co/microsoft/phi-2) | Microsoft's small language models | N/A | MIT |
| [ChatGLM](https://github.com/THUDM/ChatGLM-6B) | Bilingual conversational language model |  | Apache 2.0 |
| [Alpaca](https://github.com/tatsu-lab/stanford_alpaca) | Stanford's instruction-following model |  | Apache 2.0 |
| [Vicuna](https://github.com/lm-sys/FastChat) | Open chatbot trained by fine-tuning LLaMA |  | Apache 2.0 |
| [BELLE](https://github.com/LianjiaTech/BELLE) | Chinese language model based on LLaMA |  | Apache 2.0 |
| [Falcon](https://huggingface.co/tiiuae/falcon-40b) | TII's high-performance open models | N/A | Apache 2.0 |
| [Bloom](https://github.com/bigscience-workshop/model_card) | Multilingual LLM from BigScience |  | RAIL |
### Multimodal Models
| Model | Description | Stars |
|-------|-------------|-------|
| [LLaVA](https://github.com/haotian-liu/LLaVA) | Large Language and Vision Assistant |  |
| [MiniCPM-V](https://github.com/OpenBMB/MiniCPM-V) | Efficient multimodal model |  |
| [Qwen-VL](https://github.com/QwenLM/Qwen-VL) | Vision-language model from Alibaba |  |
### Audio Foundation Models
| Model | Description | Stars |
|-------|-------------|-------|
| [Whisper](https://github.com/openai/whisper) | OpenAI's speech recognition model |  |
| [Faster Whisper](https://github.com/guillaumekln/faster-whisper) | Fast inference engine for Whisper |  |
---
## Inference & Serving
### Inference Engines
| Tool | Description | Stars |
|------|-------------|-------|
| [vLLM](https://github.com/vllm-project/vllm) | High-throughput and memory-efficient inference engine |  |
| [llama.cpp](https://github.com/ggerganov/llama.cpp) | LLM inference in C/C++ |  |
| [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) | NVIDIA's optimized inference library |  |
| [LMDeploy](https://github.com/InternLM/lmdeploy) | Toolkit for compressing and deploying LLMs |  |
| [DeepSpeed-MII](https://github.com/microsoft/DeepSpeed-MII) | Low-latency inference powered by DeepSpeed |  |
| [CTranslate2](https://github.com/OpenNMT/CTranslate2) | Fast inference engine for Transformer models |  |
| [Cortex.cpp](https://github.com/janhq/cortex.cpp) | Local AI API Platform |  |
| [LoRAX](https://github.com/predibase/lorax) | Multi-LoRA inference server |  |
| [MInference](https://github.com/microsoft/minference) | Speed up long-context LLM inference |  |
| [ipex-llm](https://github.com/intel-analytics/ipex-llm) | Accelerate LLM inference on Intel hardware |  |
### Inference Platforms
| Platform | Description | Stars |
|----------|-------------|-------|
| [Ollama](https://github.com/ollama/ollama) | Run LLMs locally with ease |  |
| [LocalAI](https://github.com/mudler/LocalAI) | OpenAI-compatible API for local models |  |
| [LM Studio](https://lmstudio.ai/) | Desktop app for running LLMs locally | N/A |
| [GPUStack](https://github.com/gpustack/gpustack) | Manage GPU clusters for LLM inference |  |
| [OpenLLM](https://github.com/bentoml/OpenLLM) | Operating LLMs in production |  |
| [Ray Serve](https://github.com/ray-project/ray) | Scalable model serving with Ray |  |
### Model Serving Frameworks
| Framework | Description | Stars |
|-----------|-------------|-------|
| [BentoML](https://github.com/bentoml/BentoML) | Unified model serving framework |  |
| [Triton Inference Server](https://github.com/triton-inference-server/server) | NVIDIA's optimized inference solution |  |
| [TorchServe](https://github.com/pytorch/serve) | Serve PyTorch models in production |  |
| [TensorFlow Serving](https://github.com/tensorflow/serving) | Flexible ML serving system |  |
| [Jina](https://github.com/jina-ai/jina) | Build multimodal AI services |  |
| [Mosec](https://github.com/mosecorg/mosec) | Model serving with dynamic batching |  |
| [Infinity](https://github.com/michaelfeil/infinity) | REST API for text embeddings |  |
---
## Orchestration
### Application Frameworks
| Framework | Description | Stars |
|-----------|-------------|-------|
| [LangChain](https://github.com/langchain-ai/langchain) | Framework for developing LLM applications |  |
| [LlamaIndex](https://github.com/run-llama/llama_index) | Data framework for LLM applications |  |
| [Haystack](https://github.com/deepset-ai/haystack) | End-to-end NLP framework |  |
| [Semantic Kernel](https://github.com/microsoft/semantic-kernel) | Microsoft's SDK for AI orchestration |  |
| [Langfuse](https://github.com/langfuse/langfuse) | Open-source LLM engineering platform |  |
| [Neurolink](https://github.com/juspay/neurolink) | Universal AI development platform |  |
### Agent Frameworks
| Framework | Description | Stars |
|-----------|-------------|-------|
| [AutoGPT](https://github.com/Significant-Gravitas/AutoGPT) | Autonomous AI agent framework |  |
| [CrewAI](https://github.com/joaomdmoura/crewAI) | Framework for orchestrating AI agents |  |
| [AutoGen](https://github.com/microsoft/autogen) | Multi-agent conversation framework |  |
| [LangGraph](https://github.com/langchain-ai/langgraph) | Build stateful multi-actor applications |  |
| [AgentMark](https://github.com/puzzlet-ai/agentmark) | Type-safe Markdown-based agents |  |
### Workflow Management
| Tool | Description | Stars |
|------|-------------|-------|
| [Prefect](https://github.com/PrefectHQ/prefect) | Modern workflow orchestration |  |
| [Airflow](https://github.com/apache/airflow) | Platform to programmatically author workflows |  |
| [Flyte](https://github.com/flyteorg/flyte) | Kubernetes-native workflow automation |  |
| [Flowise](https://github.com/FlowiseAI/Flowise) | Drag & drop UI for LLM flows |  |
---
## Training & Fine-Tuning
### Training Frameworks
| Framework | Description | Stars |
|-----------|-------------|-------|
| [DeepSpeed](https://github.com/microsoft/DeepSpeed) | Deep learning optimization library |  |
| [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) | Large-scale transformer training |  |
| [PyTorch FSDP](https://pytorch.org/docs/stable/fsdp.html) | Fully Sharded Data Parallel | N/A |
| [Colossal-AI](https://github.com/hpcaitech/ColossalAI) | Unified deep learning system |  |
| [Accelerate](https://github.com/huggingface/accelerate) | Simple way to train on distributed setups |  |
### Fine-Tuning Tools
| Tool | Description | Stars |
|------|-------------|-------|
| [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) | Streamlined LLM fine-tuning |  |
| [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) | Unified fine-tuning framework |  |
| [PEFT](https://github.com/huggingface/peft) | Parameter-Efficient Fine-Tuning |  |
| [Unsloth](https://github.com/unslothai/unsloth) | 2x faster LLM fine-tuning |  |
| [TRL](https://github.com/huggingface/trl) | Transformer Reinforcement Learning |  |
| [LitGPT](https://github.com/Lightning-AI/litgpt) | Pretrain, fine-tune, deploy LLMs |  |
### Experiment Tracking
| Tool | Description | Stars |
|------|-------------|-------|
| [Weights & Biases](https://github.com/wandb/wandb) | ML experiment tracking |  |
| [MLflow](https://github.com/mlflow/mlflow) | Open-source ML lifecycle platform |  |
| [TensorBoard](https://github.com/tensorflow/tensorboard) | TensorFlow's visualization toolkit |  |
| [Aim](https://github.com/aimhubio/aim) | Easy-to-use experiment tracker |  |
---
## Prompt Engineering
### Tools & Platforms
| Tool | Description | Link |
|------|-------------|------|
| [PromptBase](https://promptbase.com/) | Marketplace for prompt engineering | 🔗 |
| [PromptHero](https://prompthero.com/) | Prompt engineering resources | 🔗 |
| [Prompt Perfect](https://promptperfect.jina.ai/) | Auto prompt optimizer | 🔗 |
| [Learn Prompting](https://learnprompting.org/) | Prompt engineering tutorials | 🔗 |
| [LangSmith](https://www.langchain.com/langsmith) | Debug and test LLM applications | 🔗 |
| [PromptLayer](https://promptlayer.com/) | Prompt engineering platform | 🔗 |
### Resources
- [Exploring Prompt Injection Attacks](https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/)
- [Prompt Leaking Guide](https://learnprompting.org/docs/prompt_hacking/leaking)
- [Prefix-Tuning Paper](https://aclanthology.org/2021.acl-long.353.pdf)
---
## Vector Search & RAG
| Tool | Description | Stars |
|------|-------------|-------|
| [Chroma](https://github.com/chroma-core/chroma) | AI-native embedding database |  |
| [Weaviate](https://github.com/weaviate/weaviate) | Vector search engine |  |
| [Qdrant](https://github.com/qdrant/qdrant) | Vector similarity search engine |  |
| [Milvus](https://github.com/milvus-io/milvus) | Cloud-native vector database |  |
| [Pinecone](https://www.pinecone.io/) | Managed vector database | N/A |
| [FAISS](https://github.com/facebookresearch/faiss) | Efficient similarity search library |  |
| [pgvector](https://github.com/pgvector/pgvector) | Vector similarity search for Postgres |  |
| [LanceDB](https://github.com/lancedb/lancedb) | Developer-friendly vector database |  |
---
## Observability & Monitoring
| Tool | Description | Stars |
|------|-------------|-------|
| [Langfuse](https://github.com/langfuse/langfuse) | Open-source LLM observability |  |
| [Phoenix](https://github.com/Arize-ai/phoenix) | AI observability & evaluation |  |
| [Helicone](https://github.com/Helicone/helicone) | Open-source LLM observability |  |
| [Lunary](https://lunary.ai) | Production toolkit for LLMs | N/A |
| [OpenLIT](https://github.com/openlit/openlit) | OpenTelemetry-native LLM observability |  |
| [Evidently](https://github.com/evidentlyai/evidently) | ML and LLM observability framework |  |
| [DeepEval](https://github.com/confident-ai/deepeval) | LLM evaluation framework |  |
| [PostHog](https://github.com/PostHog/posthog) | Product analytics and feature flags |  |
---
## Security & Safety
| Tool | Description | Stars |
|------|-------------|-------|
| [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) | Programmable guardrails for LLM apps |  |
| [Guardrails AI](https://github.com/guardrails-ai/guardrails) | Add guardrails to LLM applications |  |
| [LLM Guard](https://github.com/protectai/llm-guard) | Security toolkit for LLM interactions |  |
| [Rebuff](https://github.com/protectai/rebuff) | Prompt injection detection |  |
| [LangKit](https://github.com/whylabs/langkit) | LLM monitoring toolkit |  |
---
## Data Management
| Tool | Description | Stars |
|------|-------------|-------|
| [DVC](https://github.com/iterative/dvc) | Data version control |  |
| [LakeFS](https://github.com/treeverse/lakeFS) | Git for data lakes |  |
| [Pachyderm](https://github.com/pachyderm/pachyderm) | Data versioning and pipelines |  |
| [Delta Lake](https://github.com/delta-io/delta) | Storage framework for data lakes |  |
---
## Optimization & Performance
| Tool | Description | Stars |
|------|-------------|-------|
| [ONNX Runtime](https://github.com/microsoft/onnxruntime) | Cross-platform ML accelerator |  |
| [TVM](https://github.com/apache/tvm) | ML compiler framework |  |
| [BitsAndBytes](https://github.com/TimDettmers/bitsandbytes) | 8-bit optimizers and quantization |  |
| [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) | Easy-to-use LLM quantization |  |
| [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa) | 4-bit quantization for LLaMA |  |
---
## Development Tools
### IDEs & Code Assistants
| Tool | Description | Stars |
|------|-------------|-------|
| [GitHub Copilot](https://github.com/features/copilot) | AI pair programmer | N/A |
| [Cursor](https://cursor.sh/) | AI-first code editor | N/A |
| [Continue](https://github.com/continuedev/continue) | Open-source AI code assistant |  |
| [Cody](https://sourcegraph.com/cody) | AI coding assistant | N/A |
| [Tabby](https://github.com/TabbyML/tabby) | Self-hosted AI coding assistant |  |
### Notebooks & Workspaces
| Tool | Description | Stars |
|------|-------------|-------|
| [Jupyter](https://github.com/jupyter/notebook) | Interactive computing environment |  |
| [Google Colab](https://colab.research.google.com/) | Free cloud notebooks | N/A |
| [Gradient](https://gradient.run/) | Managed notebooks and workflows | N/A |
---
## LLMOps Platforms
| Platform | Description | Stars |
|----------|-------------|-------|
| [Agenta](https://github.com/Agenta-AI/agenta) | LLMOps platform for building robust apps |  |
| [Dify](https://github.com/langgenius/dify) | LLM app development platform |  |
| [Pezzo](https://github.com/pezzolabs/pezzo) | Open-source LLMOps platform |  |
| [Humanloop](https://humanloop.com/) | Prompt management and evaluation | N/A |
| [PromptLayer](https://promptlayer.com/) | Prompt engineering platform | N/A |
| [Weights & Biases](https://wandb.ai/) | ML platform with LLM support | N/A |
---
## Resources & Learning
### Documentation & Guides
- [OpenAI Cookbook](https://github.com/openai/openai-cookbook) - Examples and guides for OpenAI API
- [LLM University](https://docs.cohere.com/docs/llmu) - Cohere's LLM learning resources
- [Hugging Face Course](https://huggingface.co/learn/nlp-course) - NLP with Transformers
- [Full Stack LLM Bootcamp](https://fullstackdeeplearning.com/llm-bootcamp/) - Comprehensive LLM course
### Awesome Lists
- [Awesome LLM](https://github.com/Hannibal046/Awesome-LLM) - Curated list of LLM resources
- [Awesome ChatGPT Prompts](https://github.com/f/awesome-chatgpt-prompts) - Prompt examples
- [Awesome AI Agents](https://github.com/e2b-dev/awesome-ai-agents) - AI agent resources
- [Awesome LangChain](https://github.com/kyrolabs/awesome-langchain) - LangChain resources
### Papers & Research
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762) - Original Transformer paper
- [BERT: Pre-training of Deep Bidirectional Transformers](https://arxiv.org/abs/1810.04805)
- [GPT-3: Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
- [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)
---
## Contributing
We welcome contributions from the community! Here's how you can help:
### How to Contribute
1. **Fork the repository**
2. **Create a new branch** (`git checkout -b feature/amazing-tool`)
3. **Add your contribution** following our guidelines
4. **Commit your changes** (`git commit -m 'Add amazing tool'`)
5. **Push to the branch** (`git push origin feature/amazing-tool`)
6. **Open a Pull Request**
### Contribution Guidelines
- **Quality over quantity**: Only add tools/resources you've personally used or thoroughly researched
- **Keep descriptions concise**: 1-2 sentences maximum
- **Include GitHub stars badge**: Use the format shown in existing entries
- **Maintain alphabetical order**: Within each category
- **Check for duplicates**: Search before adding
- **Update the Table of Contents**: If adding new sections
- **Follow the existing format**: Match the style of current entries
### What to Contribute
- ✅ New tools, frameworks, or platforms
- ✅ Useful resources, tutorials, or guides
- ✅ Bug fixes or improvements to existing entries
- ✅ Better descriptions or categorizations
- ❌ Promotional content or spam
- ❌ Outdated or unmaintained projects (unless historically significant)
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
---
## License
[](https://creativecommons.org/publicdomain/zero/1.0/)
This project is licensed under CC0 1.0 Universal. See [LICENSE](LICENSE) for details.
---
## Star History
[](https://star-history.com/#pmady/llmops&Date)
---
## Acknowledgments
This repository is inspired by and builds upon several excellent awesome lists:
- [tensorchord/Awesome-LLMOps](https://github.com/tensorchord/Awesome-LLMOps)
- [KennethanCeyer/awesome-llmops](https://github.com/KennethanCeyer/awesome-llmops)
- [InftyAI/Awesome-LLMOps](https://github.com/InftyAI/Awesome-LLMOps)
Special thanks to all contributors who help maintain and improve this resource!
---
If you find this repository helpful, please consider giving it a ⭐️
Made with ❤️ by the community