Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
Awesome-LLMOps
🎉 An awesome & curated list of best LLMOps tools.
https://github.com/InftyAI/Awesome-LLMOps
Last synced: 3 days ago
JSON representation
-
LLMOps
- BentoML - Grade AI Applications | |
- Dify
- FastChat - sys/fastchat.svg) |  |  | An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena. | |
- Flowise
- Haystack - ai/haystack.svg) |  |  | 🔍 LLM orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots. | |
- LangChain - ai/langchain.svg) |  |  | ⚡ Building applications with LLMs through composability ⚡ | |
- LiteLLM - Azure, OpenAI, Cohere, Anthropic, Replicate. Manages input/output translation | |
- LLaMa-Factory - factory.svg) |  |  | Easy-to-use LLM fine-tuning framework (LLaMA, BLOOM, Mistral, Baichuan, Qwen, ChatGLM) | |
- LlamaIndex - llama/llama_index.svg) |  |  | LlamaIndex is a data framework for your LLM applications | |
- Mem0
- Open WebUI - webui/open-webui.svg) |  |  | User-friendly WebUI for LLMs (Formerly Ollama WebUI) | |
- PrivateGPUT - ai/private-gpt.svg) |  |  | Interact with your documents using the power of GPT, 100% privately, no data leaks | |
- Swift - swift) |  |  | SWIFT supports training(PreTraining/Fine-tuning/RLHF), inference, evaluation and deployment of 350+ LLMs and 90+ MLLMs (multimodal large models). | |
- BentoML - Grade AI Applications | |
- Dify
- FastChat - sys/fastchat.svg) |  |  | An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena. | |
- Flowise
- Haystack - ai/haystack.svg) |  |  | 🔍 LLM orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots. | |
- LangChain - ai/langchain.svg) |  |  | ⚡ Building applications with LLMs through composability ⚡ | |
- LiteLLM - Azure, OpenAI, Cohere, Anthropic, Replicate. Manages input/output translation | |
- LLaMa-Factory - factory.svg) |  |  | Easy-to-use LLM fine-tuning framework (LLaMA, BLOOM, Mistral, Baichuan, Qwen, ChatGLM) | |
- LlamaIndex - llama/llama_index.svg) |  |  | LlamaIndex is a data framework for your LLM applications | |
- Mem0
- PrivateGPUT - ai/private-gpt.svg) |  |  | Interact with your documents using the power of GPT, 100% privately, no data leaks | |
- Swift - swift) |  |  | SWIFT supports training(PreTraining/Fine-tuning/RLHF), inference, evaluation and deployment of 350+ LLMs and 90+ MLLMs (multimodal large models). | |
-
MLOps
- Flyte
- Kubeflow
- Metaflow - life data science projects with ease! | |
- MLflow
- Seldon-Core - core.svg) |  |  | An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models. | cloud |
- Flyte
- Kubeflow
- Metaflow - life data science projects with ease! | |
- MLflow
- Seldon-Core - core.svg) |  |  | An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models. | cloud |
- ZenML - io/zenml.svg) |  |  | ZenML 🙏: Build portable, production-ready MLOps pipelines. <https://zenml.io>. | |
- ZenML - io/zenml.svg) |  |  | ZenML 🙏: Build portable, production-ready MLOps pipelines. <https://zenml.io>. | |
-
Inference
- DeepSpeed-MII - mii.svg) |  |  | MII makes low-latency and high-throughput inference possible, powered by DeepSpeed. | |
- Inference - to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models. | vision |
- ipex-llm - analytics/ipex-llm.svg) |  |  | Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, MiniCPM, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, GraphRAG, DeepSpeed, vLLM, FastChat, Axolotl, etc. | device |
- llmaz - of-the-art LLMs on Kubernetes. | |
- LMDeploy
- MaxText
- LMDeploy
- DeepSpeed-MII - mii.svg) |  |  | MII makes low-latency and high-throughput inference possible, powered by DeepSpeed. | |
- MaxText
- Inference - to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models. | vision |
- llama.cpp
- MInference - context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy. | |
- ipex-llm - analytics/ipex-llm.svg) |  |  | Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, MiniCPM, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, GraphRAG, DeepSpeed, vLLM, FastChat, Axolotl, etc. | device |
- llmaz - of-the-art LLMs on Kubernetes. | |
- MLC LLM - ai/mlc-llm.svg) |  |  | Universal LLM Deployment Engine with ML Compilation | |
- Nanoflow - oriented high-performance serving framework for LLMs | |
- Ollama
- OpenLLM
- Ratchet - platform browser ML framework. | browser |
- RayServe - project/ray.svg) |  |  | Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads. | |
- RouteLLM - sys/routellm.svg) |  |  | A framework for serving and evaluating LLM routers - save LLM costs without compromising quality. | cost |
- SGLang - project/sglang.svg) |  |  | SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable. | |
- transformers.js - of-the-art Machine Learning for the web. Run 🤗 Transformers directly in your browser, with no need for a server! | browser |
- llama.cpp
- MInference - context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy. | |
- MLC LLM - ai/mlc-llm.svg) |  |  | Universal LLM Deployment Engine with ML Compilation | |
- MLServer
- Nanoflow - oriented high-performance serving framework for LLMs | |
- Triton Inference Server - inference-server/server.svg) |  |  | The Triton Inference Server provides an optimized cloud and edge inferencing solution. | |
- Ollama
- OpenLLM
- OpenVINO - source toolkit for optimizing and deploying AI inference | |
- Ratchet - platform browser ML framework. | browser |
- SGLang - project/sglang.svg) |  |  | SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable. | |
- transformers.js - of-the-art Machine Learning for the web. Run 🤗 Transformers directly in your browser, with no need for a server! | browser |
- Triton Inference Server - inference-server/server.svg) |  |  | The Triton Inference Server provides an optimized cloud and edge inferencing solution. | |
- RayServe - project/ray.svg) |  |  | Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads. | |
- RouteLLM - sys/routellm.svg) |  |  | A framework for serving and evaluating LLM routers - save LLM costs without compromising quality. | cost |
- TensorRT-LLM - LLM) |  |  | TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.||
- vLLM - project/vllm.svg) |  |  | A high-throughput and memory-efficient inference and serving engine for LLMs | |
- web-llm - ai/web-llm.svg) |  |  | A high-throughput and memory-efficient inference and serving engine for LLMs | browser |
- zml
- TensorRT-LLM - LLM) |  |  | TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.||
- Text Generation Inference - generation-inference.svg) |  |  | Large Language Model Text Generation Inference | |
- vLLM - project/vllm.svg) |  |  | A high-throughput and memory-efficient inference and serving engine for LLMs | |
- web-llm - ai/web-llm.svg) |  |  | A high-throughput and memory-efficient inference and serving engine for LLMs | browser |
- zml
-
Training
- ColossalAI
- Ludwig - ai/ludwig.svg) |  |  | Low-code framework for building custom LLMs, neural networks, and other AI models | |
- MLX - explore/mlx.svg) |  |  | MLX: An array framework for Apple silicon | |
- ColossalAI
- Ludwig - ai/ludwig.svg) |  |  | Low-code framework for building custom LLMs, neural networks, and other AI models | |
- MLX - explore/mlx.svg) |  |  | MLX: An array framework for Apple silicon | |
-
FineTune
- Axolotl - ai-cloud/axolotl.svg) |  |  | Go ahead and axolotl questions | |
- torchtune - PyTorch Library for LLM Fine-tuning | |
- unsloth - 5x faster with 80% less memory | |
- Axolotl - ai-cloud/axolotl.svg) |  |  | Go ahead and axolotl questions | |
- torchtune - PyTorch Library for LLM Fine-tuning | |
- unsloth - 5x faster with 80% less memory | |
-
Agent
- AutoGPT - Gravitas/AutoGPT.svg) |  |  | An experimental open-source attempt to make GPT-4 fully autonomous. | |
- MetaGPT - Agent Framework: Given one line Requirement, return PRD, Design, Tasks, Repo | |
- PydanticAI - ai.svg) |  |  | Agent Framework / shim to use Pydantic with LLMs | |
- Swarm - agent systems. Managed by OpenAI Solutions team. Experimental framework. | |
- XAgent
- AutoGPT - Gravitas/AutoGPT.svg) |  |  | An experimental open-source attempt to make GPT-4 fully autonomous. | |
- MetaGPT - Agent Framework: Given one line Requirement, return PRD, Design, Tasks, Repo | |
- PydanticAI - ai.svg) |  |  | Agent Framework / shim to use Pydantic with LLMs | |
- Swarm - agent systems. Managed by OpenAI Solutions team. Experimental framework. | |
- XAgent
-
Evaluation
- AgentBench
- lm-evaluation-harness - evaluation-harness.svg) |  |  | A framework for few-shot evaluation of language models. | |
- LongBench - context |
- AgentBench
- lm-evaluation-harness - evaluation-harness.svg) |  |  | A framework for few-shot evaluation of language models. | |
-
DB Store
- chroma - core/chroma.svg) |  |  | the AI-native open-source embedding database | vector |
- deeplake - time to PyTorch/TensorFlow. <https://activeloop.ai> | |
- Faiss
- milvus - io/milvus.svg) |  |  | A cloud-native vector database, storage for next generation AI applications | cloud,vector |
- deeplake - time to PyTorch/TensorFlow. <https://activeloop.ai> | |
- chroma - core/chroma.svg) |  |  | the AI-native open-source embedding database | vector |
- Faiss
- milvus - io/milvus.svg) |  |  | A cloud-native vector database, storage for next generation AI applications | cloud,vector |
- weaviate - source vector database that stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native database. | cloud,vector |
- weaviate - source vector database that stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native database. | cloud,vector |
-
Observation
- OpenLLMetry - source observability for your LLM application, based on OpenTelemetry | |
- Helicone AI - source LangSmith alternative for logging, monitoring, and debugging AI applications.| |
- phoenix - ai/phoenix.svg) |  |  | ML Observability in a Notebook - Uncover Insights, Surface Problems, Monitor, and Fine Tune your Generative LLM, CV and Tabular Models | |
- wandb
- OpenLLMetry - source observability for your LLM application, based on OpenTelemetry | |
- Helicone AI - source LangSmith alternative for logging, monitoring, and debugging AI applications.| |
- phoenix - ai/phoenix.svg) |  |  | ML Observability in a Notebook - Uncover Insights, Surface Problems, Monitor, and Fine Tune your Generative LLM, CV and Tabular Models | |
- wandb
-
Alignment
- OpenRLHF - to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & Mixtral) | |
- Self-RLHF - Alignment/safe-rlhf.svg) |  |  | Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback | |
- OpenRLHF - to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & Mixtral) | |
- Self-RLHF - Alignment/safe-rlhf.svg) |  |  | Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback | |
-
Outputs
- Instructor
- Outlines - dev/outlines.svg) |  |  | Structured Text Generation | |
- Instructor
- Outlines - dev/outlines.svg) |  |  | Structured Text Generation | |
Programming Languages
Categories
Sub Categories
Keywords
llm
57
ai
30
machine-learning
29
python
26
llama
22
mlops
22
deep-learning
22
pytorch
21
llmops
20
inference
16
ml
16
llama3
14
data-science
14
openai
13
gpt
13
large-language-models
12
fine-tuning
12
transformers
11
llms
11
rag
11
llama2
10
agent
10
llm-serving
10
language-model
10
vector-database
10
mistral
10
tensorflow
10
chatgpt
10
transformer
9
kubernetes
8
langchain
8
generative-ai
7
computer-vision
7
llm-inference
7
llava
6
deployment
6
golang
6
reinforcement-learning
6
workflow
6
datasets
6
gpt-4
6
lora
6
vector-search
6
embeddings
4
data
4
qlora
4
application
4
grpc
4
rlhf
4
gpu
4