An open API service indexing awesome lists of open source software.

awesome-production-machine-learning

A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning
https://github.com/EthicalML/awesome-production-machine-learning

Last synced: 27 days ago
JSON representation

Filtered by llm. Clear filters

Agentic Framework

  • OpenAGI - OpenAGI is used as the agent creation package to build agents for AIOS.
  • LangGraph - ai/langgraph.svg?style=social) - LangGraph is a library for building stateful, multi-actor applications with LLMs, used to create agent and multi-agent workflows.
  • Composio - Composio equip's your AI agents & LLMs with 100+ high-quality integrations via function calling.
  • AgentOps - AI/agentops.svg?style=social) - AgentOps helps developers build, evaluate, and monitor AI agents from prototype to production.
  • PydanticAI - ai.svg?style=social) - PydanticAI is a Python agent framework designed to make it less painful to build production grade applications with Generative AI.
  • TensorZero - TensorZero is an open-source framework for building production-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluations, and experimentation.

AutoML

  • Katib - A Kubernetes-based system for Hyperparameter Tuning and Neural Architecture Search.
  • Upgini - Free automated data & feature enrichment library for machine learning: automatically searches through thousands of ready-to-use features from public and community shared data sources and enriches your training dataset with only the accuracy improving features.
  • AIDE - AIDE is an open-source ML engineering agent that uses a tree search algorithm to autonomously explore, implement, and evaluate solution strategies for machine learning tasks.

Computation and Communication Optimisation

  • Ray - project/ray.svg?style=social) - Ray is a flexible, high-performance distributed execution framework for machine learning.
  • PEFT - Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient adaptation of pre-trained language models (PLMs) to various downstream applications without fine-tuning all the model's parameters.
  • bitsandbytes - foundation/bitsandbytes.svg?style=social) - Bitsandbytes library is a lightweight Python wrapper around CUDA custom functions, in particular 8-bit optimizers, matrix multiplication (LLM.int8()), and 8 & 4-bit quantization functions.
  • GPUStack - GPUStack is an open-source GPU cluster manager for running AI models.

Data Annotation and Synthesis

  • Argilla - io/argilla.svg?style=social) - Argilla helps domain experts and data teams to build better NLP datasets in less time.
  • NeMo Curator - Curator.svg?style=social) - NeMo Curator is a GPU-accelerated framework for efficient large language model data curation.
  • NeMo Curator - Curator.svg?style=social) - NeMo Curator is a GPU-accelerated framework for efficient large language model data curation.

Data Pipeline

  • Towhee - io/towhee.svg?style=social) - General-purpose machine learning pipeline for generating embedding vectors using one or many ML models.
  • ZenML - io/zenml.svg?style=social) - ZenML is an extensible, open-source MLOps framework to create reproducible ML pipelines with a focus on automated metadata tracking, caching, and many integrations to other tools.
  • Instill VDP - ai/instill-core.svg?style=social) - Instill VDP (Versatile Data Pipeline) aims to streamline the data processing pipelines from inception to completion.
  • Sycamore - ai/sycamore.svg?style=social) - Sycamore is an open source, AI-powered document processing engine for ETL, RAG, LLM-based applications, and analytics on unstructured data.
  • unstructured - IO/unstructured.svg?style=social) - unstructured streamlines and optimizes the data processing workflow for LLMs, ingesting and pre-processing images and text documents, such as PDFs, HTML, Word docs, and many more.
  • Pixeltable - source Python library providing declarative, incremental data infrastructure for building and managing multimodal AI workloads.

Data Storage Optimisation

  • GPTCache - GPTCache is a library for creating semantic cache for large language model queries.
  • Milvus - io/milvus.svg?style=social) Milvus is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks.
  • PostgresML
  • Casibase - Casibase is a LangChain-like RAG (Retrieval-Augmented Generation) knowledge database with web UI and Enterprise SSO.

Deployment and Serving

  • BentoML - BentoML is an open source framework for high performance ML model serving.
  • LightLLM - LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its * llama.cpp - llama.cpp is an open source software library that performs inference on various large language models such as Llama.
  • LocalAI - LocalAI is a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing.
  • Mosec - A rust-powered and multi-stage pipelined model server which offers dynamic batching and more. Super easy to implement and deploy as micro-services.
  • PowerInfer - IPADS/PowerInfer.svg?style=social) - PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device.
  • vLLM - project/vllm.svg?style=social) - vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.
  • OpenLLM - OpenLLM allows developers to run any open-source LLMs (Llama 3.1, Qwen2, Phi3 and more) or custom models as OpenAI-compatible APIs with a single command.
  • Vercel AI - Vercel AI is a TypeScript toolkit designed to help you build AI-powered applications using popular frameworks like Next.js, React, Svelte, Vue and runtimes like Node.js.
  • AirLLM - AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning.
  • Infinity - Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models and clip.
  • LM Studio - ai/lms.svg?style=social) - LM Studio is a tool for deploying LLM models locally on the computer, even on a relatively modest machine, provided it meets the minimum requirements.
  • Agenta - AI/agenta.svg?style=social) - Agenta provides end-to-end tools for the entire LLMOps workflow: building (LLM playground, evaluation), deploying (prompt and configuration management), and (LLM observability and tracing).
  • BISHENG - BISHENG is an open LLM application devops platform, focusing on enterprise scenarios.
  • Genkit - Genkit is an open source framework for building AI-powered apps with familiar code-centric patterns. Genkit makes it easy to develop, integrate, and test AI features with observability and evaluations.
  • IPEX-LLM - llm.svg?style=social) - IPEX-LLM is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency.
  • Langtrace - Labs/langtrace.svg?style=social) - Langtrace is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.
  • LMDeploy - LMDeploy is a toolkit for compressing, deploying, and serving LLM.
  • Open WebUI - webui/open-webui.svg?style=social) - Open WebUI is an extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It supports various LLM runners like Ollama and OpenAI-compatible APIs, with built-in inference engine for RAG, making it a powerful AI deployment solution.
  • SGLang - project/sglang.svg?style=social) - SGLang is a fast serving framework for large language models and vision language models.
  • LiteLLM - LiteLLM is a Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq.
  • nndeploy - An Easy-to-Use and High-Performance AI deployment framework.
  • OptiLLM - OptiLLM is an OpenAI API-compatible optimizing inference proxy that implements 20+ state-of-the-art techniques to dramatically improve LLM accuracy and performance on reasoning tasks - without requiring any model training or fine-tuning.
  • RamaLama - RamaLama is an open-source tool that simplifies the local use and serving of AI models for inference through OCI containers, eliminating the need to configure the host system.
  • llmfit - A terminal tool that right-sizes LLM models to your system's RAM, CPU, and GPU. Detects your hardware, scores each model across quality, speed, fit, and context dimensions, and tells you which ones will actually run well on your machine.
  • LMCache - LMCache is a high-performance KV cache layer that accelerates LLM inference.
  • SIE - Open-source inference server and production cluster for embeddings, reranking, and extraction. 85+ pre-configured models across dense, sparse, multi-vector, vision, rerankers, and extractors. Ships with Helm, KEDA autoscaling, Grafana dashboards, and Terraform.
  • LMDeploy - LMDeploy is a toolkit for compressing, deploying, and serving LLM.
  • Genkit - ai/genkit.svg?cacheSeconds=172800) - Genkit is an open source framework for building AI-powered apps with familiar code-centric patterns. Genkit makes it easy to develop, integrate, and test AI features with observability and evaluations.

Evaluation and Monitoring

  • Evidently - Evidently is an open-source framework to evaluate, test and monitor ML and LLM-powered systems.
  • Giskard - AI/giskard.svg?style=social) - Giskard is an open-source Python library that automatically detects performance, bias & security issues in AI applications.
  • Helicone - Helicone is the all-in-one, open-source LLM developer platform.
  • Langfuse - Langfuse is an observability & analytics solution for LLM-based applications.
  • EvalScope - EvalScope is a streamlined and customizable framework for efficient large model evaluation and performance benchmarking.
  • OpenCompass - compass/OpenCompass.svg?style=social) - OpenCompass is an LLM evaluation platform, supporting a wide range of models (LLaMA, LLaMa2, ChatGLM2, ChatGPT, Claude, etc) over 50+ datasets.
  • PhaseLLM - PhaseLLM is a large language model evaluation and workflow framework.
  • Prometheus-Eval - eval/prometheus-eval.svg?style=social) - RagaAI Catalyst is a comprehensive platform designed to enhance the management and optimization of LLM projects.
  • Ragas - Ragas is a framework to evaluate RAG pipelines.
  • Rageval - community/rageval.svg?style=social) - Rageval is a tool to evaluate RAG system.
  • TrustLLM - TrustLLM is a comprehensive framework to evaluate the trustworthiness of large language models, which includes principles, surveys, and benchmarks.
  • VLMEvalKit - compass/VLMEvalKit.svg?style=social) - VLMEvalKit is an open-source evaluation toolkit of large vision-language models (LVLMs).
  • LLMonitor - ai/lunary.svg?style=social) - LLMonitor is an observability & analytics for AI apps and agents.
  • Tonic Validate - Tonic Validate is a high-performance evaluation framework for LLM/RAG outputs.
  • Opik - ml/opik.svg?style=social) - Opik is an open-source platform for evaluating, testing and monitoring LLM applications.
  • LangTest - LangTest is a comprehensive evaluation toolkit for NLP models.
  • BEIR - cellar/beir.svg?style=social) - BEIR is a heterogeneous benchmark containing diverse IR tasks. It also provides a common and easy framework for evaluation of your NLP-based retrieval models within the benchmark.
  • LangWatch - LangWatch is a visual interface for DSPy and a complete LLM Ops platform for monitoring, experimenting, measuring and improving LLM pipelines, with a fair-code distribution model.
  • Ollama Grid Search - grid-search.svg?style=social) - Ollama Grid Search automates the process of selecting the best models, prompts, or inference parameters for a given use-case, allowing you to iterate over their combinations and to visually inspect the results.
  • SwanLab - SwanLab is an AI training tracking and visualization tool.
  • OpenLLMetry - OpenLLMetry provides developers with deep visibility into Large Language Model applications through performance monitoring, execution tracing, and debugging capabilities.
  • Promptfoo - Promptfoo is a developer-friendly local tool for testing LLM applications.
  • LangTest - LangTest is a comprehensive evaluation toolkit for NLP models.
  • Ragas - Ragas is a framework to evaluate RAG pipelines.

Industry Strength Computer Vision

  • Deep Lake - Deep Lake is a data infrastructure optimized for computer vision.

Industry Strength Information Retrieval

  • Cognita - Cognita is a RAG framework for building modular and production-ready applications.
  • llmware - ai/llmware.svg?style=social) - llmware provides a unified framework for building LLM-based applications (e.g, RAG, Agents), using small, specialized models that can be deployed privately, integrated with enterprise knowledge sources safely and securely, and cost-effectively tuned and adapted for any business process.
  • Mem0 - Mem0 enhances AI assistants and agents with an intelligent memory layer, enabling personalized AI interactions.
  • RAGxplorer - RAGxplorer is a tool to build RAG visualisations.
  • Vanna - ai/vanna.svg?style=social) - Vanna is a RAG framework for SQL generation and related functionality.
  • fastRAG - fastRAG is a research framework for efficient and optimized retrieval augmented generative pipelines, incorporating state-of-the-art LLMs and Information Retrieval.
  • GraphRAG - GraphRAG is a data pipeline and transformation suite that is designed to extract meaningful, structured data from unstructured text using the power of LLMs.
  • LightRAG - A simple and fast retrieval-augmented generation framework.
  • JamAI Base - JamAI Base is an open-source RAG (Retrieval-Augmented Generation) backend platform that integrates an embedded database (SQLite) and an embedded vector database (LanceDB) with managed memory and RAG capabilities. It features built-in LLM, vector embeddings, and reranker orchestration and management, all accessible through a convenient, intuitive, spreadsheet-like UI and a simple REST API.
  • BGE - BGE builds one-stop retrieval toolkit for search and RAG.
  • LangExtract - LangExtract is a Python library that uses LLMs to extract structured information from unstructured text documents based on user-defined instructions. It processes materials such as clinical notes or reports, identifying and organizing key details while ensuring the extracted data corresponds to the source text.

Industry Strength Natural Language Processing

  • Dust - tt/dust.svg?style=social) - Dust assists in the design and deployment of large language model apps.
  • h2oGPT - h2oGPT is an open source generative AI, gives organizations like yours the power to own large language models while preserving your data ownership.
  • Haystack - ai/haystack.svg?style=social) - Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs (GPT-3 and alike). Haystack offers production-ready tools to quickly build ChatGPT-like question answering, semantic search, text generation, and more.
  • LangChain - ai/langchain.svg?style=social) - LangChain assists in building applications with LLMs through composability.
  • LlamaIndex - llama/llama_index.svg?style=social) - LlamaIndex (GPT Index) is a data framework for your LLM application.
  • MLC LLM - ai/mlc-llm.svg?style=social) - MLC LLM is a universal solution that allows any language models to be deployed natively on a diverse set of hardware backends and native applications, plus a productive framework for everyone to further optimize model performance for their own use cases.
  • Ollama - Get up and running with large language models, locally.
  • Transformers - Huggingface's library of state-of-the-art pretrained models for Natural Language Processing (NLP).
  • SWIFT - swift.svg?style=social) - SWIFT is a scalable lightweight infrastructure for deep learning model fine-tuning.
  • PaddleNLP - PaddleNLP is a Large Language Model (LLM) development suite based on the PaddlePaddle deep learning framework, supporting efficient large model training, lossless compression, and high-performance inference on various hardware devices.
  • Semantic Kernel - kernel.svg?style=social) - Semantic Kernel is an SDK that integrates Large Language Models (LLMs) like OpenAI, Azure OpenAI, and Hugging Face with conventional programming languages like C#, Python, and Java. Semantic Kernel achieves this by allowing you to define plugins that can be chained together in just a few lines of code.
  • MindNLP - lab/mindnlp.svg?style=social) - MindNLP is an easy-to-use and high-performance NLP and LLM framework based on MindSpore, compatible with models and datasets of Huggingface.
  • LLaMA-Factory - Factory.svg?style=social) - LLaMA-Factory makes it easy to fine-tunes 100+ large language models with zero-code CLI and Web UI
  • Dify - Dify is an open-source LLM app development platform whose intuitive interface combines agentic AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.
  • Jan - Jan is an open-source ChatGPT alternative that runs 100% offline on your computer, allowing you to download and run LLMs locally with full control and privacy.
  • Context7 - Context7 provides up-to-date code docs for prompts and AI coding agents.
  • MindNLP - lab/mindnlp.svg?cacheSeconds=172800) - MindNLP is an easy-to-use and high-performance NLP and LLM framework based on MindSpore, compatible with models and datasets of Huggingface.

Industry Strength Reinforcement Learning

  • AReaL - project/AReaL.svg?cacheSeconds=172800) - AReaL is a reinforcement learning library.

Industry Strength Visualisation

  • Transformer Explainer - explainer.svg?style=social) - Transformer Explainer is an interactive visualization tool designed to help anyone learn how Transformer-based models like GPT work.

Model, Data and Experiment Management

  • Neptune - ai/neptune-client.svg?style=social) - Neptune is a scalable experiment tracker for teams that train foundation models.
  • HuggingFace Model Downloader - HuggingFace Model Downloader is a utility tool for downloading models and datasets from the HuggingFace website. It offers multithreaded downloading for LFS files and ensures the integrity of downloaded models with SHA256 checksum verification.

Model Training and Orchestration

  • Ludwig - ai/ludwig.svg?style=social) - Ludwig is a low-code framework for building custom AI models like LLMs and other deep neural networks.
  • Axolotl - ai-cloud/axolotl.svg?style=social) - Axolotl is a tool designed to streamline the fine-tuning of various AI models, offering support for multiple configurations and architectures.
  • unsloth - Fine-tuning & Reinforcement Learning for LLMs. Train OpenAI gpt-oss, DeepSeek-R1, Qwen3, Gemma 3, TTS 2x faster with 70% less VRAM.

Neural Search and Retrieval

  • BeyondLLM - Beyond LLM offers an all-in-one toolkit for experimentation, evaluation, and deployment of RAG systems, simplifying the process with automated integration, customizable evaluation metrics, and support for various LLMs tailored to specific needs, ultimately aiming to reduce LLM hallucination risks and enhance reliability.

Privacy and Safety

  • Guardrails - ai/guardrails.svg?style=social) - Guardrails is a package that lets a user add structure, type and quality guarantees to the outputs of large language models.
  • CipherChat - CipherChat is a framework to evaluate the generalization capability of safety alignment for LLMs
  • AI Gateway - ai/gateway.svg?style=social) - The AI Gateway is a blazing fast AI Gateway with integrated guardrails.
  • Awesome RAG Production - Curated list of production-grade tools and best practices for building scalable RAG systems.
  • Awesome AI Tokenomics - ai-tokenomics.svg?cacheSeconds=172800) - Covers token cost and efficiency in AI systems, including monitoring, optimization, caching, model selection, and context management.