https://github.com/tensorchord/Awesome-LLMOps
An awesome & curated list of best LLMOps tools for developers
https://github.com/tensorchord/Awesome-LLMOps
List: awesome-llmops
ai-development-tools awesome-list llmops mlops
Last synced: about 1 year ago
JSON representation
An awesome & curated list of best LLMOps tools for developers
- Host: GitHub
- URL: https://github.com/tensorchord/Awesome-LLMOps
- Owner: tensorchord
- License: cc0-1.0
- Created: 2022-04-15T01:56:44.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2024-10-10T02:23:34.000Z (over 1 year ago)
- Last Synced: 2024-10-29T15:09:50.282Z (over 1 year ago)
- Topics: ai-development-tools, awesome-list, llmops, mlops
- Language: Shell
- Homepage:
- Size: 200 KB
- Stars: 3,917
- Watchers: 65
- Forks: 373
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Contributing: contributing.md
- License: LICENSE
Awesome Lists containing this project
- awesome-gpt-prompt-engineering - tensorchord Awesome-LLMOps
- awesome-llm-eval - Awesome-LLMOps - An awesome & curated list of the best LLMOps tools for developers. (Other-Awesome-Lists / Popular-LLM)
- Awesome-TimeSeries-SpatioTemporal-LM-LLM - \[link\
- StarryDivineSky - tensorchord/Awesome-LLMOps
- awesome-awesome-artificial-intelligence - Awesome LLMOps - LLMOps?style=social) | (System & Production)
- awesome-data-analysis - Awesome LLMOps - An awesome & curated list of best LLMOps tools for developers. (🚀 MLOps / Resources)
- awesome - tensorchord/Awesome-LLMOps - An awesome & curated list of best LLMOps tools for developers (Shell)
- awesome-ai-list-guide - Awesome-LLMOps
- awesome_ai_agents - Awesome-Llmops - An awesome & curated list of best LLMOps tools for developers (Building / Tools)
- awesome-llm-cost - Awesome-LLMOps - Broader LLMOps tooling. (Related Lists / Speculative decoding)
- awesome-awesome-llm - tensorchord/Awesome-LLMOps - LLMOps.svg) | Resources for LLM operations and deployment | | ★★★★★ | (Topics / LLM Applications)
- awesome - Awesome LLMOps - A curated collection of tools, frameworks, platforms, and best practices for operationalizing Large Language Models, covering deployment, monitoring, evaluation, and production workflows. ([Read more](/details/awesome-llmops.md)) `Llm` `Mlops` `Deployment` (Machine Learning & AI)
- llmops - tensorchord/Awesome-LLMOps
- awesome-ai-tools - Awesome LLMOps
- stars - Awesome-LLMOps
- my-awesomes-collection - Awesome-LLMOps - LLMOps) |LLMOps 工具的精选合集，覆盖模型训练、部署、监控全流程| (AI 相关 / AI 资源合集 Awesome List)
- jimsghstars - tensorchord/Awesome-LLMOps - An awesome & curated list of best LLMOps tools for developers (Shell)
README
# Awesome LLMOps
An awesome & curated list of the best LLMOps tools for developers.
> [!NOTE]
> Contributions are most welcome, please adhere to the [contribution guidelines](contributing.md).
## Table of Contents
- [Awesome LLMOps](#awesome-llmops)
- [Table of Contents](#table-of-contents)
- [Model](#model)
- [Large Language Model](#large-language-model)
- [CV Foundation Model](#cv-foundation-model)
- [Audio Foundation Model](#audio-foundation-model)
- [Serving](#serving)
- [Large Model Serving](#large-model-serving)
- [Frameworks/Servers for Serving](#frameworksservers-for-serving)
- [Security](#security)
- [Frameworks for LLM security](#frameworks-for-llm-security)
- [Observability](#observability)
- [LLMOps](#llmops)
- [Search](#search)
- [Vector search](#vector-search)
- [Code AI](#code-ai)
- [Training](#training)
- [IDEs and Workspaces](#ides-and-workspaces)
- [Foundation Model Fine Tuning](#foundation-model-fine-tuning)
- [Frameworks for Training](#frameworks-for-training)
- [Experiment Tracking](#experiment-tracking)
- [Visualization](#visualization)
- [Model Editing](#model-editing)
- [Data](#data)
- [Data Management](#data-management)
- [Data Storage](#data-storage)
- [Data Tracking](#data-tracking)
- [Feature Engineering](#feature-engineering)
- [Data/Feature enrichment](#datafeature-enrichment)
- [Large Scale Deployment](#large-scale-deployment)
- [ML Platforms](#ml-platforms)
- [Workflow](#workflow)
- [Scheduling](#scheduling)
- [Model Management](#model-management)
- [Performance](#performance)
- [ML Compiler](#ml-compiler)
- [Profiling](#profiling)
- [AutoML](#automl)
- [Optimizations](#optimizations)
- [Federated ML](#federated-ml)
- [Awesome Lists](#awesome-lists)
## Model
### Large Language Model
| Project | Details | Repository |
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------- |
| [Alpaca](https://github.com/tatsu-lab/stanford_alpaca) | Code and documentation to train Stanford's Alpaca models, and generate the data. |  |
| [BELLE](https://github.com/LianjiaTech/BELLE) | A 7B Large Language Model fine-tune by 34B Chinese Character Corpus, based on LLaMA and Alpaca. |  |
| [Bloom](https://github.com/bigscience-workshop/model_card) | BigScience Large Open-science Open-access Multilingual Language Model |  |
| [dolly](https://github.com/databrickslabs/dolly) | Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform |  |
| [Falcon 40B](https://huggingface.co/tiiuae/falcon-40b-instruct) | Falcon-40B-Instruct is a 40B parameters causal decoder-only model built by TII based on Falcon-40B and finetuned on a mixture of Baize. It is made available under the Apache 2.0 license. | |
| [FastChat (Vicuna)](https://github.com/lm-sys/FastChat) | An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and FastChat-T5. |  |
| [Gemma](https://www.kaggle.com/models/google/gemma) | Gemma is a family of lightweight, open models built from the research and technology that Google used to create the Gemini models. | |
| [GLM-6B (ChatGLM)](https://github.com/THUDM/ChatGLM-6B) | An Open Bilingual Pre-Trained Model, quantization of ChatGLM-130B, can run on consumer-level GPUs. |  |
| [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B) | ChatGLM2-6B is the second-generation version of the open-source bilingual (Chinese-English) chat model [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B). |  |
| [GLM-130B (ChatGLM)](https://github.com/THUDM/GLM-130B) | An Open Bilingual Pre-Trained Model (ICLR 2023) |  |
| [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) | An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library. |  |
| [Luotuo](https://github.com/LC1332/Luotuo-Chinese-LLM) | A Chinese LLM, Based on LLaMA and fine tune by Stanford Alpaca, Alpaca LoRA, Japanese-Alpaca-LoRA. |  |
| [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) | The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts. | |
| [StableLM](https://github.com/Stability-AI/StableLM) | StableLM: Stability AI Language Models |  |
**[⬆ back to ToC](#table-of-contents)**
### CV Foundation Model
| Project | Details | Repository |
| ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| [disco-diffusion](https://github.com/alembics/disco-diffusion) | A frankensteinian amalgamation of notebooks, models and techniques for the generation of AI Art and Animations. |  |
| [midjourney](https://www.midjourney.com/home/) | Midjourney is an independent research lab exploring new mediums of thought and expanding the imaginative powers of the human species. | |
| [segment-anything (SAM)](https://github.com/facebookresearch/segment-anything) | produces high quality object masks from input prompts such as points or boxes, and it can be used to generate masks for all objects in an image. |  |
| [stable-diffusion](https://github.com/CompVis/stable-diffusion) | A latent text-to-image diffusion model |  |
| [stable-diffusion v2](https://github.com/Stability-AI/stablediffusion) | High-Resolution Image Synthesis with Latent Diffusion Models |  |
**[⬆ back to ToC](#table-of-contents)**
### Audio Foundation Model
| Project | Details | Repository |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| [bark](https://github.com/suno-ai/bark) | Bark is a transformer-based text-to-audio model created by Suno. Bark can generate highly realistic, multilingual speech as well as other audio - including music, background noise and simple sound effects. |  |
| [whisper](https://github.com/openai/whisper) | Robust Speech Recognition via Large-Scale Weak Supervision |  |
## Serving
### Large Model Serving
| Project | Details | Repository |
| ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| [Alpaca-LoRA-Serve](https://github.com/deep-diver/Alpaca-LoRA-Serve) | Alpaca-LoRA as Chatbot service |  |
| [CTranslate2](https://github.com/OpenNMT/CTranslate2) | fast inference engine for Transformer models in C++ |  |
| [Clip-as-a-service](https://github.com/jina-ai/clip-as-service) | serving the OpenAI CLIP model |  |
| [DeepSpeed-MII](https://github.com/microsoft/DeepSpeed-MII) | MII makes low-latency and high-throughput inference possible, powered by DeepSpeed. |  |
| [Faster Whisper](https://github.com/guillaumekln/faster-whisper) | fast inference engine for whisper in C++ using CTranslate2. |  |
| [FlexGen](https://github.com/FMInference/FlexGen) | Running large language models on a single GPU for throughput-oriented scenarios. |  |
| [Flowise](https://github.com/FlowiseAI/Flowise) | Drag & drop UI to build your customized LLM flow using LangchainJS. |  |
| [llama.cpp](https://github.com/ggerganov/llama.cpp) | Port of Facebook's LLaMA model in C/C++ |  |
| [Infinity](https://github.com/michaelfeil/infinity) | Rest API server for serving text-embeddings |  |
| [Modelz-LLM](https://github.com/tensorchord/modelz-llm) | OpenAI compatible API for LLMs and embeddings (LLaMA, Vicuna, ChatGLM and many others) |  |
| [Ollama](https://github.com/jmorganca/ollama) | Serve Llama 2 and other large language models locally from command line or through a browser interface. |  |
| [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) | Inference engine for TensorRT on Nvidia GPUs |  |
| [text-generation-inference](https://github.com/huggingface/text-generation-inference) | Large Language Model Text Generation Inference |  |
| [text-embeddings-inference](https://github.com/huggingface/text-embeddings-inference) | Inference for text-embedding models |  |
| [tokenizers](https://github.com/huggingface/tokenizers) | 💥 Fast State-of-the-Art Tokenizers optimized for Research and Production |  |
| [vllm](https://github.com/vllm-project/vllm) | A high-throughput and memory-efficient inference and serving engine for LLMs. |  |
| [whisper.cpp](https://github.com/ggerganov/whisper.cpp) | Port of OpenAI's Whisper model in C/C++ |  |
| [x-stable-diffusion](https://github.com/stochasticai/x-stable-diffusion) | Real-time inference for Stable Diffusion - 0.88s latency. Covers AITemplate, nvFuser, TensorRT, FlashAttention. |  |
**[⬆ back to ToC](#table-of-contents)**
### Frameworks/Servers for Serving
| Project | Details | Repository |
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| [BentoML](https://github.com/bentoml/BentoML) | The Unified Model Serving Framework |  |
| [Jina](https://github.com/jina-ai/jina) | Build multimodal AI services via cloud native technologies · Model Serving · Generative AI · Neural Search · Cloud Native |  |
| [Mosec](https://github.com/mosecorg/mosec) | A machine learning model serving framework with dynamic batching and pipelined stages, provides an easy-to-use Python interface. |  |
| [TFServing](https://github.com/tensorflow/serving) | A flexible, high-performance serving system for machine learning models. |  |
| [Torchserve](https://github.com/pytorch/serve) | Serve, optimize and scale PyTorch models in production |  |
| [Triton Server (TRTIS)](https://github.com/triton-inference-server/server) | The Triton Inference Server provides an optimized cloud and edge inferencing solution. |  |
| [langchain-serve](https://github.com/jina-ai/langchain-serve) | Serverless LLM apps on Production with Jina AI Cloud |  |
| [lanarky](https://github.com/ajndkr/lanarky) | FastAPI framework to build production-grade LLM applications |  |
| [ray-llm](https://github.com/ray-project/ray-llm) | LLMs on Ray - RayLLM |  |
| [Xinference](https://github.com/xorbitsai/inference) | Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop. |  |
**[⬆ back to ToC](#table-of-contents)**
## Security
### Frameworks for LLM security
| Project | Details | Repository |
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| [Plexiglass](https://github.com/kortex-labs/plexiglass) | A Python Machine Learning Pentesting Toolbox for Adversarial Attacks. Works with LLMs, DNNs, and other machine learning algorithms. |  |
**[⬆ back to ToC](#table-of-contents)**
### Observability
| Project | Details | Repository |
| ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| [Azure OpenAI Logger](https://github.com/aavetis/azure-openai-logger) | "Batteries included" logging solution for your Azure OpenAI instance. |  |
| [Deepchecks](https://github.com/deepchecks/deepchecks) | Tests for Continuous Validation of ML Models & Data. Deepchecks is a Python package for comprehensively validating your machine learning models and data with minimal effort. |  |
| [Evidently](https://github.com/evidentlyai/evidently) | An open-source framework to evaluate, test and monitor ML and LLM-powered systems. |  |
| [Fiddler AI](https://github.com/fiddler-labs/fiddler-auditor) | Evaluate, monitor, analyze, and improve machine learning and generative models from pre-production to production. Ship more ML and LLMs into production, and monitor ML and LLM metrics like hallucination, PII, and toxicity. |  |
| [Giskard](https://github.com/Giskard-AI/giskard) | Testing framework dedicated to ML models, from tabular to LLMs. Detect risks of biases, performance issues and errors in 4 lines of code. |  |
| [Great Expectations](https://github.com/great-expectations/great_expectations) | Always know what to expect from your data. |  |
| [Helicone](https://github.com/Helicone/helicone) | Open source LLM observability platform. One line of code to monitor, evaluate, and experiment with features like prompt management, agent tracing, and evaluations. |  |
| [Traceloop OpenLLMetry](https://github.com/traceloop/openllmetry) | OpenTelemetry-based observability and monitoring for LLM and agents workflows. | 
| [whylogs](https://github.com/whylabs/whylogs) | The open standard for data logging |  |
**[⬆ back to ToC](#table-of-contents)**
## LLMOps
| Project | Details | Repository |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| [agenta](https://github.com/Agenta-AI/agenta) | The LLMOps platform to build robust LLM apps. Easily experiment and evaluate different prompts, models, and workflows to build robust apps. |  |
| [AgentMark](https://github.com/puzzlet-ai/agentmark) | Type-Safe Markdown-based Agents |  |
| [AI studio](https://github.com/missingstudio/ai) | A Reliable Open Source AI studio to build core infrastructure stack for your LLM Applications. It allows you to gain visibility, make your application reliable, and prepare it for production with features such as caching, rate limiting, exponential retry, model fallback, and more. |  |
| [Arize-Phoenix](https://github.com/Arize-ai/phoenix) | ML observability for LLMs, vision, language, and tabular models. |  |
| [BudgetML](https://github.com/ebhy/budgetml) | Deploy a ML inference service on a budget in less than 10 lines of code. |  |
| [Cheshire Cat AI](https://github.com/cheshire-cat-ai/core) | Web framework to create vertical AI agents. FastAPI based, plugin system inspired to WordPress, admin panel, vector DB included |  |
| [Dataoorts](https://dataoorts.com/ai) | Enjoy unlimited API calls with Serverless AI Workers/LLMs for just $25 per month. No rate or concurrency limits. | |
| [deeplake](https://github.com/activeloopai/deeplake) | Stream large multimodal datasets to achieve near 100% GPU utilization. Query, visualize, & version control data. Access data w/o the need to recompute the embeddings for the model finetuning. |  |
| [Dify](https://github.com/langgenius/dify) | Open-source framework aims to enable developers (and even non-developers) to quickly build useful applications based on large language models, ensuring they are visual, operable, and improvable. |  |
| [Dstack](https://github.com/dstackai/dstack) | Cost-effective LLM development in any cloud (AWS, GCP, Azure, Lambda, etc). |  |
| [Embedchain](https://github.com/embedchain/embedchain) | Framework to create ChatGPT like bots over your dataset. |  |
| [Epsilla](https://epsilla.com) | An all-in-one platform to create vertical AI agents powered by your private data and knowledge. | |
| [Evidently](https://github.com/evidentlyai/evidently) | An open-source framework to evaluate, test and monitor ML and LLM-powered systems. |  |
| [Fiddler AI](https://www.fiddler.ai/llmops) | Evaluate, monitor, analyze, and improve MLOps and LLMOps from pre-production to production. | |
| [Glide](https://github.com/EinStack/glide) | Cloud-Native LLM Routing Engine. Improve LLM app resilience and speed. |  |
| [gotoHuman](https://www.gotohuman.com) | Bring a **human into the loop** in your LLM-based and agentic workflows. Prompt users to approve actions, select next steps, or review and validate generated results. |
| [GPTCache](https://github.com/zilliztech/GPTCache) | Creating semantic cache to store responses from LLM queries. |  |
| [GPUStack](https://github.com/gpustack/gpustack) | An open-source GPU cluster manager for running and managing LLMs |  |
| [Haystack](https://github.com/deepset-ai/haystack) | Quickly compose applications with LLM Agents, semantic search, question-answering and more. |  |
| [Helicone](https://github.com/Helicone/helicone) | Open-source LLM observability platform for logging, monitoring, and debugging AI applications. Simple 1-line integration to get started. |  |
| [Humanloop](https://humanloop.com) | The LLM evals platform for enterprises, providing tools to develop, evaluate, and observe AI systems. | |
| [Izlo](https://getizlo.com/) | Prompt management tools for teams. Store, improve, test, and deploy your prompts in one unified workspace. | |
| [Keywords AI](https://keywordsai.co/) | A unified DevOps platform for AI software. Keywords AI makes it easy for developers to build LLM applications. | |
| [MLflow](https://github.com/mlflow/mlflow/tree/master) | An open-source framework for the end-to-end machine learning lifecycle, helping developers track experiments, evaluate models/prompts, deploy models, and add observability with tracing. |  |
| [Laminar](https://github.com/lmnr-ai/lmnr) | Open-source all-in-one platform for engineering AI products. Traces, Evals, Datasets, Labels. |  |
| [langchain](https://github.com/hwchase17/langchain) | Building applications with LLMs through composability |  |
| [LangFlow](https://github.com/logspace-ai/langflow) | An effortless way to experiment and prototype LangChain flows with drag-and-drop components and a chat interface. |  |
| [Langfuse](https://github.com/langfuse/langfuse) | Open Source LLM Engineering Platform: Traces, evals, prompt management and metrics to debug and improve your LLM application. |  |
| [LangKit](https://github.com/whylabs/langkit) | Out-of-the-box LLM telemetry collection library that extracts features and profiles prompts, responses and metadata about how your LLM is performing over time to find problems at scale. |  |
| [LangWatch](https://github.com/langwatch/langwatch) | LLM Ops platform with Analytics, Monitoring, Evaluations and an LLM Optimization Studio powered by DSPy |  |
| [LiteLLM 🚅](https://github.com/BerriAI/litellm/) | A simple & light 100 line package to **standardize LLM API calls** across OpenAI, Azure, Cohere, Anthropic, Replicate API Endpoints |  |
| [Literal AI](https://literalai.com/) | Multi-modal LLM observability and evaluation platform. Create prompt templates, deploy prompts versions, debug LLM runs, create datasets, run evaluations, monitor LLM metrics and collect human feedback. | |
| [LlamaIndex](https://github.com/jerryjliu/llama_index) | Provides a central interface to connect your LLMs with external data. |  |
| [LLMApp](https://github.com/pathwaycom/llm-app) | LLM App is a Python library that helps you build real-time LLM-enabled data pipelines with few lines of code. |  |
| [LLMFlows](https://github.com/stoyan-stoyanov/llmflows) | LLMFlows is a framework for building simple, explicit, and transparent LLM applications such as chatbots, question-answering systems, and agents. |  |
| [Lunary](https://github.com/lunary-ai/lunary) | Observability and prompt management for LLM chabots and agents. Debug agents with powerful tracing and logging. Usage analytics and dive deep into the history of your requests. Developer friendly modules with plug-and-play integration into LangChain. |  |
| [magentic](https://github.com/jackmpcollins/magentic) | Seamlessly integrate LLMs as Python functions. Use type annotations to specify structured output. Mix LLM queries and function calling with regular Python code to create complex LLM-powered functionality. |  |
| [Manag.ai](https://www.manag.ai) | Your all-in-one prompt management and observability platform. Craft, track, and perfect your LLM prompts with ease. | |
| [Mirascope](https://github.com/Mirascope/mirascope) | Intuitive convenience tooling for lightning-fast, efficient development and ensuring quality in LLM-based applications |  |
| [OpenLIT](https://github.com/openlit/openlit) | OpenLIT is an OpenTelemetry-native GenAI and LLM Application Observability tool and provides OpenTelmetry Auto-instrumentation for monitoring LLMs, VectorDBs and Frameworks. It provides valuable insights into token & cost usage, user interaction, and performance related metrics. |  |
| [Opik](https://github.com/comet-ml/opik) | Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle. |  |
| [Parea AI](https://www.parea.ai/) | Platform and SDK for AI Engineers providing tools for LLM evaluation, observability, and a version-controlled enhanced prompt playground. |  |
| [Pezzo 🕹️](https://github.com/pezzolabs/pezzo) | Pezzo is the open-source LLMOps platform built for developers and teams. In just two lines of code, you can seamlessly troubleshoot your AI operations, collaborate and manage your prompts in one place, and instantly deploy changes to any environment. |  |
| [PromptDX](https://github.com/puzzlet-ai/promptdx) | A declarative, extensible, and composable approach for developing LLM prompts using Markdown and JSX. |  |
| [PromptHub](https://www.prompthub.us) | Full stack prompt management tool designed to be usable by technical and non-technical team members. Test, version, collaborate, deploy, and monitor, all from one place. | |
| [promptfoo](https://github.com/typpo/promptfoo) | Open-source tool for testing & evaluating prompt quality. Create test cases, automatically check output quality and catch regressions, and reduce evaluation cost. |  |
| [PromptFoundry](https://www.promptfoundry.ai) | The simple prompt engineering and evaluation tool designed for developers building AI applications. |  |
| [PromptLayer 🍰](https://www.promptlayer.com) | Prompt Engineering platform. Collaborate, test, evaluate, and monitor your LLM applications |  |
| [PromptMage](https://github.com/tsterbak/promptmage) | Open-source tool to simplify the process of creating and managing LLM workflows and prompts as a self-hosted solution. |  |
| [PromptSite](https://github.com/dkuang1980/promptsite) | A lightweight Python library for prompt lifecycle management that helps you version control, track, experiment and debug with your LLM prompts with ease. Minimal setup, no servers, databases, or API keys required - works directly with your local filesystem, ideal for data scientists and engineers to easily integrate into existing LLM workflows | |
| [Prompteams](https://www.prompteams.com) | Prompt management system. Version, test, collaborate, and retrieve prompts through real-time APIs. Have GitHub style with repos, branches, and commits (and commit history). | |
| [prompttools](https://github.com/hegelai/prompttools) | Open-source tools for testing and experimenting with prompts. The core idea is to enable developers to evaluate prompts using familiar interfaces like code and notebooks. In just a few lines of codes, you can test your prompts and parameters across different models (whether you are using OpenAI, Anthropic, or LLaMA models). You can even evaluate the retrieval accuracy of vector databases. |  |
| [Puzzlet AI](https://www.puzzlet.ai) | The Git-Based LLM Engineering Platform. Achieve more from GenAI: Manage, evaluate, and improve your full-stack LLM application - with version control, type-safety, and local development built-in. | |
| [systemprompt.io](https://systemprompt.io) | Systemprompt.io is a Rest API with quality tooling to enable the creation, use and observability of prompts in any AI system. Control every detail of your prompt for a SOTA prompt management experience. | |
| [TreeScale](https://treescale.com) | All In One Dev Platform For LLM Apps. Deploy LLM-enhanced APIs seamlessly using tools for prompt optimization, semantic querying, version management, statistical evaluation, and performance tracking. As a part of the developer friendly API implementation TreeScale offers Elastic LLM product, which makes a unified API Endpoint for all major LLM providers and open source models. | |
| [TrueFoundry](https://www.truefoundry.com/) | Deploy LLMOps tools like Vector DBs, Embedding server etc on your own Kubernetes (EKS,AKS,GKE,On-prem) Infra including deploying, Fine-tuning, tracking Prompts and serving Open Source LLM Models with full Data Security and Optimal GPU Management. Train and Launch your LLM Application at Production scale with best Software Engineering practices. | |
| [ReliableGPT 💪](https://github.com/BerriAI/reliableGPT/) | Handle OpenAI Errors (overloaded OpenAI servers, rotated keys, or context window errors) for your production LLM Applications. |  |
| [Portkey](https://portkey.ai/) | Control Panel with an observability suite & an AI gateway — to ship fast, reliable, and cost-efficient apps. | |
| [Vellum](https://www.vellum.ai/) | An AI product development platform to experiment with, evaluate, and deploy advanced LLM apps. | |
| [Weights & Biases (Prompts)](https://docs.wandb.ai/guides/prompts) | A suite of LLMOps tools within the developer-first W&B MLOps platform. Utilize W&B Prompts for visualizing and inspecting LLM execution flow, tracking inputs and outputs, viewing intermediate results, securely managing prompts and LLM chain configurations. | |
| [Wordware](https://www.wordware.ai) | A web-hosted IDE where non-technical domain experts work with AI Engineers to build task-specific AI agents. It approaches prompting as a new programming language rather than low/no-code blocks. | |
| [xTuring](https://github.com/stochasticai/xturing) | Build and control your personal LLMs with fast and efficient fine-tuning. |  |
| [ZenML](https://github.com/zenml-io/zenml) | Open-source framework for orchestrating, experimenting and deploying production-grade ML solutions, with built-in `langchain` & `llama_index` integrations. |  |
**[⬆ back to ToC](#table-of-contents)**
## Search
### Vector search
| Project | Details | Repository |
| --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| [AquilaDB](https://github.com/Aquila-Network/AquilaDB) | An easy to use Neural Search Engine. Index latent vectors along with JSON metadata and do efficient k-NN search. |  |
| [Awadb](https://github.com/awa-ai/awadb) | AI Native database for embedding vectors |  |
| [Chroma](https://github.com/chroma-core/chroma) | the open source embedding database |  |
| [Epsilla](https://github.com/epsilla-cloud/vectordb) | A 10x faster, cheaper, and better vector database |  |
| [Infinity](https://github.com/infiniflow/infinity) | The AI-native database built for LLM applications, providing incredibly fast vector and full-text search |  |
| [Lancedb](https://github.com/lancedb/lancedb) | Developer-friendly, serverless vector database for AI applications. Easily add long-term memory to your LLM apps! |  |
| [Marqo](https://github.com/marqo-ai/marqo) | Tensor search for humans. |  |
| [Milvus](https://github.com/milvus-io/milvus) | Vector database for scalable similarity search and AI applications. |  |
| [Pinecone](https://www.pinecone.io/) | The Pinecone vector database makes it easy to build high-performance vector search applications. Developer-friendly, fully managed, and easily scalable without infrastructure hassles. | |
| [pgvector](https://github.com/pgvector/pgvector) | Open-source vector similarity search for Postgres. |  |
| [pgvecto.rs](https://github.com/tensorchord/pgvecto.rs) | Vector database plugin for Postgres, written in Rust, specifically designed for LLM. |  |
| [Qdrant](https://github.com/qdrant/qdrant) | Vector Search Engine and Database for the next generation of AI applications. Also available in the cloud |  |
| [txtai](https://github.com/neuml/txtai) | Build AI-powered semantic search applications |  |
| [Vald](https://github.com/vdaas/vald) | A Highly Scalable Distributed Vector Search Engine |  |
| [Vearch](https://github.com/vearch/vearch) | A distributed system for embedding-based vector retrieval |  |
| [VectorDB](https://github.com/jina-ai/vectordb) | A Python vector database you just need - no more, no less. |  |
| [Vellum](https://www.vellum.ai/products/retrieval) | A managed service for ingesting documents and performing hybrid semantic/keyword search across them. Comes with out-of-box support for OCR, text chunking, embedding model experimentation, metadata filtering, and production-grade APIs. | |
| [Weaviate](https://github.com/semi-technologies/weaviate) | Weaviate is an open source vector search engine that stores both objects and vectors, allowing for combining vector search with structured filtering with the fault-tolerance and scalability of a cloud-native database, all accessible through GraphQL, REST, and various language clients. |  |
**[⬆ back to ToC](#table-of-contents)**
## Code AI
| Project | Details | Repository |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| [CodeGeeX](https://github.com/THUDM/CodeGeeX) | CodeGeeX: An Open Multilingual Code Generation Model (KDD 2023) |  |
| [CodeGen](https://github.com/salesforce/CodeGen) | CodeGen is an open-source model for program synthesis. Trained on TPU-v4. Competitive with OpenAI Codex. |  |
| [CodeT5](https://github.com/salesforce/CodeT5) | Open Code LLMs for Code Understanding and Generation. |  |
| [Continue](https://github.com/continuedev/continue) | ⏩ the open-source autopilot for software development—bring the power of ChatGPT to VS Code |  |
| [fauxpilot](https://github.com/fauxpilot/fauxpilot) | An open-source alternative to GitHub Copilot server |  |
| [tabby](https://github.com/TabbyML/tabby) | Self-hosted AI coding assistant. An opensource / on-prem alternative to GitHub Copilot. |  |
## Training
### IDEs and Workspaces
| Project | Details | Repository |
| -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| [code server](https://github.com/coder/code-server) | Run VS Code on any machine anywhere and access it in the browser. |  |
| [conda](https://github.com/conda/conda) | OS-agnostic, system-level binary package manager and ecosystem. |  |
| [Docker](https://github.com/moby/moby) | Moby is an open-source project created by Docker to enable and accelerate software containerization. |  |
| [envd](https://github.com/tensorchord/envd) | 🏕️ Reproducible development environment for AI/ML. |  |
| [Jupyter Notebooks](https://github.com/jupyter/notebook) | The Jupyter notebook is a web-based notebook environment for interactive computing. |  |
| [Kurtosis](https://github.com/kurtosis-tech/kurtosis) | A build, packaging, and run system for ephemeral multi-container environments. |  |
| [Wordware](https://www.wordware.ai) | A web-hosted IDE where non-technical domain experts work with AI Engineers to build task-specific AI agents. It approaches prompting as a new programming language rather than low/no-code blocks. | |
**[⬆ back to ToC](#table-of-contents)**
### Foundation Model Fine Tuning
| Project | Details | Repository |
| -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| [alpaca-lora](https://github.com/tloen/alpaca-lora) | Instruct-tune LLaMA on consumer hardware |  |
| [finetuning-scheduler](https://github.com/speediedan/finetuning-scheduler) | A PyTorch Lightning extension that accelerates and enhances foundation model experimentation with flexible fine-tuning schedules. |  |
| [Flyflow](https://github.com/flyflow-devs) | Open source, high performance fine tuning as a service for GPT4 quality models with 5x lower latency and 3x lower cost |  |
| [LMFlow](https://github.com/OptimalScale/LMFlow) | An Extensible Toolkit for Finetuning and Inference of Large Foundation Models |  |
| [Lora](https://github.com/cloneofsimo/lora) | Using Low-rank adaptation to quickly fine-tune diffusion models. |  |
| [peft](https://github.com/huggingface/peft) | State-of-the-art Parameter-Efficient Fine-Tuning. |  |
| [p-tuning-v2](https://github.com/THUDM/P-tuning-v2) | An optimized prompt tuning strategy achieving comparable performance to fine-tuning on small/medium-sized models and sequence tagging challenges. [(ACL 2022)](https://arxiv.org/abs/2110.07602) |  |
| [QLoRA](https://github.com/artidoro/qlora) | Efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. |  |
| [TRL](https://github.com/huggingface/trl) | Train transformer language models with reinforcement learning. |  |
**[⬆ back to ToC](#table-of-contents)**
### Frameworks for Training
| Project | Details | Repository |
| -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| [Accelerate](https://github.com/huggingface/accelerate) | 🚀 A simple way to train and use PyTorch models with multi-GPU, TPU, mixed-precision. |  |
| [Apache MXNet](https://github.com/apache/mxnet) | Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler. |  |
| [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) | A tool designed to streamline the fine-tuning of various AI models, offering support for multiple configurations and architectures. |  |
| [Caffe](https://github.com/BVLC/caffe) | A fast open framework for deep learning. |  |
| [Candle](https://github.com/huggingface/candle) | Minimalist ML framework for Rust . |  |
| [ColossalAI](https://github.com/hpcaitech/ColossalAI) | An integrated large-scale model training system with efficient parallelization techniques. |  |
| [DeepSpeed](https://github.com/microsoft/DeepSpeed) | DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. |  |
| [Horovod](https://github.com/horovod/horovod) | Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet. |  |
| [Jax](https://github.com/google/jax) | Autograd and XLA for high-performance machine learning research. |  |
| [Kedro](https://github.com/kedro-org/kedro) | Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code. |  |
| [Keras](https://github.com/keras-team/keras) | Keras is a deep learning API written in Python, running on top of the machine learning platform TensorFlow. |  |
| [LightGBM](https://github.com/microsoft/LightGBM) | A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks. |  |
| [MegEngine](https://github.com/MegEngine/MegEngine) | MegEngine is a fast, scalable and easy-to-use deep learning framework, with auto-differentiation. |  |
| [metric-learn](https://github.com/scikit-learn-contrib/metric-learn) | Metric Learning Algorithms in Python. |  |
| [MindSpore](https://github.com/mindspore-ai/mindspore) | MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios. |  |
| [Oneflow](https://github.com/Oneflow-Inc/oneflow) | OneFlow is a performance-centered and open-source deep learning framework. |  |
| [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) | Machine Learning Framework from Industrial Practice. |  |
| [PyTorch](https://github.com/pytorch/pytorch) | Tensors and Dynamic neural networks in Python with strong GPU acceleration. |  |
| [PyTorch Lightning](https://github.com/lightning-AI/lightning) | Deep learning framework to train, deploy, and ship AI products Lightning fast. |  |
| [XGBoost](https://github.com/dmlc/xgboost) | Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library. |  |
| [scikit-learn](https://github.com/scikit-learn/scikit-learn) | Machine Learning in Python. |  |
| [TensorFlow](https://github.com/tensorflow/tensorflow) | An Open Source Machine Learning Framework for Everyone. |  |
| [VectorFlow](https://github.com/Netflix/vectorflow) | A minimalist neural network library optimized for sparse data and single machine environments. |  |
**[⬆ back to ToC](#table-of-contents)**
### Experiment Tracking
| Project | Details | Repository |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| [Aim](https://github.com/aimhubio/aim) | an easy-to-use and performant open-source experiment tracker. |  |
| [ClearML](https://github.com/allegroai/clearml) | Auto-Magical CI/CD to streamline your ML workflow. Experiment Manager, MLOps and Data-Management |  |
| [Comet](https://github.com/comet-ml/comet-examples) | Comet is an MLOps platform that offers experiment tracking, model production management, a model registry, and full data lineage from training straight through to production. Comet plays nicely with all your favorite tools, so you don't have to change your existing workflow. Comet Opik to confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle! |  |
| [Guild AI](https://github.com/guildai/guildai) | Experiment tracking, ML developer tools. |  |
| [MLRun](https://github.com/mlrun/mlrun) | Machine Learning automation and tracking. |  |
| [Kedro-Viz](https://github.com/kedro-org/kedro-viz) | Kedro-Viz is an interactive development tool for building data science pipelines with Kedro. Kedro-Viz also allows users to view and compare different runs in the Kedro project. |  |
| [LabNotebook](https://github.com/henripal/labnotebook) | LabNotebook is a tool that allow