An open API service indexing awesome lists of open source software.

awesome-llmops

An awesome & curated list of best LLMOps tools for developers
https://github.com/tensorchord/awesome-llmops

Last synced: 1 day ago
JSON representation

  • Large Scale Deployment

    • Workflow

      • Airflow - square) |
      • Metaflow - life data science projects with ease! | ![GitHub Badge](https://img.shields.io/github/stars/Netflix/metaflow.svg?style=flat-square) |
      • Kubeflow Pipelines - square) |
      • Argo Workflows - workflows.svg?style=flat-square) |
      • Prefect - square) |
      • Flyte - native workflow automation platform for complex, mission-critical data and ML processes at scale. | ![GitHub Badge](https://img.shields.io/github/stars/flyteorg/flyte.svg?style=flat-square) |
      • Hamilton - inc/hamilton.svg?style=flat-square) |
      • ZenML - io/zenml.svg?style=flat-square) |
      • Ploomber - square) |
      • aqueduct - Source Platform for Production Data Science | ![GitHub Badge](https://img.shields.io/github/stars/aqueducthq/aqueduct.svg?style=flat-square) |
      • LangFlow - and-drop components and a chat interface. | ![GitHub Badge](https://img.shields.io/github/stars/logspace-ai/langflow.svg?style=flat-square) |
    • ML Platforms

      • OpenLLM - tune, serve, deploy, and monitor any LLMs with ease. | ![GitHub Badge](https://img.shields.io/github/stars/bentoml/OpenLLM.svg?style=flat-square) |
      • MLflow - square) |
      • Kserve - square) |
      • Kubeflow - square) |
      • Polyaxon - square) |
      • ModelFox - square) |
      • Seldon-core - core.svg?style=flat-square) |
      • Hopsworks - tuning and serving LLMs. Hopsworks includes both a feature store and vector database for RAG. | ![GitHub Badge](https://img.shields.io/github/stars/logicalclocks/hopsworks.svg?style=flat-square) |
      • Weights & Biases - powered applications, featuring W&B Prompts for LLM execution flow visualization, input and output monitoring, and secure management of prompts and LLM chain configurations. | ![GitHub Badge](https://img.shields.io/github/stars/wandb/wandb.svg?style=flat-square) |
      • MLRun - square) |
      • Primehub - square) |
      • OpenModelZ - click machine learning deployment (LLM, text-to-image and so on) at scale on any cluster (GCP, AWS, Lambda labs, your home lab, or even a single machine). | ![GitHub Badge](https://img.shields.io/github/stars/tensorchord/openmodelz.svg?style=flat-square) |
      • Starwhale - tuning. | ![GitHub Badge](https://img.shields.io/github/stars/star-whale/starwhale.svg?style=flat-square) |
      • ClearML - Magical CI/CD to streamline your ML workflow. Experiment Manager, MLOps and Data-Management. | ![GitHub Badge](https://img.shields.io/github/stars/allegroai/clearml.svg?style=flat-square) |
      • TrueFoundry - tune and serve LLM Models on a company’s own Infrastructure with Data Security and Optimal GPU and Cost Management. Launch your LLM Application at Production scale with best DevSecOps practices. | |
    • Model Management

      • dvc - Data Version Control - Git for Data & Models | ![GitHub Badge](https://img.shields.io/github/stars/iterative/dvc.svg?style=flat-square) |
      • MLEM - square) |
      • ormb - square) |
      • Comet - ml/comet-examples.svg?style=flat-square) |
      • ModelDB - square) |
    • Scheduling

      • Kueue - native Job Queueing. | ![GitHub Badge](https://img.shields.io/github/stars/kubernetes-sigs/kueue.svg?style=flat-square) |
      • Volcano - sh/volcano.svg?style=flat-square) |
      • Slurm - square) |
      • PAI - sourced by Microsoft). | ![GitHub Badge](https://img.shields.io/github/stars/microsoft/pai.svg?style=flat-square) |
      • Yunikorn - weight, universal resource scheduler for container orchestrator systems. | ![GitHub Badge](https://img.shields.io/github/stars/apache/yunikorn-core.svg?style=flat-square) |
      • Pinecone - performance vector search applications. Developer-friendly, fully managed, and easily scalable without infrastructure hassles. | |
      • Vellum - of-box support for OCR, text chunking, embedding model experimentation, metadata filtering, and production-grade APIs. | |
      • pgvector - source vector similarity search for Postgres. | ![GitHub Badge](https://img.shields.io/github/stars/pgvector/pgvector.svg?style=flat-square) |
      • Milvus - io/milvus.svg?style=flat-square) |
      • txtai - powered semantic search applications | ![GitHub Badge](https://img.shields.io/github/stars/neuml/txtai.svg?style=flat-square) |
      • Qdrant - square) |
      • Marqo - ai/marqo.svg?style=flat-square) |
      • Vald - square) |
      • Chroma - core/chroma.svg?style=flat-square) |
      • Lancedb - friendly, serverless vector database for AI applications. Easily add long-term memory to your LLM apps! | ![GitHub Badge](https://img.shields.io/github/stars/lancedb/lancedb.svg?style=flat-square) |
      • pgvecto.rs - square) |
      • Infinity - native database built for LLM applications, providing incredibly fast vector and full-text search | ![GitHub Badge](https://img.shields.io/github/stars/infiniflow/infinity.svg?style=flat-square) |
      • ParadeDB - square) |
      • Vearch - based vector retrieval | ![GitHub Badge](https://img.shields.io/github/stars/vearch/vearch.svg?style=flat-square) |
      • Epsilla - cloud/vectordb.svg?style=flat-square) |
      • Awadb - ai/awadb.svg?style=flat-square) |
      • VectorDB - no more, no less. | ![GitHub Badge](https://img.shields.io/github/stars/jina-ai/vectordb.svg?style=flat-square) |
      • VectorChord - friendly vector search in Postgres, the successor of `pgvecto.rs`. | ![GitHub Badge](https://img.shields.io/github/stars/tensorchord/VectorChord.svg?style=flat-square) |
      • Weaviate - tolerance and scalability of a cloud-native database, all accessible through GraphQL, REST, and various language clients. | ![GitHub Badge](https://img.shields.io/github/stars/semi-technologies/weaviate.svg?style=flat-square) |
      • AquilaDB - NN search. | ![GitHub Badge](https://img.shields.io/github/stars/Aquila-Network/AquilaDB.svg?style=flat-square) |
  • Model

    • CV Foundation Model

      • midjourney
      • stable-diffusion - to-image diffusion model | ![GitHub Badge](https://img.shields.io/github/stars/CompVis/stable-diffusion.svg?style=flat-square) |
      • stable-diffusion v2 - Resolution Image Synthesis with Latent Diffusion Models | ![GitHub Badge](https://img.shields.io/github/stars/Stability-AI/stablediffusion.svg?style=flat-square) |
      • segment-anything (SAM) - anything.svg?style=flat-square) |
      • disco-diffusion - diffusion.svg?style=flat-square) |
    • Large Language Model

      • Mixtral-8x7B-v0.1 - 8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts. | |
      • Falcon 40B - 40B-Instruct is a 40B parameters causal decoder-only model built by TII based on Falcon-40B and finetuned on a mixture of Baize. It is made available under the Apache 2.0 license. | |
      • Gemma
      • FastChat (Vicuna) - T5. | ![GitHub Badge](https://img.shields.io/github/stars/lm-sys/FastChat.svg?style=flat-square) |
      • Alpaca - lab/stanford_alpaca.svg?style=flat-square) |
      • BELLE - tune by 34B Chinese Character Corpus, based on LLaMA and Alpaca. | ![GitHub Badge](https://img.shields.io/github/stars/LianjiaTech/BELLE.svg?style=flat-square) |
      • StableLM - AI/StableLM.svg?style=flat-square) |
      • GLM-6B (ChatGLM) - Trained Model, quantization of ChatGLM-130B, can run on consumer-level GPUs. | ![GitHub Badge](https://img.shields.io/github/stars/THUDM/ChatGLM-6B.svg?style=flat-square) |
      • dolly - square) |
      • Luotuo - Alpaca-LoRA. | ![GitHub Badge](https://img.shields.io/github/stars/LC1332/Luotuo-Chinese-LLM.svg?style=flat-square) |
      • ChatGLM2-6B - 6B is the second-generation version of the open-source bilingual (Chinese-English) chat model [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B). | ![GitHub Badge](https://img.shields.io/github/stars/THUDM/ChatGLM2-6B.svg?style=flat-square) |
      • GLM-130B (ChatGLM) - Trained Model (ICLR 2023) | ![GitHub Badge](https://img.shields.io/github/stars/THUDM/GLM-130B.svg?style=flat-square) |
      • GPT-NeoX - neox.svg?style=flat-square) |
      • Bloom - science Open-access Multilingual Language Model | ![GitHub Badge](https://img.shields.io/github/stars/bigscience-workshop/model_card.svg?style=flat-square) |
    • Audio Foundation Model

      • bark - based text-to-audio model created by Suno. Bark can generate highly realistic, multilingual speech as well as other audio - including music, background noise and simple sound effects. | ![GitHub Badge](https://img.shields.io/github/stars/suno-ai/bark.svg?style=flat-square) |
      • whisper - Scale Weak Supervision | ![GitHub Badge](https://img.shields.io/github/stars/openai/whisper.svg?style=flat-square) |
  • LLMOps

    • Observability

      • TrueFoundry - prem) Infra including deploying, Fine-tuning, tracking Prompts and serving Open Source LLM Models with full Data Security and Optimal GPU Management. Train and Launch your LLM Application at Production scale with best Software Engineering practices. | |
      • Portkey - efficient apps. | |
      • Fiddler AI - production to production. | |
      • Parea AI - controlled enhanced prompt playground. | ![GitHub Badge](https://img.shields.io/github/stars/parea-ai/parea-sdk-py?style=flat-square) |
      • Vellum
      • Izlo
      • Keywords AI
      • Literal AI - modal LLM observability and evaluation platform. Create prompt templates, deploy prompts versions, debug LLM runs, create datasets, run evaluations, monitor LLM metrics and collect human feedback. | |
      • agenta - AI/agenta.svg?style=flat-square) |
      • Dify - source framework aims to enable developers (and even non-developers) to quickly build useful applications based on large language models, ensuring they are visual, operable, and improvable. | ![GitHub Badge](https://img.shields.io/github/stars/langgenius/dify.svg?style=flat-square) |
      • Pezzo 🕹️ - source LLMOps platform built for developers and teams. In just two lines of code, you can seamlessly troubleshoot your AI operations, collaborate and manage your prompts in one place, and instantly deploy changes to any environment. | ![GitHub Badge](https://img.shields.io/github/stars/pezzolabs/pezzo.svg?style=flat-square) |
      • Langfuse - square) |
      • Evidently - source framework to evaluate, test and monitor ML and LLM-powered systems. | ![GitHub Badge](https://img.shields.io/github/stars/evidentlyai/evidently.svg?style=flat-square) |
      • Haystack - answering and more. | ![GitHub Badge](https://img.shields.io/github/stars/deepset-ai/haystack.svg?style=flat-square) |
      • deeplake - square) |
      • Cheshire Cat AI - cat-ai/core.svg?style=flat-square) |
      • GPTCache - square) |
      • LLMApp - time LLM-enabled data pipelines with few lines of code. | ![GitHub Badge](https://img.shields.io/github/stars/pathwaycom/llm-app.svg?style=flat-square) |
      • Arize-Phoenix - ai/phoenix.svg?style=flat-square) |
      • LangKit - of-the-box LLM telemetry collection library that extracts features and profiles prompts, responses and metadata about how your LLM is performing over time to find problems at scale. | ![GitHub Badge](https://img.shields.io/github/stars/whylabs/langkit.svg?style=flat-square) |
      • Glide - Native LLM Routing Engine. Improve LLM app resilience and speed. | ![GitHub Badge](https://img.shields.io/github/stars/einstack/glide.svg?style=flat-square) |
      • xTuring - tuning. | ![GitHub Badge](https://img.shields.io/github/stars/stochasticai/xturing.svg?style=flat-square) |
      • Helicone - source LLM observability platform for logging, monitoring, and debugging AI applications. Simple 1-line integration to get started. | ![GitHub Badge](https://img.shields.io/github/stars/helicone/helicone.svg?style=flat-square) |
      • prompttools - source tools for testing and experimenting with prompts. The core idea is to enable developers to evaluate prompts using familiar interfaces like code and notebooks. In just a few lines of codes, you can test your prompts and parameters across different models (whether you are using OpenAI, Anthropic, or LLaMA models). You can even evaluate the retrieval accuracy of vector databases. | ![GitHub Badge](https://img.shields.io/github/stars/hegelai/prompttools.svg?style=flat-square) |
      • magentic - powered functionality. | ![GitHub Badge](https://img.shields.io/github/stars/jackmpcollins/magentic.svg?style=flat-square) |
      • BudgetML - square) |
      • Lunary - and-play integration into LangChain. | ![GitHub Badge](https://img.shields.io/github/stars/lunary-ai/lunary.svg?style=flat-square) |
      • LLMFlows - answering systems, and agents. | ![GitHub Badge](https://img.shields.io/github/stars/stoyan-stoyanov/llmflows.svg?style=flat-square) |
      • Mirascope - fast, efficient development and ensuring quality in LLM-based applications | ![GitHub Badge](https://img.shields.io/github/stars/Mirascope/mirascope.svg?style=flat-square) |
      • OpenLIT - native GenAI and LLM Application Observability tool and provides OpenTelmetry Auto-instrumentation for monitoring LLMs, VectorDBs and Frameworks. It provides valuable insights into token & cost usage, user interaction, and performance related metrics. | ![GitHub Badge](https://img.shields.io/github/stars/dokulabs/doku.svg?style=flat-square) |
      • AI studio - square) |
      • Dstack - effective LLM development in any cloud (AWS, GCP, Azure, Lambda, etc). | ![GitHub Badge](https://img.shields.io/github/stars/dstackai/dstack.svg?style=flat-square) |
      • PromptMage - source tool to simplify the process of creating and managing LLM workflows and prompts as a self-hosted solution. | ![GitHub Badge](https://img.shields.io/github/stars/tsterbak/promptmage.svg?style=flat-square) |
      • GPUStack - source GPU cluster manager for running and managing LLMs | ![GitHub Badge](https://img.shields.io/github/stars/gpustack/gpustack.svg?style=flat-square) |
      • PromptFoundry - foundry/python-sdk.svg?style=flat-square) |
      • Opik - ml/opik.svg?style=flat-square) |
      • gotoHuman - based and agentic workflows. Prompt users to approve actions, select next steps, or review and validate generated results. |
      • Laminar - source all-in-one platform for engineering AI products. Traces, Evals, Datasets, Labels. | ![GitHub Badge](https://img.shields.io/github/stars/lmnr-ai/lmnr.svg?style=flat-square) |
      • PromptLayer 🍰 - layer-library.svg?style=flat-square) |
      • TensorZero - source framework for building production-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluations, and experimentation. | ![GitHub Badge](https://img.shields.io/github/stars/tensorzero/tensorzero.svg?style=flat-square) |
      • Dataoorts
      • PromptDX - ai/promptdx.svg?style=flat-square) |
      • systemprompt.io
      • MLflow - source framework for the end-to-end machine learning lifecycle, helping developers track experiments, evaluate models/prompts, deploy models, and add observability with tracing. | ![GitHub Badge](https://img.shields.io/github/stars/mlflow/mlflow.svg?style=flat-square) |
      • Epsilla - in-one platform to create vertical AI agents powered by your private data and knowledge. | |
      • PromptSite - works directly with your local filesystem, ideal for data scientists and engineers to easily integrate into existing LLM workflows | |
      • AgentMark - Safe Markdown-based Agents | ![GitHub Badge](https://img.shields.io/github/stars/Puzzlet-ai/agentmark.svg?style=flat-square) |
      • AI studio - square) |
      • LiteLLM 🚅 - square) |
      • LlamaIndex - square) |
      • langchain - square) |
      • Manag.ai - in-one prompt management and observability platform. Craft, track, and perfect your LLM prompts with ease. | |
      • Neurolink - provider AI agent framework that unifies 12+ LLM providers (OpenAI, Google, Anthropic, AWS, Azure, Groq, etc.) with workflow orchestration. Production-grade platform for building LLM applications with streaming, tool calling, caching, and enterprise features. Battle-tested at 15M+ requests/month. | ![GitHub Badge](https://img.shields.io/github/stars/juspay/neurolink.svg?style=flat-square) |
      • Manag.ai - in-one prompt management and observability platform. Craft, track, and perfect your LLM prompts with ease. | |
      • Dataoorts
      • Hypersigil - source prompt lifecycle management and gateway with a Web UI. | ![GitHub Badge](https://img.shields.io/github/stars/hypersigilhq/hypersigil.svg?style=flat-square) |
      • Roundtable - configuration unified AI assistant management built on the FastMCP framework. Provides seamless integration with Claude, ChatGPT, and other AI assistants through a single MCP interface with session management, logging, and production-ready operations. | ![GitHub Badge](https://img.shields.io/github/stars/askbudi/roundtable.svg?style=flat-square) |
      • Weights & Biases (Prompts) - first W&B MLOps platform. Utilize W&B Prompts for visualizing and inspecting LLM execution flow, tracking inputs and outputs, viewing intermediate results, securely managing prompts and LLM chain configurations. | |
      • Embedchain - square) |
      • Epsilla - in-one platform to create vertical AI agents powered by your private data and knowledge. | |
      • gotoHuman - based and agentic workflows. Prompt users to approve actions, select next steps, or review and validate generated results. |
      • Keywords AI
      • Literal AI - modal LLM observability and evaluation platform. Create prompt templates, deploy prompts versions, debug LLM runs, create datasets, run evaluations, monitor LLM metrics and collect human feedback. | |
      • PromptDX - ai/promptdx.svg?style=flat-square) |
      • PromptFoundry - foundry/python-sdk.svg?style=flat-square) |
      • Prompteams - time APIs. Have GitHub style with repos, branches, and commits (and commit history). | |
      • Puzzlet AI - Based LLM Engineering Platform. Achieve more from GenAI: Manage, evaluate, and improve your full-stack LLM application - with version control, type-safety, and local development built-in. | |
      • systemprompt.io
      • TreeScale - enhanced APIs seamlessly using tools for prompt optimization, semantic querying, version management, statistical evaluation, and performance tracking. As a part of the developer friendly API implementation TreeScale offers Elastic LLM product, which makes a unified API Endpoint for all major LLM providers and open source models. | |
  • AutoML

    • Profiling

      • TPOT - source software packages. | ![GitHub Badge](https://img.shields.io/github/stars/EpistasisLab/tpot.svg?style=flat-square) |
      • auto-sklearn - in replacement for a scikit-learn estimator. | ![GitHub Badge](https://img.shields.io/github/stars/automl/auto-sklearn.svg?style=flat-square) |
      • Goptuna - bata/goptuna.svg?style=flat-square) |
      • Hyperopt - square) |
      • FLAML - us/research/publication/flaml-a-fast-and-lightweight-automl-library/)). | ![GitHub Badge](https://img.shields.io/github/stars/microsoft/FLAML.svg?style=flat-square) |
      • Pycaret - source, low-code machine learning library in Python that automates machine learning workflows. | ![GitHub Badge](https://img.shields.io/github/stars/pycaret/pycaret.svg?style=flat-square) |
      • AutoRAG - Boost your LLM app performance with your own data | ![GitHub Badge](https://img.shields.io/github/stars/Marker-Inc-Korea/AutoRAG.svg?style=flat-square) |
      • autokeras - team/autokeras.svg?style=flat-square) |
      • Optuna - square) |
      • Determined - ai/determined.svg?style=flat-square) |
      • Model Search - square) |
      • Auto-PyTorch - PyTorch.svg?style=flat-square) |
      • automl-gs - gs.svg?style=flat-square) |
      • AutoGL - square) |
      • Torchmeta - Learning library for PyTorch. | ![GitHub Badge](https://img.shields.io/github/stars/tristandeleu/pytorch-meta.svg?style=flat-square) |
      • learn2learn - learning Framework for Researchers. | ![GitHub Badge](https://img.shields.io/github/stars/learnables/learn2learn.svg?style=flat-square) |
      • Keras Tuner - team/keras-tuner.svg?style=flat-square) |
      • Dragonfly - square) |
      • Archai - square) |
      • MOE - square) |
      • Hyperband - square) |
      • autoai - square) |
      • DEvol (DeepEvolution) - square) |
      • EvalML - square) |
      • FEDOT - itmo/FEDOT.svg?style=flat-square) |
      • HpBandSter - square) |
      • Hypernets - square) |
      • hyperunity - box hyperparameter optimisation. | ![GitHub Badge](https://img.shields.io/github/stars/gdikov/hypertunity.svg?style=flat-square) |
      • Intelli - square) |
      • Katib - native project for automated machine learning (AutoML). | ![GitHub Badge](https://img.shields.io/github/stars/kubeflow/katib.svg?style=flat-square) |
      • NASGym - of-concept OpenAI Gym environment for Neural Architecture Search (NAS). | ![GitHub Badge](https://img.shields.io/github/stars/gomerudo/nas-env.svg?style=flat-square) |
      • NNI - parameter tuning. | ![GitHub Badge](https://img.shields.io/github/stars/Microsoft/nni.svg?style=flat-square) |
      • REMBO - dimensions via random embedding. | ![GitHub Badge](https://img.shields.io/github/stars/ziyuw/rembo.svg?style=flat-square) |
      • RoBO - square) |
      • scikit-optimize(skopt) - based optimization with a `scipy.optimize` interface. | ![GitHub Badge](https://img.shields.io/github/stars/scikit-optimize/scikit-optimize.svg?style=flat-square) |
      • Spearmint - square) |
      • Vegas - noah/vega.svg?style=flat-square) |
      • AutoGluon - square) |
      • Ludwig - square) |
      • HPOlib2 - square) |
  • Observability

    • PromptHub - Full stack prompt management tool designed to be usable by technical and non-technical team members. Test, version, collaborate, deploy, and monitor, all from one place.
    • Prompteams - Prompt management system. Version, test, collaborate, and retrieve prompts through real-time APIs. Have GitHub style with repos, branches, and commits (and commit history).
    • Doku - An open-source LLM Observability platform streamlining the monitoring of LLM applications with just two lines of code. It provides valuable insights into token usage and user engagement, tracks API usage for providers like OpenAI, and facilitates easy data export to observability platforms like Grafana and DataDog.
  • Training

    • Visualization

      • Fiddler AI
      • netron - square) |
      • TensorBoard - square) |
      • TensorSpace - trained deep learning models from TensorFlow, Keras, TensorFlow.js. | ![GitHub Badge](https://img.shields.io/github/stars/tensorspace-team/tensorspace.svg?style=flat-square) |
      • Zetane Viewer - square) |
      • Maniford - agnostic visual debugging tool for machine learning. | ![GitHub Badge](https://img.shields.io/github/stars/uber/manifold.svg?style=flat-square) |
      • dtreeviz - square) |
      • OpenOps - square) |
      • Zeno - ml/zeno.svg?style=flat-square) |
      • OpenOps - square) |
    • Foundation Model Fine Tuning

      • Flyflow - devs/flyflow.svg?style=flat-square) |
      • alpaca-lora - tune LLaMA on consumer hardware | ![GitHub Badge](https://img.shields.io/github/stars/tloen/alpaca-lora.svg?style=flat-square) |
      • peft - of-the-art Parameter-Efficient Fine-Tuning. | ![GitHub Badge](https://img.shields.io/github/stars/huggingface/peft.svg?style=flat-square) |
      • TRL - square) |
      • p-tuning-v2 - tuning on small/medium-sized models and sequence tagging challenges. [(ACL 2022)](https://arxiv.org/abs/2110.07602) | ![GitHub Badge](https://img.shields.io/github/stars/THUDM/P-tuning-v2.svg?style=flat-square) |
      • QLoRA - bit finetuning task performance. | ![GitHub Badge](https://img.shields.io/github/stars/artidoro/qlora.svg?style=flat-square) |
      • LMFlow - square) |
      • Lora - rank adaptation to quickly fine-tune diffusion models. | ![GitHub Badge](https://img.shields.io/github/stars/cloneofsimo/lora.svg?style=flat-square) |
      • finetuning-scheduler - tuning schedules. | ![GitHub Badge](https://img.shields.io/github/stars/speediedan/finetuning-scheduler.svg?style=flat-square) |
    • Frameworks for Training

      • LightGBM - square) |
      • TensorFlow - square) |
      • Keras - team/keras.svg?style=flat-square) |
      • Horovod - square) |
      • scikit-learn - learn/scikit-learn.svg?style=flat-square) |
      • Apache MXNet - aware Dataflow Dep Scheduler. | ![GitHub Badge](https://img.shields.io/github/stars/apache/mxnet.svg?style=flat-square) |
      • Caffe - square) |
      • PyTorch - square) |
      • Kedro - source Python framework for creating reproducible, maintainable and modular data science code. | ![GitHub Badge](https://img.shields.io/github/stars/kedro-org/kedro.svg?style=flat-square) |
      • XGBoost - square) |
      • PaddlePaddle - square) |
      • ColossalAI - scale model training system with efficient parallelization techniques. | ![GitHub Badge](https://img.shields.io/github/stars/hpcaitech/ColossalAI.svg?style=flat-square) |
      • MindSpore - ai/mindspore.svg?style=flat-square) |
      • MegEngine - to-use deep learning framework, with auto-differentiation. | ![GitHub Badge](https://img.shields.io/github/stars/MegEngine/MegEngine.svg?style=flat-square) |
      • Oneflow - centered and open-source deep learning framework. | ![GitHub Badge](https://img.shields.io/github/stars/Oneflow-Inc/oneflow.svg?style=flat-square) |
      • Accelerate - GPU, TPU, mixed-precision. | ![GitHub Badge](https://img.shields.io/github/stars/huggingface/accelerate.svg?style=flat-square) |
      • Candle - square`) |
      • metric-learn - learn-contrib/metric-learn.svg?style=flat-square) |
      • VectorFlow - square) |
      • Jax - performance machine learning research. | ![GitHub Badge](https://img.shields.io/github/stars/google/jax.svg?style=flat-square) |
      • DeepSpeed - square) |
      • axolotl - tuning of various AI models, offering support for multiple configurations and architectures. | ![GitHub Badge](https://img.shields.io/github/stars/OpenAccess-AI-Collective/axolotl.svg?style=flat-square) |
    • IDEs and Workspaces

      • Docker - source project created by Docker to enable and accelerate software containerization. | ![GitHub Badge](https://img.shields.io/github/stars/moby/moby.svg?style=flat-square) |
      • code server - server.svg?style=flat-square) |
      • conda - agnostic, system-level binary package manager and ecosystem. | ![GitHub Badge](https://img.shields.io/github/stars/conda/conda.svg?style=flat-square) |
      • Kurtosis - container environments. | ![GitHub Badge](https://img.shields.io/github/stars/kurtosis-tech/kurtosis.svg?style=flat-square) |
      • Jupyter Notebooks - based notebook environment for interactive computing. | ![GitHub Badge](https://img.shields.io/github/stars/jupyter/notebook.svg?style=flat-square) |
      • envd - square) |
    • Experiment Tracking

      • Aim - to-use and performant open-source experiment tracker. | ![GitHub Badge](https://img.shields.io/github/stars/aimhubio/aim.svg?style=flat-square) |
      • Kedro-Viz - Viz is an interactive development tool for building data science pipelines with Kedro. Kedro-Viz also allows users to view and compare different runs in the Kedro project. | ![GitHub Badge](https://img.shields.io/github/stars/kedro-org/kedro-viz.svg?style=flat-square) |
      • Guild AI - square) |
      • LabNotebook - square) |
      • Sacred - square) |
    • Model Editing

  • ML Platforms

    • TrueFoundry - A PaaS to deploy, Fine-tune and serve LLM Models on a company’s own Infrastructure with Data Security and Optimal GPU and Cost Management. Launch your LLM Application at Production scale with best DevSecOps practices.
  • Awesome Lists

  • Serving

    • Frameworks/Servers for Serving

      • BentoML - square) |
      • TFServing - performance serving system for machine learning models. | ![GitHub Badge](https://img.shields.io/github/stars/tensorflow/serving.svg?style=flat-square) |
      • Triton Server (TRTIS) - inference-server/server.svg?style=flat-square) |
      • Xinference - source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop. | ![GitHub Badge](https://img.shields.io/github/stars/xorbitsai/inference.svg?style=flat-square) |
      • Torchserve - square) |
      • lanarky - grade LLM applications | ![GitHub Badge](https://img.shields.io/github/stars/ajndkr/lanarky.svg?style=flat-square) |
      • ray-llm - RayLLM | ![GitHub Badge](https://img.shields.io/github/stars/ray-project/ray-llm.svg?style=flat-square) |
      • langchain-serve - ai/langchain-serve.svg?style=flat-square) |
      • Mosec - to-use Python interface. | ![GitHub Badge](https://img.shields.io/github/stars/mosecorg/mosec?style=flat-square) |
      • KubeAI - to-text. | ![GitHub Badge](https://img.shields.io/github/stars/substratusai/kubeai.svg?style=flat-square) |
      • Kaito - 3) using container images and GPU auto-provisioning. Includes an OpenAI-compatible server for inference and preset configurations for popular runtimes such as vLLM and transformers. | ![GitHub Badge](https://img.shields.io/github/stars/kaito-project/kaito.svg?style=flat-square) |
      • Open Responses - source platform for building long-running LLM agents with tool use. | ![GitHub Badge](https://img.shields.io/github/stars/julep-ai/julep.svg?style=flat-square) |
      • Open Responses - source platform for building long-running LLM agents with tool use. | ![GitHub Badge](https://img.shields.io/github/stars/julep-ai/julep.svg?style=flat-square) |
      • Open Responses - source platform for building long-running LLM agents with tool use. | ![GitHub Badge](https://img.shields.io/github/stars/julep-ai/julep.svg?style=flat-square) |
      • Jina - ai/jina.svg?style=flat-square) |
    • Large Model Serving

      • whisper.cpp - square) |
      • text-generation-inference - generation-inference.svg?style=flat-square) |
      • Clip-as-a-service - ai/clip-as-service.svg?style=flat-square) |
      • text-embeddings-inference - embedding models | ![GitHub Badge](https://img.shields.io/github/stars/huggingface/text-embeddings-inference.svg?style=flat-square) |
      • Infinity - embeddings | ![GitHub Badge](https://img.shields.io/github/stars/michaelfeil/infinity.svg?style=flat-square) |
      • vllm - throughput and memory-efficient inference and serving engine for LLMs. | ![GitHub stars](https://img.shields.io/github/stars/vllm-project/vllm.svg?style=flat-square) |
      • TensorRT-LLM - LLM.svg?style=flat-square) |
      • Flowise - square) |
      • tokenizers - of-the-Art Tokenizers optimized for Research and Production | ![GitHub Badge](https://img.shields.io/github/stars/huggingface/tokenizers.svg?style=flat-square) |
      • CTranslate2 - square) |
      • Modelz-LLM - llm.svg?style=flat-square) |
      • x-stable-diffusion - time inference for Stable Diffusion - 0.88s latency. Covers AITemplate, nvFuser, TensorRT, FlashAttention. | ![GitHub Badge](https://img.shields.io/github/stars/stochasticai/x-stable-diffusion.svg?style=flat-square) |
      • DeepSpeed-MII - latency and high-throughput inference possible, powered by DeepSpeed. | ![GitHub Badge](https://img.shields.io/github/stars/microsoft/DeepSpeed-MII.svg?style=flat-square) |
      • prima.cpp - square) |
      • Shimmy - free Rust inference server with OpenAI API compatibility and hot model swapping | ![GitHub Badge](https://img.shields.io/github/stars/Michael-A-Kuykendall/shimmy.svg?style=flat-square) |
      • FlexGen - oriented scenarios. | ![GitHub Badge](https://img.shields.io/github/stars/FMInference/FlexGen.svg?style=flat-square) |
      • Ollama - square) |
      • llama.cpp - square) |
  • Data

    • Data Management

      • Quilt - organizing data hub for S3. | ![GitHub Badge](https://img.shields.io/github/stars/quiltdata/quilt.svg?style=flat-square) |
      • Dolt - square) |
      • Pachyderm - square) |
      • Delta-Lake - io/delta.svg?style=flat-square) |
      • ArtiVC - square) |
    • Data Storage

      • JuiceFS - square) |
      • LakeFS - like capabilities for your object storage. | ![GitHub Badge](https://img.shields.io/github/stars/treeverse/lakeFS.svg?style=flat-square) |
      • Lance - ai/lance.svg?style=flat-square) |
    • Data Tracking

      • LUX - org/lux.svg?style=flat-square) |
      • Piperider - square) |
    • Data/Feature enrichment

      • Feast - dev/feast.svg?style=flat-square) |
      • Upgini - to-use features from public and community shared data sources and enriches your training dataset with only the accuracy improving features | ![GitHub Badge](https://img.shields.io/github/stars/upgini/upgini.svg?style=flat-square) |
      • distilabel - quality outputs, full data ownership, and overall efficiency. | ![GitHub Badge](https://img.shields.io/github/stars/argilla-io/distilabel.svg?style=flat-square) |
      • FastDatasets - quality training datasets for Large Language Models. | ![GitHub Badge](https://img.shields.io/github/stars/ZhuLinsen/FastDatasets.svg?style=flat-square) |
    • Feature Engineering

  • Optimizations

  • Code AI

      • CodeT5 - square) |
      • Continue - source autopilot for software development—bring the power of ChatGPT to VS Code | ![GitHub Badge](https://img.shields.io/github/stars/continuedev/continue.svg?style=flat-square) |
      • tabby - hosted AI coding assistant. An opensource / on-prem alternative to GitHub Copilot. | ![GitHub Badge](https://img.shields.io/github/stars/TabbyML/tabby.svg?style=flat-square) |
      • fauxpilot - source alternative to GitHub Copilot server | ![GitHub Badge](https://img.shields.io/github/stars/fauxpilot/fauxpilot.svg?style=flat-square) |
      • CodeGeeX - square) |
      • CodeGen - source model for program synthesis. Trained on TPU-v4. Competitive with OpenAI Codex. | ![GitHub Badge](https://img.shields.io/github/stars/salesforce/CodeGen.svg?style=flat-square) |
      • promptext - square) |
  • Performance

    • ML Compiler

      • ONNX-MLIR - mlir.svg?style=flat-square) |
      • TVM - square) |
      • bitsandbytes - bit quantization for PyTorch. | ![GitHub Badge](https://img.shields.io/github/stars/bitsandbytes-foundation/bitsandbytes?style=flat-square) |
    • Profiling

      • scalene - performance, high-precision CPU, GPU, and memory profiler for Python | ![GitHub Badge](https://img.shields.io/github/stars/plasma-umass/scalene.svg?style=flat-square) |
      • octoml-profile - profile is a python library and cloud service designed to provide the simplest experience for assessing and optimizing the performance of PyTorch models on cloud hardware with state-of-the-art ML acceleration technology. | ![GitHub Badge](https://img.shields.io/github/stars/octoml/octoml-profile.svg?style=flat-square) |
  • Security

    • Observability

      • Great Expectations - expectations/great_expectations.svg?style=flat-square) |
      • Deepchecks - square) |
      • Traceloop OpenLLMetry - based observability and monitoring for LLM and agents workflows. | ![GitHub Badge](https://img.shields.io/github/stars/traceloop/openllmetry.svg?style=flat-square)
      • whylogs - square) |
      • Giskard - AI/giskard.svg?style=flat-square) |
      • Azure OpenAI Logger - openai-logger?style=flat-square) |
      • Fiddler AI - production to production. Ship more ML and LLMs into production, and monitor ML and LLM metrics like hallucination, PII, and toxicity. | ![GitHub Badge](https://img.shields.io/github/stars/fiddler-labs/fiddler-auditor.svg?style=flat-square) |
      • Maxim AI
    • Frameworks for LLM security

      • Plexiglass - labs/plexiglass?style=flat-square) |
      • Plexiglass - labs/plexiglass?style=flat-square) |
  • Federated ML

    • Profiling

      • FATE - square) |
      • Flower - square) |
      • FedML - scale cross-silo federated learning, cross-device federated learning on smartphones/IoTs, and research simulation. | ![GitHub Badge](https://img.shields.io/github/stars/FedML-AI/FedML.svg?style=flat-square) |
      • EasyFL - to-use Federated Learning Platform | ![GitHub Badge](https://img.shields.io/github/stars/EasyFL-AI/EasyFL.svg?style=flat-square) |
      • Harmonia - source project aiming at developing systems/infrastructures and libraries to ease the adoption of federated learning (abbreviated to FL) for researches and production usage. | ![GitHub Badge](https://img.shields.io/github/stars/ailabstw/harmonia.svg?style=flat-square) |