Inference

Inference Engine

• OpenVINO - commit/openvinotoolkit/openvino?color=green)
• Triton Inference Server - inference-server/server.svg?style=flat&color=green)
• Text Generation Inference - generation-inference.svg?style=flat&color=green)
• web-llm - performance In-browser LLM Inference Engine.
• DeepSpeed-MII - latency and high-throughput inference possible, powered by DeepSpeed.
• Petals - style. Fine-tuning and inference up to 10x faster than offloading

Inference Platform

• BentoML - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

Orchestration

Agent Framework

• CAMEL - agent framework. Finding the Scaling Law of Agents.

Runtime

Chatbot

• Gradio - app/gradio.svg?style=flat&color=green)

Database

• deeplake - time to PyTorch/TensorFlow.

Observation

• wandb - tune models, and manage models from experimentation to production.

Training

FineTune

• EasyLM - training, finetuning, evaluating and serving LLMs in JAX/Flax.
• LMFlow - commit/optimalscale/lmflow?color=green)

Framework

• ColossalAI - commit/hpcaitech/ColossalAI?color=green)
• Ludwig - code framework for building custom LLMs, neural networks, and other AI models.
• AXLearn - commit/apple/axlearn?color=green)

Workflow

• ZenML - io/zenml.svg?style=flat&color=green)
• Ray - project/ray.svg?style=flat&color=green)
• Polyaxon - commit/polyaxon/polyaxon?color=green)