Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

awesome-private-ai

Curated list of tools, frameworks, and resources for running, building, and deploying AI privately — on-prem, air-gapped, or self-hosted.
https://github.com/tdi/awesome-private-ai

Last synced: 26 days ago
JSON representation

Agents & Orchestration

  • AG2 - Open-source operating system for agentic AI with native Ollama support for local model deployment and multi-agent collaboration.
  • LangChain - Agent and LLM orchestration framework.
  • Langflow - Visual workflow builder for creating and deploying AI-powered agents and workflows with built-in API servers.
  • Flowise - No-code LangChain UI.
  • Trae Agent - Privacy-friendly agent framework for orchestrating LLMs and tools, designed for secure, local, and scalable AI workflows.
  • Qwen-Agent - Open-source, privacy-friendly agent framework for orchestrating LLMs and tools, designed for secure, local, and scalable AI workflows.
  • Crush - Privacy-first, open-source agentic coding and automation platform for local AI workflows.
  • OpenCode AI - Open-source agentic coding platform for private, local, and secure AI-powered development workflows.
  • PydanticAI - Python agent framework by the Pydantic team, model-agnostic with Ollama support for local deployment.
  • dspy - Modular, open-source agent framework for building composable, private LLM applications and workflows.
  • CUA - enables AI agents to control full operating systems in virtual containers and deploy them locally or to the cloud.
  • Bytebot - A desktop agent is an AI that has its own computer. Unlike browser-only agents or traditional RPA tools, Bytebot comes with a full virtual desktop.
  • agentgateway - Gateway for managing and orchestrating AI agents with support for local deployment.
  • MetaGPT - Multi-agent framework for building collaborative AI systems with role-based agents that can work together on complex tasks.
  • Herdr - Self-hosted runtime and terminal multiplexer for coding agents. Runs the agent CLIs on your own machine or a box you control, grouping them into workspaces you can detach from and reattach to over SSH.
  • Aider - Terminal-based pair programming agent that edits code in your local git repo, with support for local models via Ollama and OpenAI-compatible servers.
  • MFS - Exposes your code, docs, chat (Slack/Gmail/Jira), databases and object stores as one file-like, searchable namespace for agents (ls/cat/grep + semantic search); runs fully local with on-device ONNX embeddings on Milvus, no API key.
  • DeepCode - Open agentic coding framework that turns papers and specs into working code (Paper2Code, Text2Web, Text2Backend), running against local Ollama or vLLM backends.
  • Skales - Source-available (BSL 1.1) local-first desktop AI agent that runs fully on-device, offline via Ollama or with 15+ providers; your files never leave your machine, no cloud required.
  • Orkas - Open-source, local-first desktop AI workforce whose Commander coordinates specialist agents through one chat; model calls can use a compatible local endpoint.
  • Haystack - End-to-end RAG pipelines.
  • LlamaIndex - Data framework for LLM apps.

AI Routers & API Aggregators

  • Nexus - Open-source AI router to aggregate Model Context Protocol (MCP) servers, intelligently route requests to the best LLMs, and provide security, governance, observability, and simplified architecture for private AI deployments. Blog
  • LiteLLM - Self-hosted proxy exposing 100+ model backends — including Ollama, vLLM, and any OpenAI-compatible server — behind one API, with keys, budgets, routing, and logging.
  • Bifrost - Self-hosted, open-source AI gateway for multi-provider routing, failover, observability, guardrails, and budget controls.

Datasets & Data Prep

  • OpenWebText - Open dataset similar to GPT training data.
  • Docling - Parse PDF, DOCX, PPTX, and HTML into structured formats for RAG and training, running entirely on your own hardware.
  • MinerU - High-quality PDF-to-Markdown and JSON extraction, including formulas and tables, for building private corpora.
  • Marker - Fast, accurate document-to-Markdown conversion across PDFs, images, and office formats.
  • Unstructured - Ingestion and preprocessing library for turning messy documents into model-ready chunks.
  • Label Studio - Self-hosted data labelling platform for text, image, audio, and video annotation.
  • Argilla - Collaborative tool for curating, annotating, and quality-checking datasets for fine-tuning and evaluation.

Fine-Tuning & Adapters

  • LoRA - Low-rank adaptation technique.
  • PEFT - Parameter-efficient fine-tuning.
  • QLoRA - Memory-efficient LoRA on quantized models.
  • LLaMA-Factory - Unified fine-tuning for 100+ models with a web UI, covering SFT, DPO, PPO, and reward modelling on your own hardware.
  • Unsloth - Fine-tune and reinforcement-train LLMs 2x faster with substantially less VRAM, on a single local GPU.
  • Axolotl - YAML-driven post-training framework covering full fine-tunes, LoRA, QLoRA, and multi-GPU sharding.
  • TRL - Hugging Face's library for SFT, DPO, GRPO, and reward-model training on top of transformers.
  • ms-swift - Training and deployment toolkit covering 500+ LLMs and 200+ multimodal models, from PEFT through to quantized export.
  • torchtune - Native PyTorch library for fine-tuning and experimenting with LLMs, with readable single-file recipes.

Image & Video Generation

  • ComfyUI - Node-based interface and backend for diffusion models, running image, video, and audio pipelines entirely locally.
  • AUTOMATIC1111 Stable Diffusion WebUI - The most widely deployed self-hosted Stable Diffusion interface, with a large extension ecosystem.
  • InvokeAI - Professional-grade local generative image toolkit with a unified canvas and workflow editor.

Inference Runtimes & Backends

  • vLLM - High-throughput, low-latency inference engine for LLMs.
  • mlx-lm - Fast, Apple Silicon-optimized LLM inference engine for running models locally and privately.
  • LM Studio - Cross-platform desktop app for running local LLMs with an easy-to-use interface.
  • Cherry Studio - Powerful and customizable cross-platform desktop app for LLM inference with built in web search, RAG, MCP support, and a quick assistant hotkey to summon your LLM from anywhere. Supports a wide variety of providers and OpenAI compatible endpoints for local inference.
  • LLM-D - Privacy-first, distributed LLM inference engine for scalable, local deployments.
  • llama.cpp - Portable, CPU/GPU-friendly LLM inference, good for GPU + CPU hybrid inference.
  • ik_llama.cpp - Fork of llama.cpp with bleeding edge feature implementations and quantization improvements.
  • text-generation-inference - Optimized serving stack from Hugging Face.
  • exo - Run your own AI cluster at home with everyday devices. Dynamic model partitioning across multiple devices like iPhones, Macs, and Linux machines.
  • exllama3 - An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs. Use TabbyAPI for an API server.
  • tabbyAPI - Official API server for running exllamav2 and exllamav3 models. Aims to be a friendly backend with high customizablity and an idiotmatic OAI compatible API for users.
  • llama-swap - Model swapping for llama.cpp (or any local OpenAPI compatible server).
  • sglang - Fast serving engine for LLMs and vision-language models, with RadixAttention prefix caching and a structured generation language.
  • Jan - Privacy-first, offline AI assistant and LLM runtime for local, secure inference.
  • oMLX - macOS-native MLX inference server with paged SSD KV caching and continuous batching. Serves LLM, VLM, embedding, and reranker models over OpenAI- and Anthropic-compatible endpoints for local coding agents on Apple Silicon.
  • llamafile - Distribute and run an entire LLM as a single executable file that works across six operating systems, with no install step.
  • LocalAI - Drop-in OpenAI-compatible API for local inference across text, image, audio, and embedding models, with no GPU required.
  • KTransformers - Optimized framework for running very large MoE models on limited hardware via GPU/CPU offloading and kernel injection.
  • MLC LLM - Compiler and runtime that deploys LLMs natively to GPUs, phones, and browsers via machine learning compilation.
  • RamaLama - Runs models as OCI containers, pulling from registries you control — a good fit for air-gapped and enterprise workflows.
  • YALS (Yet another llamacpp server) - TabbyAPI's sister project, adapted for llama.cpp and GGUF models. Built from the ground up using libllama instead of wrapping llama-server.
  • Ollama - Local LLM runner with model packaging. Uses llama.cpp backend to serve cautious model defaults.

Learning Resources & Research

  • LLMs from Scratch - Build a GPT-style model step by step in PyTorch, on your own machine.
  • ML Engineering Open Book - Field-tested notes on training and serving large models: hardware, parallelism, throughput, and debugging.
  • LLM Course - Roadmap and notebooks covering LLM fundamentals, fine-tuning, quantization, and deployment.
  • Smol Course - Hugging Face's practical course on aligning and fine-tuning small models that fit on local hardware.

Model Management & Serving

  • Ray Serve - Scalable Python model serving.
  • Seldon Core - Kubernetes-native model deployment.
  • KServe - Serverless model inference on Kubernetes.
  • BentoML - Model packaging & serving framework.
  • vLLM Production Stack - End-to-end stack for deploying vLLM in production, including orchestration, monitoring, autoscaling, and best practices for private LLM serving.
  • Triton Inference Server - NVIDIA's multi-framework inference server, supporting TensorRT, PyTorch, ONNX, and vLLM backends behind one endpoint.
  • Xinference - Serve and manage LLM, embedding, rerank, image, and audio models in one self-hosted cluster with an OpenAI-compatible API.
  • OME (Open Model Engine) - Unified, open-source engine for serving, managing, and scaling LLMs and multimodal models privately. Supports sglang, vLLM, and more.

Models for Private Deployment

  • Qwen 3 - A wide variety of general and specialized models in both dense and "Mixture of Experts" formats.
  • Kimi K2 - Mixture-of-Experts model with 32 billion activated parameters and 1 trillion total parameters.
  • Phi-4 - Small, high-quality models from Microsoft.
  • Gemma3 - Open source model from Google.
  • MLX Community - Community-driven Hugging Face page for open MLX models, optimized for Apple Silicon and private deployment.
  • Bielik - An open source project that provides data, tools and LLMs for the development of the Polish artificial intelligence landscape
  • LLaMA 3 - Meta’s open-weight language model.
  • Mistral 7B - Dense 7B parameter model.
  • Mixtral - Mixture-of-experts model.
  • Falcon - Open-source model from TII.

Observability & Evaluation

  • Langfuse - Self-hostable LLM observability, tracing, prompt management, and evaluation.
  • promptfoo - Local-first evaluation and red-teaming for prompts and models, runnable entirely offline in CI.
  • DeepEval - Unit-testing framework for LLM outputs, with metrics that can run against locally hosted judge models.
  • lm-evaluation-harness - The standard harness for benchmarking language models, supporting local vLLM and Hugging Face backends.
  • Phoenix - Self-hosted tracing, evaluation, and experiment tracking for LLM applications.
  • OrcaReplay - Records an agent at the HTTP boundary and serves the recording back, so a run reproduces with the model server switched off; traces are files in the project and nothing is uploaded.

Privacy, Security & Governance

  • BlindAI - Confidential AI inference using TEEs.
  • Concrete - Fully homomorphic encryption for AI.
  • OpenFL - Federated learning framework.
  • Presidio - Detect and de-identify PII in text and images before it ever reaches a model.
  • garak - LLM vulnerability scanner that probes local models for prompt injection, jailbreaks, and data leakage.
  • NeMo Guardrails - Add programmable topical and safety rails to LLM applications, running alongside self-hosted models.
  • LLM Guard - Input and output scanning toolkit covering prompt injection, PII, toxicity, and secrets leakage.

Quantization & Compression

  • llm-compressor - Apply GPTQ, SmoothQuant, SparseGPT, and FP8/INT4 weight-activation quantization, exporting straight to vLLM.
  • bitsandbytes - 8-bit and 4-bit quantization primitives that underpin QLoRA and much of the local fine-tuning ecosystem.
  • GPTQModel - Actively maintained GPTQ toolkit for producing and running 4-bit quantized models.

Speech & Audio

  • whisper.cpp - C++ port of OpenAI's Whisper automatic speech recognition model, optimized for local, CPU/GPU inference without internet connectivity.
  • Whisper - The original open-weight speech recognition model, runnable fully offline.
  • WhisperX - Whisper with word-level timestamps, speaker diarization, and much faster batched transcription.
  • RealtimeSTT - Low-latency local speech-to-text with voice activity detection, for building private voice interfaces.
  • F5-TTS - Local text-to-speech with voice cloning from a short reference sample.
  • speaches - Self-hosted OpenAI-compatible server for transcription, translation, and speech generation.

UI & Interaction Layers

  • Chatbot UI - Open-source ChatGPT clone.
  • LibreChat - Enhanced web UI for LLMs.
  • AnythingLLM - Full-stack private LLM workspace.
  • Open WebUI - Commonly recommended Web UI frontend which features built in search, web scrape, RAG, and optional user authentication.
  • Lobe Chat - Modern self-hosted chat framework with plugin and multimodal support, deployable in one click against local backends.
  • text-generation-webui - Long-standing Gradio web UI for local text generation, supporting llama.cpp, ExLlama, and transformers backends.
  • SillyTavern - Self-hosted, highly customizable frontend for local models, with extensive character, prompt, and context management.
  • Screenpipe - Source-available screen/audio history with local search and MCP; supports local Whisper transcription and Ollama, while configured cloud AI, transcription, sync, or integrations can send context off-device.

Vector Databases & Embeddings

  • Qdrant - High-performance Vector Database and Vector Search Engine.
  • Chroma - Local-first vector database.
  • FAISS - Facebook AI Similarity Search.
  • Milvus - Scalable vector database.
  • pgvector - Vector similarity search inside PostgreSQL, keeping embeddings in the database you already self-host.
  • LanceDB - Embedded, serverless vector database that stores vectors and metadata as files on your own disk or object store.
  • text-embeddings-inference - Fast local serving for embedding and reranker models, so retrieval never leaves your network.
  • Weaviate - Open-source semantic search engine.

VS Code Plugins & Extensions

  • Roo Code - Privacy-first, open-source agentic coding platform for secure, local AI development (VS Code extension).
  • cline - Privacy-first, open-source agentic coding platform for local AI workflows and automation (VS Code extension).
  • Continue - Open-source autocomplete and chat assistant for VS Code and JetBrains, configurable against Ollama, llama.cpp, vLLM, and other local endpoints.
  • Tabby - Self-hosted AI coding assistant with its own inference server, offering a private alternative to hosted completion services.