Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/a16z-infra/llm-app-stack
https://github.com/a16z-infra/llm-app-stack
Last synced: 28 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/a16z-infra/llm-app-stack
- Owner: a16z-infra
- Created: 2023-07-27T00:31:17.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-07-26T13:43:18.000Z (5 months ago)
- Last Synced: 2024-11-09T20:38:02.032Z (about 1 month ago)
- Size: 177 KB
- Stars: 1,020
- Watchers: 26
- Forks: 98
- Open Issues: 19
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-repositories - a16z-infra/llm-app-stack - (Others)
- jimsghstars - a16z-infra/llm-app-stack - (Others)
README
# LLM App Stack
*aka Emerging Architectures for LLM Applications*
This is a list of available tools, projects, and vendors at each layer of the LLM app stack.
Our [original article](https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/) included only the most popular options, based on user interviews. This repo is meant to be more comprehensive, covering all available options in each category. We probably still missed some important projects, so please open a PR if you see anything missing.
We also included [Perplexity and Cursor.sh prompts](#formatting-prompt-templates) to make searching and markdown table formatting easier.
## Table of Contents
1. [Data Pipelines](#data-pipelines)
2. [Embedding Models](#embedding-models)
3. [Vector Databases](#vector-databases)
4. [Playgrounds](#playgrounds)
5. [Orchestrators](#orchestrators)
6. [APIs / Plugins](#apis--plugins)
7. [LLM Caches](#llm-caches)
8. [Logging / Monitoring / Eval](#logging--monitoring--eval)
9. [Validators](#validators)
10. [LLM APIs (proprietary)](#llm-apis-proprietary)
11. [LLM APIs (open source)](#llm-apis-open-source)
12. [App Hosting Platforms](#app-hosting-platforms)
13. [Cloud Providers](#cloud-providers)
14. [Opinionated Clouds](#opinionated-clouds)
## Project List
### Data Pipelines
| Name (site) | Description | Github | Pip Installs |
|------------|------------|------------|------------|
| [Databricks](https://databricks.com/) | A unified data platform for building, deploying, and maintaining enterprise data solutions, including products (like MosaicML and MLflow) purpose-built for AI | 
| 
|
| [Airflow](https://airflow.apache.org/) | A data pipeline framework to programmatically author, schedule, and monitor data pipelines and workflows, including for LLMs | 
| 
|
| [Unstructured.io](https://unstructured.io/) | Open-source components for pre-processing documents such as PDFs, HTML, and Word documents for usage with LLM apps | 
| 
|
| [Fivetran](https://www.fivetran.com/) | A platform that extracts, loads, and transforms data from various sources for analytics, AI, and operations | N/A | 
|
| [Airbyte](https://www.airbyte.com/) | An open-source data integration engine that helps consolidate data in data warehouses, lakes, and databases | 
| 
|
| [Anyscale](https://www.anyscale.com/) | An AI compute platform that allows developers to scale data ingest, preprocessing, embedding, and inference computations using Ray | 
| 
|
| [Alluxio](https://www.alluxio.io/) | An open-source data platform at the intersection of compute and storage, bringing data closer to compute, to accelerate model training and serving, boost GPU utilization, and reduce costs for AI workloads | 
| 
|
### Embedding Models
| Name (site) | Description | Github | Pip Installs |
|------------|------------|------------|------------|
| [OpenAI Ada Embedding 2](https://platform.openai.com/docs/guides/embeddings) | OpenAI's most popular embedding model for capturing semantic relationships in text | n/a | 
|
| [Cohere AI](https://docs.cohere.com/docs/embeddings) | An independent commerical provider of LLMs, with particular focus on embeddings for semantic search, topic clustering, and vertical applications | 
| 
|
| [Sentence Transformers](https://www.sbert.net/) | An open-source Python framework for sentence, text, and image embeddings | 
| 
|
### Vector Databases
| Name (site) | Description | Github | Pip Installs |
|------------|------------|------------|------------|
| [Pinecone](https://pinecone.io/) | A managed, cloud-native vector database with a simple API for high-performance AI applications | n/a | 
|
| [Weaviate](https://weaviate.io/) | An open-source vector database that stores both objects and vectors | 
| 
|
| [ChromaDB](https://trychroma.com/) | An AI-native, open-source embedding database platform for developers | 
| 
|
| [Pgvector](https://github.com/pgvector/pgvector) | An open-source vector similarity search for Postgres, allowing for exact and approximate nearest-neighbor search | 
| 
|
| [Zilliz (Milvus)](https://milvus.io/) | An open-source vector database, built for developing and maintaining AI applications | 
| 
|
| [Qdrant](https://qdrant.tech/) | A vector database and vector similarity search engine | 
| 
|
| [Metal io](https://getmetal.io/) | A managed service for developers to build applications with ML embeddings | N/A | 
|
| [LanceDB](https://lancedb.com/) | A serverless vector database for AI applications | 
| 
|
### Playgrounds
| Name (site) | Description | Github | Pip Installs |
|---------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------|
| [OpenAI Playground](https://platform.openai.com/) | A web-based platform for experimenting with various machine-learning models developed by OpenAI | N/A | N/A |
| [nat.dev](https://nat.dev) | A platform that allows users to test prompts with multiple language models and compare their performance | 
| 
|
| [Humanloop](https://humanloop.com/) | A platform that helps developers build applications on top of LLMs | 
| 
|
| [Parea AI](https://www.parea.ai/) | Platform and SDK for AI Engineers providing tools for LLM evaluation, observability, and a version-controlled enhanced prompt playground. | 
| 
|
### Orchestrators
| Name (site) | Description | Github | Pip Installs |
|------------|------------|------------|------------|
| [Langchain](https://langchain.com/) | An open-source library that gives developers the tools to build applications powered by LLMs | 
| 
|
| [LlamaIndex](https://llamaindex.ai/) | A data framework for LLM applications to ingest, structure, and access private or domain-specific data | 
| 
|
| [Autogen](https://microsoft.github.io/autogen/) | A framework for automating and streamlining LLM workflows using customizable, conversable agents for complex AI applications | 
| 
|
| [Microsoft Semantic Kernel](https://github.com/microsoft/semantic-kernel) | A lightweight open-source orchestration SDK | 
| 
|
| [Haystack](https://haystack.deepset.ai/) | LLM orchestration framework to build customizable, production-ready LLM applications | 
| 
|
| [Vercel AI SDK](https://sdk.vercel.ai/docs) | An open-source library for developers to build streaming UIs in JavaScript and TypeScript | 
| 
(node/npm)|
| [Vectara AI](https://vectara.com/) | A search and discovery platform for AI conversations utilizing your own data | 
| N/A |
| [ChatGPT](https://chat.openai.com) | An AI chatbot that uses natural language processing to create humanlike conversational dialogue | N/A| N/A |
### APIs / Plugins
| Name (site) | Description | Github | Pip Installs |
|------------|------------|------------|------------|
| [Serp API](https://serpapi.com/) | A real-time API to access Google search results, as well as handling proxies, solving captchas, and parsing structured data | 
| 
|
| [Wolfram Alpha API](https://wolframalpha.com/) | A web-based API providing computational and presentation capabilities for integration into various applications | N/A | 
|
| [Zapier API AI Plugin](https://zapier.com/) | A plugin that allows you to connect 5,000+ apps and interact with them directly inside ChatGPT | N/A | N/A |
### LLM Caches
| Name (site) | Description | Github | Pip Installs |
|------------|------------|------------|------------|
| [Redis](https://redis.io/) | An in-memory data structure store used as a database, cache, message broker, and streaming engine | 
| 
|
| [SQLite](https://sqlite.org/) | A self-contained, serverless, zero-configuration, transactional SQL database engine | 
| 
|
| [GPTCache](https://github.com/zilliztech/GPTCache) | An open-source tool for improving the efficiency and speed of GPT-based applications by implementing a cache to store the responses | 
| N/A |
### Logging / Monitoring / Eval
| Name (site) | Description | Github | Pip Installs |
|------------|------------|------------|------------|
| [Braintrust Data](https://www.braintrustdata.com/) | An AI product stack featuring evaluations, prompt playgrounds, continuous integration, dataset management, and access to various AI models through a single API | 
| 
|
| [Arize AI](https://arize.com/) | An observability platform for both LLMs and supervised ML | 
| 
|
| [Weights & Biases](https://wandb.ai) | An MLOps platform for streamlining ML workflows | 
| 
|
| [MLflow](https://mlflow.org/docs/latest/llms/index.html#) | A platform to streamline ML development | 
| 
|
| [PromptLayer](https://promptlayer.com/) | A platform for tracking, managing, and sharing LLM prompt engineering | 
| 
|
| [Helicone](https://helicone.ai/) | An open-source observability platform for LLMs | 
| 
|
| [Quotient AI](https://www.quotientai.co/) | Quotient AI is a platform for evaluating AI products on real-world use-cases, during research, development, and in production | N/A | N/A |
| [Portkey AI](https://portkey.ai/) | A platform to develop, launch, maintain, and iterate generative AI apps and features | N/A | N/A |
| [Freeplay AI](https://freeplay.ai/) | A platform to prototype, test, and optimize LLM features for customers | N/A | N/A |
| [Gentrace](https://gentrace.ai/) | An API and SDKs for evaluating and observing generative data, with features like AI, heuristic, and human grading evaluations, as well as production data observation | N/A | 
|
| [Patronus AI](https://www.patronus.ai/) | An automated evaluation and benchmarking platform for LLMs, providing tools for testing, scoring, and evaluating LLMs in real-world scenarios | N/A | N/A |
| [Autoblocks AI](https://www.autoblocks.ai/) | A collaborative cloud-based workspace designed for rapid iteration on GenAI products, offering features like prompt management, observability, continuous evaluations, fine-tuning, prototyping, debugging, and scalable data ingestion & search, all in a provider-agnostic environment | N/A | 
|
| [Context AI](https://context.ai/) | Tools for pre-launch LLM evaluations and post-launch analytics, with features such as testing, performance monitoring, user conversation analysis, and support for various models and libraries | N/A | 
|
| [E2b dev](https://e2b.dev/) | Services to deploy, test, and monitor AI agents, including a sandbox with a secure, long-running cloud environment for various LLMs with features like internet access | 
| 
|
| [Agentops](https://www.agentops.ai/) | Toolkit for evaluating and developing AI agents, providing tools for agent development, monitoring capabilities, and replay analytics | 
| 
|
| [Zenoml](https://zenoml.com/) | AI evaluation platform that enables data visualization, model performance analysis, and the creation of interactive reports for various data types | 
| 
|
| [Baserun](https://baserun.ai/welcome) | Tools for model configuration, prompt playground, monitoring, and prototype workflow, as well as features for full visibility into LLM workflows and end-to-end testing | 
| 
|
| [WhyLabs](https://whylabs.ai/) | AI Observability platform for ML and GenAI including LLM monitoring, guardrails and security | 
| 
|
| [Log10](https://log10.io/) | AI-powered LLMOps platform that automatically optimizes prompts and models with built-in logging, debugging, metrics, feedback, evaluations and fine-tuning | 
| 
|
| [promptfoo](https://www.promptfoo.dev/) | Open-source LLM eval framework with support for model/prompt/RAG eval, dataset generation, local models, and self-hosting. | 
| 
(node/npm) |
| [Parea AI](https://www.parea.ai/) | Platform and SDK for AI Engineers providing tools for LLM evaluation, observability, and a version-controlled enhanced prompt playground. | | |
| [Galileo](https://www.rungalileo.io/) | Galileo is a platform for evaluation, fine-tuning and real-time observability, powered by high-accuracy hallucination guardrails. | N/A | N/A |
### Validators
| Name (site) | Description | Github | Pip Installs |
|------------|------------|------------|------------|
| [Guardrails AI](https://shreyar.github.io/guardrails/) | An open-source Python package for specifying structure and type, validating, and correcting the outputs of LLMs | 
| 
|
| [Rebuff](https://github.com/woop/rebuff) | An open-source framework designed to detect and protect against prompt injection attacks in LLM apps | 
| 
|
| [Microsoft Guidance](https://github.com/microsoft/guidance) | A guidance language for controlling LLMs, providing a syntax for architecting LLM workflows | 
| 
|
| [LMQL](https://lmql.ai/) | An open-source programming language and platform for language model interaction | 
| 
|
| [Outlines](https://outlines-dev.github.io/outlines/) | A tool for helping developers guide text generation to build robust interfaces with external systems and guarantee that outputs match a regex or JSON schema | 
| 
|
| [LLM Guard](https://github.com/laiyer-ai/llm-guard) | An open-source, comprehensive tool designed to fortify the security of Large Language Models (LLMs). | 
| 
|
### LLM APIs (proprietary)
| Name (site) | Description | Github | Pip Installs |
|------------|------------|------------|------------|
| [OpenAI](https://openai.com) | A company providing many leading LLMs, including the GPT-3.5 and GPT-4 families | N/A | |
| [Anthropic](https://anthropic.com) | The developer of Claude, an AI assistant based on Anthropic’s research | N/A | 
|
| [Cohere AI](https://docs.cohere.com/docs/embeddings) | An LLM vendor with particular focus on embeddings for semantic search, topic clustering, and vertical applications | | |
| [LLM](https://llm.datasette.io/en/stable/) | A CLI utility and Python library for interacting with Large Language Models, both via remote APIs and models that can be installed and run on your own machine. | 
| 
|
### LLM APIs (open source)
| Name (site) | Description | Github | Pip Installs |
|------------|------------|------------|------------|
| [Hugging Face](https://huggingface.co/) | A hub for open-source AI models and inference endpoints, including leading base LLMs and LoRAs/fine-tunes | 
| 
|
| [Replicate](https://replicate.com/) | An AI hosting platform and model inference hub that allows software developers to integrate AI models into their apps | 
| 
|
| [Anyscale](https://www.anyscale.com/) | An AI API and compute platform that allows developers to scale inference, training, and embedding computations with any model using Ray | | |
| [Ollama](https://ollama.ai/) | Get up and running with large language models locally | 
| 
|
| [GPT4ALL](https://gpt4all.io/index.html) | An ecosystem of open-source on-edge large language models. | 
| 
|
### App Hosting Platforms
| Name (site) | Description |
|------------|------------|
| [Vercel](https://vercel.com/) | A cloud platform designed for front-end engineers, built with first-class support for LLM apps |
| [Netlify](https://www.netlify.com/) | An enterprise cloud computing company that offers a development platform for web applications and dynamic websites |
| [Steamship](https://steamship.com/) | An SDK and hosting platform for AI agents and tools, both a package manager and package hosting service for AI |
| [Streamlit](https://streamlit.io/) | An open-source Python library designed for creating and sharing custom web apps for ML and data science |
| [Modal](https://modal.com) | A platform that enables running distributed applications using the modal Python package |
### Cloud Providers
| Name (site) | Description |
|------------|------------|
| [Amazon Web Services](https://aws.amazon.com/) | A cloud computing platform, offering services from data centers globally |
| [Google Cloud Platform](https://cloud.google.com/) | A cloud computing platform, offering services from data centers globally |
| [Microsoft Azure](https://azure.microsoft.com/) | A cloud computing platform, offering services from data centers globally |
| [CoreWeave](https://coreweave.com/) | A specialized cloud provider that delivers GPUs on top of flexible deployment infrastructure |
### Opinionated Clouds
| Name (site) | Description |
|------------|------------|
| [Databricks (MosaicML)](https://databricks.com/) | Databricks acquired Mosaic ML in 2023, along with its tooling and platform for efficient pre-trainining, fine-tuning and inferencing LLMs |
| [Anyscale](https://anyscale.com/) | An AI compute platform that enables developers to scale inference, training, and embedding computations with any model using Ray |
| [Modal](https://modal.com) | A platform that eables running distributed applications using the Modal Python package |
| [Runpod](https://runpod.io/) | A cloud computing platform designed for AI and ML applications |
| [OctoML](https://octoml.ai/) | A compute service that allows users to run, tune, and scale generative models |
| [Baseten](https://baseten.co/) | A inference service that allows users to deploy, serve, and scale custom and open-source models |
| [E2B](https://github.com/e2b-dev/e2b) | Secure sandboxed cloud environments made for AI agents and AI apps |
## Formatting Prompt Templates
We were able to partialy automate this - particularly finding Github and PyPI links - using this [Perplexity search prompt](https://github.com/a16z-infra/llm-app-stack/blob/main/table_construction_prompts/prompt_1_search.txt). It worked roughly ~75% of the time and could handle ~3 projects at a time, pulling data from 20-30 sources in each iteration.
Once you have the data you would like to add, if you don't want deal with the markdown formatting here, it is easy to correctly format using a tool like [Cursor](https://cursor.sh/).
See the prompt below that works as an inline edit, just make sure you highlight 4-5 previous examples so Cursor can infer the format itself:
