https://github.com/Arize-ai/phoenix
AI Observability & Evaluation
https://github.com/Arize-ai/phoenix
agents ai-monitoring ai-observability aiengineering anthropic datasets evals langchain llamaindex llm-eval llm-evaluation llmops llms openai prompt-engineering smolagents
Last synced: about 1 year ago
JSON representation
AI Observability & Evaluation
- Host: GitHub
- URL: https://github.com/Arize-ai/phoenix
- Owner: Arize-ai
- License: other
- Created: 2022-11-09T23:44:35.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2025-02-22T03:19:49.000Z (about 1 year ago)
- Last Synced: 2025-02-23T09:31:38.133Z (about 1 year ago)
- Topics: agents, ai-monitoring, ai-observability, aiengineering, anthropic, datasets, evals, langchain, llamaindex, llm-eval, llm-evaluation, llmops, llms, openai, prompt-engineering, smolagents
- Language: Jupyter Notebook
- Homepage: https://docs.arize.com/phoenix
- Size: 261 MB
- Stars: 4,803
- Watchers: 33
- Forks: 352
- Open Issues: 287
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
- free-for-dev - Phoenix - Open source platform for tracing, model evaluators, experiments for AI agents from the team at Arize AI. All features open source under elastic License 2.0 (ELv2). (Generative AI)
- awesome-llm-interpretability - Phoenix - AI Observability & Evaluation - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (Table of Contents / LLM Interpretability Tools)
- Awesome-RAG-Production - Arize Phoenix
- awesome-llmops - Arize-Phoenix - ai/phoenix.svg?style=flat-square) | (LLMOps / Observability)
- awesome-llm-tools - Arize Phoenix
- awesome-llm-eval - Arize-Phoenix - ai/phoenix.svg?style=social) | 用于LLMs、视觉、语言和表格模型的ML可观测性。 | (LLMOps / Popular-LLM)
- awesome-ai-engineering-reads - AI Observability & Evaluation - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook.
- AwesomeResponsibleAI - phoenix
- awesome-production-machine-learning - Phoenix - ai/phoenix.svg?style=social) - Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. (Evaluation and Monitoring)
- StarryDivineSky - Arize-ai/phoenix
- awesome-ai-research-tools - Arize Phoenix - source observability and evaluation toolkit. (Evaluation & Benchmarking)
- awesome-oss-saas - Arize-Phoenix - ai/phoenix><img src="https://img.shields.io/github/stars/arize-ai/phoenix?style=social" width=150></img></a> | [Arize](https://arize.com/) | [Docs](https://docs.arize.com/phoenix/), [Twitter](https://twitter.com/ArizePhoenix) | (Data Validation)
- Awesome-Prompt-Engineering - [Github
- awesome-github-repos - Arize-ai/phoenix - AI Observability & Evaluation (Python)
- awesome-mistral - Phoenix
- awesome-mcp - Arize-ai/phoenix - Phoenix is an open-source AI observability platform for experimentation, evaluation, and troubleshooting of large language model applications, supporting multiple frameworks and LLM providers with flexible deployment options. (MCP Frameworks and libraries / Python)
- Awesome-LLM-RAG-Application - phoenix - ai/phoenix) (开源工具 / 可观测性)
- awesome-agents - Arize Phoenix - Open-source AI observability tool for monitoring and evaluating LLM applications in real time. (Monitoring and Observability)
- awesome-safety-critical-ai - `Arize-ai/phoenix` - source AI observability platform designed for experimentation, evaluation, and troubleshooting (<a id="tools"></a>🛠️ Tools / Bleeding Edge ⚗️)
- awesome-ai-agents-2026 - Arize Phoenix
- metorial-index - Phoenix - Provides advanced observability and evaluation tools for AI applications, facilitating performance tracking, dataset management, and prompt engineering. Integrates with various frameworks and large language model providers for seamless operation. (Monitoring and Logging)
- awesome-hacking-lists - Arize-ai/phoenix - AI Observability & Evaluation (Jupyter Notebook)
- awesome_ai_agents - Phoenix - Open source tool for testing changes in AI agent or application (Building / Testing)
- Awesome-AI-Evaluation-Guide - Arize Phoenix - OpenTelemetry-based observability with embedding visualization (Tools & Platforms / Open Source Frameworks)
- awesome-ai-benchmarks-evaluation - Arize Phoenix - source toolkit for LLM/RAG evals and trace analysis. (Evaluation Frameworks)
- awesome-opensource-ai - Phoenix (Arize) - ai/phoenix?style=social) - AI observability & evaluation platform. (📋 Contents / 📊 8. MLOps / LLMOps & Production)
- awesome-nlp-llm-spanish-espa-ol - Phoenix
- awesome-ai-agents - Arize-ai/phoenix - Phoenix is an open-source AI observability platform that enables tracing, evaluation, and optimization of large language model applications across various deployment environments. (Agent Integration & Deployment Tools / AI Agent Gateway)
- jimsghstars - Arize-ai/phoenix - AI Observability & Evaluation (Jupyter Notebook)
- awesome-agents - Arize-Phoenix - Phoenix is an open source library for agent testing, evaluation and observability.  (Testing and Evaluation)
- awesome-llms-fine-tuning - Phoenix - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (1596 stars) (GitHub projects)
- awesome-production-llm - phoenix - ai/phoenix.svg?style=social) AI Observability & Evaluation (LLM Testing / Monitoring)
- awesome-ai-eval - **Arize Phoenix** - ai/phoenix?style=social&label=github.com) - OpenTelemetry-native observability and evaluation toolkit for RAG, LLMs, and agents. (Platforms / Open Source Platforms)
- awesome-open-data-centric-ai - Arize-Phoenix - Phoenix is a Python library for ML observability (monitoring + root-cause analysis) for tabular, CV, NLP, and LLM models. |  | <a href="https://github.com/Arize-ai/phoenix/blob/main/LICENSE"><img src="https://img.shields.io/github/license/Arize-AI/phoenix" height="15"/></a> | (Observability and Monitoring)
- awesome-llm-tools - Phoenix - source AI observability platform for tracing, evaluating, and monitoring LLM apps |  | Apache-2.0 | (12. Miscellaneous / Data & Alignment Tools)
- llmops - Phoenix - ai/phoenix?style=flat-square) | (Observability & Monitoring / Resources)
- awesome-agent-cortex - Phoenix - Open-source AI observability platform from Arize. (Agent Observability and Testing / Benchmark Reality Check (real-world tool use))
- awesome-data-analysis - Phoenix - AI observability platform. Tracing, datasets, experiments, and playground for troubleshooting and evaluating LLM apps. (🧠 AI Applications & Platforms / Tools)
- awesome-agent-collaborate-tools - Arize Phoenix - Open-source LLM observability. Trace agent runs, visualize spans, and evaluate outputs locally or in the cloud. Built on OpenTelemetry. (Observability & Debugging)
- awesome-harness-engineering - Arize-ai/phoenix
- awesome-ai-agents - Arize Phoenix
- awesome-rag - Arize Phoenix - Open-source observability platform (Evaluation Metrics and Benchmarks / Comparison Guides)
- awesome-observability - Arize Phoenix - Open-source AI observability platform for tracing, evaluation, datasets, experiments, prompt management and playground. Built on OpenTelemetry with Python and TypeScript support. (10. LLM & AI Observability / Platforms)
- awesome-ai-engineering - Phoenix
- awesome-LLMs-finetuning - Phoenix - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (1596 stars) (4. Fine-Tuning / Frameworks)
README
Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. It provides:
- [**_Tracing_**](https://docs.arize.com/phoenix/tracing/integrations-tracing/llm-traces) - Trace your LLM application's runtime using OpenTelemetry-based instrumentation.
- [**_Evaluation_**](https://docs.arize.com/phoenix/evaluation/llm-evals) - Leverage LLMs to benchmark your application's performance using response and retrieval evals.
- [**_Datasets_**](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets) - Create versioned datasets of examples for experimentation, evaluation, and fine-tuning.
- [**_Experiments_**](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets#experiments) - Track and evaluate changes to prompts, LLMs, and retrieval.
Phoenix is vendor and language agnostic with out-of-the-box support for popular frameworks (🦙[LlamaIndex](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/llamaindex), 🦜⛓[LangChain](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/langchain), [Haystack](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/haystack), 🧩[DSPy](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/dspy)) and LLM providers ([OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/openai), [Bedrock](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/bedrock), [MistralAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/mistralai), [VertexAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/vertexai), [LiteLLM](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/litellm), and more). For details on auto-instrumentation, check out the [OpenInference](https://github.com/Arize-ai/openinference) project.
Phoenix runs practically anywhere, including your Jupyter notebook, local machine, containerized deployment, or in the cloud.
## Installation
Install Phoenix via `pip` or `conda`
```shell
pip install arize-phoenix
```
Phoenix container images are available via [Docker Hub](https://hub.docker.com/r/arizephoenix/phoenix) and can be deployed using Docker or Kubernetes.
## Features
| Key Features | Availability |
| ---------------------------------------------------------------------------------------------------------------- | -------------- |
| [Tracing](https://docs.arize.com/phoenix/tracing/concepts-tracing/what-are-traces) | ✅ |
| [Evaluation](https://docs.arize.com/phoenix/evaluation/llm-evals) | ✅ |
| [Retrieval (RAG) Analysis](https://docs.arize.com/phoenix/tracing/use-cases-tracing/rag-evaluation) | ✅ |
| [Datasets](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets) | ✅ |
| [Fine-Tuning Export](https://docs.arize.com/phoenix/datasets-and-experiments/how-to-datasets/exporting-datasets) | ✅ |
| [Annotations](https://docs.arize.com/phoenix/tracing/concepts-tracing/how-to-annotate-traces) | ✅ |
| [Human Feedback](https://docs.arize.com/phoenix/tracing/how-to-tracing/capture-feedback) | ✅ |
| [Experiments](https://docs.arize.com/phoenix/datasets-and-experiments/how-to-experiments/run-experiments) | ✅ |
| [Embeddings Analysis](https://docs.arize.com/phoenix/inferences/phoenix-inferences) | ✅ |
| [Data Export](https://docs.arize.com/phoenix/tracing/how-to-tracing/extract-data-from-spans) | ✅ |
| REST API | ✅ |
| GraphQL API | ✅ |
| Data Retention | Customizable |
| Authentication | ✅ |
| Social Login | ✅ |
| RBAC | ✅ |
| Projects | ✅ |
| [Self-Hosting](https://docs.arize.com/phoenix/deployment) | ✅ |
| Jupyter Notebooks | ✅ |
| [Sessions](https://github.com/Arize-ai/phoenix/issues/2619) | In Progress 🚧 |
| [Prompt Playground](https://github.com/Arize-ai/phoenix/issues/3435) | In Progress 🚧 |
| Prompt Management | Coming soon ⏱️ |
## Tracing Integrations
Phoenix is built on top of OpenTelemetry and is vendor, language, and framework agnostic.
**Python**
| Integration | Package | Version Badge |
|------------------|-----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| [OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/openai) | `openinference-instrumentation-openai` | [](https://pypi.python.org/pypi/openinference-instrumentation-openai) |
| [LlamaIndex](https://docs.arize.com/phoenix/tracing/integrations-tracing/llamaindex) | `openinference-instrumentation-llama-index` | [](https://pypi.python.org/pypi/openinference-instrumentation-llama-index) |
| [DSPy](https://docs.arize.com/phoenix/tracing/integrations-tracing/dspy) | `openinference-instrumentation-dspy` | [](https://pypi.python.org/pypi/openinference-instrumentation-dspy) |
| [AWS Bedrock](https://docs.arize.com/phoenix/tracing/integrations-tracing/bedrock) | `openinference-instrumentation-bedrock` | [](https://pypi.python.org/pypi/openinference-instrumentation-bedrock) |
| [LangChain](https://docs.arize.com/phoenix/tracing/integrations-tracing/langchain) | `openinference-instrumentation-langchain` | [](https://pypi.python.org/pypi/openinference-instrumentation-langchain) |
| [MistralAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/mistralai) | `openinference-instrumentation-mistralai` | [](https://pypi.python.org/pypi/openinference-instrumentation-mistralai) |
| [Guardrails](https://docs.arize.com/phoenix/tracing/integrations-tracing/guardrails) | `openinference-instrumentation-guardrails` | [](https://pypi.python.org/pypi/openinference-instrumentation-guardrails) |
| [VertexAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/vertexai) | `openinference-instrumentation-vertexai` | [](https://pypi.python.org/pypi/openinference-instrumentation-vertexai) |
| [CrewAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/crewai) | `openinference-instrumentation-crewai` | [](https://pypi.python.org/pypi/openinference-instrumentation-crewai) |
| [Haystack](https://docs.arize.com/phoenix/tracing/integrations-tracing/haystack) | `openinference-instrumentation-haystack` | [](https://pypi.python.org/pypi/openinference-instrumentation-haystack) |
| [LiteLLM](https://docs.arize.com/phoenix/tracing/integrations-tracing/litellm) | `openinference-instrumentation-litellm` | [](https://pypi.python.org/pypi/openinference-instrumentation-litellm) |
| [Groq](https://docs.arize.com/phoenix/tracing/integrations-tracing/groq) | `openinference-instrumentation-groq` | [](https://pypi.python.org/pypi/openinference-instrumentation-groq) |
| [Instructor](https://docs.arize.com/phoenix/tracing/integrations-tracing/instructor) | `openinference-instrumentation-instructor` | [](https://pypi.python.org/pypi/openinference-instrumentation-instructor) |
| [Anthropic](https://docs.arize.com/phoenix/tracing/integrations-tracing/anthropic) | `openinference-instrumentation-anthropic` | [](https://pypi.python.org/pypi/openinference-instrumentation-anthropic) |
### JavaScript
| Integration | Package | Version Badge |
| ------------------------------------------------------------------------------------------ | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/openai-node-sdk) | `@arizeai/openinference-instrumentation-openai` | [](https://www.npmjs.com/package/@arizeai/openinference-instrumentation-openai) |
| [LangChain.js](https://docs.arize.com/phoenix/tracing/integrations-tracing/langchain.js) | `@arizeai/openinference-instrumentation-langchain` | [](https://www.npmjs.com/package/@arizeai/openinference-instrumentation-langchain) |
| [Vercel AI SDK](https://docs.arize.com/phoenix/tracing/integrations-tracing/vercel-ai-sdk) | `@arizeai/openinference-vercel` | [](https://www.npmjs.com/package/@arizeai/openinference-vercel) |
For details about tracing integrations and example applications, see the [OpenInference](https://github.com/Arize-ai/openinference) project.
## Community
Join our community to connect with thousands of AI builders.
- 🌍 Join our [Slack community](https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q).
- 📚 Read our [documentation](https://docs.arize.com/phoenix).
- 💡 Ask questions and provide feedback in the _#phoenix-support_ channel.
- 🌟 Leave a star on our [GitHub](https://github.com/Arize-ai/phoenix).
- 🐞 Report bugs with [GitHub Issues](https://github.com/Arize-ai/phoenix/issues).
- 𝕏 Follow us on [𝕏](https://twitter.com/ArizePhoenix).
- 💌️ Sign up for our [mailing list](https://phoenix.arize.com/#updates).
- 🗺️ Check out our [roadmap](https://github.com/orgs/Arize-ai/projects/45) to see where we're heading next.
## Breaking Changes
See the [migration guide](./MIGRATION.md) for a list of breaking changes.
## Copyright, Patent, and License
Copyright 2024 Arize AI, Inc. All Rights Reserved.
Portions of this code are patent protected by one or more U.S. Patents. See [IP_NOTICE](https://github.com/Arize-ai/phoenix/blob/main/IP_NOTICE).
This software is licensed under the terms of the Elastic License 2.0 (ELv2). See [LICENSE](https://github.com/Arize-ai/phoenix/blob/main/LICENSE).