https://github.com/Arize-ai/phoenix
AI Observability & Evaluation
https://github.com/Arize-ai/phoenix
agents ai-monitoring ai-observability aiengineering anthropic datasets evals langchain llamaindex llm-eval llm-evaluation llmops llms openai prompt-engineering smolagents
Last synced: 3 months ago
JSON representation
AI Observability & Evaluation
- Host: GitHub
- URL: https://github.com/Arize-ai/phoenix
- Owner: Arize-ai
- License: other
- Created: 2022-11-09T23:44:35.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-02-22T03:19:49.000Z (4 months ago)
- Last Synced: 2025-02-23T09:31:38.133Z (4 months ago)
- Topics: agents, ai-monitoring, ai-observability, aiengineering, anthropic, datasets, evals, langchain, llamaindex, llm-eval, llm-evaluation, llmops, llms, openai, prompt-engineering, smolagents
- Language: Jupyter Notebook
- Homepage: https://docs.arize.com/phoenix
- Size: 261 MB
- Stars: 4,803
- Watchers: 33
- Forks: 352
- Open Issues: 287
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
- awesome-llm-interpretability - Phoenix - AI Observability & Evaluation - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (Table of Contents / LLM Interpretability Tools)
- awesome-LLMs-finetuning - Phoenix - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (1596 stars) (4. Fine-Tuning / Frameworks)
- awesome-llmops - Arize-Phoenix - ai/phoenix.svg?style=flat-square) | (LLMOps / Observability)
- awesome-llm-eval - Arize-Phoenix - ai/phoenix.svg?style=social) | 用于LLMs、视觉、语言和表格模型的ML可观测性。 | (LLMOps / Popular-LLM)
- awesome-ai-engineering-reads - AI Observability & Evaluation - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook.
- awesome-open-data-centric-ai - Arize-Phoenix - Phoenix is a Python library for ML observability (monitoring + root-cause analysis) for tabular, CV, NLP, and LLM models. |  | <a href="https://github.com/Arize-ai/phoenix/blob/main/LICENSE"><img src="https://img.shields.io/github/license/Arize-AI/phoenix" height="15"/></a> | (Observability and Monitoring)
- AwesomeResponsibleAI - phoenix
- awesome-production-machine-learning - Phoenix - ai/phoenix.svg?style=social) - Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. (Evaluation and Monitoring)
- StarryDivineSky - Arize-ai/phoenix
- awesome-oss-saas - Arize-Phoenix - ai/phoenix><img src="https://img.shields.io/github/stars/arize-ai/phoenix?style=social" width=150></img></a> | [Arize](https://arize.com/) | [Docs](https://docs.arize.com/phoenix/), [Twitter](https://twitter.com/ArizePhoenix) | (Data Validation)
- Awesome-Prompt-Engineering - [Github
- awesome-llms-fine-tuning - Phoenix - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (1596 stars) (GitHub projects)
- jimsghstars - Arize-ai/phoenix - AI Observability & Evaluation (Jupyter Notebook)
- Awesome-LLM-RAG-Application - phoenix
- awesome-safety-critical-ai - `Arize-ai/phoenix` - source AI observability platform designed for experimentation, evaluation, and troubleshooting (<a id="tools"></a>🛠️ Tools / Bleeding Edge ⚗️)
- awesome-safety-critical-ai - `Arize-ai/phoenix` - source AI observability platform designed for experimentation, evaluation, and troubleshooting (<a id="tools"></a>🛠️ Tools / Bleeding Edge ⚗️)
- mcp-index - Phoenix - Provides advanced observability and evaluation tools for AI applications, facilitating performance tracking, dataset management, and prompt engineering. Integrates with various frameworks and large language model providers for seamless operation. (Monitoring and Logging)
- awesome-hacking-lists - Arize-ai/phoenix - AI Observability & Evaluation (Jupyter Notebook)
README
Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. It provides:
- [**_Tracing_**](https://docs.arize.com/phoenix/tracing/integrations-tracing/llm-traces) - Trace your LLM application's runtime using OpenTelemetry-based instrumentation.
- [**_Evaluation_**](https://docs.arize.com/phoenix/evaluation/llm-evals) - Leverage LLMs to benchmark your application's performance using response and retrieval evals.
- [**_Datasets_**](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets) - Create versioned datasets of examples for experimentation, evaluation, and fine-tuning.
- [**_Experiments_**](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets#experiments) - Track and evaluate changes to prompts, LLMs, and retrieval.
Phoenix is vendor and language agnostic with out-of-the-box support for popular frameworks (🦙[LlamaIndex](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/llamaindex), 🦜⛓[LangChain](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/langchain), [Haystack](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/haystack), 🧩[DSPy](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/dspy)) and LLM providers ([OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/openai), [Bedrock](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/bedrock), [MistralAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/mistralai), [VertexAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/vertexai), [LiteLLM](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/litellm), and more). For details on auto-instrumentation, check out the [OpenInference](https://github.com/Arize-ai/openinference) project.
Phoenix runs practically anywhere, including your Jupyter notebook, local machine, containerized deployment, or in the cloud.
## Installation
Install Phoenix via `pip` or `conda`
```shell
pip install arize-phoenix
```
Phoenix container images are available via [Docker Hub](https://hub.docker.com/r/arizephoenix/phoenix) and can be deployed using Docker or Kubernetes.
## Features
| Key Features | Availability |
| ---------------------------------------------------------------------------------------------------------------- | -------------- |
| [Tracing](https://docs.arize.com/phoenix/tracing/concepts-tracing/what-are-traces) | ✅ |
| [Evaluation](https://docs.arize.com/phoenix/evaluation/llm-evals) | ✅ |
| [Retrieval (RAG) Analysis](https://docs.arize.com/phoenix/tracing/use-cases-tracing/rag-evaluation) | ✅ |
| [Datasets](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets) | ✅ |
| [Fine-Tuning Export](https://docs.arize.com/phoenix/datasets-and-experiments/how-to-datasets/exporting-datasets) | ✅ |
| [Annotations](https://docs.arize.com/phoenix/tracing/concepts-tracing/how-to-annotate-traces) | ✅ |
| [Human Feedback](https://docs.arize.com/phoenix/tracing/how-to-tracing/capture-feedback) | ✅ |
| [Experiments](https://docs.arize.com/phoenix/datasets-and-experiments/how-to-experiments/run-experiments) | ✅ |
| [Embeddings Analysis](https://docs.arize.com/phoenix/inferences/phoenix-inferences) | ✅ |
| [Data Export](https://docs.arize.com/phoenix/tracing/how-to-tracing/extract-data-from-spans) | ✅ |
| REST API | ✅ |
| GraphQL API | ✅ |
| Data Retention | Customizable |
| Authentication | ✅ |
| Social Login | ✅ |
| RBAC | ✅ |
| Projects | ✅ |
| [Self-Hosting](https://docs.arize.com/phoenix/deployment) | ✅ |
| Jupyter Notebooks | ✅ |
| [Sessions](https://github.com/Arize-ai/phoenix/issues/2619) | In Progress 🚧 |
| [Prompt Playground](https://github.com/Arize-ai/phoenix/issues/3435) | In Progress 🚧 |
| Prompt Management | Coming soon ⏱️ |
## Tracing Integrations
Phoenix is built on top of OpenTelemetry and is vendor, language, and framework agnostic.
**Python**
| Integration | Package | Version Badge |
|------------------|-----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| [OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/openai) | `openinference-instrumentation-openai` | [](https://pypi.python.org/pypi/openinference-instrumentation-openai) |
| [LlamaIndex](https://docs.arize.com/phoenix/tracing/integrations-tracing/llamaindex) | `openinference-instrumentation-llama-index` | [](https://pypi.python.org/pypi/openinference-instrumentation-llama-index) |
| [DSPy](https://docs.arize.com/phoenix/tracing/integrations-tracing/dspy) | `openinference-instrumentation-dspy` | [](https://pypi.python.org/pypi/openinference-instrumentation-dspy) |
| [AWS Bedrock](https://docs.arize.com/phoenix/tracing/integrations-tracing/bedrock) | `openinference-instrumentation-bedrock` | [](https://pypi.python.org/pypi/openinference-instrumentation-bedrock) |
| [LangChain](https://docs.arize.com/phoenix/tracing/integrations-tracing/langchain) | `openinference-instrumentation-langchain` | [](https://pypi.python.org/pypi/openinference-instrumentation-langchain) |
| [MistralAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/mistralai) | `openinference-instrumentation-mistralai` | [](https://pypi.python.org/pypi/openinference-instrumentation-mistralai) |
| [Guardrails](https://docs.arize.com/phoenix/tracing/integrations-tracing/guardrails) | `openinference-instrumentation-guardrails` | [](https://pypi.python.org/pypi/openinference-instrumentation-guardrails) |
| [VertexAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/vertexai) | `openinference-instrumentation-vertexai` | [](https://pypi.python.org/pypi/openinference-instrumentation-vertexai) |
| [CrewAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/crewai) | `openinference-instrumentation-crewai` | [](https://pypi.python.org/pypi/openinference-instrumentation-crewai) |
| [Haystack](https://docs.arize.com/phoenix/tracing/integrations-tracing/haystack) | `openinference-instrumentation-haystack` | [](https://pypi.python.org/pypi/openinference-instrumentation-haystack) |
| [LiteLLM](https://docs.arize.com/phoenix/tracing/integrations-tracing/litellm) | `openinference-instrumentation-litellm` | [](https://pypi.python.org/pypi/openinference-instrumentation-litellm) |
| [Groq](https://docs.arize.com/phoenix/tracing/integrations-tracing/groq) | `openinference-instrumentation-groq` | [](https://pypi.python.org/pypi/openinference-instrumentation-groq) |
| [Instructor](https://docs.arize.com/phoenix/tracing/integrations-tracing/instructor) | `openinference-instrumentation-instructor` | [](https://pypi.python.org/pypi/openinference-instrumentation-instructor) |
| [Anthropic](https://docs.arize.com/phoenix/tracing/integrations-tracing/anthropic) | `openinference-instrumentation-anthropic` | [](https://pypi.python.org/pypi/openinference-instrumentation-anthropic) |
### JavaScript
| Integration | Package | Version Badge |
| ------------------------------------------------------------------------------------------ | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/openai-node-sdk) | `@arizeai/openinference-instrumentation-openai` | [](https://www.npmjs.com/package/@arizeai/openinference-instrumentation-openai) |
| [LangChain.js](https://docs.arize.com/phoenix/tracing/integrations-tracing/langchain.js) | `@arizeai/openinference-instrumentation-langchain` | [](https://www.npmjs.com/package/@arizeai/openinference-instrumentation-langchain) |
| [Vercel AI SDK](https://docs.arize.com/phoenix/tracing/integrations-tracing/vercel-ai-sdk) | `@arizeai/openinference-vercel` | [](https://www.npmjs.com/package/@arizeai/openinference-vercel) |
For details about tracing integrations and example applications, see the [OpenInference](https://github.com/Arize-ai/openinference) project.
## Community
Join our community to connect with thousands of AI builders.
- 🌍 Join our [Slack community](https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q).
- 📚 Read our [documentation](https://docs.arize.com/phoenix).
- 💡 Ask questions and provide feedback in the _#phoenix-support_ channel.
- 🌟 Leave a star on our [GitHub](https://github.com/Arize-ai/phoenix).
- 🐞 Report bugs with [GitHub Issues](https://github.com/Arize-ai/phoenix/issues).
- 𝕏 Follow us on [𝕏](https://twitter.com/ArizePhoenix).
- 💌️ Sign up for our [mailing list](https://phoenix.arize.com/#updates).
- 🗺️ Check out our [roadmap](https://github.com/orgs/Arize-ai/projects/45) to see where we're heading next.
## Breaking Changes
See the [migration guide](./MIGRATION.md) for a list of breaking changes.
## Copyright, Patent, and License
Copyright 2024 Arize AI, Inc. All Rights Reserved.
Portions of this code are patent protected by one or more U.S. Patents. See [IP_NOTICE](https://github.com/Arize-ai/phoenix/blob/main/IP_NOTICE).
This software is licensed under the terms of the Elastic License 2.0 (ELv2). See [LICENSE](https://github.com/Arize-ai/phoenix/blob/main/LICENSE).