https://github.com/Arize-ai/phoenix
AI Observability & Evaluation
https://github.com/Arize-ai/phoenix
agents ai-monitoring ai-observability aiengineering anthropic datasets evals langchain llamaindex llm-eval llm-evaluation llmops llms openai prompt-engineering smolagents
Last synced: 2 months ago
JSON representation
AI Observability & Evaluation
- Host: GitHub
- URL: https://github.com/Arize-ai/phoenix
- Owner: Arize-ai
- License: other
- Created: 2022-11-09T23:44:35.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-02-22T03:19:49.000Z (3 months ago)
- Last Synced: 2025-02-23T09:31:38.133Z (3 months ago)
- Topics: agents, ai-monitoring, ai-observability, aiengineering, anthropic, datasets, evals, langchain, llamaindex, llm-eval, llm-evaluation, llmops, llms, openai, prompt-engineering, smolagents
- Language: Jupyter Notebook
- Homepage: https://docs.arize.com/phoenix
- Size: 261 MB
- Stars: 4,803
- Watchers: 33
- Forks: 352
- Open Issues: 287
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
- awesome-llm-interpretability - Phoenix - AI Observability & Evaluation - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (Table of Contents / LLM Interpretability Tools)
- awesome-LLMs-finetuning - Phoenix - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (1596 stars) (4. Fine-Tuning / Frameworks)
- awesome-llmops - Arize-Phoenix - ai/phoenix.svg?style=flat-square) | (LLMOps / Observability)
- awesome-llm-eval - Arize-Phoenix - ai/phoenix.svg?style=social) | 用于LLMs、视觉、语言和表格模型的ML可观测性。 | (LLMOps / Popular-LLM)
- awesome-ai-engineering-reads - AI Observability & Evaluation - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook.
- awesome-open-data-centric-ai - Arize-Phoenix - Phoenix is a Python library for ML observability (monitoring + root-cause analysis) for tabular, CV, NLP, and LLM models. |  | <a href="https://github.com/Arize-ai/phoenix/blob/main/LICENSE"><img src="https://img.shields.io/github/license/Arize-AI/phoenix" height="15"/></a> | (Observability and Monitoring)
- AwesomeResponsibleAI - phoenix
- awesome-production-machine-learning - Phoenix - ai/phoenix.svg?style=social) - Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. (Evaluation and Monitoring)
- StarryDivineSky - Arize-ai/phoenix
- awesome-oss-saas - Arize-Phoenix - ai/phoenix><img src="https://img.shields.io/github/stars/arize-ai/phoenix?style=social" width=150></img></a> | [Arize](https://arize.com/) | [Docs](https://docs.arize.com/phoenix/), [Twitter](https://twitter.com/ArizePhoenix) | (Data Validation)
- Awesome-Prompt-Engineering - [Github
- awesome-llms-fine-tuning - Phoenix - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (1596 stars) (GitHub projects)
- jimsghstars - Arize-ai/phoenix - AI Observability & Evaluation (Jupyter Notebook)
- Awesome-LLM-RAG-Application - phoenix
- awesome-safety-critical-ai - `Arize-ai/phoenix` - source AI observability platform designed for experimentation, evaluation, and troubleshooting (<a id="tools"></a>🛠️ Tools / Bleeding Edge ⚗️)
- awesome-safety-critical-ai - `Arize-ai/phoenix` - source AI observability platform designed for experimentation, evaluation, and troubleshooting (<a id="tools"></a>🛠️ Tools / Bleeding Edge ⚗️)
- awesome-hacking-lists - Arize-ai/phoenix - AI Observability & Evaluation (Jupyter Notebook)
README
Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. It provides:
- [**_Tracing_**](https://docs.arize.com/phoenix/tracing/integrations-tracing/llm-traces) - Trace your LLM application's runtime using OpenTelemetry-based instrumentation.
- [**_Evaluation_**](https://docs.arize.com/phoenix/evaluation/llm-evals) - Leverage LLMs to benchmark your application's performance using response and retrieval evals.
- [**_Datasets_**](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets) - Create versioned datasets of examples for experimentation, evaluation, and fine-tuning.
- [**_Experiments_**](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets#experiments) - Track and evaluate changes to prompts, LLMs, and retrieval.
Phoenix is vendor and language agnostic with out-of-the-box support for popular frameworks (🦙[LlamaIndex](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/llamaindex), 🦜⛓[LangChain](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/langchain), [Haystack](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/haystack), 🧩[DSPy](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/dspy)) and LLM providers ([OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/openai), [Bedrock](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/bedrock), [MistralAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/mistralai), [VertexAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/vertexai), [LiteLLM](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/litellm), and more). For details on auto-instrumentation, check out the [OpenInference](https://github.com/Arize-ai/openinference) project.
Phoenix runs practically anywhere, including your Jupyter notebook, local machine, containerized deployment, or in the cloud.
## Installation
Install Phoenix via `pip` or `conda`
```shell
pip install arize-phoenix
```
Phoenix container images are available via [Docker Hub](https://hub.docker.com/r/arizephoenix/phoenix) and can be deployed using Docker or Kubernetes.
## Features
| Key Features | Availability |
| ---------------------------------------------------------------------------------------------------------------- | -------------- |
| [Tracing](https://docs.arize.com/phoenix/tracing/concepts-tracing/what-are-traces) | ✅ |
| [Evaluation](https://docs.arize.com/phoenix/evaluation/llm-evals) | ✅ |
| [Retrieval (RAG) Analysis](https://docs.arize.com/phoenix/tracing/use-cases-tracing/rag-evaluation) | ✅ |
| [Datasets](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets) | ✅ |
| [Fine-Tuning Export](https://docs.arize.com/phoenix/datasets-and-experiments/how-to-datasets/exporting-datasets) | ✅ |
| [Annotations](https://docs.arize.com/phoenix/tracing/concepts-tracing/how-to-annotate-traces) | ✅ |
| [Human Feedback](https://docs.arize.com/phoenix/tracing/how-to-tracing/capture-feedback) | ✅ |
| [Experiments](https://docs.arize.com/phoenix/datasets-and-experiments/how-to-experiments/run-experiments) | ✅ |
| [Embeddings Analysis](https://docs.arize.com/phoenix/inferences/phoenix-inferences) | ✅ |
| [Data Export](https://docs.arize.com/phoenix/tracing/how-to-tracing/extract-data-from-spans) | ✅ |
| REST API | ✅ |
| GraphQL API | ✅ |
| Data Retention | Customizable |
| Authentication | ✅ |
| Social Login | ✅ |
| RBAC | ✅ |
| Projects | ✅ |
| [Self-Hosting](https://docs.arize.com/phoenix/deployment) | ✅ |
| Jupyter Notebooks | ✅ |
| [Sessions](https://github.com/Arize-ai/phoenix/issues/2619) | In Progress 🚧 |
| [Prompt Playground](https://github.com/Arize-ai/phoenix/issues/3435) | In Progress 🚧 |
| Prompt Management | Coming soon ⏱️ |
## Tracing Integrations
Phoenix is built on top of OpenTelemetry and is vendor, language, and framework agnostic.
**Python**
| Integration | Package | Version Badge |
|------------------|-----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| [OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/openai) | `openinference-instrumentation-openai` | [](https://pypi.python.org/pypi/openinference-instrumentation-openai) |
| [LlamaIndex](https://docs.arize.com/phoenix/tracing/integrations-tracing/llamaindex) | `openinference-instrumentation-llama-index` | [](https://pypi.python.org/pypi/openinference-instrumentation-llama-index) |
| [DSPy](https://docs.arize.com/phoenix/tracing/integrations-tracing/dspy) | `openinference-instrumentation-dspy` | [](https://pypi.python.org/pypi/openinference-instrumentation-dspy) |
| [AWS Bedrock](https://docs.arize.com/phoenix/tracing/integrations-tracing/bedrock) | `openinference-instrumentation-bedrock` | [](https://pypi.python.org/pypi/openinference-instrumentation-bedrock) |
| [LangChain](https://docs.arize.com/phoenix/tracing/integrations-tracing/langchain) | `openinference-instrumentation-langchain` | [](https://pypi.python.org/pypi/openinference-instrumentation-langchain) |
| [MistralAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/mistralai) | `openinference-instrumentation-mistralai` | [](https://pypi.python.org/pypi/openinference-instrumentation-mistralai) |
| [Guardrails](https://docs.arize.com/phoenix/tracing/integrations-tracing/guardrails) | `openinference-instrumentation-guardrails` | [](https://pypi.python.org/pypi/openinference-instrumentation-guardrails) |
| [VertexAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/vertexai) | `openinference-instrumentation-vertexai` | [](https://pypi.python.org/pypi/openinference-instrumentation-vertexai) |
| [CrewAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/crewai) | `openinference-instrumentation-crewai` | [](https://pypi.python.org/pypi/openinference-instrumentation-crewai) |
| [Haystack](https://docs.arize.com/phoenix/tracing/integrations-tracing/haystack) | `openinference-instrumentation-haystack` | [](https://pypi.python.org/pypi/openinference-instrumentation-haystack) |
| [LiteLLM](https://docs.arize.com/phoenix/tracing/integrations-tracing/litellm) | `openinference-instrumentation-litellm` | [](https://pypi.python.org/pypi/openinference-instrumentation-litellm) |
| [Groq](https://docs.arize.com/phoenix/tracing/integrations-tracing/groq) | `openinference-instrumentation-groq` | [](https://pypi.python.org/pypi/openinference-instrumentation-groq) |
| [Instructor](https://docs.arize.com/phoenix/tracing/integrations-tracing/instructor) | `openinference-instrumentation-instructor` | [](https://pypi.python.org/pypi/openinference-instrumentation-instructor) |
| [Anthropic](https://docs.arize.com/phoenix/tracing/integrations-tracing/anthropic) | `openinference-instrumentation-anthropic` | [](https://pypi.python.org/pypi/openinference-instrumentation-anthropic) |
### JavaScript
| Integration | Package | Version Badge |
| ------------------------------------------------------------------------------------------ | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/openai-node-sdk) | `@arizeai/openinference-instrumentation-openai` | [](https://www.npmjs.com/package/@arizeai/openinference-instrumentation-openai) |
| [LangChain.js](https://docs.arize.com/phoenix/tracing/integrations-tracing/langchain.js) | `@arizeai/openinference-instrumentation-langchain` | [](https://www.npmjs.com/package/@arizeai/openinference-instrumentation-langchain) |
| [Vercel AI SDK](https://docs.arize.com/phoenix/tracing/integrations-tracing/vercel-ai-sdk) | `@arizeai/openinference-vercel` | [](https://www.npmjs.com/package/@arizeai/openinference-vercel) |
For details about tracing integrations and example applications, see the [OpenInference](https://github.com/Arize-ai/openinference) project.
## Community
Join our community to connect with thousands of AI builders.
- 🌍 Join our [Slack community](https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q).
- 📚 Read our [documentation](https://docs.arize.com/phoenix).
- 💡 Ask questions and provide feedback in the _#phoenix-support_ channel.
- 🌟 Leave a star on our [GitHub](https://github.com/Arize-ai/phoenix).
- 🐞 Report bugs with [GitHub Issues](https://github.com/Arize-ai/phoenix/issues).
- 𝕏 Follow us on [𝕏](https://twitter.com/ArizePhoenix).
- 💌️ Sign up for our [mailing list](https://phoenix.arize.com/#updates).
- 🗺️ Check out our [roadmap](https://github.com/orgs/Arize-ai/projects/45) to see where we're heading next.
## Breaking Changes
See the [migration guide](./MIGRATION.md) for a list of breaking changes.
## Copyright, Patent, and License
Copyright 2024 Arize AI, Inc. All Rights Reserved.
Portions of this code are patent protected by one or more U.S. Patents. See [IP_NOTICE](https://github.com/Arize-ai/phoenix/blob/main/IP_NOTICE).
This software is licensed under the terms of the Elastic License 2.0 (ELv2). See [LICENSE](https://github.com/Arize-ai/phoenix/blob/main/LICENSE).