Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/Arize-ai/phoenix

AI Observability & Evaluation
https://github.com/Arize-ai/phoenix

ai-monitoring ai-observability ai-roi aiengineering datasets hacktoberfest llm-eval llmops ml-observability mlops model-observability

Last synced: 3 months ago
JSON representation

AI Observability & Evaluation

Host: GitHub
URL: https://github.com/Arize-ai/phoenix
Owner: Arize-ai
License: other
Created: 2022-11-09T23:44:35.000Z (about 2 years ago)
Default Branch: main
Last Pushed: 2024-10-29T08:17:12.000Z (3 months ago)
Last Synced: 2024-10-29T12:30:40.033Z (3 months ago)
Topics: ai-monitoring, ai-observability, ai-roi, aiengineering, datasets, hacktoberfest, llm-eval, llmops, ml-observability, mlops, model-observability
Language: Jupyter Notebook
Homepage: https://docs.arize.com/phoenix
Size: 92.5 MB
Stars: 3,801
Watchers: 30
Forks: 282
Open Issues: 225
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md

Awesome Lists containing this project

awesome-llm-interpretability - Phoenix - AI Observability & Evaluation - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (Table of Contents / LLM Interpretability Tools)
awesome-LLMs-finetuning - Phoenix - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (1596 stars) (4. Fine-Tuning / Frameworks)
awesome-llmops - Arize-Phoenix - ai/phoenix.svg?style=flat-square) | (LLMOps / Observability)
awesome-llm-eval - Arize-Phoenix - ai/phoenix.svg?style=social) | 用于LLMs、视觉、语言和表格模型的ML可观测性。 | (LLMOps / Popular-LLM)
awesome-ai-engineering-reads - AI Observability & Evaluation - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook.
awesome-open-data-centric-ai - Arize-Phoenix - Phoenix is a Python library for ML observability (monitoring + root-cause analysis) for tabular, CV, NLP, and LLM models. | ![GitHub stars](https://img.shields.io/github/stars/Arize-AI/phoenix?style=social) | <a href="https://github.com/Arize-ai/phoenix/blob/main/LICENSE"><img src="https://img.shields.io/github/license/Arize-AI/phoenix" height="15"/></a> | (Observability and Monitoring)
AwesomeResponsibleAI - phoenix
awesome-production-machine-learning - Phoenix - ai/phoenix.svg?style=social) - Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. (Evaluation and Monitoring)
StarryDivineSky - Arize-ai/phoenix
awesome-oss-saas - Arize-Phoenix - ai/phoenix><img src="https://img.shields.io/github/stars/arize-ai/phoenix?style=social" width=150></img></a> | [Arize](https://arize.com/) | [Docs](https://docs.arize.com/phoenix/), [Twitter](https://twitter.com/ArizePhoenix) | (Data Validation)
Awesome-Prompt-Engineering - [Github
awesome-llms-fine-tuning - Phoenix - Evaluate, troubleshoot, and fine tune your LLM, CV, and NLP models in a notebook. (1596 stars) (GitHub projects)
jimsghstars - Arize-ai/phoenix - AI Observability & Evaluation (Jupyter Notebook)
Awesome-LLM-RAG-Application - phoenix

README

        


    

        

    

    


    


    

        

    

    

        

    

    

        

    

    

        

    

    

        

    

    

        

    

    

        

    



Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. It provides:

- [**_Tracing_**](https://docs.arize.com/phoenix/tracing/integrations-tracing/llm-traces) - Trace your LLM application's runtime using OpenTelemetry-based instrumentation.

- [**_Evaluation_**](https://docs.arize.com/phoenix/evaluation/llm-evals) - Leverage LLMs to benchmark your application's performance using response and retrieval evals.

- [**_Datasets_**](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets) - Create versioned datasets of examples for experimentation, evaluation, and fine-tuning.

- [**_Experiments_**](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets#experiments) - Track and evaluate changes to prompts, LLMs, and retrieval.

Phoenix is vendor and language agnostic with out-of-the-box support for popular frameworks (🦙[LlamaIndex](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/llamaindex), 🦜⛓[LangChain](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/langchain), [Haystack](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/haystack), 🧩[DSPy](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/dspy)) and LLM providers ([OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/openai), [Bedrock](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/bedrock), [MistralAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/mistralai), [VertexAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/vertexai), [LiteLLM](https://docs.arize.com/phoenix/tracing/integrations-tracing/integrations-tracing/litellm), and more). For details on auto-instrumentation, check out the [OpenInference](https://github.com/Arize-ai/openinference) project.

Phoenix runs practically anywhere, including your Jupyter notebook, local machine, containerized deployment, or in the cloud.



## Installation

Install Phoenix via `pip` or `conda`

```shell

pip install arize-phoenix

```

Phoenix container images are available via [Docker Hub](https://hub.docker.com/r/arizephoenix/phoenix) and can be deployed using Docker or Kubernetes.

## Features

| Key Features                                                                                                     | Availability   |

| ---------------------------------------------------------------------------------------------------------------- | -------------- |

| [Tracing](https://docs.arize.com/phoenix/tracing/concepts-tracing/what-are-traces)                               | ✅             |

| [Evaluation](https://docs.arize.com/phoenix/evaluation/llm-evals)                                                | ✅             |

| [Retrieval (RAG) Analysis](https://docs.arize.com/phoenix/tracing/use-cases-tracing/rag-evaluation)              | ✅             |

| [Datasets](https://docs.arize.com/phoenix/datasets-and-experiments/overview-datasets)                            | ✅             |

| [Fine-Tuning Export](https://docs.arize.com/phoenix/datasets-and-experiments/how-to-datasets/exporting-datasets) | ✅             |

| [Annotations](https://docs.arize.com/phoenix/tracing/concepts-tracing/how-to-annotate-traces)                    | ✅             |

| [Human Feedback](https://docs.arize.com/phoenix/tracing/how-to-tracing/capture-feedback)                         | ✅             |

| [Experiments](https://docs.arize.com/phoenix/datasets-and-experiments/how-to-experiments/run-experiments)        | ✅             |

| [Embeddings Analysis](https://docs.arize.com/phoenix/inferences/phoenix-inferences)                              | ✅             |

| [Data Export](https://docs.arize.com/phoenix/tracing/how-to-tracing/extract-data-from-spans)                     | ✅             |

| REST API                                                                                                         | ✅             |

| GraphQL API                                                                                                      | ✅             |

| Data Retention                                                                                                   | Customizable   |

| Authentication                                                                                                   | ✅             |

| Social Login                                                                                                     | ✅             |

| RBAC                                                                                                             | ✅             |

| Projects                                                                                                         | ✅             |

| [Self-Hosting](https://docs.arize.com/phoenix/deployment)                                                        | ✅             |

| Jupyter Notebooks                                                                                                | ✅             |

| [Sessions](https://github.com/Arize-ai/phoenix/issues/2619)                                                      | In Progress 🚧 |

| [Prompt Playground](https://github.com/Arize-ai/phoenix/issues/3435)                                             | In Progress 🚧 |

| Prompt Management                                                                                                | Coming soon ⏱️ |

## Tracing Integrations

Phoenix is built on top of OpenTelemetry and is vendor, language, and framework agnostic.

**Python**

| Integration | Package | Version Badge |

|------------------|-----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|

| [OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/openai) | `openinference-instrumentation-openai` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-openai.svg)](https://pypi.python.org/pypi/openinference-instrumentation-openai) |

| [LlamaIndex](https://docs.arize.com/phoenix/tracing/integrations-tracing/llamaindex) | `openinference-instrumentation-llama-index` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-llama-index.svg)](https://pypi.python.org/pypi/openinference-instrumentation-llama-index) |

| [DSPy](https://docs.arize.com/phoenix/tracing/integrations-tracing/dspy) | `openinference-instrumentation-dspy` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-dspy.svg)](https://pypi.python.org/pypi/openinference-instrumentation-dspy) |

| [AWS Bedrock](https://docs.arize.com/phoenix/tracing/integrations-tracing/bedrock) | `openinference-instrumentation-bedrock` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-bedrock.svg)](https://pypi.python.org/pypi/openinference-instrumentation-bedrock) |

| [LangChain](https://docs.arize.com/phoenix/tracing/integrations-tracing/langchain) | `openinference-instrumentation-langchain` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-langchain.svg)](https://pypi.python.org/pypi/openinference-instrumentation-langchain) |

| [MistralAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/mistralai) | `openinference-instrumentation-mistralai` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-mistralai.svg)](https://pypi.python.org/pypi/openinference-instrumentation-mistralai) |

| [Guardrails](https://docs.arize.com/phoenix/tracing/integrations-tracing/guardrails) | `openinference-instrumentation-guardrails` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-guardrails.svg)](https://pypi.python.org/pypi/openinference-instrumentation-guardrails) |

| [VertexAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/vertexai) | `openinference-instrumentation-vertexai` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-vertexai.svg)](https://pypi.python.org/pypi/openinference-instrumentation-vertexai) |

| [CrewAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/crewai) | `openinference-instrumentation-crewai` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-crewai.svg)](https://pypi.python.org/pypi/openinference-instrumentation-crewai) |

| [Haystack](https://docs.arize.com/phoenix/tracing/integrations-tracing/haystack) | `openinference-instrumentation-haystack` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-haystack.svg)](https://pypi.python.org/pypi/openinference-instrumentation-haystack) |

| [LiteLLM](https://docs.arize.com/phoenix/tracing/integrations-tracing/litellm) | `openinference-instrumentation-litellm` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-litellm.svg)](https://pypi.python.org/pypi/openinference-instrumentation-litellm) |

| [Groq](https://docs.arize.com/phoenix/tracing/integrations-tracing/groq) | `openinference-instrumentation-groq` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-groq.svg)](https://pypi.python.org/pypi/openinference-instrumentation-groq) |

| [Instructor](https://docs.arize.com/phoenix/tracing/integrations-tracing/instructor) | `openinference-instrumentation-instructor` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-instructor.svg)](https://pypi.python.org/pypi/openinference-instrumentation-instructor) |

| [Anthropic](https://docs.arize.com/phoenix/tracing/integrations-tracing/anthropic) | `openinference-instrumentation-anthropic` | [![PyPI Version](https://img.shields.io/pypi/v/openinference-instrumentation-anthropic.svg)](https://pypi.python.org/pypi/openinference-instrumentation-anthropic) |

### JavaScript

| Integration                                                                                | Package                                            | Version Badge                                                                                                                                                                       |

| ------------------------------------------------------------------------------------------ | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

| [OpenAI](https://docs.arize.com/phoenix/tracing/integrations-tracing/openai-node-sdk)      | `@arizeai/openinference-instrumentation-openai`    | [![NPM Version](https://img.shields.io/npm/v/@arizeai/openinference-instrumentation-openai.svg)](https://www.npmjs.com/package/@arizeai/openinference-instrumentation-openai)       |

| [LangChain.js](https://docs.arize.com/phoenix/tracing/integrations-tracing/langchain.js)   | `@arizeai/openinference-instrumentation-langchain` | [![NPM Version](https://img.shields.io/npm/v/@arizeai/openinference-instrumentation-langchain.svg)](https://www.npmjs.com/package/@arizeai/openinference-instrumentation-langchain) |

| [Vercel AI SDK](https://docs.arize.com/phoenix/tracing/integrations-tracing/vercel-ai-sdk) | `@arizeai/openinference-vercel`                    | [![NPM Version](https://img.shields.io/npm/v/@arizeai/openinference-vercel)](https://www.npmjs.com/package/@arizeai/openinference-vercel)                                           |

For details about tracing integrations and example applications, see the [OpenInference](https://github.com/Arize-ai/openinference) project.

## Community

Join our community to connect with thousands of AI builders.

- 🌍 Join our [Slack community](https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q).

- 📚 Read our [documentation](https://docs.arize.com/phoenix).

- 💡 Ask questions and provide feedback in the _#phoenix-support_ channel.

- 🌟 Leave a star on our [GitHub](https://github.com/Arize-ai/phoenix).

- 🐞 Report bugs with [GitHub Issues](https://github.com/Arize-ai/phoenix/issues).

- 𝕏 Follow us on [𝕏](https://twitter.com/ArizePhoenix).

- 💌️ Sign up for our [mailing list](https://phoenix.arize.com/#updates).

- 🗺️ Check out our [roadmap](https://github.com/orgs/Arize-ai/projects/45) to see where we're heading next.

## Breaking Changes

See the [migration guide](./MIGRATION.md) for a list of breaking changes.

## Copyright, Patent, and License

Copyright 2024 Arize AI, Inc. All Rights Reserved.

Portions of this code are patent protected by one or more U.S. Patents. See [IP_NOTICE](https://github.com/Arize-ai/phoenix/blob/main/IP_NOTICE).

This software is licensed under the terms of the Elastic License 2.0 (ELv2). See [LICENSE](https://github.com/Arize-ai/phoenix/blob/main/LICENSE).