https://github.com/truera/trulens

Evaluation and Tracking for LLM Experiments
https://github.com/truera/trulens

explainable-ml llm llmops machine-learning neural-networks

Last synced: 4 days ago
JSON representation

Evaluation and Tracking for LLM Experiments

• Host: GitHub
• URL: https://github.com/truera/trulens
• Owner: truera
• License: mit
• Created: 2020-11-02T21:56:45.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2024-12-10T02:16:32.000Z (12 days ago)
• Last Synced: 2024-12-10T02:18:56.897Z (12 days ago)
• Topics: explainable-ml, llm, llmops, machine-learning, neural-networks
• Language: Python
• Homepage: https://www.trulens.org/
• Size: 296 MB
• Stars: 2,226
• Watchers: 19
• Forks: 195
• Open Issues: 37
• Metadata Files:
  • Readme: README.md
  • Contributing: docs/contributing/design.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-production-machine-learning - TruLens - TruLens provides a set of tools for evaluating and tracking LLM experiments. (Evaluation and Monitoring)
• StarryDivineSky - truera/trulens - Eval 评估LLMs和LLM基于应用程序的工具，以及使用 TruLens-Explain 的深度学习可解释性。TruLens-Eval 和 TruLens-Explain 装在单独的封装中，可以独立使用。更快地创建可靠且功能强大的LLM应用程序。TruLens 是一种软件工具，可帮助您使用反馈功能客观地衡量基于应用程序LLM的质量和有效性。反馈函数有助于以编程方式评估输入、输出和中间结果的质量，以便您可以加快和扩大实验评估。将其用于各种用例，包括问答、摘要、检索增强生成和基于代理的应用程序。 (A01_文本生成_文本对话 / 大语言对话模型及数据)
• Awesome-LLM-RAG-Application - trulens

README

        
![PyPI - Version](https://img.shields.io/pypi/v/trulens?label=trulens&link=https%3A%2F%2Fpypi.org%2Fproject%2Ftrulens%2F)

[![Azure Build Status](https://dev.azure.com/truera/trulens/_apis/build/status%2FTruLens%20E2E%20Tests?branchName=main)](https://dev.azure.com/truera/trulens/_build/latest?definitionId=8&branchName=main)

![GitHub](https://img.shields.io/github/license/truera/trulens)

![PyPI - Downloads](https://img.shields.io/pypi/dm/trulens)

[![Slack](https://img.shields.io/badge/slack-join-green?logo=slack)](https://communityinviter.com/apps/aiqualityforum/josh)

[![Docs](https://img.shields.io/badge/docs-trulens.org-blue)](https://www.trulens.org/getting_started/)

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/truera/trulens/blob/main/examples/quickstart/langchain_quickstart.ipynb)

# 🦑 Welcome to TruLens!

![TruLens](https://www.trulens.org/assets/images/Neural_Network_Explainability.png)

**Don't just vibe-check your llm app!** Systematically evaluate and track your

LLM experiments with TruLens. As you develop your app including prompts, models,

retrievers, knowledge sources and more, *TruLens* is the tool you need to

understand its performance.

Fine-grained, stack-agnostic instrumentation and comprehensive evaluations help

you to identify failure modes & systematically iterate to improve your

application.

Read more about the core concepts behind TruLens including [Feedback Functions](https://www.trulens.org/getting_started/core_concepts/feedback_functions/),

[The RAG Triad](https://www.trulens.org/getting_started/core_concepts/rag_triad/),

and [Honest, Harmless and Helpful Evals](https://www.trulens.org/getting_started/core_concepts/honest_harmless_helpful_evals/).

## TruLens in the development workflow

Build your first prototype then connect instrumentation and logging with

TruLens. Decide what feedbacks you need, and specify them with TruLens to run

alongside your app. Then iterate and compare versions of your app in an

easy-to-use user interface 👇

![Architecture

Diagram](https://www.trulens.org/assets/images/TruLens_Architecture.png)

## Installation and Setup

Install the trulens pip package from PyPI.

```bash

    pip install trulens

```

## Quick Usage

Walk through how to instrument and evaluate a RAG built from scratch with

TruLens.

[![Open In

Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/truera/trulens/blob/main/examples/quickstart/quickstart.ipynb)

### 💡 Contributing & Community

Interested in contributing? See our [contributing

guide](https://www.trulens.org/trulens/contributing/) for more details.

The best way to support TruLens is to give us a ⭐ on

[GitHub](https://www.github.com/truera/trulens) and join our [slack

community](https://communityinviter.com/apps/aiqualityforum/josh)!