Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/raga-ai-hub/raga-llm-hub
Framework for LLM evaluation, guardrails and security
https://github.com/raga-ai-hub/raga-llm-hub
guardrails llm-evaluation llm-security llmops
Last synced: 6 days ago
JSON representation
Framework for LLM evaluation, guardrails and security
- Host: GitHub
- URL: https://github.com/raga-ai-hub/raga-llm-hub
- Owner: raga-ai-hub
- License: other
- Created: 2024-03-02T18:32:13.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-09-09T10:53:31.000Z (2 months ago)
- Last Synced: 2024-11-03T09:47:40.681Z (13 days ago)
- Topics: guardrails, llm-evaluation, llm-security, llmops
- Language: Python
- Homepage: https://www.raga.ai/llms
- Size: 1.22 MB
- Stars: 96
- Watchers: 2
- Forks: 9
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-llm-eval - raga-llm-hub - llm-hub是一个全面的语言和学习模型（LLM）评估工具包。它拥有超过100个精心设计的评价指标，是允许开发者和组织有效评估和比较LLM的最具综合性的平台，并为LLM和检索增强生成（RAG）应用建立基本的防护措施。这些测试评估包括相关性与理解、内容质量、幻觉、安全与偏见、上下文相关性、防护措施以及漏洞扫描等多个方面，同时提供一系列基于指标的测试用于定量分析 (2024-03-10) | (Datasets-or-Benchmark / RAG检索增强生成评估)
README
Raga LLM Hub
Raga AI |
Documentation |
Getting Started
[](https://badge.fury.io/py/raga-llm-hub) [](https://colab.research.google.com/drive/1PQGqDGdcSUxhSvpSQYX8ZdHf5r90WSYf?usp=sharing)
[](https://pypi.org/project/raga-llm-hub/) []()
Welcome to RagaAI LLM Hub, a comprehensive evaluation toolkit for Language and Learning Models (LLMs). With over 100 meticulously designed metrics, it is the most comprehensive platform that allows developers and organizations to evaluate and compare LLMs effectively and establish essential guardrails for LLMs and Retrieval Augmented Generation(RAG) applications. These tests assess various aspects including Relevance & Understanding, Content Quality, Hallucination, Safety & Bias, Context Relevance, Guardrails, and Vulnerability scanning, along with a suite of Metric-Based Tests for quantitative analysis.
The RagaAI LLM Hub is uniquely designed to help teams identify issues and fix them throughout the LLM lifecycle, by identifying issues across the entire RAG pipeline. This is pivotal for understanding the root cause of failures within an LLM application and addressing them at their source, revolutionizing the approach to ensuring reliability and trustworthiness.
## Installation
### Via pip
```bash
# Create and activate a new Python environment
python -m venv venv
source venv/bin/activate
# Install Raga LLM Hub
pip install raga-llm-hub
```
### Via conda
```py
# Create and activate a new Conda environment
conda create --name myenv python=3.11
conda activate myenv
# Install Raga LLM Hub
python -m pip install raga-llm-hub
```
## Quick Tour
### Initialization
```py
from raga_llm_hub import RagaLLMEval
# Initialize the evaluator with your API key
evaluator = RagaLLMEval("OPENAI_API_KEY"="your_api_key")
```
### Run Tests
```py
# Add and run a custom test
evaluator.add_test(
test_name="relevancy_test",
data={
"prompt": "How are you?",
"context": "Responding as a student to a teacher.",
"response": "I am well, thank you.",
},
arguments={"model": "gpt-4", "threshold": 0.5},
).run()
# Review the results
evaluator.print_results()
```
## Managing Results
- **Instant Overview**: Quickly view your test results directly.
- **Save for Detailed Analysis**: Export your results for comprehensive examination or sharing with your team.
- **In-depth Access**: Utilize the app for advanced result processing and visualization.
- **Historical Comparisons**: Leverage past evaluations for ongoing performance tracking.
```py
# Printing Results: View your test results immediately for a quick analysis
evaluator.print_results()
# Saving Results: Export your results to a JSON file for in-depth analysis
evaluator.save_results("my_test_results.json")
# Accessing Results: Utilize the fetched detailed results and metrics for further processing or visualization
detailed_results = evaluator.get_results()
# Re-using Previous Results: If you have an evaluation ID from a previous run, you can load and compare those results
previous_eval_id = "your_previous_eval_id_here"
evaluator.load_eval(previous_eval_id)
# After loading, you can print, save, or further analyze these results
evaluator.print_results()
```
## Examples
- **Evaluation Tests**: Evaluation Tests assesse a Large Language Model's (LLM's) performance in generating responses that are accurate, relevant, and linguistically coherent to a wide array of prompts. This evaluation is pivotal in determining the model's ability to understand and respond appropriately to diverse user inputs, ranging from simple queries to complex, context-rich requests.
[](https://colab.research.google.com/drive/1cY5eN5w7bK1CH8L8MYEAfrfzaVZ7LNS5?usp=sharing)
- **Guardrail Tests**: Guardrails ensure that the models operate within predefined ethical, legal, and safety boundaries. These mechanisms are implemented to prevent the generation of biased, offensive, or harmful content, making sure that the outputs align with societal norms and values.
[](https://colab.research.google.com/drive/1TAX2PeicBBWHtdiZZelpXN5YcOB7WnEk?usp=sharing)
## Enterprise
Enterprise Version
Introducing raga-llm-platform,(enterprise version of raga-llm-hub) for Large Language Model (LLM) evaluation and guardrails, designed to empower organizations to harness the full potential of LLMs securely and efficiently. Here’s what sets raga-llm-platform apart:
1. **Production Scale Analysis**
2. **State-of-the-Art Evaluation Methods and Metrics**
3. **Issue Diagnosis and Remediation**
4. **On-Prem/Private Cloud Deployment with Real-Time Streaming Support**
5. **Real-Time Evaluation and Guardrails**
To learn more and see how raga-llm-platform can benefit your organization, [book a call with our team today](https://calendly.com/vijay-srinivas/ragaai-product-offering?month=2024-03). Discover the value of enterprise-grade LLM management tailored to your needs.
## Learn More
For those who wish to dive deeper, we encourage exploring [our extensive documentation](https://docs.raga.ai)
For more details and the latest news from RagaAI, visit [our official website](https://raga.ai).