https://github.com/mervinpraison/praisonaitest
https://github.com/mervinpraison/praisonaitest
Last synced: 4 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/mervinpraison/praisonaitest
- Owner: MervinPraison
- Created: 2025-05-31T10:48:08.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-05-31T12:18:57.000Z (about 1 year ago)
- Last Synced: 2025-09-24T01:27:54.366Z (8 months ago)
- Language: Python
- Size: 209 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# PraisonAITest
A minimal and simple testing framework for Large Language Models (LLMs) using LiteLLM.
## Features
- ✅ Simple, clean interface for testing LLMs
- ✅ Supports multiple LLM providers via LiteLLM
- ✅ String-based or function-based validation
- ✅ Performance tracking and metrics
## Installation
1. Clone the repository:
```bash
git clone https://github.com/MervinPraison/PraisonAITest.git
cd PraisonAITest
```
2. Create and activate a conda environment:
```bash
zsh -c "source $(conda info --base)/etc/profile.d/conda.sh && conda create -n windsurf python=3.11 -y && conda activate windsurf"
```
3. Install dependencies using uv package manager:
```bash
pip install uv
uv pip install -r requirements.txt
# Install the package in development mode
uv pip install -e ./libs/praisonaitest
```
4. Copy the example environment file and add your API keys:
```bash
cp .env.example .env
```
Then edit the `.env` file and add your API keys.
## Usage
### Basic Usage
```python
from libs.praisonaitest.praisonaitest import Test
# Simple test with string validation
test = Test(
model="openai/gpt-4o-mini",
instruction="What is the capital of France?",
validate="Paris"
)
result = test.run()
print(result)
```
### Using a Custom Validation Function
```python
from libs.praisonaitest.praisonaitest import Test
# Define a custom validation function
def validate_response(response):
return "Paris" in response and "France" in response
# Test with function validation
test = Test(
model="openai/gpt-4o-mini",
instruction="Tell me about France's capital city",
validate=validate_response
)
result = test.run()
# Check the result
if result["passed"]:
print("Test passed!")
else:
print("Test failed!")
print(f"Response: {result['response']}")
```
## Result Dictionary
The `run()` method returns a dictionary with the following keys:
- `success`: Whether the API call was successful (boolean)
- `passed`: Whether the validation passed (boolean)
- `response`: The text response from the LLM (string)
- `latency`: Time taken to get the response in seconds (float)
- `model`: The model used for the test (string)
- `token_usage`: Token usage information if available (dictionary or None)
- `error`: Error message if an error occurred (string or None)
## Environment Variables
Create a `.env` file with your API keys:
```
LITELLM_API_KEY=your_api_key_here
OPENAI_API_KEY=your_openai_key_here # Optional, if using OpenAI directly
```
## Examples
See the `test_gpt4o.py` file for a complete example.
## License
MIT