Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/wronai/allama

For All LLM benchmark
https://github.com/wronai/allama

benchmark chatbot cline ide llm ollama openapi plugin windsurf

Last synced: 3 days ago
JSON representation

For All LLM benchmark

Awesome Lists containing this project

README 

          
![allama-logo.svg](allama-logo.svg)



# Allama - LLM Testing and Benchmarking Suite 



A comprehensive testing and benchmarking suite for Large Language Models (LLMs) focused on Python code generation. The project enables automatic quality assessment of generated code through various metrics and generates detailed HTML reports.



## Features



- **Automated Testing** of multiple LLM models with configurable prompts

- **Code Quality Assessment** - syntax checking, execution, style, and functionality

- **Detailed HTML Reports** with metrics, charts, and comparisons

- **Interactive Code Diff** - visual comparison of code generated by different models

- **Results Export** to CSV and JSON for further analysis

- **Highly Configurable** - easily add new models and tests

- **Multiple API Support** - Ollama, local servers, cloud services

- **Model Ranking** based on performance and quality metrics

- **Zero Configuration** - automatically generates default config files when needed

- **Benchmark Publishing** - share your results with the community via the Allama server

- **Local Results Storage** - automatically saves results in timestamped folders

- **Prompt Analysis** - detailed information about prompts used in benchmarks

- **Radar Charts** - visual comparison of model performance across multiple metrics



## Quick Start

![img.png](img.png)

### 1. Installation



#### Using Poetry (recommended)

```bash

# Clone the repository

git clone https://github.com/wronai/allama.git

cd allama



# Install dependencies

pip install poetry

poetry install



# Activate the virtual environment

poetry shell

```



#### Using pip

```bash

# Clone the repository

git clone https://github.com/wronai/allama.git

cd allama



pip install .

```



### 2. Model Configuration



Create or edit the `models.csv` file to configure your models:



```csv

model_name,url,auth_header,auth_value,think,description

mistral:latest,http://localhost:11434/api/chat,,,false,Mistral Latest on Ollama

llama3:8b,http://localhost:11434/api/chat,,,false,Llama 3 8B

gpt-4,https://api.openai.com/v1/chat/completions,Authorization,Bearer sk-...,false,OpenAI GPT-4

```



**CSV Columns:**

- `model_name` - Name of the model (e.g., mistral:latest, gpt-4)

- `url` - API endpoint URL

- `auth_header` - Authorization header (if required, e.g., "Authorization")

- `auth_value` - Authorization value (e.g., "Bearer your-api-key")

- `think` - Whether the model supports "think" parameter (true/false)

- `description` - Description of the model



## Configuration



The application is configured using external files, with `config.json` being the primary configuration file.



### Main Configuration (`config.json`)



This file, located in the root directory, contains all the main settings for the application:



- **`prompts_file`**: Path to the file containing test prompts (e.g., `prompts.json`).

- **`evaluation_weights`**: Points awarded for different code quality metrics.

- **`timeouts`**: Time limits for API requests and code execution.

- **`report_config`**: Settings for the generated HTML report, such as the title.

- **`colors`**: Color scheme used in the HTML report.



You can create your own configuration file (e.g., `my_config.yaml`) and use it with the `--config` flag during runtime.



### Prompts Configuration (`prompts.json`)



This file contains a list of test cases (prompts) that will be sent to the language models. Each prompt is a JSON object with the following keys:



- **`name`**: A descriptive name for the test (e.g., "Simple Addition Function").

- **`prompt`**: The full text of the prompt to be sent to the model.

- **`expected_keywords`**: A list of keywords that are expected to be present in the generated code.



### Automatic Configuration Generation



The system will automatically generate default configuration files (`config.json` and `prompts.json`) if they don't exist when you run the tool. This means you can simply run the `allama` command without any setup, and the necessary configuration files will be created for you with sensible defaults.



### Using Custom Configuration



You can run tests with a custom configuration file (in either JSON or YAML format) using the `--config` or `-c` flag. The settings from your custom file will be merged with the defaults.



**Example with JSON:**

```bash

allama --config my_config.json

```



**Example with YAML:**

```bash

allama --config custom_settings.yaml

```



## Reports and Output



Allama generates comprehensive reports to help you analyze and compare model performance:



### HTML Report (`allama.html`)



An interactive HTML report is generated after each test run, containing:



- **Summary Dashboard** - Overview of test results with key metrics

- **Model Ranking** - Performance comparison of all tested models

- **Detailed Results** - In-depth analysis of each model's performance

- **Code Comparison** - Interactive diff viewer to compare code generated by different models



### JSON Data (`allama.json`)



All test results are also saved in a structured JSON format for:

- Further analysis with external tools

- Integration with other systems

- Custom visualization and reporting



The JSON file contains complete information about:

- Test configuration

- Model responses

- Evaluation metrics

- Generated code



### CSV Summary (`*_summary.csv`)



A CSV summary file is also generated with key metrics for quick analysis in spreadsheet applications.



### Example Report Usage



The HTML report allows you to:

1. View overall model rankings

2. Examine detailed results for each model and prompt

3. Compare code generated by different models using the interactive diff tool

4. Filter and sort results based on various metrics



To view the report, simply open `allama.html` in any modern web browser after running tests.



### Publishing Results Online



Allama allows you to publish your benchmark results to a central repository at `allama.sapletta.com`, making it easy to share and compare results with others:



```bash

# Run benchmark and publish results

allama --benchmark --publish



# Specify a custom server URL

allama --benchmark --publish --server-url https://your-server.com/upload.php

```



The publishing system includes:

- **Rate limiting**: Maximum 3 uploads per day per IP address

- **Request throttling**: Minimum 1 second between requests

- **Automatic organization**: Results are stored in timestamped directories

- **Web interface**: Browse and compare published benchmarks

- **Responsive design**: Optimized for both desktop and mobile devices

- **Radar charts**: Visual comparison of model performance across multiple metrics

- **Badge-style metrics**: Quick overview of key benchmark statistics



After publishing, you'll receive a URL where you can view your results online.



### Local Results Storage



All benchmark results are automatically saved locally in a timestamped folder structure:

```

data/

└── test_YYYYMMDD_HHMMSS/

    ├── allama.json       # Complete benchmark results

    ├── allama.html       # HTML report

    └── prompts.json      # Detailed prompt information

```



This allows you to:

- Keep a history of all benchmark runs

- Compare results over time

- Share specific benchmark results with others



### Benchmark Visualization



The benchmark server provides several visualization features:

- **Responsive 3-column layout**: Displays benchmarks in an easy-to-scan grid (collapses to single column on mobile)

- **Radar charts**: Each benchmark includes a radar chart showing model performance across 6 key metrics:

  - Success rate

  - Response speed

  - Syntax correctness

  - Execution success

  - Keywords presence

  - Code quality

- **Badge-style metrics**: Key statistics displayed as GitHub-style badges for quick reference

- **Model comparison**: Easy visual comparison of multiple models within each benchmark



## Usage



### Using Makefile (recommended)

```bash

# Install dependencies and setup

make install



# Run tests

make test



# Run all tests including end-to-end

make test-all



# Run benchmark suite

make benchmark



# Test a single model (set MODEL=name)

make single-model



# Generate HTML report

make report



# Run code formatters

make format



# Run linters

make lint

```



### Basic Command-Line Usage

```bash

# Run all tests with default configuration

allama



# Run benchmark suite

allama --benchmark



# Test specific models

allama --models "mistral:latest,llama3:8b,gemma2:2b"



# Test a single model

allama --single-model "mistral:latest"



# Compare specific models

allama --compare "mistral:latest" "llama3:8b"



# Generate HTML report

allama --output benchmark_report.html



# Run with verbose output

allama --verbose

```



### Advanced Usage

```bash

# Run with custom configuration

allama --config custom_config.json



# Test with a specific prompt

allama --single-model "mistral:latest" --prompt-index 0



# Set request timeout (in seconds)

allama --timeout 60

```



## Evaluation Metrics



The system evaluates generated code based on the following criteria:



### Basic Metrics (automatic)

- Correct Syntax - whether the code compiles without errors

- Executability - whether the code runs without runtime errors

- Keyword Matching - whether the code contains expected elements from the prompt



### Code Quality Metrics

- Function/Class Definitions - proper code structure

- Error Handling - try/except blocks, input validation

- Documentation - docstrings, comments

- Imports - proper library usage

- Code Length - reasonable number of lines



### Scoring System

- Correct Syntax: **3 points**

- Runs without errors: **2 points**

- Contains expected elements: **2 points**

- Has function/class definitions: **1 point**

- Has error handling: **1 point**

- Has documentation: **1 point**

- **Maximum: 10 points**



## Ansible Configuration



Create `tests/ansible/inventory.ini` with:



```ini

[all]

localhost ansible_connection=local

```



## API Integration Examples



### Ollama (local)

```csv

llama3:8b,http://localhost:11434/api/chat,,,false,Llama 3 8B

```



### OpenAI API

```csv

gpt-4,https://api.openai.com/v1/chat/completions,Authorization,Bearer sk-your-key,false,OpenAI GPT-4

```



### Anthropic Claude

```csv

claude-3,https://api.anthropic.com/v1/messages,x-api-key,your-key,false,Claude 3

```



### Local Server

```csv

local-model,http://localhost:8080/generate,,,false,Local Model

```



## Project Structure



```

allama/

├── allama/               # Main package

│   ├── __init__.py      # Package initialization

│   ├── main.py          # Main module

│   ├── config_loader.py # Configuration loading and generation

│   └── runner.py        # Test runner implementation

├── tests/               # Test files

│   └── test_allama.py   # Unit tests

├── models.csv           # Model configurations

├── config.json          # Main configuration (auto-generated if missing)

├── prompts.json         # Test prompts (auto-generated if missing)

├── pyproject.toml       # Project metadata and dependencies

├── Makefile             # Common tasks

└── README.md            # This file

```



## Example Output



After running the benchmark, you'll get:



1. **Console Output**: Summary of test results

2. **HTML Report**: Detailed report with code examples and metrics

3. **CSV/JSON**: Raw data for further analysis



## Getting Help



If you encounter any issues or have questions:



1. Check the [issues](https://github.com/wronai/allama/issues) page

2. Create a new issue with detailed information about your problem



## Contributing



Contributions are welcome! Please read our [Contributing Guidelines](CONTRIBUTING.md) for details on how to contribute to this project.



## License



This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.



## Acknowledgments



- Thanks to all the open-source projects that made this possible

- Special thanks to the Ollama team for their amazing work



---





  Made with ❤️ by the Allama team