https://github.com/tinybirdco/llm-benchmark
We assessed the ability of popular LLMs to generate accurate and efficient SQL from natural language prompts. Using a 200 million record dataset from the GH Archive uploaded to Tinybird, we asked the LLMs to generate SQL based on 50 prompts.
https://github.com/tinybirdco/llm-benchmark
Last synced: 11 months ago
JSON representation
We assessed the ability of popular LLMs to generate accurate and efficient SQL from natural language prompts. Using a 200 million record dataset from the GH Archive uploaded to Tinybird, we asked the LLMs to generate SQL based on 50 prompts.
- Host: GitHub
- URL: https://github.com/tinybirdco/llm-benchmark
- Owner: tinybirdco
- Created: 2025-05-01T11:29:39.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-07-14T10:56:07.000Z (11 months ago)
- Last Synced: 2025-07-14T12:17:37.713Z (11 months ago)
- Language: TypeScript
- Homepage:
- Size: 9.46 MB
- Stars: 38
- Watchers: 2
- Forks: 4
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# LLM SQL Generation Benchmark
A tool for benchmarking various Large Language Models (LLMs) on their ability to generate correct analytical SQL queries for Tinybird.
See results: https://llm-benchmark.tinybird.live/
## Overview
This benchmark evaluates how well different LLMs can generate analytical SQL queries based on natural language questions about data in Tinybird. It measures:
- SQL query correctness
- Execution success
- Performance metrics (time to first token, total duration, token usage)
- Error handling and recovery
The benchmark includes an automated retry mechanism that feeds execution errors back to the model for correction.
## Supported Providers & Models
The benchmark currently supports the following providers and models through [OpenRouter](https://openrouter.ai/):
- **X.AI**: Grok-3 Beta, Grok-3 Mini Beta, Grok-4
- **Anthropic**: Claude 3.5 Sonnet, Claude 3.7 Sonnet, Claude Opus 4, Claude Sonnet 4
- **DeepSeek**: DeepSeek Chat v3 0324, DeepSeek Chat v3 0324 Free
- **Google**: Gemini 2.0 Flash 001, Gemini 2.5 Flash, Gemini 2.5 Pro
- **Meta**: Llama 4 Maverick, Llama 4 Scout, Llama 3.3 70B Instruct
- **Mistral**: Ministral 8B, Mistral Small 3.1 24B Instruct, Mistral Nemo, Magistral Small 2506, Devstral Medium, Devstral Small
- **OpenAI**: GPT-4.1, GPT-4.1 Nano, GPT-4o Mini, O3, O3 Mini, O3 Pro, O4 Mini, O4 Mini High
It can be extended to other models, see [how to benchmark a new model](#how-to-benchmark-a-new-model)
## Methodology
The benchmark is based on this [github_events](https://github.com/tinybirdco/llm-benchmark/blob/main/src/tinybird/datasources/github_events.datasource) table schema deployed to a [free Tinybird account](https://www.tinybird.co/pricing).
The corpus consists of 200M rows from the public [GitHub archive](https://www.gharchive.org/). For ease of ingestion, they are provided as [Parquet files](https://storage.googleapis.com/dev-alrocar-public/github). Each file has 50M rows, so just 4 files are ingested for the benchmark. For this specific benchmark, scale is not critical since we are comparing models against each other, and performance data can be easily extrapolated.
Each LLM must produce SQL from 50 natural language prompts or questions about public GitHub activity. You can find the complete list of questions in the `DESCRIPTION` of each Tinybird endpoint [here](https://github.com/tinybirdco/llm-benchmark/tree/main/src/tinybird/endpoints). These endpoints are deployed to Tinybird to be used as a baseline for output correctness.
The [benchmark](https://github.com/tinybirdco/llm-benchmark/blob/96a738aafafbae32a0a72e6f149da7ebab452130/src/benchmark/index.ts#L181) is a Node.js application that:
1. Runs the Tinybird endpoints to get a [results-human.json](https://github.com/tinybirdco/llm-benchmark/blob/main/src/benchmark/results-human.json) as output baseline.
2. Iterates through a [list of models](https://github.com/tinybirdco/llm-benchmark/blob/main/src/benchmark-config.json) to extract SQL generation performance metrics.
3. Runs the generated queries and validates output correctness.
Each model receives a [system prompt](https://github.com/tinybirdco/llm-benchmark/blob/main/src/benchmark/prompt.ts) and the [github_events.datasource](https://github.com/tinybirdco/llm-benchmark/blob/main/src/tinybird/datasources/github_events.datasource) schema as context and must produce SQL that returns an output when executed over the SQL API of Tinybird.
If the SQL produced is not valid, the LLM call is retried up to three times, passing the error from the SQL API as context. Output is stored in a [results.json](https://github.com/tinybirdco/llm-benchmark/blob/main/src/benchmark/results.json) file.
Once all models have been processed, `results-human.json` and `results.json` are compared to extract a metric for output correctness and stored in [validation-results.json](https://github.com/tinybirdco/llm-benchmark/blob/main/src/benchmark/validation-results.json).
Read this [blog post](https://tbrd.co/LKHKD7c) to learn more about how the benchmark measures output performance and correctness.
## Results presentation
Results produced by the benchmark are stored in JSON files and presented in a web application deployed to https://llm-benchmark.tinybird.live/
You can find the source code [here](https://github.com/tinybirdco/llm-benchmark/tree/main/src/src)
## How to benchmark a new model
Run the benchmark locally and extend it to any other model following these instructions.
### Prerequisites
- Node.js 18+ and npm
- OpenRouter API key
- Tinybird workspace token and API access
### Installation
1. Clone this repository
2. Install dependencies:
```bash
cd llm-benchmark/src
npm install
```
3. Prepare the Tinybird Workspace:
```bash
curl https://tinybird.co | sh
cd llm-benchmark/src/tinybird
tb login
tb --cloud deploy
tb --cloud datasource append github_events https://storage.googleapis.com/dev-alrocar-public/github/01.parquet
```
4. Create a `.env` file with required credentials:
```
OPENROUTER_API_KEY=your_openrouter_api_key
TINYBIRD_WORKSPACE_TOKEN=your_tinybird_token
TINYBIRD_API_HOST=your_tinybird_api_host
```
### Usage
Run the benchmark:
```bash
npm run benchmark
```
This will:
1. Load the configured models from `benchmark-config.json`
2. Run each model against a set of predefined questions
3. Execute generated SQL queries against your Tinybird workspace
4. Store results in `benchmark/results.json`
### Test a new model
Edit `benchmark-config.json` to customize which providers and models to test.
Run the new model so the results file is updated:
```bash
npm run benchmark -- --model="openai/o3" --debug
```
## Results Analysis
Results are saved in JSON format with detailed information about each query inside the `benchmark` folder in this repository. To visualize the results, you can start the Next.js application:
```bash
cd llm-benchmark/src
npm install
npm run dev
```
## Attribution
The GitHub dataset used in this benchmark is based on work by:
Milovidov A., 2020. Everything You Ever Wanted To Know About GitHub (But Were Afraid To Ask), https://ghe.clickhouse.tech/
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request. You can propose changes, extend the documentation, and share ideas by creating pull requests and issues on the GitHub repository.
## License
This project is open-source and available under the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) license or [Apache 2](https://www.apache.org/licenses/LICENSE-2.0) license. Attribution is required when using or adapting this content.