{"id":28923683,"url":"https://github.com/lucianosarno/llms-benchmarks-costs-parser","last_synced_at":"2025-08-30T07:05:12.017Z","repository":{"id":298970882,"uuid":"1001545880","full_name":"lucianosarno/llms-benchmarks-costs-parser","owner":"lucianosarno","description":"The `vals_parser.py` script aims to parse LLM costs/latency/benchmarks data from the website vals.ai and make it available through the charts in the README file and the 'benchmarks_data.json' file","archived":false,"fork":false,"pushed_at":"2025-08-24T20:10:22.000Z","size":104,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"develop","last_synced_at":"2025-08-24T23:27:54.751Z","etag":null,"topics":["chrome-driver","cost-benchmarking","cost-benefit","information-parsing","llm","llm-benchmarking","llms","parser","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucianosarno.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-13T15:07:32.000Z","updated_at":"2025-08-24T20:09:33.000Z","dependencies_parsed_at":"2025-06-13T22:35:13.024Z","dependency_job_id":"faea1b57-d18f-43c7-9d30-fd8240ef1455","html_url":"https://github.com/lucianosarno/llms-benchmarks-costs-parser","commit_stats":null,"previous_names":["lucianosarno/llms-benchmarks-costs-parser"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lucianosarno/llms-benchmarks-costs-parser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucianosarno%2Fllms-benchmarks-costs-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucianosarno%2Fllms-benchmarks-costs-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucianosarno%2Fllms-benchmarks-costs-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucianosarno%2Fllms-benchmarks-costs-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucianosarno","download_url":"https://codeload.github.com/lucianosarno/llms-benchmarks-costs-parser/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucianosarno%2Fllms-benchmarks-costs-parser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272815862,"owners_count":24997673,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chrome-driver","cost-benchmarking","cost-benefit","information-parsing","llm","llm-benchmarking","llms","parser","python"],"created_at":"2025-06-22T09:39:54.332Z","updated_at":"2025-08-30T07:05:12.009Z","avatar_url":"https://github.com/lucianosarno.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vals.ai LLM Benchmarks Data Parser\n\n[![Python 3.x](https://img.shields.io/badge/Python-3.x-blue.svg)](https://www.python.org/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n\n## Description\n\nThis repository contains a Python script to programmatically scrape benchmark data from the [Vals.ai Benchmarks](https://www.vals.ai/benchmarks) website. The goal is to provide a structured, machine-readable dataset of LLM performance metrics (Accuracy, Cost Input/Output, Latency) across various benchmarks, updated daily.\n\nThe scraped data is stored in a JSON file (`benchmarks_data.json`) in this repository, allowing for easier analysis, comparison, and visualization of different Large Language Models and their providers.\n\n## Features\n\n*   Scrapes all available benchmark links from the main Vals.ai benchmarks page.\n*   Visits each benchmark page to extract detailed model data.\n*   Uses Selenium to dynamically load and capture all available benchmark entries on each page, ensuring comprehensive data collection beyond initial view limits.\n*   Parses Model Name, Company (from SVG icon filename), Accuracy, Cost Input, Cost Output, and Latency for each model.\n*   Stores the collected data in a structured JSON format.\n*   Includes a timestamp for when the data was last updated.\n*   Automated daily updates via GitHub Actions (see [Automation](#automation)).\n\n## Getting Started\n\n### Prerequisites\n\n*   Python 3.x installed.\n*   `pip` (Python package installer).\n\n### Installation\n\n1.  Clone this repository:\n    ```bash\n    git clone https://github.com/lucianosarno/llm-benchmarks-costs-parser.git\n    cd llm-benchmarks-costs-parser\n    ```\n2.  (Recommended) Create and activate a Python virtual environment:\n    ```bash\n    python3 -m venv .venv\n    source .venv/bin/activate # On Windows use `.venv\\Scripts\\activate`\n    ```\n3.  Install the required Python libraries:\n    ```bash\n    pip install -r requirements.txt\n    ```\n\n## Usage\n\nTo run the scraping script manually:\n\n```bash\nsource .venv/bin/activate # Activate virtual environment if you created one\npython vals_parser.py\n```\n\nThis script will fetch the latest data and update the benchmarks_data.json file in the root of the repository.\nData Output: benchmarks_data.json\n\nThe core output of the script is the benchmarks_data.json file. This file contains a JSON object with the following structure:\n\n```json\n{\n    \"timestamp_utc\": \"YYYY-MM-DDTHH:MM:SS.ssssss\", // UTC timestamp of the data scrape\n    \"benchmarks\": [\n        {\n            \"benchmark\": \"benchmark-name-from-url\", // e.g., \"legal-qa\"\n            \"model\": \"Model Name\",                // e.g., \"Grok 3 Beta\"\n            \"company\": \"Company Name\",            // e.g., \"xAI\", \"OpenAI\"\n            \"accuracy\": \"Percentage%\",            // e.g., \"88.1%\"\n            \"cost_input\": \"$Value\",               // e.g., \"\\$3.00\"\n            \"cost_output\": \"$Value\",              // e.g., \"\\$15.00\"\n            \"latency\": \"Value s\"                  // e.g., \"3.91 s\"\n        },\n        // ... more model entries ...\n    ]\n}\n```\n\nYou can view the latest version of this file directly on GitHub or access its raw content via:\n\nhttps://raw.githubusercontent.com/lucianosarno/llms-benchmarks-costs-parser/main/benchmarks_data.json\n\n## Data Visualization\n\nThe data collected by this script is used to power interactive dashboards hosted on Zoho Analytics, providing visual insights into LLM performance across benchmarks and providers.\n\n### LLM Providers vs. Benchmarks vs. Cost\nhttps://analytics.zoho.com/open-view/2732937000006457007\n\nThis chart aggregates data by provider, showing average costs across different benchmarks.\n\n### Individual LLM Models vs. Benchmarks vs. Cost\nhttps://analytics.zoho.com/open-view/2732937000006459493\nThis chart provides a detailed view of individual model performance and costs per benchmark.\n\n### LLM Performance Grouped by Benchmarks\nhttps://analytics.zoho.com/open-view/2732937000006462636\nThis chart groups the data by the benchmark categories themselves (e.g., Legal QA, Coding, Reasoning), allowing comparison of how models perform on average within specific domains.\n\n### Chart Filters and Cost Explanation\n\nBoth embedded charts offer interactive filters to refine the data being displayed:\n\n#### Minimum Accuracy Filter\nLocated on the chart interface, this filter allows you to set a minimum accuracy threshold. Only models that achieved an accuracy score above this minimum percentage in all benchmarks included in the current view will be shown. This helps focus on top-performing models.\n#### Timestamp Filter\nThis date filter allows you to select the data from a specific date's scrape. The script saves data with a timestamp_utc, and this filter uses that timestamp to let you view historical snapshots of the benchmark results.\n#### Costs\nThe charts primarily visualize costs based on the 'Cost Input' metric from the scraped data. It's important to note that the raw data in benchmarks_data.json includes both 'Cost Input' and 'Cost Output', which are often proportional or related based on the model's pricing structure. The charts may simplify by focusing on one metric, but the full cost breakdown is available in the source JSON.\n\n## Automation\n\nThe script is configured to run daily using GitHub Actions. This ensures the benchmarks_data.json file is automatically updated with the latest information from Vals.ai without manual intervention.\n\n(Note: This README assumes you will set up the GitHub Action workflow. You'll need to create the .github/workflows directory and a .yml file for the workflow configuration.)\nContributing\n\nContributions are welcome! If you find issues or have suggestions for improvements (e.g., adding more data points, improving parsing robustness), please open an issue or submit a pull request.\nLicense\n\nThis project is licensed under the MIT License.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucianosarno%2Fllms-benchmarks-costs-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucianosarno%2Fllms-benchmarks-costs-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucianosarno%2Fllms-benchmarks-costs-parser/lists"}