{"id":13717569,"url":"https://github.com/ossf-cve-benchmark/ossf-cve-benchmark","last_synced_at":"2025-05-07T07:31:45.498Z","repository":{"id":36969544,"uuid":"308246564","full_name":"ossf-cve-benchmark/ossf-cve-benchmark","owner":"ossf-cve-benchmark","description":"The OpenSSF CVE Benchmark consists of code and metadata for over 200 real life CVEs, as well as tooling to analyze the vulnerable codebases using a variety of static analysis security testing (SAST) tools and generate reports to evaluate those tools.","archived":false,"fork":false,"pushed_at":"2024-03-12T10:24:51.000Z","size":821,"stargazers_count":138,"open_issues_count":26,"forks_count":39,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-08-04T00:13:45.285Z","etag":null,"topics":["benchmark","cve","open-source","security","vulnerability"],"latest_commit_sha":null,"homepage":"https://github.com/ossf-cve-benchmark/ossf-cve-benchmark","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ossf-cve-benchmark.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/security.md","support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-10-29T07:11:58.000Z","updated_at":"2024-06-07T14:57:30.000Z","dependencies_parsed_at":"2023-11-29T15:41:35.936Z","dependency_job_id":null,"html_url":"https://github.com/ossf-cve-benchmark/ossf-cve-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossf-cve-benchmark%2Fossf-cve-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossf-cve-benchmark%2Fossf-cve-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossf-cve-benchmark%2Fossf-cve-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossf-cve-benchmark%2Fossf-cve-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ossf-cve-benchmark","download_url":"https://codeload.github.com/ossf-cve-benchmark/ossf-cve-benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224573504,"owners_count":17333804,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","cve","open-source","security","vulnerability"],"created_at":"2024-08-03T00:01:24.213Z","updated_at":"2024-11-14T05:32:07.447Z","avatar_url":"https://github.com/ossf-cve-benchmark.png","language":"TypeScript","funding_links":[],"categories":["Dependency intelligence"],"sub_categories":["Vulnerability information exchange"],"readme":"# Welcome to the OpenSSF CVE Benchmark project!\n\n**The OpenSSF CVE Benchmark consists of code and metadata for over 200 real life CVEs, as well as tooling to analyze the vulnerable codebases using a variety of static analysis security testing (SAST) tools and generate reports to evaluate those tools.**\n\nThe benchmark addresses two problems that security teams face today when assessing security tools. First, rather than using synthetic test code, the OpenSSF CVE Benchmark uses real life CVEs that have been validated and fixed in open source projects. Using this approach, security tools are tested on *real* codebases that contain validated *real* vulnerabilities. \nSecond, by also analyzing the patched versions of the same codebases, false positive rates of these tools can be measured more accurately and based on real validated fixes. \n\nFor each CVE in the in the dataset (200+ CVEs so far), the CVE Benchmark determines:\n\n 1. Is the security tool able to detect the vulnerability, or does it produce a false negative?\n 2. When run against the patched codebase, does it recognize the validated patch, or does it produce a false positive?\n\nThe CVE Benchmark was developed by the [OpenSSF Security Tooling working group](https://github.com/ossf/wg-security-tooling) and announced at [Black Hat Europe 2020](https://www.blackhat.com/eu-20/briefings/schedule/#fps-are-cheap-show-me-the-cves-21345). At this time, it covers over 200 JavaScript and TypeScript CVEs. The tooling has integration for benchmarking ESLint, NodeJSScan, and CodeQL. We'd love your help with [expanding the dataset](CONTRIBUTING.md#contributing-benchmark-data), and [building integration](CONTRIBUTING.md#contributing-analysis-tool-drivers) for more security tools! \n\n![](docs/screenshot.png)\n\n## Using the CVE Benchmark\n\n### Step 1: install the tooling and metadata\nSimply clone this repo and use `npm` to set up the tooling (requires npm \u0026GreaterEqual; 7 and Node.js \u0026GreaterEqual; 12):\n\n```\n$ git clone git@github.com:ossf-cve-benchmark/ossf-cve-benchmark.git\n$ cd ossf-cve-benchmark\n$ npm i\n$ npm run-script build\n```\n(a warning about \"DoubleScrollbar.js\" can be ignored, if it appears)\n\nOn Linux and macOS, you can now run `bin/cli` with the appropriate command-line options. \nOn Windows, you need to run `bin/cli.cmd`.\n\nTo confirm that everything is working as it should,\nrun `bin/cli -h` to display usage and basic help\ninformation:\n\n```\n$ bin/cli -h\nusage: bin/cli [-h] { ... }\n\nCLI for ossf-cve-benchmark\n\n...\n```\n\n### Step 2: run the security tools against the benchmark\n\nThe tooling included in the OpenSSF CVE Benchmark will help you evaluate the ability of security tools to generate alerts for the vulnerabilities (CVEs) in the dataset. The benchmark tooling will run the security tools on the source code of the commits associated with CVEs. Using the tools' output, you can then generate benchmark reports that show how the security tools performed. \nFor more information about generating benchmark reports, see \n[Step 3](#step-3-generate-benchmark-reports) below.\n\nBefore running an analysis, you must install the security tools you wish to benchmark and configure the *tool drivers* accordingly.\nA tool driver is a script that acts as the interface\nbetween the analysis tools and the CVE Benchmark.\nFor reference information about analysis tool drivers, including information \nabout adding drivers for new tools, see \n[Analysis tools](docs/analysis-tools.md). \nTo see how to set up ESLint to run an analysis, see \n[Generating a benchmark report for `eslint`](docs/eslint-example.md)\n\nTo run an analysis, use the `bin/cli run` command \nspecifying:\n\n- the identifier of the driver for your analysis tool, \n  as given in [`config.json`](docs/configuration.md), using the `--tool` option. \n- one or more CVEs to run the analysis on. For more information, \n  see [Selecting CVEs of interest](docs/configuration.md#selecting-cves-of-interest).\n\nFor example, to run an analysis using ESLint over CVE-2018-16492 \nand CVE-2020-4066 you would use:\n\n```\n$ bin/cli run --tool eslint-default CVE-2018-16492 CVE-2020-4066\n```\n\nwhere `eslint-default` is the name of the driver configured for ESLint\nanalysis in the `config.json` file:\n\n```json\n{\n  \"tools\": {\n    \"eslint-default\": {\n      \"bin\": \"node\",\n      \"args\": [\n        \"/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js\"\n      ],\n      \"options\": {\n        \"eslintDir\": \"/home/user-name/analysis-tools/eslint-2020-12-08\"\n      }\n    }\n  }\n}\n```\n\nTo run ESLint against all CVEs in the dataset, you can run:\n\n```\n$ bin/cli run --tool eslint-default '*'\n```\n\n### Step 3: generate benchmark reports\n\nAfter running one or more security tools against the dataset (using `bin/cli run`), you can generate benchmark reports to see how well different security tools performed at detecting vulnerabilities. To do this, you can run `bin/cli` with the `report` command, which can generate either an [interactive browser-based report](#interactive-report-in-your-web-browser), or a [text-based report](#text-based-reporting).\n\n#### Interactive report in your web browser\n\nTo view an interactive benchmark report in your browser, you can use the `report` command with the `--kind server` option. Just like the `run` command, it expects you to specify one or more CVEs. You can also pass it `'*'` to generate a report for all CVEs in the dataset. \n\nFor example, to browse a report that compares ESLint to NodeJSScan on all CVEs in the dataset, you would use:\n\n```\n$ bin/cli report --kind server --tool eslint-default --tool nodejsscan '*'\n```\n\nAfter running this command, you can browse the report at http://localhost:8080.\n\nThere are some special ways to [select groups of CVEs](docs/configuration.md#selecting-cves-of-interest). For example, if you're interested in a report for *just* [MITRE's Top-25 CWEs](https://cwe.mitre.org/top25/archive/2020/2020_cwe_top25.html), you could run:\n\n```\n$ bin/cli report --kind server --tool eslint-default --tool nodejsscan mitre-cwe-top:25:2020\n```\n\n#### Text-based reporting\nUse `--kind txt` to create a static text-based report. For example:\n\n```\n$ bin/cli report --kind txt --tool eslint-default CVE-2018-16492 CVE-2020-4066\n```\n\n## Status and roadmap\n\nThis project has only just started and we'd love your help! The CVE Benchmark currently supports benchmarking\nthree JavaScript security analysis tools on just over 200 CVEs. \n\nIn time, we'll work on the following high-level improvements:\n\n- additional CVE data, for both JavaScript and other languages\n- support for additional security analysis tools, for both JavaScript and other languages\n- more advanced reports\n\nAt the meta-level, we expect to soon formalize versioning\nand release strategies, and to make the workflows for external\ncontributions as seamless as possible.\n\n### We don't aim to be ...\n\n- A way of organising and presenting data on all CVEs in existence (such as https://cve.mitre.org)\n  - rather, this project hopes that the information in this project makes its way to such views\n- A code viewer for browsing alerts (such as a [SARIF viewer](https://sarifweb.azurewebsites.net/#viewersSectionTitle))\n  - rather, this project aims to have a simple analysis result format in order to have a low bar of entry for new analysis tools\n- A platform for disclosing new security vulnerabilities (such as https://www.hackerone.com)\n  - rather, this project contains simple descriptions of the weaknesses of the above vulnerabilities\n\n\nWe do :heart: all of the initiatives mentioned above!\n\n\n## Contributing to the project\n\nWe welcome contributions to the OpenSSF CVE Benchmark project.  If you want\nto contribute, then please go ahead and open a pull request! See\nadditional details in our [contributing guidelines](CONTRIBUTING.md) and our [FAQ](docs/FAQ.md).\n\n## About us: the OpenSSF Security Tooling working group\n\nThe OpenSSF CVE Benchmark project was initiated by the [OpenSSF](https://openssf.org/)'s [Security Tooling working group](https://github.com/ossf/wg-security-tooling). The working group contains a wide variety of members; some contribute independently, others join through their roles at organizations like Google, GitHub, Mozilla, Tencent, and OWASP. The working group typically meets (virtually) once every two week; [all are welcome](https://github.com/ossf/wg-security-tooling#meeting-times). \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fossf-cve-benchmark%2Fossf-cve-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fossf-cve-benchmark%2Fossf-cve-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fossf-cve-benchmark%2Fossf-cve-benchmark/lists"}