Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ossf-cve-benchmark/ossf-cve-benchmark

The OpenSSF CVE Benchmark consists of code and metadata for over 200 real life CVEs, as well as tooling to analyze the vulnerable codebases using a variety of static analysis security testing (SAST) tools and generate reports to evaluate those tools.
https://github.com/ossf-cve-benchmark/ossf-cve-benchmark

benchmark cve open-source security vulnerability

Last synced: 2 months ago
JSON representation

The OpenSSF CVE Benchmark consists of code and metadata for over 200 real life CVEs, as well as tooling to analyze the vulnerable codebases using a variety of static analysis security testing (SAST) tools and generate reports to evaluate those tools.

Awesome Lists containing this project

README 

        
# Welcome to the OpenSSF CVE Benchmark project!



**The OpenSSF CVE Benchmark consists of code and metadata for over 200 real life CVEs, as well as tooling to analyze the vulnerable codebases using a variety of static analysis security testing (SAST) tools and generate reports to evaluate those tools.**



The benchmark addresses two problems that security teams face today when assessing security tools. First, rather than using synthetic test code, the OpenSSF CVE Benchmark uses real life CVEs that have been validated and fixed in open source projects. Using this approach, security tools are tested on *real* codebases that contain validated *real* vulnerabilities. 

Second, by also analyzing the patched versions of the same codebases, false positive rates of these tools can be measured more accurately and based on real validated fixes. 



For each CVE in the in the dataset (200+ CVEs so far), the CVE Benchmark determines:



 1. Is the security tool able to detect the vulnerability, or does it produce a false negative?

 2. When run against the patched codebase, does it recognize the validated patch, or does it produce a false positive?



The CVE Benchmark was developed by the [OpenSSF Security Tooling working group](https://github.com/ossf/wg-security-tooling) and announced at [Black Hat Europe 2020](https://www.blackhat.com/eu-20/briefings/schedule/#fps-are-cheap-show-me-the-cves-21345). At this time, it covers over 200 JavaScript and TypeScript CVEs. The tooling has integration for benchmarking ESLint, NodeJSScan, and CodeQL. We'd love your help with [expanding the dataset](CONTRIBUTING.md#contributing-benchmark-data), and [building integration](CONTRIBUTING.md#contributing-analysis-tool-drivers) for more security tools! 



![](docs/screenshot.png)



## Using the CVE Benchmark



### Step 1: install the tooling and metadata

Simply clone this repo and use `npm` to set up the tooling (requires npm ≥ 7 and Node.js ≥ 12):



```

$ git clone [email protected]:ossf-cve-benchmark/ossf-cve-benchmark.git

$ cd ossf-cve-benchmark

$ npm i

$ npm run-script build

```

(a warning about "DoubleScrollbar.js" can be ignored, if it appears)



On Linux and macOS, you can now run `bin/cli` with the appropriate command-line options. 

On Windows, you need to run `bin/cli.cmd`.



To confirm that everything is working as it should,

run `bin/cli -h` to display usage and basic help

information:



```

$ bin/cli -h

usage: bin/cli [-h] { ... }



CLI for ossf-cve-benchmark



...

```



### Step 2: run the security tools against the benchmark



The tooling included in the OpenSSF CVE Benchmark will help you evaluate the ability of security tools to generate alerts for the vulnerabilities (CVEs) in the dataset. The benchmark tooling will run the security tools on the source code of the commits associated with CVEs. Using the tools' output, you can then generate benchmark reports that show how the security tools performed. 

For more information about generating benchmark reports, see 

[Step 3](#step-3-generate-benchmark-reports) below.



Before running an analysis, you must install the security tools you wish to benchmark and configure the *tool drivers* accordingly.

A tool driver is a script that acts as the interface

between the analysis tools and the CVE Benchmark.

For reference information about analysis tool drivers, including information 

about adding drivers for new tools, see 

[Analysis tools](docs/analysis-tools.md). 

To see how to set up ESLint to run an analysis, see 

[Generating a benchmark report for `eslint`](docs/eslint-example.md)



To run an analysis, use the `bin/cli run` command 

specifying:



- the identifier of the driver for your analysis tool, 

  as given in [`config.json`](docs/configuration.md), using the `--tool` option. 

- one or more CVEs to run the analysis on. For more information, 

  see [Selecting CVEs of interest](docs/configuration.md#selecting-cves-of-interest).



For example, to run an analysis using ESLint over CVE-2018-16492 

and CVE-2020-4066 you would use:



```

$ bin/cli run --tool eslint-default CVE-2018-16492 CVE-2020-4066

```



where `eslint-default` is the name of the driver configured for ESLint

analysis in the `config.json` file:



```json

{

  "tools": {

    "eslint-default": {

      "bin": "node",

      "args": [

        "/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js"

      ],

      "options": {

        "eslintDir": "/home/user-name/analysis-tools/eslint-2020-12-08"

      }

    }

  }

}

```



To run ESLint against all CVEs in the dataset, you can run:



```

$ bin/cli run --tool eslint-default '*'

```



### Step 3: generate benchmark reports



After running one or more security tools against the dataset (using `bin/cli run`), you can generate benchmark reports to see how well different security tools performed at detecting vulnerabilities. To do this, you can run `bin/cli` with the `report` command, which can generate either an [interactive browser-based report](#interactive-report-in-your-web-browser), or a [text-based report](#text-based-reporting).



#### Interactive report in your web browser



To view an interactive benchmark report in your browser, you can use the `report` command with the `--kind server` option. Just like the `run` command, it expects you to specify one or more CVEs. You can also pass it `'*'` to generate a report for all CVEs in the dataset. 



For example, to browse a report that compares ESLint to NodeJSScan on all CVEs in the dataset, you would use:



```

$ bin/cli report --kind server --tool eslint-default --tool nodejsscan '*'

```



After running this command, you can browse the report at http://localhost:8080.



There are some special ways to [select groups of CVEs](docs/configuration.md#selecting-cves-of-interest). For example, if you're interested in a report for *just* [MITRE's Top-25 CWEs](https://cwe.mitre.org/top25/archive/2020/2020_cwe_top25.html), you could run:



```

$ bin/cli report --kind server --tool eslint-default --tool nodejsscan mitre-cwe-top:25:2020

```



#### Text-based reporting

Use `--kind txt` to create a static text-based report. For example:



```

$ bin/cli report --kind txt --tool eslint-default CVE-2018-16492 CVE-2020-4066

```



## Status and roadmap



This project has only just started and we'd love your help! The CVE Benchmark currently supports benchmarking

three JavaScript security analysis tools on just over 200 CVEs. 



In time, we'll work on the following high-level improvements:



- additional CVE data, for both JavaScript and other languages

- support for additional security analysis tools, for both JavaScript and other languages

- more advanced reports



At the meta-level, we expect to soon formalize versioning

and release strategies, and to make the workflows for external

contributions as seamless as possible.



### We don't aim to be ...



- A way of organising and presenting data on all CVEs in existence (such as https://cve.mitre.org)

  - rather, this project hopes that the information in this project makes its way to such views

- A code viewer for browsing alerts (such as a [SARIF viewer](https://sarifweb.azurewebsites.net/#viewersSectionTitle))

  - rather, this project aims to have a simple analysis result format in order to have a low bar of entry for new analysis tools

- A platform for disclosing new security vulnerabilities (such as https://www.hackerone.com)

  - rather, this project contains simple descriptions of the weaknesses of the above vulnerabilities



We do :heart: all of the initiatives mentioned above!



## Contributing to the project



We welcome contributions to the OpenSSF CVE Benchmark project.  If you want

to contribute, then please go ahead and open a pull request! See

additional details in our [contributing guidelines](CONTRIBUTING.md) and our [FAQ](docs/FAQ.md).



## About us: the OpenSSF Security Tooling working group



The OpenSSF CVE Benchmark project was initiated by the [OpenSSF](https://openssf.org/)'s [Security Tooling working group](https://github.com/ossf/wg-security-tooling). The working group contains a wide variety of members; some contribute independently, others join through their roles at organizations like Google, GitHub, Mozilla, Tencent, and OWASP. The working group typically meets (virtually) once every two week; [all are welcome](https://github.com/ossf/wg-security-tooling#meeting-times).