Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/vulsio/go-cve-dictionary

Build a local copy of CVE (NVD and Japanese JVN). Server mode for easy querying.
https://github.com/vulsio/go-cve-dictionary

Last synced: 3 months ago
JSON representation

Build a local copy of CVE (NVD and Japanese JVN). Server mode for easy querying.

Host: GitHub
URL: https://github.com/vulsio/go-cve-dictionary
Owner: vulsio
License: apache-2.0
Created: 2016-03-24T09:05:49.000Z (almost 9 years ago)
Default Branch: master
Last Pushed: 2024-07-15T07:54:22.000Z (6 months ago)
Last Synced: 2024-08-11T17:10:13.734Z (5 months ago)
Language: Go
Homepage:
Size: 992 KB
Stars: 361
Watchers: 20
Forks: 109
Open Issues: 7
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE

Awesome Lists containing this project

awesome-repositories - vulsio/go-cve-dictionary - Build a local copy of CVE (NVD and Japanese JVN). Server mode for easy querying. (Go)

README

# go-cve-dictionary

This is tool to build a local copy of the NVD (National Vulnerabilities Database) [1]
and the Japanese JVN [2], which contain security vulnerabilities according to their
CVE identifiers [3] including exhaustive information and a risk score. The local
copy is generated in sqlite format, and the tool has a server mode for easy querying.

[1] https://en.wikipedia.org/wiki/National_Vulnerability_Database
[2] https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures
[3] http://jvndb.jvn.jp/apis/termsofuse.html

## Installation

### Install requirements

go-cve-dictionary requires the following packages.

- SQLite3, MySQL, PostgreSQL or Redis
- git
- gcc
- go v1.7.1 or later
- https://golang.org/doc/install

Here's an example for Amazon EC2 server.

```bash
$ ssh [email protected] -i ~/.ssh/private.pem
$ sudo yum -y install sqlite git gcc
$ wget https://storage.googleapis.com/golang/go1.7.1.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go1.7.1.linux-amd64.tar.gz
$ mkdir $HOME/go
```

Put these lines into /etc/profile.d/goenv.sh

```bash
export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
```

Set the OS environment variable to current shell

```bash
$ source /etc/profile.d/goenv.sh
```

### Deploy go-cve-dictionary

To install:

```bash
$ mkdir -p $GOPATH/src/github.com/vulsio
$ cd $GOPATH/src/github.com/vulsio
$ git clone https://github.com/vulsio/go-cve-dictionary.git
$ cd go-cve-dictionary
$ make install
```

Create a log output directory.
You can use another directory on the command line option (--log-dir).

```bash
$ sudo mkdir /var/log/go-cve-dictionary
$ sudo chown ec2-user /var/log/go-cve-dictionary
$ sudo chmod 700 /var/log/go-cve-dictionary
```

Fetch vulnerability data from NVD.
```bash
$ go-cve-dictionary fetch nvd
... snip ...
$ ls -alh cve.sqlite3
-rw-r--r-- 1 ec2-user ec2-user 7.0M Mar 24 13:20 cve.sqlite3
```

Now we have vulnerability data.
Start go-cve-dictionary as server mode.

```bash
$ go-cve-dictionary server
[Mar 24 15:21:55] INFO Opening DB. datafile: /home/ec2-user/cve.sqlite3
[Mar 24 15:21:55] INFO Migrating DB
[Mar 24 15:21:56] INFO Starting HTTP Sever...
[Mar 24 15:21:56] INFO Listening on 127.0.0.1:1323
```

### Update go-cve-dictionary

If the DB schema was changed, please specify new SQLite3, MySQL, PostgreSQL or Redis DB file.

```bash
$ cd $GOPATH/src/github.com/vulsio/go-cve-dictionary
$ git pull
$ rm -r vendor
$ make install
```

Binary files are created under $GOPATH/bin

----

## Sample data sources

### Hello HeartBleed

```bash
$ curl http://127.0.0.1:1323/cves/CVE-2014-0160 | jq "."
{
"CveID": "CVE-2014-0160",
"Nvd": {
"Summary": "The (1) TLS and (2) DTLS implementations in OpenSSL 1.0.1 before 1.0.1g do not properly handle Heartbeat Extension packets, which allows remote attackers to obtain sensitive information from process memory via crafted packets that trigger a buffer over-read, as demonstrated by reading private keys, related to d1_both.c and t1_lib.c, aka the Heartbleed bug.",
"Score": 5,
"AccessVector": "NETWORK",
"AccessComplexity": "LOW",
"Authentication": "NONE",
"ConfidentialityImpact": "PARTIAL",
"IntegrityImpact": "NONE",
"AvailabilityImpact": "NONE",
"Cpes": null,
"References": [
{
"Source": "CERT",
"Link": "http://www.us-cert.gov/ncas/alerts/TA14-098A"
},
...snip...
],
"PublishedDate": "2014-04-07T18:55:03.893-04:00",
"LastModifiedDate": "2015-10-22T10:19:38.453-04:00"
},
"Jvn": {
"Title": "OpenSSL の heartbeat 拡張に情報漏えいの脆弱性",
"Summary": "OpenSSL の heartbeat 拡張の実装には、情報漏えいの脆弱性が存在します。TLS や DTLS 通信において OpenSSL のコードを実行しているプロセスのメモリ内容が通信相手に漏えいする可能性があります。",
"JvnLink": "http://jvndb.jvn.jp/ja/contents/2014/JVNDB-2014-001920.html",
"JvnID": "JVNDB-2014-001920",
"Score": 5,
"Severity": "Medium",
"Vector": "(AV:N/AC:L/Au:N/C:P/I:N/A:N)",
"References": [
{
"Source": "AT-POLICE",
"Link": "http://www.npa.go.jp/cyberpolice/detect/pdf/20140410.pdf"
},
...snip...
],
"Cpes": null,
"PublishedDate": "2014-04-08T16:13:59+09:00",
"LastModifiedDate": "2014-04-08T16:13:59+09:00"
}
}

```

### Hello Ruby on Rails 4.0.2

```bash
$ curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"name": "cpe:/a:rubyonrails:ruby_on_rails:4.0.2:-"}' http://localhost:1323/cpes | jq "."
[
{
"CveID": "CVE-2016-0751",
"Nvd": {
"CveDetailID": 345,
"Summary": "actionpack/lib/action_dispatch/http/mime_type.rb in Action Pack in Ruby on Rails before 3.2.22.1, 4.0.x and 4.1.x before 4.1.14.1, 4.2.x before 4.2.5.1, and 5.x before 5.0.0.beta1.1 does not properly restrict use of the MIME type cache, which allows remote attackers to cause a denial of service (memory consumption) via a crafted HTTP Accept header.",
"Score": 5,
"AccessVector": "NETWORK",
"AccessComplexity": "LOW",
"Authentication": "NONE",
"ConfidentialityImpact": "NONE",
"IntegrityImpact": "NONE",
"AvailabilityImpact": "PARTIAL",
"Cpes": null,
"References": [
{
"Source": "MLIST",
"Link": "https://groups.google.com/forum/message/raw?msg=ruby-security-ann/9oLY_FCzvoc/5CDXbvpYEgAJ"
},
{
"Source": "MLIST",
"Link": "http://www.openwall.com/lists/oss-security/2016/01/25/9"
}
],
"PublishedDate": "2016-02-15T21:59:05.877-05:00",
"LastModifiedDate": "2016-03-18T21:02:43.817-04:00"
},
"Jvn": {
"Title": "",
"Summary": "",
"JvnLink": "",
"JvnID": "",
"Score": 0,
"Severity": "",
"Vector": "",
"References": null,
"Cpes": null,
"PublishedDate": "0001-01-01T00:00:00Z",
"LastModifiedDate": "0001-01-01T00:00:00Z"
}
},
... snip ...
]
```

----

## Usage
```bash
$ go-cve-dictionary --help
GO CVE Dictionary

Usage:
go-cve-dictionary [command]

Available Commands:
completion generate the autocompletion script for the specified shell
fetch Fetch Vulnerability dictionary
help Help about any command
search Search for Vulnerability in the dictionary
server Start CVE dictionary HTTP Server
version Show version

Flags:
--config string config file (default is $HOME/.go-cve-dictionary.yaml)
--dbpath string /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
--dbtype string Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
--debug debug mode (default: false)
--debug-sql SQL debug mode
-h, --help help for go-cve-dictionary
--http-proxy string http://proxy-url:port (default: empty)
--log-dir string /path/to/log (default "/var/log/go-cve-dictionary")
--log-json output log as JSON
--log-to-file output log to file

Use "go-cve-dictionary [command] --help" for more information about a command.
```
----

### Usage: Fetch Command

```bash
$ go-cve-dictionary fetch --help
Fetch Vulnerability dictionary

Usage:
go-cve-dictionary fetch [command]

Available Commands:
fortinet Fetch Vulnerability dictionary from Fortinet Advisories
jvn Fetch Vulnerability dictionary from JVN
mitre Fetch Vulnerability dictionary from MITRE
nvd Fetch Vulnerability dictionary from NVD

Flags:
--batch-size int The number of batch size to insert. (default 5)
-h, --help help for fetch

Global Flags:
--config string config file (default is $HOME/.go-cve-dictionary.yaml)
--dbpath string /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
--dbtype string Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
--debug debug mode (default: false)
--debug-sql SQL debug mode
--http-proxy string http://proxy-url:port (default: empty)
--log-dir string /path/to/log (default "/var/log/go-cve-dictionary")
--log-json output log as JSON
--log-to-file output log to file

Use "go-cve-dictionary fetch [command] --help" for more information about a command.
```

#### Fetch NVD data
- to fetch all years
```bash
$ go-cve-dictionary fetch nvd
```

- to fetch specific years
```bash
$ go-cve-dictionary fetch nvd 2021
```

#### Fetch JVN data
- to fetch all years
```bash
$ go-cve-dictionary fetch jvn
```

- to fetch specific years
```bash
$ go-cve-dictionary fetch jvn 2021
```

#### Fetch Fortinet data
```bash
$ go-cve-dictionary fetch fortinet
```

#### Fetch MITRE data
- to fetch all years
```bash
$ go-cve-dictionary fetch mitre
```

- to fetch specific years
```bash
$ go-cve-dictionary fetch mitre 2021
```

----

### Usage: Run HTTP Server

```bash
$ go-cve-dictionary server --help
Start CVE dictionary HTTP Server

Usage:
go-cve-dictionary server [flags]

Flags:
--bind string HTTP server bind to IP address (default "127.0.0.1")
-h, --help help for server
--port string HTTP server port number (default "1323")

----

### Usage: Search Command

```bash
$ go-cve-dictionary search --help
Search for Vulnerability in the dictionary

Usage:
go-cve-dictionary search [command]

Available Commands:
cpe Search for Vulnerability in the dictionary by CPE
cve Search for Vulnerability in the dictionary by CVEID

Flags:
-h, --help help for search

Use "go-cve-dictionary search [command] --help" for more information about a command.
```

#### Search All CVE IDs
```bash
$ go-cve-dictionary search cve
[
"CVE-2023-38624",
"CVE-2024-20750",
"CVE-2024-21101",
"CVE-2023-27427",
"CVE-2023-30445",
...
```

#### Search by CVE ID(s)
```bash
$ go-cve-dictionary search cve CVE-2024-3400
{
"CveID": "CVE-2024-3400",
"Nvds": [
{
"CveID": "CVE-2024-3400",
"Descriptions": [
{
"Lang": "en",
"Value": "A command injection as a result of arbitrary file creation vulnerability in the GlobalProtect feature of Palo Alto Networks PAN-OS software for specific PAN-OS versions and distinct feature configurations may enable an unauthenticated attacker to execute arbitrary code with root privileges on the firewall.\n\nCloud NGFW, Panorama appliances, and Prisma Access are not impacted by this vulnerability."
},
...

$ go-cve-dictionary search cve CVE-2023-48783 CVE-2024-3400
{
"CVE-2023-48783": {
"CveID": "CVE-2023-48783",
"Nvds": [
{
"CveID": "CVE-2023-48783",
...
}
],
"Jvns": [],
"Fortinets": [
{
"AdvisoryID": "FG-IR-23-408",
"CveID": "CVE-2023-48783",
...
}
]
},
"CVE-2024-3400": {
"CveID": "CVE-2024-3400",
"Nvds": [
{
"CveID": "CVE-2024-3400",
...
```

#### Search by CPE
```bash
$ go-cve-dictionary search cpe "cpe:/a:fortinet:fortiportal"
[
{
"CveID": "CVE-2017-7337",
"Nvds": [],
"Jvns": [],
"Fortinets": [
{
"AdvisoryID": "FG-IR-17-114",
"CveID": "CVE-2017-7337",
"Title": "FortiPortal Multiple Vulnerabilities",
...
```

#### Search CVE IDs by CPE
```bash
$ go-cve-dictionary search cpe --cveid-only "cpe:/a:fortinet:fortiportal"
{
"Fortinet": [
"CVE-2023-46712",
"CVE-2023-48791",
"CVE-2017-7339",
"CVE-2017-7343",
"CVE-2022-27490",
"CVE-2017-7342",
"CVE-2017-7731",
"CVE-2023-41842",
"CVE-2023-48783",
"CVE-2024-21761",
"CVE-2017-7337",
"CVE-2017-7338",
"CVE-2017-7340"
],
"JVN": [],
"NVD": [
"CVE-2023-46712",
"CVE-2023-48791",
"CVE-2023-41842",
"CVE-2023-48783",
"CVE-2024-21761"
]
}
```

----

### Usage: Use MySQL as a DB storage back-end

- fetch nvd

```bash
$ go-cve-dictionary fetch nvd \
--dbtype mysql \
--dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"
```

- fetch jvn

```bash
$ go-cve-dictionary fetch jvn \
--dbtype mysql \
--dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"
```

- fetch fortinet

```bash
$ go-cve-dictionary fetch fortinet \
--dbtype mysql \
--dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"
```

- fetch mitre

```bash
$ go-cve-dictionary fetch mitre \
--dbtype mysql \
--dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"
```

- server

```bash
$ go-cve-dictionary server \
--dbtype mysql \
--dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"
```

### Usage: Use Postgres as a DB storage back-end

- fetch nvd

```bash
$ go-cve-dictionary fetch nvd \
--dbtype postgres \
--dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"
```

- fetch jvn

```bash
$ go-cve-dictionary fetch jvn \
--dbtype postgres \
--dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"
```

- fetch fortinet

```bash
$ go-cve-dictionary fetch fortinet \
--dbtype postgres \
--dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"
```

- fetch mitre

```bash
$ go-cve-dictionary fetch mitre \
--dbtype postgres \
--dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"
```

- server

```bash
$ go-cve-dictionary server \
--dbtype postgres \
--dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"
```

### Usage: Use Redis as a DB storage back-end

- fetch nvd

```bash
$ go-cve-dictionary fetch nvd \
--dbtype redis \
--dbpath "redis://localhost/0"
```

- fetch jvn

```bash
$ go-cve-dictionary fetch jvn \
--dbtype redis \
--dbpath "redis://localhost/0"
```

- fetch fortinet

```bash
$ go-cve-dictionary fetch fortinet \
--dbtype redis \
--dbpath "redis://localhost/0"
```

- fetch mitre

```bash
$ go-cve-dictionary fetch mitre \
--dbtype redis \
--dbpath "redis://localhost/0"
```

- server

```bash
$ go-cve-dictionary server \
--dbtype redis \
--dbpath "redis://localhost/0"
```

----

## Misc

- HTTP Proxy Support
If your system at behind HTTP proxy, you have to specify -http-proxy option.

- How to daemonize go-cve-dictionary
Use Systemd, Upstart or supervisord, daemontools...

- How to cross compile

```bash
$ cd /path/to/your/local-git-repository/go-cve-dictionary
$ GOOS=linux GOARCH=amd64 go build -o cvedict.amd64
```

- Logging
go-cve-dictionary writes a log under -log-path specified directory (default is /var/log/go-cve-dictionary/).

- Debug
Run with --debug, --debug-sql option.

- Completion Support
- bash
```console
$ go-cve-dictionary completion bash > /usr/share/bash-completion/completions/go-cve-dictionary
```
- zsh
```console
$ go-cve-dictionary completion zsh > ~/.zsh-completions/go-cve-dictionary
```
- fish
```console
$ go-cve-dictionary completion fish > ~/.config/fish/completions/go-cve-dictionary.fish
```
----

## Data Source

- [NVD](https://nvd.nist.gov/)
- [JVN(Japanese)](http://jvndb.jvn.jp/apis/myjvn/)
- [Fortinet(https://www.fortiguard.com/psirt)](https://github.com/vulsio/vuls-data-raw-fortinet)

----

## Authors

kotakanbe ([@kotakanbe](https://twitter.com/kotakanbe)) created go-cve-dictionary and [these fine people](https://github.com/future-architect/go-cve-dictionary/graphs/contributors) have contributed.

----

## How to Contribute

1. fork a repository: github.com/vulsio/go-cve-dictionary to github.com/you/repository
1. get original code: github.com/vulsio/go-cve-dictionary
1. work on original code
1. add remote to your repository: git remote add myfork https://github.com/you/repo.git
1. push your changes: git push myfork
1. create a new Pull Request

- see [GitHub and Go: forking, pull requests, and go-getting](http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html)

----

## Licence

Please see [LICENSE](https://github.com/vulsio/go-cve-dictionary/blob/master/LICENSE).

----

## Additional License

- [NVD](https://nvd.nist.gov/faq)

> How can my organization use the NVD data within our own products and services?
> All NVD data is freely available from our XML Data Feeds. There are no fees, licensing restrictions, or even a requirement to register. All NIST publications are available in the public domain according to Title 17 of the United States Code. Acknowledgment of the NVD when using our information is appreciated. In addition, please email [email protected] to let us know how the information is being used.

- [JVN](http://jvndb.jvn.jp/apis/termsofuse.html)