Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/influxdata/sedg

Tools for tracking security issues flexibly in version control
https://github.com/influxdata/sedg

cve-tracking cve-triage security vulnerability-tracking

Last synced: 2 months ago
JSON representation

Tools for tracking security issues flexibly in version control

Awesome Lists containing this project

README 

        
# Using



 1. Clone this repo (https://github.com/influxdata/sedg)



 2. Create a venv for it (a venv is recommended to ensure you have the proper

    versions of dependencies to avoid mismatches and bugs (eg, requests_cache

    1.0.1 is known to have issues with system python packages from Ubuntu 20.04

    LTS)):



    ```

    $ cd /path/to/sedg

    $ make install-venv

    ```



    `make install-venv` does the following:

    ```

    $ python3 -m venv .venv

    $ . ./.venv/bin/activate

    (venv) $ pip install ./requirements.txt

    ```



 3. Clone CVE data into `/path/to/cve-data`



 4. Optionally create a shell function to setup your environment for using sedg

    with your CVE data. Eg, in `~/.bashrc`:



    ```

    cve_env() {

        sedgpath="/path/to/sedg"



        test "$VIRTUAL_ENV" = "$sedgpath/.venv" && return  # already activated



        echo "Entering venv for 'sedg' tooling. Use 'deactivate' when done."

        VIRTUAL_ENV_DISABLE_PROMPT=1 . "$sedgpath/.venv/bin/activate"



        cd "/path/to/cve-data"

    }

    ```



 4. Create `~/.config/sedg/sedg.conf` to have:



    ```

    [Locations]

    cve-data = /path/to/cve-data

    ```



    Optionally add any template URLs for GitHub alert template text to the

    `Behavior` section (comma-separated):



    ```

    [Behavior]

    template-urls = https://url1,https://url2

    ```



    Optionally add OCI product overrides. By default, `cve-report gar|quay`

    security scan reports will use `oci/gar-.` or

    `oci/quay-` depending on where the security report is coming

    from. Use `oci-cve-override-where` to override this behavior to always use

    `oci/gar-` or `oci/quay-`. Eg:



    ```

    [Behavior]

    oci-cve-override-where = myorg

    ```



 5. Do stuff



    ```

    # if added shell function

    $ cve_env

    # otherwise

    $ cd /path/to/cve-data

    $ VIRTUAL_ENV_DISABLE_PROMPT=1 . /path/to/sedg/.venv/bin/activate



    # create a CVE against a particular package

    $ cve-add --cve CVE-2020-1234 -p git/foo-org_foo



    # create a placeholder CVE against a particular package

    $ cve-add -c CVE-2020-NNN1 -p git/foo-org_foo



    # create a new placeholder CVE for this year against a particular package

    # (creates CVE--NNN1 if it doesn't exist or incremements the highest

    # found for the year and adds it (eg, if CVE--NNN9 exists,

    # CVE--NN10 is created))

    $ cve-add -c next -p git/foo-org_foo



    # create a GitHub placholder CVE from the GitHub url using this year

    $ cve-add --cve https://github.com/...



    # create a GitHub placeholder CVE with a particular id and package

    # template

    $ cve-add -c CVE-2020-GH1234#foo -p git/foo-org_foo --package-template=bar



    $ 

    $ cve-check-syntax



    # various reports for humans (add --help to see additional options)

    $ cve-report summary			# summary

    $ cve-report todo           		# todo list



    # various machine reports

    $ cve-report influxdb               	# InfluxDB line protocol



    # GitHub-specific reports

    $ cve-report gh --org  --status=active

    $ cve-report gh --org  --alerts --since YYYY-MM-DD



    # GAR container security reports

    $ cve-report gar --list /

    $ cve-report gar --list-digest ///

    $ cve-report gar --alerts --name ///@



    # quay.io container security reports

    $ cve-report quay --list 

    $ cve-report quay --list-digest /

    $ cve-report quay --alerts --name /@



    # Docker DSO container security reports

    $ cve-report dso --list-digest [:|@]

    $ cve-report dso --alerts --name [:|@]



    # if desired, leave the venv

    $ deactivate

    ```



 6. (Optional) If vim user, symlink `extras/vim/syntax/cve.vim` in

    `~/.vim/syntax` or `~/.config/nvim/syntax`, symlink

    `extras/vim/indent/cve.vim` in `~/.vim/indent` or `~/.config/nvim/indent`

    and add to VIMRC:

    ```

    " CVEs

    augroup cve

      autocmd!

      autocmd BufNewFile,BufRead CVE-[0-9][0-9][0-9][0-9]-[0-9GN]* set ft=cve

      autocmd BufNewFile,BufRead CVE-[0-9][0-9][0-9][0-9]-[0-9GN]* set syntax=cve

      autocmd BufNewFile,BufRead CVE-[0-9][0-9][0-9][0-9]-[0-9GN]* set autoindent

      autocmd BufNewFile,BufRead CVE-[0-9][0-9][0-9][0-9]-[0-9GN]* setlocal indentexpr=GetCVEIndent()

      autocmd BufNewFile,BufRead */path/to/cve-data-dir/templates/* set ft=cve

      autocmd BufNewFile,BufRead */path/to/cve-data-dir/templates/* set syntax=cve

      autocmd BufNewFile,BufRead */path/to/cve-data-dir/templates/* set autoindent

      autocmd BufNewFile,BufRead */path/to/cve-data-dir/templates/* setlocal indentexpr=GetCVEIndent()

    augroup END

    ```



# Staying up to date



Assuming you've setup sedg as described in 'Using', can stay up to date with:



    ```

    $ cd /path/to/sedg

    $ git checkout main

    $ git fetch && git pull --ff-only

    $ make install-venv

    ```



# Uninstalling



Since `sedg` is installed to a venv, uninstall is as simple as:



    ```

    $ rm -rf /path/to/sedg/.venv

    ```



# Tests



Development should be done in a venv, see 'Using' above to set one up. Once

setup, develop like so:



    $ cd /path/to/sedg

    $ VIRTUAL_ENV_DISABLE_PROMPT=1 . ./venv/bin/activate

    $ pip install .[dev]   # install extra development dependencies

    $

    

    $ deactivate



Run all checks:



    $ make check



Run unittests:



    $ make test

    ...

    Ran 116 tests in 0.219s

    OK



or a single test file:



    # template

    $ python3 -m unittest tests.test_foo

    # example

    $ python3 -m unittest tests.test_cve



or a single test:



    # template

    $ python3 -m unittest tests.test_foo.TestFoo.test_bar

    # example

    $ python3 -m unittest tests.test_cve.TestCve.test___init__valid



# CVE File format



The CVE file format follows RFC5322. In essence:



```

    Field1: single-line value

    Field2:

     multi-line values start on the following line with each line always

     preceded by a space

```



This file format is compliant with various language libraries, such as Python's

email.policy.Compat32 (compat since it doesn't conform to Python's shipping

policies). Since RFC5322 doesn't support utf-8, utf-8 is not properly supported

by the file format. The format could be moved to RFC6532 at a future date.

Items within `[]` are optional.



```

    Candidate: CVE-- | CVE--GH# | CVE--NNNX

    OpenDate: YYYY-MM-DD [ HH:MM:SS [ TZ|+-N ] ]

    CloseDate: YYYY-MM-DD [ HH:MM:SS [ TZ|+-N ] ]

    PublicDate: YYYY-MM-DD [ HH:MM:SS [ TZ|+-N ] ]

    CRD: YYYY-MM-DD [ HH:MM:SS [ TZ|+-N ] ]

    References:

     

      (with comment)

    Description:

     Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod

     tempor incididunt ut labore et dolore magna aliqua. Aliquam sem et tortor

     consequat id porta nibh venenatis.

     .

     Tellus orci ac auctor augue mauris augue neque.

    [GitHub-Advanced-Security:

     - type: dependabot

       dependency: <3rd party software>

       detectedIn: path/to/file

       advisory: https://github.com/advisories/GHSA-... | unavailable

       severity: low | moderate | high | critical

       status: needs-triage | needed | released | removed | dismissed (; )

       url: https://github.com/ORG/REPO/security/dependabot/N | unavailable

     - type: secret

       secret: 

       detectedIn: path/to/file

       status: needs-triage |needed | released | dismissed (; )

       url: https://github.com/ORG/REPO/security/secret-scanning/N | unavailable]

    [Scan-Reports:

     - type: oci

       component: 

       detectedIn: 

       advisory:  | unavailable

       fixedBy:  | unavailable

       severity: low | moderate | high | critical

       status: needs-triage|needed|released|dismissed (; )

       url:  | unavailable]

    Notes:

     person> One line note

     person> Multi-line note. Lorem ipsum dolor sit amet, consectetur adipiscing

      elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

      Aliquam sem et tortor consequat id porta nibh venenatis.

      .

      Tellus orci ac auctor augue mauris augue neque.

    Mitigation: 

    Bugs:

     

     

    Priority: negligible | low | medium | high | critical

    Discovered-by: First Last [ @ |  ], ...

    Assigned-to: First Last [ @ |  ]

    CVSS:

     :  [( [low|medium|high|critical)]



    Patches_:

     upstream | vendor | distro | other: 

    [CloseDate_: YYYY-MM-DD [ HH:MM:SS [ TZ|+-N ] ]

    [CloseDate_[/modifier]: YYYY-MM-DD [ HH:MM:SS [ TZ|+-N ] ]

    [Tags_:  ]

    [Tags_[/]:  ]

    [Priority_: negligible | low | medium | high | critical]

    [Priority_[/modifier]: negligible | low | medium | high | critical]

    [/]_[/]:  [()]

    [/]_[/]:  [()]



    Patches_:

     upstream | vendor | distro | other: 

    [CloseDate_: YYYY-MM-DD [ HH:MM:SS [ TZ|+-N ] ]

    [CloseDate_[/modifier]: YYYY-MM-DD [ HH:MM:SS [ TZ|+-N ] ]

    [Tags_:  ]

    [Tags_[/modifier]:  ]

    [Priority_: negligible | low | medium | high | critical]

    [Priority_[/]: negligible | low | medium | high | critical]

    [/]_[/]:  [()]

    [/]_[/]:  [()]



    ...  ...

```



The CVE format can be thought of in two sections: the global section and the

software section. Since CVE identifiers often apply to more than one product,

software version, etc, the global section is meant to generally apply to all

software declared in the CVE, whereas the software section lists specifics for

each software product, version, etc. In this manner, a single CVE file may

contain all pertinent information for the CVE on the whole with per-software

entries showing status for this issue.



## Global section



For each field in the global section:

 * `Candidate` is the CVE identifier for this issue. 3 formats are supported:

   * `CVE--` (eg, `CVE-2020-1234`) is used for tracking publicly

     assigned issues

   * `CVE--NNNX` (eg, `CVE-2020-NNN1` or `CVE-2020-NNN2`) is used for

     tracking non-public issues. This might be useful for tracking before a

     public assignment or when tracking a private issue. By convention, when

     first creating a non-public CVE, choose the current year and increment the

     value of `X` like so: `CVE-2020-NNN1`, `CVE-2020-NNN2`, ...,

     `CVE-2020-NNN9`, `CVE-2020-NN10`, ..., `CVE-2020-N999`, `CVE-2020-N1000`,

     etc)

   * `CVE--GH#` (eg, `CVE-2020-GH1234#foo`) is used

     for tracking a non-public or a not-yet-assigned-CVE identifier issue in

     GitHub. It encodes both the GitHub repository and issue number/pull

     request, such that `CVE-2020-GH1234#foo` corresponds with

     `https://github.com//foo/issues/1234`. This is particularly

     useful for organizations with lots of repositories.

 * `OpenDate` is the date the CVE was created

 * `CloseDate` is the date the CVE was closed (nothing 'need-triage', 'needed'

   or 'pending'). Per-software close date overrides are possible in the

   software section (below).

 * `PublicDate` is the date that a private CVE was made public

 * `CRD` is the Coordinated Release Date. This can be used for coordinating

   with other parties to make a CVE public at a specific time

 * `References` contains a list of URLs, one per line, relevant for the CVE

 * `Description` contains the general description for the CVE. It may contain

   details for a specific piece of software (but typically not details on how

   other software might use it)

 * `GitHub-Advanced-Security` is optional and may be used to track GitHub

   dependabot alerts and secret scanning

 * `Scan-Reports` is optional and may be used to track OCI scan reports

 * `Notes` contains information from triagers, developers, etc to give more

   insight into the vulnerability, particularly how it pertains to individual

   pieces of software, configuration, build artifacts, etc. It will often

   contain rationale for entries in the software section (below), but may

   contain anything that is relevant

 * `Mitigation` contains information on how available mitigations may be

   applied before patching

 * `Bugs` contains a list of Bug tracker URLs, one per line. These may also be

   listed in the References section

 * `Priority` contains the priority for fixing the CVE, typically as it pertains

   to the item described in the `Description` (above). Per-software priority

   overrides are possible in the software section (below). Supported

   priorities are:

   * `negligible`

   * `low`

   * `medium`

   * `high`

   * `critical`

 * `Discovered-by` is a comma-separated list of names, slack handles, GitHub

   usernames, etc and used for attribution. `gh-dependabot` and `gh-secret` are

   used when using `GitHub-Advanced-Security` (above). `quay.io`, `gar` and

   `docker` are used when using `Scan-Reports` (above) with specific

   report URLs.

 * `Assigned-to` contains the name, slack handle, GitHub username, etc of the

   person assigned to this issue

 * `CVSS` contains a list of CVSS scores as assigned by different entities (eg,

   'nvd', 'redhat', 'ubuntu', etc)



## Software section



By convention, the software section is typically divided into stanzas grouped

by software. The blank line before each software section is not required but is

normally used since it makes the CVE easier to read.



For each field in the software section:

 * `Patches_` (eg, `Patches_foo` or `Patches_bar`) contains a list

   of patch URLs, one per line, with provenance prefixed

 * `CloseDate_[/modifier]` (eg, `CloseDate_foo` or

   `CloseDate_bar/v1`) is optional and when specified contains a close date

   override for `[/modifier]`

 * `Tags_[/modifier]` (eg, `Tags_foo` or `Tags_bar/v1`) is optional

   and when specified contains a comma-separated list of tags. Tags have

   historically been used to describe a mitigating factor provided by the OS

   (ie, without user intervention, so different than the `Mitigation` section,

   above). The `limit-report` tag is used to manipulate reports in different

   ways. Currently supported tags are:

   * `apparmor`

   * `fortify-source`

   * `hardlink-restriction`

   * `heap-protector`

   * `limit-report`

   * `pie`

   * `stack-protector`

   * `symlink-restriction`

 * `Priority_[/modifier]` (eg, `Priority_foo` or `Priority_bar/v1`)

   is optional and when specified contains a priority override for

   `[/modifier]`

 * `` is the supporting technology (eg, a VCS system, build artifact

   or operating system) or simply `upstream`. Supported ``s:

   * VCS: bzr, cvs, git, hg or svn

   * build artifacts: appimage, archive, deb, dmg, exe, flatpak, oci, rpm,

     shell, snap

   * OS: alpine, android, centos, debian, distroless, flatcar, ios, opensuse,

     osx, rhel, suse, ubuntu, windows

 * `` indicates where the software lives or in the case of technologies

   with a concept of publishers, who the publisher is. For OS (eg, `ubuntu`,

   `debian`, `suse`, etc), `` indicates the release of the distribution

   (eg, `ubuntu/focal` indicates 20.04 for Ubuntu where `ubuntu` is the

   `` (distro) and `focal` is the `` (distro release)). For

   `git`, `` could indicate the source code hosting provider (eg,

   `github`, `gitlab`, `launchpad`, etc) or could be used to indicate the

   organization/user in a hosting provider. Eg, `git/github_foo` indicates that

   `foo` is somewhere in GitHub whereas `git/bar_foo` could be used to indicate

   that `foo` is in some provider's organization/user. How `` is used is

   not prescribed and the `/` combination is not meant to

   capture everything where the software lives. It is meant to be flexible to

   allow triagers a means to discern these differences as appropriate for their

   requirements (eg, perhaps the triager wants to track the status of the

   `targz` vs `zip`, one might use `archive/targz` and `archive/zip`).

 * `` is the name of the software as dictated by the product (eg, the

   name of the github project, the name of the OCI image, the deb source

   package, the name of the build artifact, etc)

 * `` is an optional key for grouping collections of packages (eg,

   'v1' for the project's `v1` branch, `v2`, etc)

 * `` indicates the status of fixing the issue for this software (eg,

   `needs-triage`, `needed`, `pending`, `released`, `not-affected` and

   `deferred`).

 * `` is optional and when specified occurs within parentheses and

   indicates when the software will be/was fixed when used with the `pending`,

   `released` or `not-affected` status. When may be a version number (for

   software with releases), a git hash (for continuous development), a revision

   number or a date (eg, for an OCI image).

   * As a special case for `not-affected` and `deferred` the ``

     parenthetical might give a hint as to why (eg `code-not-present` with

     `not-affected`). Typical hints for 'not-affected':

     * `code-not-present`: project does not contain the vulnerable code path

     * `code-not-compiled`: project does not compile the vulnerable code path

     * `code-not-imported`: project does not import the vulnerable code path

     * `code-not-used`: project contains/imports/etc the code but does use it

     * YYYY-MM-DD with deferred: date when entry was put into the deferred

       status

   * `not-affected` and `released` are similar but convey different things. Eg

     `not-affected (1.2.3)` vs `released (1.2.1+patch1)` is saying that `1.2.3`

     wasn't ever affected (ie, didn't require anything to be done) but

     `1.2.1+patch1` was affected but is now fixed. This can be useful for

     software following a release schedule or tracking updates within a Linux

     distribution.



Typical software stanza examples:

```

    # Simple example for some upstream project

    Patches_foo:

    upstream_foo: released (1.2)



    # github-hosted example for https://github.com/barproj/bar

    Patches_bar:

    git/barprog_bar: needed



    # github-hosted example for https://github.com/barproj/baz with different

    # branches for different releases for v1/v2 and continuous development on

    # main

    Patches_baz:

     upstream: https://github.com/org/baz/pull/123

    git/barproj_baz/v1: pending

    git/barproj_baz/v2: released (2.0.13)

    git/barproj_baz/main: released (907e560b)



    # OCI images for different registries

    Patches_norf:

    Tags_norf: pie

    oci/dockerhub_norf: needs-triage



    # Linux distribution

    Patches_corge:

    upstream_corge: released (1.2)

    debian/buster_corge: needed

    ubuntu/xenial_corge: not-affected (code-not-present)

    ubuntu/bionic_corge: needed

    ubuntu/focal_corge: not-affected (1.2-3)

    ubuntu/groovy_corge: not-affected (1.4-1)



    # Snap for different publishers

    Patches_qux:

    snap/pub1_qux: released (1234)

    snap/pub2_qux: not-affected (code-not-compiled)

```



The format offers considerable flexibility. For example, to capture upstream vs

GitLab vs GitHub (where, for example, GitHub and GitLab might be forks of

upstream), one might use:



```

    Patches_foo:

    upstream_foo: needed

    github/someacct_foo: needed

    gitlab/otheracct_foo: needed

```



## Ubuntu compatibility

The file format was initially defined in Ubuntu and the syntax above is largely

compatible with the Ubuntu CVE tracker (UCT) and a longer term goal of this

project is to push this tooling to UCT. The above format is more generalized,

strict and applicable to other projects. If using this tooling with UCT data,

then adjust `~/.config/sedg_ubuntu.conf` to contain:



```

    [Behavior] compat-ubuntu = yes

```



When specifying compat mode:

* `CVSS` is single line instead of multiline list. Eg:

  `CVSS:  [\[ LOW|MEDIUM|HIGH|CRITICAL\]]`

* `` may specify Ubuntu releases as a shorthand (eg, `focal` instead

  of `ubuntu/focal`)

* patches can specify various other types (eg, in addition to `distro`,

  `other`, `upstream` and `vendor`, allow `debdiff`, `diff`, `fork`, `merge`,

  etc)

* For `Tags...` and `Priority...`, disallow `_` in ``, allow `/` in

  `` and allow `_` as the delimiter for their use of

  `_[/]` (eg, `Tags_foo_precise/esm`)



Package stanzas then become:

```

    Patches_:

     upstream | vendor | debdiff | other | debdiff | diff | ...: 

    [Tags_:  ]

    [Tags_[_[/]]:  ]

    [Priority_: negligible | low | medium | high | critical]

    [Priority_[_[/]]: negligible | low | medium | high | critical]

    _[/]:  [()]

    [/]_[/]:  [()]

```



# Monitoring



`bin/cve-report` can be used to generate human-readable output or line protocol

suitable for sending to InfluxDB. Eg:



```

$ cve-report influxdb

cvelog,priority=medium,status=needed,product=git id="CVE-2020-1234",software="foo",modifier="" 1633040641675003246

...



# or pipe into 'influx write' (InfluxData Cloud 2 or InfluxDB 2.x):

$ cve-report influxdb | influx write --host $URL --org-id $ORGID --bucket-id $BUCKETID --token $TOKEN



# or pipe into 'curl' (omit '&org=$INFLUX_ORG' when not using InfluxData Cloud

# 2 and InfluxDB 2.x (eg, InfluxData Cloud Dedicated))

$ export INFLUX_URL=https://...

$ export INFLUX_BUCKET=...

$ export INFLUX_ORG=...

$  export INFLUX_TOKEN=...

$ cve-report influxdb | curl --header "Authorization: Bearer $INFLUX_TOKEN" --header "Content-Type: text/plain; charset=utf-8" --header "Accept: application/json" --request POST "$INFLUX_HOST/api/v2/write?bucket=$INFLUX_BUCKET&org=$INFLUX_ORG" --data-binary @-

```



For now, paste this into 'Add Data/Line Protocol' whenever you make relevant

changes to the CVE data. A tool will eventually be provided to make this

easier.



This line protocol can then be queried in various ways. Since CVE data is not

expected to be updated on a daily basis, the following techniques of using two

`aggregateWindow()` functions, the first with `createEmpty: false` and the

second with `createEmpty: true` (the default) and a `noop`, allows for graphs

to be filled in for any days that are missing (both in the middle and at the

end). The start time must necessarily have at least one point to work. If

sending data in daily, you can skip the `noop()` function and at the end use a

single `aggregateWindow(every: 1d, count)` without a `fill()`.



It is possible to backfill by checking out a commit and running `cve-report`

with `--starttime`. Eg:

```

$ cve-report influxdb --starttime $(date --date "8 days ago" "+%s")

```



TODO: there is also a telegraf/github plugin that could be investigated.



## Total unique open issues

```

from(bucket: "sec-issues")

  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)

  |> filter(fn: (r) => r["_measurement"] == "cvelog" and r["_field"] == "id")

  |> keep(columns: ["_time", "_value", "_field", "priority"])

  |> window(every: 1d)

  |> unique()

  |> group(columns: ["priority"])

  // https://community.influxdata.com/t/advice-how-to-carry-forward-data-from-the-previous-day/21895

  // get all the counts, but don't create any empty data since count() has the

  // side-effect of turning nulls to 0s

  |> aggregateWindow(every: 1d, createEmpty: false, fn: count)

  // create the empty data with nulls intact

  |> aggregateWindow(every: 1d, fn: (tables=<-, column="_value") => tables)

  // convert nulls in "_value" to the previous row

  |> fill(usePrevious: true)

```



Best viewed as stacked graph with static legend.



## Open issues in affected software



Grouped by software:

```

import "strings"

from(bucket: "sec-issues")

  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)

  |> filter(fn: (r) => r["_measurement"] == "cvelog" and (r["_field"] == "id" or r["_field"] == "software"))

  |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")

  |> map(fn: (r) => ({

    r with

    _value: strings.joinStr(arr: [r.software, r.id, r.priority], v: ":")

  }))

  |> drop(columns: ["id", "product", "status"])

  |> group(columns: ["software"])

  |> window(every: 1d)

  |> unique()

  // https://community.influxdata.com/t/advice-how-to-carry-forward-data-from-the-previous-day/21895

  |> aggregateWindow(every: 1d, createEmpty: false, fn: count)

  |> aggregateWindow(every: 1d, fn: (tables=<-, column="_value") => tables)

  |> fill(usePrevious: true)

```



Best viewed as stacked graph with static legend.



Old (grouped by priority):

```

import "strings"

from(bucket: "sec-issues")

  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)

  |> filter(fn: (r) => r["_measurement"] == "cvelog" and (r["_field"] == "id" or r["_field"] == "software"))

  |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")

  |> map(fn: (r) => ({

    r with

    _value: strings.joinStr(arr: [r.software, r.id, r.priority], v: ":")

  }))

  |> drop(columns: ["id", "product", "status", "software"])

  |> group(columns: ["priority"])

  |> window(every: 1d)

  |> unique()

  |> aggregateWindow(every: 1d, fn: count)

```



### Open issues by software/priority

```

import "strings"

from(bucket: "sec-issues")

  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)

  |> filter(fn: (r) => r["_measurement"] == "cvelog" and (r["_field"] == "id" or r["_field"] == "software"))

  |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")

  |> map(fn: (r) => ({

    r with

    tuple: strings.joinStr(arr: [r.software, r.priority], v: ":"),

    _value: strings.joinStr(arr: [r.software, r.priority, r.id], v: ":")

  }))

  |> drop(columns: ["id", "product", "status", "software"])

  |> group(columns: ["tuple"])

  |> window(every: 1d)

  |> unique()

  // https://community.influxdata.com/t/advice-how-to-carry-forward-data-from-the-previous-day/21895

  |> aggregateWindow(every: 1d, createEmpty: false, fn: count)

  |> aggregateWindow(every: 1d, fn: (tables=<-, column="_value") => tables)

  |> fill(usePrevious: true)

```



Best viewed as stacked graph with static legend.



### Alert on open issues

```

import "influxdata/influxdb/secrets"

import "slack"



// secret is https://hooks.slack.com/services/X/Y/Z

webhook_url = secrets.get(key: "my-webhook-url")

endpoint = slack.endpoint(url: webhook_url)



mapFnCrit = (r) => ({

  text: if r._value == 1 then "${r._value} open critical security issue" else "${r._value} open critical security issues",

  color: "danger",

  channel: "",

})

toSlackCrit = endpoint(mapFn: mapFnCrit)



mapFnHigh = (r) => ({

  text: if r._value == 1 then "${r._value} open high security issue" else "${r._value} open high security issues",

  color: "danger",

  channel: "",

})

toSlackHigh = endpoint(mapFn: mapFnHigh)



critlvl = 0

highlvl = 0



// grab everything from the last 30 days and filter down once so we don't

// have to keep doing it. We'll filter down more as we go.

data = from(bucket: "jdstrand-sec-stats")

  |> range(start: -30d, stop: now())

  |> filter(fn: (r) => r["_measurement"] == "cvelog" and r["_field"] == "id")



getLatest = (tables=<-) => {

  row = tables

    |> group(columns: ["_time"])

    |> aggregateWindow(every: 1d, createEmpty: false, fn: count)

    |> group(columns: ["_time"], mode: "except")

    |> last()

    |> limit(n: 1)

    // we only have 1 row at this point, but fn must be specified

    |> findRecord(fn: (key) => true, idx: 0)

  return row._time

}



latest = data

  |> getLatest()



// check for statuses for the 24 hour period before time of the latest entry

checkStatus = (tables=<-, priority, threshold) =>

  tables

    |> range(start: -24h, stop: latest)

    |> filter(fn: (r) => r["priority"] == priority)

    |> group(columns: ["priority"])

    |> window(every: 1d)

    |> unique()

    |> aggregateWindow(every: 1d, createEmpty: false, fn: count)

    |> last()

    |> limit(n: 1)

    |> filter(fn: (r) => r["_value"] > threshold)



crit = data

  |> checkStatus(priority: "critical", threshold: critlvl)

  |> toSlackCrit()

  |> yield(name: "critical")



high = data

  |> checkStatus(priority: "high", threshold: highlvl)

  |> toSlackHigh()

  |> yield(name: "high")

```



# GitHub and CVE data

GitHub does not provide easy mechanisms to subscribe to labels in repos or

across the org and also doesn't provide issue label information in their email

headers for bug comments. Combined, we must poll GitHub for information to

detect new issues or issues that have received updates. The

`cve-report` tool aims to address this gap. Example usage:

```

    # first export a GitHub Personal Access Token that can read issues:

    $  export GHTOKEN=...



    # Show issues that are referenced in open CVE data that have been

    # updated since last week

    $ cve-report gh --updated --org foo --since $(date --date "7 days ago" "+%s")

    Updated issues:

     https://github.com/foo/bar/issues/123

     https://github.com/foo/baz/issues/234



    # Show list of issues for specific repos in an org with different

    # labels, but without label 'skipme'

    $ cve-report gh --missing \

        --org foo \

        --labels="bar:baz" \

        --excluded-labels="skipme" \

        --repos=norf,corge,qux

    Fetching list of repos: ...... done!

    Fetching list of issues for:

     foo/corge: .. done!

     foo/norf: .. done!

     foo/qux: . done!

     ...

    Issues missing from CVE data:

     https://github.com/foo/corge/issues/345

     https://github.com/foo/quz/issues/456



    # Show dependabot alerts since last stamp file update (does not filter on

    # 'active' status)

    $ cve-report gh --alerts \

        --org foo \

        --since-stamp /path/to/stamp

    Collecting repo status: [#########################################] 100%

    Collecting alerts: [##############################################] 100%

    Updated vulnerability alerts:

     bar (https://github.com/foo/bar/security/dependabot):

      lodash

      - severity: high

      - yarn.lock

      - https://github.com/advisories/GHSA-35jh-r3h4-6jhm

```



`cve-report gh --updated` also supports `--since-stamp` as a convenience and

will set the since time to the `mtime` of the specified file. Eg, to bootstrap

and then just use the stamp file, use:

```

    # first time only

    $ cve-report gh --updated --org foo --since $(date --date "7 days ago" "+%s")



    # hereafter

    $ cve-report gh --updated \

        --org foo \

        --since-stamp /path/to/stamp

```



The `cve-report` script can also be used to show a few things about GitHub

repos. Eg, to show archived repos:



```

$ cve-report gh --org foo --status=archived

norf

```



To show active repos:

```

$ cve-report gh --org foo --status=active

bar

baz

```



This can, for example, by used to manage a file of active repos that can be fed

into other tools. Eg:

```

# for private repos

$  export GHUSER=...

$  export GHTOKEN=...

$ cve-report gh --org foo --status=active | sort -f > foo-repos.txt

$ cve-report summary --software="foo-repos.txt"

```