Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/nvidia-ai-blueprints/vulnerability-analysis

Rapidly identify and mitigate container security vulnerabilities with generative AI.
https://github.com/nvidia-ai-blueprints/vulnerability-analysis

Last synced: 5 months ago
JSON representation

Rapidly identify and mitigate container security vulnerabilities with generative AI.

Awesome Lists containing this project

README 

          





NVIDIA AI Blueprint: Vulnerability Analysis for Container Security



## Table of Contents

- [Table of Contents](#table-of-contents)

- [Overview](#overview)

- [Software components](#software-components)

- [Target audience](#target-audience)

- [Prerequisites](#prerequisites)

- [Hardware requirements](#hardware-requirements)

- [API definition](#api-definition)

- [Use case description](#use-case-description)

  - [How it works](#how-it-works)

  - [Key components](#key-components)

  - [NIM microservices](#nim-microservices)

- [Getting started](#getting-started)

  - [Install system requirements](#install-system-requirements)

  - [Obtain API keys](#obtain-api-keys)

  - [Set up the workflow repository](#set-up-the-workflow-repository)

  - [Set up the environment file](#set-up-the-environment-file)

  - [Authenticate Docker with NGC](#authenticate-docker-with-ngc)

  - [Build the Docker containers](#build-the-docker-containers)

  - [Start the Docker containers](#start-the-docker-containers)

    - [Using NVIDIA-hosted NIMs](#using-nvidia-hosted-nims)

    - [Using self-hosted NIMs](#using-self-hosted-nims)

- [Running the workflow](#running-the-workflow)

  - [From the quick start user guide notebook](#from-the-quick-start-user-guide-notebook)

  - [From the command line](#from-the-command-line)

    - [Configuration files](#configuration-files)

    - [Example command: `from_manual.json`](#example-command-from_manualjson)

      - [Reviewing the output](#reviewing-the-output)

    - [Example command: `from_http.json`](#example-command-from_httpjson)

  - [Command line interface (CLI) reference](#command-line-interface-cli-reference)

    - [Overriding options on the command line](#overriding-options-on-the-command-line)

      - [Example 1: Override `max_retries`](#example-1-override-max_retries)

      - [Example 2: Override input type](#example-2-override-input-type)

      - [Example 3: Combine multiple overrides](#example-3-combine-multiple-overrides)

    - [Full list of options](#full-list-of-options)

  - [Configuration file reference](#configuration-file-reference)

  - [NGINX caching server](#nginx-caching-server)

- [Customizing the Workflow](#customizing-the-workflow)

  - [Customizing the input](#customizing-the-input)

    - [Customizing the SBOM input](#customizing-the-sbom-input)

  - [Customizing the embedding model](#customizing-the-embedding-model)

  - [Customizing the LLM models](#customizing-the-llm-models)

    - [Supported LLM services](#supported-llm-services)

  - [Customizing the Output](#customizing-the-output)

  - [Integrating self-hosted LLM NIM](#integrating-self-hosted-llm-nim)

- [Troubleshooting](#troubleshooting)

  - [Git LFS issues](#git-lfs-issues)

  - [Container build issues](#container-build-issues)

    - [Device error](#device-error)

    - [Deploy.Resources.Reservations.devices error](#deployresourcesreservationsdevices-error)

  - [NGINX caching server](#nginx-caching-server-1)

    - [Resetting the entire cache](#resetting-the-entire-cache)

    - [Resetting just the LLM cache or the services cache](#resetting-just-the-llm-cache-or-the-services-cache)

    - [Vector databases](#vector-databases)

  - [Service errors](#service-errors)

    - [National Vulnerability Database (NVD)](#national-vulnerability-database-nvd)

    - [NVIDIA API Catalog / NVIDIA-hosted NIMs](#nvidia-api-catalog--nvidia-hosted-nims)

  - [Running out of credits](#running-out-of-credits)

- [Testing and validation](#testing-and-validation)

- [License](#license)

- [Terms of Use](#terms-of-use)



## Overview

This repository is what powers the [build experience](https://build.nvidia.com/nvidia/vulnerability-analysis-for-container-security), showcasing vulnerability analysis for container security using NVIDIA NIM microservices and [NVIDIA Morpheus](https://github.com/nv-morpheus/Morpheus).



The NVIDIA AI Blueprint demonstrates accelerated analysis on common vulnerabilities and exposures (CVE) at an enterprise scale, reducing mitigation from days and hours to just seconds. While traditional methods require substantial manual effort to pinpoint solutions for vulnerabilities, these technologies enable quick, automatic, and actionable CVE risk analysis using large language models (LLMs) and retrieval-augmented generation (RAG). With this blueprint, security analysts can expedite the process of determining whether a software package includes exploitable and vulnerable components using LLMs and event-driven RAG triggered by the creation of a new software package or the detection of a CVE.



## Software components

The following are used by this blueprint:

- [NIM of meta/llama-3.1-70b-instruct](https://build.nvidia.com/meta/llama-3_1-70b-instruct)

- [NIM of nvidia/nv-embedqa-e5-v5](https://build.nvidia.com/nvidia/nv-embedqa-e5-v5)

- [NVIDIA Morpheus Cybersecurity AI SDK](https://developer.nvidia.com/morpheus-cybersecurity)



## Target audience

This blueprint is for:

- **Security analysts and IT engineers**: People analyzing vulnerabilities and ensuring the security of containerized environments.

- **AI practitioners in cybersecurity**: People applying AI to enhance cybersecurity, particularly those interested in using the Morpheus SDK and NIMs for faster vulnerability detection and analysis.



## Prerequisites

- NVAIE developer licence

- API keys for vulnerability databases, search engines, and LLM model service(s).

  - Details can be found in this later section: [Obtain API keys](#obtain-api-keys)



## Hardware requirements

Below are the hardware requirements for each component of the vulnerability analysis pipeline.



The overall hardware requirements depend on selected pipeline configuration. At a minimum, the hardware requirements for pipeline operation must be met. The LLM NIM and Embedding NIM hardware requirements only need to be met if self-hosting these components. See [Using self-hosted NIMs](#using-self-hosted-nims), [Customizing the LLM models](#customizing-the-llm-models) and [Customizing the embedding model](#customizing-the-embedding-model) sections for more information.



- **(Required) Pipeline operation**: 1x L40 GPU or similar recommended

- **(Optional) LLM NIM**: [Meta Llama 3.1 70B Instruct Support Matrix](https://docs.nvidia.com/nim/large-language-models/latest/support-matrix.html#llama-3-1-70b-instruct)

  - For improved paralleled performance, we recommend 8x or more H100s for LLM inference.

  - The pipeline can share the GPU with the LLM NIM, but it is recommended to have a separate GPU for the LLM NIM for optimal performance.

- **(Optional) Embedding NIM**: [NV-EmbedQA-E5-v5 Support Matrix](https://docs.nvidia.com/nim/nemo-retriever/text-embedding/latest/support-matrix.html#nv-embedqa-e5-v5)

  - The pipeline can share the GPU with the Embedding NIM, but it is recommended to have a separate GPU for the Embedding NIM for optimal performance.



## API definition



[OpenAPI Specification](./configs/openapi/openapi.json)



## Use case description

Determining the impact of a documented CVE on a specific project or container is a labor-intensive and manual task, especially as the rate of new reports into the [CVE database](https://www.cve.org/) accelerates. This process involves the collection, comprehension, and synthesis of various pieces of information to ascertain whether immediate remediation is necessary upon the identification of a new CVE.



**Current challenges in CVE analysis:**

- **Information collection**: The process involves significant manual labor to collect and synthesize relevant information.

- **Decision complexity**: Decisions on whether to update a library impacted by a CVE often hinge on various considerations, including:

  - **Scan false positives**: Occasionally, vulnerability scans may incorrectly flag a library as vulnerable, leading to a false alarm.

  - **Mitigating factors**: In some cases, existing safeguards within the environment may reduce or negate the risk posed by a CVE.

  - **Lack of required environments or dependencies**: For an exploit to succeed, specific conditions must be met. The absence of these necessary elements can render a vulnerability irrelevant.

- **Manual documentation**: Once an analyst has determined the library is not affected, a Vulnerability Exploitability eXchange (VEX) document must be created to standardize and distribute the results.



The efficiency of this process can be significantly enhanced through the deployment of an automated LLM agent pipeline, leveraging generative AI to improve vulnerability defense while decreasing the load on security teams.



### How it works

The workflow operates using a Plan-and-Execute-style LLM pipeline for CVE impact analysis. The process begins with an LLM planner that generates a context-sensitive task checklist. This checklist is then executed by an LLM agent equipped with Retrieval-Augmented Generation (RAG) capabilities. The gathered information and the agent's findings are subsequently summarized and categorized by additional LLM nodes to provide a final verdict.



> [!TIP]

> The pipeline is adaptable, supporting various LLM services that conform to the `LLMService` interface, including OpenAI, NeMo, or local execution with llama-cpp-python.



### Key components

The detailed architecture consists of the following components:










- **Security scan result**: The workflow begins by inputting the identified CVEs from a container security scan as input. This can be generated from a container image scanner of your choosing such as [Anchore](https://anchore.com/container-vulnerability-scanning/).



- **PreProcessing**: All the below actions are encapsulated by multiple Morpheus preprocessing pipeline stages to prepare the data for use with the LLM engine. (See [`src/cve/pipeline/input.py`](./src/cve/pipeline/input.py).)

  - **Code repository and documentation**: The blueprint pulls code repositories and documentation provided by the user. These repositories are processed through an embedding model, and the resulting embeddings are stored in vector databases (VDBs) for the agent's reference.

    - **Vector database**: Various vector databases can be used for the embedding. We currently utilize FAISS for the VDB because it does not require an external service and is simple to use. Any vector store can be used, such as NVIDIA cuVS, which would provide accelerated indexing and search.

    - **Lexical search**: As an alternative, a lexical search is available for use cases where creating an embedding is impractical due to a large number of source files in the target container.

  - **Software Bill of Materials (SBOM)**: A Software Bill of Materials (SBOM) is a machine-readable manifest of all the dependencies of a software package or container. The blueprint cross-references every entry in the SBOM for known vulnerabilities and looks at the code implementation to see whether the implementation puts users at risk—just as a security analyst would do. For this reason, starting with an accurate SBOM is an important first step. SBOMs can be generated for any container using the open-source tool [Syft](https://github.com/anchore/syft). For more information on generating SBOMs for your containers, see the [SBOM documentation](./data/sboms/README.md).

  - **Web vulnerability intel**: The system collects detailed information about each CVE through web scraping and data retrieval from various public security databases, including GHSA, Redhat, Ubuntu, and NIST CVE records, as well as tailored threat intelligence feeds.



- **Core LLM engine**: (See [`src/cve/pipeline/engine.py`](./src/cve/pipeline/engine.py).)

  - **Checklist generation**: Leveraging the gathered information about each vulnerability, the checklist generation node creates a tailored, context-sensitive task checklist designed to guide the impact analysis. (See [`src/cve/nodes/cve_checklist_node.py`](./src/cve/nodes/cve_checklist_node.py).)



  - **Task agent**: At the core of the process is an LLM agent iterating through each item in the checklist. For each item, the agent answers the question using a set of tools which provide information about the target container. The tools tap into various data sources (web intel, vector DB, search etc.), retrieving relevant information to address each checklist item. The loop continues until the agent resolves each checklist item satisfactorily. (See [`src/cve/nodes/cve_langchain_agent_node.py`](./src/cve/nodes/cve_langchain_agent_node.py).)



  - **Summarization**: Once the agent has compiled findings for each checklist item, these results are condensed by the summarization node into a concise, human-readable paragraph. (See [`src/cve/nodes/cve_summary_node.py`](./src/cve/nodes/cve_summary_node.py).)



  - **Justification Assignment**: Given the summary, the justification status categorization node then assigns a resulting VEX (Vulnerability Exploitability eXchange) status to the CVE. We provided a set of predefined categories for the model to choose from. (See [`src/cve/nodes/cve_justification_node.py`](./src/cve/nodes/cve_justification_node.py).) If the CVE is deemed exploitable, the reasoning category is "vulnerable." If there is no vulnerable packages detected from the SBOM or insufficient intel gathered the agent is bypassed and an appropriate label is provided. If it is not exploitable, there are 10 different reasoning categories to explain why the vulnerability is not exploitable in the given environment:

      - `false_positive`

      - `code_not_present`

      - `code_not_reachable`

      - `requires_configuration`

      - `requires_dependency`

      - `requires_environment`

      - `protected_by_compiler`

      - `protected_at_runtime`

      - `protected_by_perimeter`

      - `protected_by_mitigating_control`



- **Output**: At the end of the pipeline run, an output file including all the gathered and generated information is prepared for security analysts for a final review. (See [`src/cve/pipeline/output.py`](./src/cve/pipeline/output.py).)



> [!WARNING]

> All output should be vetted by a security analyst before being used in a cybersecurity application.



### NIM microservices

The Morpheus SDK can utilize various embedding model and LLM endpoints, and is optimized to use [NVIDIA NIM microservices](https://developer.nvidia.com/nim) (NIMs). NIMs are pre-built containers for the latest AI models that provide industry-standard APIs and optimized inference for the given model and hardware. Using NIMs enables easy deployment and scaling for self-hosted model inference.



The current default embedding NIM model is `nv-embedqa-e5-v5`, which was selected to balance speed and overall pipeline accuracy. The current default LLM model is the `llama-3.1-70b-instruct` NIM, with specifically tailored prompt engineering and edge case handling. Other models are able to be substituted for either the embedding or LLM model, such as smaller, fine-tuned NIM LLM models or other external LLM inference services. Subsequent updates will provide more details about fine-tuning and data flywheel techniques.



> [!NOTE]

> The LangChain library is employed to deploy all LLM agents within a Morpheus pipeline, streamlining efficiency and reducing the need for duplicative efforts.



> [!TIP]

> Routinely checked validation datasets are critical to ensuring proper and consistent outputs. Learn more about our test-driven development approach in the section on [testing and validation](#testing-and-validation).



## Getting started



### Install system requirements



* [git](https://git-scm.com/)

* [git-lfs](https://git-lfs.com/)

* Since the workflow uses an [NVIDIA Morpheus](https://developer.nvidia.com/morpheus-cybersecurity) pipeline, the [Morpheus requirements](https://docs.nvidia.com/morpheus/getting_started.html#requirements) also need to be installed.



### Obtain API keys

To run the pipeline you need to obtain API keys for the following APIs. These will be needed in a later step to [Set up the environment file](#set-up-the-environment-file).



- **Required API Keys**: These APIs are required by the pipeline to retrieve vulnerability information from databases, perform online searches, and execute LLM queries.



  - GitHub Security Advisory (GHSA) Database

    - Follow [these](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token) instructions to create a personal access token. No repository access or permissions are required for this API.

    - This will be used in the `GHSA_API_KEY` environment variable.

  - National Vulnerability Database (NVD)

    - Follow [these](https://nvd.nist.gov/developers/request-an-api-key) instructions to create an API key.

    - This will be used in the `NVD_API_KEY` environment variable.

  - SerpApi

    - Go to https://serpapi.com/ and create a SerpApi account. Once signed in, navigate to Your Account > Api Key.

    - This will be used in the `SERPAPI_API_KEY` environment variable.

  - NVIDIA Inference Microservices (NIM)

    - There are two possible methods to generate an API key for NIM:

      - Sign in to the [NVIDIA Build](https://build.nvidia.com/explore/discover?signin=true) portal with your email.

        - Click on any [model](https://build.nvidia.com/meta/llama-3_1-70b-instruct), then click "Get API Key", and finally click "Generate Key".

      - Sign in to the [NVIDIA NGC](https://ngc.nvidia.com/) portal with your email.

        - Select your organization from the dropdown menu after logging in. You must select an organization which has NVIDIA AI Enterprise (NVAIE) enabled.

        - Click on your account in the top right, select "Setup" from the dropdown.

        - Click the "Generate Personal Key" option and then the "+ Generate Personal Key" button to create your API key.

    - This will be used in the `NVIDIA_API_KEY` environment variable.



The workflow can be configured to use other LLM services as well, see the [Customizing the LLM models](#customizing-the-llm-models) section for more info.



### Set up the workflow repository



Clone the repository and set an environment variable for the path to the repository root.



```bash

export REPO_ROOT=$(git rev-parse --show-toplevel)

```



All commands are run from the repository root unless otherwise specified.



### Set up the environment file



First we need to create an `.env` file in the `REPO_ROOT`, and add the API keys you created in the earlier [Obtain API keys](#obtain-api-keys) step.



```bash

cd $REPO_ROOT

cat < .env

GHSA_API_KEY="your GitHub personal access token"

NVD_API_KEY="your National Vulnerability Database API key"

NVIDIA_API_KEY="your NVIDIA Inference Microservices API key"

SERPAPI_API_KEY="your SerpApi API key"

EOF

```



These variables need to be exported to the environment:

```bash

export $(cat .env | xargs)

```



### Authenticate Docker with NGC



In order to pull images required by the workflow from NGC, you must first authenticate Docker with NGC. You can use same the NVIDIA API Key obtained in the [Obtain API keys](#obtain-api-keys) section (saved as `NVIDIA_API_KEY` in the `.env` file).



```bash

echo "${NVIDIA_API_KEY}" | docker login nvcr.io -u '$oauthtoken' --password-stdin

```



### Build the Docker containers

Next, build the `morpheus-vuln-analysis` container from source using the following command. This ensures that the container includes all the latest changes from the repository.

```bash

cd $REPO_ROOT



# Build the morpheus-vuln-analysis container

docker compose build morpheus-vuln-analysis

```



### Start the Docker containers



There are two supported configurations for starting the Docker containers. Both configurations utilize `docker compose` to start the service:

1. [**NVIDIA-hosted NIMs**](#using-nvidia-hosted-nims): The workflow is run with all computation being performed by NIMs hosted in NVIDIA GPU Cloud. This is the default configuration and is **recommended** for most users getting started with the workflow.

   1. When using NVIDIA-hosted NIMs, only the `docker-compose.yml` configuration file is required.

2. [**Self-hosted NIMs**](#using-self-hosted-nims): The workflow is run using self-hosted LLM NIM services. This configuration is more advanced and requires additional setup to run the NIM services locally.

   1. When using self-hosted NIMs, both the `docker-compose.yml` and `docker-compose.nim.yml` configuration files are required.



These two configurations are illustrated by the following diagram:





Workflow configurations




Before beginning, ensure that the environment variables are set correctly. Both configurations require the same environment variables to be set. More information on setting these variables can be found in the [Obtain API keys](#obtain-api-keys) section.



>[!TIP]

The container binds to port 8080 by default. If you encounter a port collision error (e.g. `Bind for 0.0.0.0:8080 failed: port is already allocated`), you can set the environment variable `NGINX_HOST_HTTP_PORT` to specify a custom port before launching `docker compose`. For example:

>

> ```bash

> export NGINX_HOST_HTTP_PORT=8081

>

> #... docker compose commands...

> ```



#### Using NVIDIA-hosted NIMs

When running the workflow in this configuration, only the `morpheus-vuln-analysis` service needs to be started since we will utilize NIMs hosted by NVIDIA. The `morpheus-vuln-analysis` container can be started using the following command:



```bash

cd ${REPO_ROOT}

docker compose up -d

```



The command above starts the container in the background using the detached mode, `-d`. We can confirm the container is running via the following command:

```bash

docker compose ps

```



Next, we need to attach to the `morpheus-vuln-analysis` container to access the environment where the workflow [command line tool](#from-the-command-line) and dependencies are installed.

```bash

docker compose exec -it morpheus-vuln-analysis bash

```



Continue to the [Running the workflow](#running-the-workflow) section to run the workflow.



#### Using self-hosted NIMs

To run the workflow using self-hosted NIMs, we use a second `docker compose` configuration file, `docker-compose.nim.yml`, which adds the self-hosted NIM services to the workflow. Utilizing a second configuration file allows for easy switching between the two configurations while keeping the base configuration file the same.



>[!NOTE]

> The self-hosted NIM services require additional GPU resources to run. With this configuration, the LLM NIM, embedding model NIM, and the `morpheus-vuln-analysis` service will all be launched on the same machine. Ensure that you have the necessary hardware requirements for all three services before proceeding (multiple services can share the same GPU).



To use multiple configuration files, we need to specify both configuration files when running the `docker compose` command. You will need to specify both configuration files for every `docker compose` command. For example:



```bash

docker compose -f docker-compose.yml -f docker-compose.nim.yml [NORMAL DOCKER COMPOSE COMMAND]

```



For example, to start the `morpheus-vuln-analysis` service with the self-hosted NIMs, you would run:

```bash

cd ${REPO_ROOT}

docker compose -f docker-compose.yml -f docker-compose.nim.yml up -d

```



Next, we need to attach to the `morpheus-vuln-analysis` container to access the environment where the workflow [command line tool](#from-the-command-line) and dependencies are installed.

```bash

docker compose -f docker-compose.yml -f docker-compose.nim.yml exec -it morpheus-vuln-analysis bash

```



Continue to the [Running the workflow](#running-the-workflow) section to run the workflow.



## Running the workflow



Once the services have been started, the workflow can be run using either the [Quick start user guide notebook](#from-the-quick-start-user-guide-notebook) for an interactive step-by-step process, or directly from the [command line](#from-the-command-line).



### From the quick start user guide notebook

To run the workflow in an interactive notebook, connect to the Jupyter notebook at http://localhost:8000/lab. Once connected, navigate to the notebook located at [`quick_start/quick_start_guide.ipynb`](./quick_start/quick_start_guide.ipynb) and follow the instructions.



> [!TIP]

> If you are running the workflow on a remote machine, you can forward the port to your local machine using SSH. For example, to forward port 8000 from the remote machine to your local machine, you can run the following command from your local machine:

>

> ```bash

> ssh -L 8000:127.0.0.1:8000 

> ```



### From the command line



The vulnerability analysis workflow is designed to be run using the command line tool installed within the `morpheus-vuln-analysis` container. This section describes how get started using the command line tool. For more detailed information about the command line interface, see the [Command line interface (CLI) reference](#command-line-interface-cli-reference) section.



#### Configuration files



The pipeline settings are controlled using configuration files. These are JSON files that define various pipeline settings, such as the input data, the LLM models used, and the output format. Several example configuration files are located in the [`configs/`](./configs/) folder. A brief description of each configuration file is as follows:

- `from_manual.json`: This configuration file starts the pipeline using manually provided data. All pipeline inputs are manually specified directly in the configuration file.

- `from_file.json`: This configuration file starts the pipeline using data fetched from a file. The pipeline fetches the input data from a file and processes it. This is very similar to `from_manual.json`, but the input data is read from a file instead of being directly specified in the config file.

- `from_http.json`: This configuration file starts an HTTP server to turn the workflow into a microservice. The pipeline fetches the input data from an HTTP source and processes it. To trigger the pipeline, you can send a POST request to the `/scan` endpoint with the input data in the request body. The pipeline will process the input data.



There are two main modalities that the pipeline can be run in. When using the `from_file.json` or `from_manual.json` configuration files, the pipeline will process the input data, then it will shut down after it is completed. This modality is suitable for rapid iteration during testing and development. When using `from_http.json`, the pipeline is turned into a microservice which will run indefinitely, which is suitable for using in production.



For a breakdown of the configuration file and available options, see the [Configuration file reference](#configuration-file-reference) section. To customize the configuration files for your use case, see [Customizing the workflow](#customizing-the-workflow).



#### Example command: `from_manual.json`



The workflow pipeline can be started using the following command:

```bash

python src/main.py --log_level DEBUG \

  cve pipeline --config_file=${CONFIG_FILE}

```



In the command, `${CONFIG_FILE}` is the path to the configuration file you want to use. For example, to run the pipeline with the `from_manual.json` configuration file, you would run:

```bash

python src/main.py --log_level DEBUG \

  cve pipeline --config_file=configs/from_manual.json

```



When the pipeline runs to completion, you should see logs similar to the following:

```

Message elapsed time: 28.240849 sec

Vulnerability 'GHSA-3f63-hfp8-52jq' affected status: FALSE. Label: code_not_reachable

Vulnerability 'CVE-2023-50782' affected status: FALSE. Label: requires_configuration

Vulnerability 'CVE-2023-36632' affected status: FALSE. Label: code_not_present

Vulnerability 'CVE-2023-43804' affected status: TRUE. Label: vulnerable

Vulnerability 'GHSA-cxfr-5q3r-2rc2' affected status: TRUE. Label: vulnerable

Vulnerability 'GHSA-554w-xh4j-8w64' affected status: TRUE. Label: vulnerable

Vulnerability 'GHSA-3ww4-gg4f-jr7f' affected status: FALSE. Label: requires_configuration

Vulnerability 'CVE-2023-31147' affected status: FALSE. Label: code_not_present

Source[Complete]: 7 messages [00:14,  2.02s/ messages]

LLM[Complete]: 7 messages [00:42,  6.05s/ messages]

====Pipeline Complete====

Total time: 45.61 sec

Pipeline runtime: 42.69 sec

```



> [!WARNING]

> The output you receive from the pipeline may not be identical as the output in the example above. The output may vary due to the non-deterministic nature of the LLM models.



##### Reviewing the output



The full pipeline JSON output is stashed by default at `.tmp/output.json`. The output JSON includes the following top level fields:



* `input`: contains the inputs that were provided to the pipeline, such as the container and repo source information, the list of vulnerabilities to scan, etc.

* `info`: contains additional information collected by the pipeline for decision making. This includes paths to the generated VDB files, intelligence from various vulnerability databases, the list of SBOM packages, and any vulnerable dependencies that were identified.

* `output`: contains the output from the core LLM Engine, including the generated checklist, analysis summary, and justification assignment.



In addition to the raw JSON output, you can also view a [Markdown-formatted report](./.tmp/vulnerability_markdown_reports/vulnerability_reports_morpheus_23_11-runtime/CVE-2023-36632.md) for each CVE in the [`.tmp/vulnerability_markdown_reports`](./.tmp/vulnerability_markdown_reports/) directory. This view is helpful for human analysts reviewing the results.



> [!TIP]

> To return detailed steps taken by the LLM agent in the output, set `return_intermediate_steps` to `true` in the configuration file. This can be helpful for explaining the output, and for troubleshooting unexpected results.



#### Example command: `from_http.json`



Similarly, to run the pipeline with the `from_http.json` configuration file, you would run:

```bash

python src/main.py --log_level DEBUG cve pipeline --config_file=configs/from_http.json

```



This command starts an HTTP server that listens on port `26466` and runs the workflow indefinitely, waiting for incoming data to process. This is useful if you want to trigger the workflow on demand via HTTP requests.



Once the server is running, you can send a `POST` request to the `/scan` endpoint with the input parameters in the request body. The pipeline will process the input data and return the output in the terminal and the given output path in the config file.



Here's an example using `curl` to send a `POST` request. From a new terminal outside of the container, go to the root of the cloned git repository, and run:

```bash

curl -X POST http://localhost:26466/scan -d @data/input_messages/morpheus:24.03-runtime.json

```

In this command:

- `http://localhost:26466/scan` is the URL of the server and endpoint.

- The `-d` option specifies the data file being sent in the request body. In this case, it's pointing to the input file `morpheus:24.03-runtime.json` under the `data/input_messages/` directory. You can refer to this file as an example of the expected data format.

    - Since it uses a relative path, it's important to run the `curl` command from the root of the git repository. Alternatively, you can modify the relative path in the command to directly reference the example json file.



Note that the results of the pipeline are not returned to the curl request. After processing the request, the server will save the results to the output path specified in the configuration file. The server will also display log and summary results from the workflow as it's running. Additional submissions to the server will append the results to the specified output file.



### Command line interface (CLI) reference



The top level entrypoint to each of the LLM example pipelines is `src/main.py`. The main entrypoint is a CLI tool with built-in documentation using the `--help` command. For example, to see what commands are available, you can run:

```

(morpheus) root@58145366033a:/workspace# python src/main.py --help

Usage: morpheus_llm [OPTIONS] COMMAND [ARGS]...



  Main entrypoint for the Vulnerability Analysis for Container Security



Options:

  --log_level [CRITICAL|FATAL|ERROR|WARN|WARNING|INFO|DEBUG]

                                  Specify the logging level to use.  [default:

                                  INFO]

  --use_cpp BOOLEAN               Whether or not to use C++ node and message

                                  types or to prefer python. Only use as a

                                  last resort if bugs are encountered

                                  [default: True]

  --version                       Show the version and exit.

  --help                          Show this message and exit.



Commands:

  cve  Run the Vulnerability Analysis for Container Security pipeline

```



#### Overriding options on the command line



It's common to want to override some options in the configuration file on the command line. This is useful to use some common options but tweak the input/output of the pipeline or to change a single setting without modifying the original config file.



##### Example 1: Override `max_retries`

For example, to override the `max_retries` option in the configuration file, you can run:

```bash

python src/main.py --log_level=DEBUG \

  cve pipeline --config_file=configs/from_manual.json \

    config \

      general --max_retries=3

```

This will run the pipeline with the `from_manual.json` configuration file, but with the `max_retries` option set to `3`.



##### Example 2: Override input type

It's also possible to change the input type. For example, to use a different input message with the `from_file.json` configuration file, you can run:

```bash

python src/main.py --log_level=DEBUG \

  cve pipeline --config_file=configs/from_file.json \

    config \

      input-file --file=data/input_messages/morpheus:24.03-runtime.json

```



##### Example 3: Combine multiple overrides

It's possible to combine multiple overrides in a single command. For example, to run the pipeline with the `from_manual.json` configuration file, but with the `max_retries` option set to `3`, the input message set to `morpheus:24.03-runtime.json`, and the output destinations set to `.tmp/output_morpheus.json` and `.tmp/morpheus_reports`, you can run:

```bash

python src/main.py --log_level=DEBUG \

  cve pipeline --config_file=configs/from_manual.json \

    config \

      general --max_retries=3 \

      input-file --file=data/input_messages/morpheus:24.03-runtime.json \

      output-file --file_path=.tmp/output_morpheus.json --markdown_dir=.tmp/morpheus_reports

```



#### Full list of options

For the full list of possible options, use the `--help` options from the CLI.



### Configuration file reference

The configuration defines how the workflow operates, including model settings, input sources, and output options.



1. **Schema**

    - `$schema`: Specifies the schema for validating the configuration file. This ensures the correct structure and data types are used throughout the config file.

2. **LLM engine configuration (`engine`)**: The `engine` section configures various models for the LLM nodes.

    - LLM processing nodes: `agent`, `checklist_model`, `justification_model`, `summary_model`

      - `model_name`: The name of the LLM model used by the node.

      - `prompt`: Manually set the prompt for the specific model in the configuration. The prompt can either be passed in as a string of text or as a path to a text file containing the desired prompting.

      - `service`: Specifies the service for running the LLM inference. (Set to `nvfoundation` if using NIM.)

      - `max_tokens`: Defines the maximum number of tokens that can be generated in one output step.

      - `temperature`: Controls randomness in the output. A lower temperature produces more deterministic results.

      - `top_p`: Limits the diversity of token sampling based on cumulative probability.

      - Settings specific to the `agent` node:

        - `verbose`: Toggles detailed logging for the agent during execution.

        - `return_intermediate_steps`: Controls whether to return intermediate steps taken by the agent, and include them in the output file. Helpful for troubleshooting agent responses.

        - `return_source_documents`: Controls whether to return source documents from the VDB tools, and include them in the intermediate steps output. Helpful for identifying the source files used in agent responses.

          - Note: enabling this will also include source documents in the agent's memory and increase the agent's prompt length.

        - `max_concurrency`: Controls the maximum number of concurrent requests to the LLM. Default is `None`, which doesn't limit concurrency.

    - Embedding model for generating VDB for RAG: `rag_embedding`

      - `_type`: Defines the source of the model used for generating embeddings (e.g., `nim`, `huggingface`, `openai`).

      - Other model-dependent parameters, such as `model`/`model_name`, `api_key`, `truncate`, or `encode_kwargs`: see the [embedding model customization](#customizing-the-embedding-model) section below for more details.

3. **General configuration**: The `general` section contains settings that influence the workflow's general behavior, including cache settings, batch sizes, and retry policies.

    - `cache_dir`: The directory where the node's cache should be stored. If None, caching is not used.

    - `base_vdb_dir`: The directory used for storing vector database files.

    - `base_git_dir`: The directory for storing pulled git repositories used for code analysis.

    - `max_retries`: Sets the number of retry attempts for failed operations.

    - `model_max_batch_size`: Specifies the maximum number of messages to send to the model for inference in a single batch.

    - `pipeline_batch_size`: Determines the number of messages per batch for the pipeline.

    - `use_uvloop`: Toggles the use of `uvloop`, an optimized event loop for improved performance.

    - `code_search_tool`: Enables or disables the use of the code search tool.

4. **Input configuration**: The `input` section defines how and where the input data (container images and vulnerabilities) is sourced.

    - `_type`: Defines the input type

      - `manual`: input data is provided manually in the config file

      - `http`: input data is provided through http `POST` call pointing to an input source file

      - `file`: input data is fetched from a provided source file

    - `message`: Contains details about the input image and its associated vulnerabilities. Required only for the `manual` input type.

      - `image`: Specifies the container image to be analyzed.

        - `name`: Specifies the name of the container image.

        - `tag`: Specifies the tag of the container image.

        - `source_info`: Specifies the sources (e.g., git repositories) used to retrieve code or documentation for analysis.

          - `include`: Specifies the file patterns to be included in the analysis.

          - `exclude`: Specifies the file patterns to exclude from analysis.

        - `sbom_info`: Specifies the Software Bill of Materials (SBOM) file to be analyzed. This can either be a local file path, or a URL pointing to the raw SBOM file data ([example](https://raw.githubusercontent.com/NVIDIA-AI-Blueprints/vulnerability-analysis/refs/heads/main/data/sboms/nvcr.io/nvidia/morpheus/morpheus%3Av23.11.01-runtime.sbom)). See [customizating the SBOM input](#customizing-the-sbom-input) section for details.

      - `scan`: Provides a list of vulnerabilities (by ID) to be analyzed for the input image.

5. **Output configuration**: The `output` section defines where the output results of the workflow should be stored and how they should be managed.

    - `_type`: Specifies the output type. Use `file` to write the results to a file.

    - `file_path`: Defines the path to the file where the output will be saved.

    - `markdown_dir`: Defines the path to the directory where the output will be saved in individual navigable markdown files per CVE-ID.

    - `overwrite`: Indicates whether the output file should be overwritten when the pipeline starts if it already exists. Will throw an error if set to `False` and the file already exists. Note that the overwrite behavior only occurs on pipeline

    initialization. For pipelines started in HTTP mode, each new request will append the existing file until the pipeline is restarted.



### NGINX caching server



The docker compose file includes an `nginx-cache` proxy server container that enables caching for API requests made by the workflow. It is highly recommend to route API requests through the proxy server to reduce API calls for duplicate requests and improve workflow speed. This is especially useful when running the pipeline multiple times with the same configuration (e.g., for debugging) and can help keep costs down when using paid APIs.



The NGINX proxy server is started by default when running the `morpheus-vuln-analysis` service. However, it can be started separately using the following command:

```bash

cd ${REPO_ROOT}

docker compose up --detach nginx-cache

```



To use the proxy server for API calls in the workflow, you can set environment variables for each base URL used by the workflow to point to `http://localhost:${NGINX_HOST_HTTP_PORT}/`. These are set automatically when running the `morpheus-vuln-analysis` service, but can be set manually in the `.env` file as follows:

```bash

CVE_DETAILS_BASE_URL="http://localhost:8080/cve-details"

CWE_DETAILS_BASE_URL="http://localhost:8080/cwe-details"

DEPSDEV_BASE_URL="http://localhost:8080/depsdev"

FIRST_BASE_URL="http://localhost:8080/first"

GHSA_BASE_URL="http://localhost:8080/ghsa"

NGC_API_BASE="http://localhost:8080/nemo/v1"

NIM_EMBED_BASE_URL="http://localhost:8080/nim_embed/v1"

NVD_BASE_URL="http://localhost:8080/nvd"

NVIDIA_API_BASE="http://localhost:8080/nim_llm/v1"

OPENAI_API_BASE="http://localhost:8080/openai/v1"

OPENAI_BASE_URL="http://localhost:8080/openai/v1"

RHSA_BASE_URL="http://localhost:8080/rhsa"

SERPAPI_BASE_URL="http://localhost:8080/serpapi"

UBUNTU_BASE_URL="http://localhost:8080/ubuntu"

```



## Customizing the Workflow



The primary method for customizing the workflow is to generate a new configuration file with new options. The configuration file defines the pipeline settings, such as the input data, the LLM models used, and the output format. The configuration file is a JSON file that can be modified to suit your needs.



### Customizing the input



Currently, there are 3 types of input sources supported by the pipeline:

- **Manual input**: The input data is directly specified in the configuration file.

- **File input**: The input data is read from a file.

- **HTTP input**: The input data is fetched from an HTTP source.



To customize the input, modify the configuration file accordingly. In any configuration file, locate the `input` section to see the input source used by the pipeline. For example, in the configuration file `configs/from_manual.json`, the following snippet defines the input source as manual:

```

    "input": {

      "_type": "manual",

      "message": {

        ...Contents of the input message...

      }

    }

```



To use a file as the input source, update the JSON object in the config file to:

```

    "input": {

      "_type": "file",

      "file": "data/input_messages/morpheus:23.11-runtime.json"

    }

```



To use an HTTP source as the input, update the JSON object in the config file to:

```

    "input": {

      "_type": "http",

      "address": "127.0.0.1",

      "endpoint": "/scan",

      "http_method": "POST",

      "port": 26466

    }

```



#### Customizing the SBOM input



To use an SBOM from a URL ([example](https://raw.githubusercontent.com/NVIDIA-AI-Blueprints/vulnerability-analysis/refs/heads/main/data/sboms/nvcr.io/nvidia/morpheus/morpheus%3Av23.11.01-runtime.sbom)), update the SBOM info configuration to:

```

        "sbom_info": {

          "_type": "http",

          "url": "https://raw.githubusercontent.com/NVIDIA-AI-Blueprints/vulnerability-analysis/refs/heads/main/data/sboms/nvcr.io/nvidia/morpheus/morpheus%3Av23.11.01-runtime.sbom"

        }

```



### Customizing the embedding model

Vector databases are used by the agent to fetch relevant information for impact analysis investigations. The embedding model used to vectorize your documents can significantly affect the agent's performance. The default embedding model used by the pipeline is the NIM [nvidia/nv-embedqa-e5-v5](https://build.nvidia.com/nvidia/nv-embedqa-e5-v5) model, but you can experiment with different embedding models of your choice.



To test a custom embedding model, modify the configuration file in the `engine.rag_embedding` section. For example, in the `from_manual.json` configuration file, the following snippet defines the settings for the default embedding model:

```

    "rag_embedding": {

      "_type": "nim",

      "model": "nvidia/nv-embedqa-e5-v5",

      "truncate": "END",

      "max_batch_size": 128

    },

```

* `rag_embedding._type`: specifies the embedding provider. The current [supported options](./src/cve/utils/embedding_loader.py) are `nim`, `huggingface` and `openai`.

* `model_name`: specifies the model name for the embedding provider. Refer to the embedding provider's documentation to determine the available models.

* `truncate`: specifies how inputs longer than the maximum token length of the model are handled. Passing `START` discards the start of the input. `END` discards the end of the input. In both cases, input is discarded until the remaining input is exactly the maximum input token length for the model. If `NONE` is selected, when the input exceeds the maximum input token length an error will be returned.

* `max_batch_size`: specifies the batch size to use when generating embeddings. We recommend setting this to 128 (default) or lower when using the cloud-hosted embedding NIM. When using a local NIM, this value can be tuned based on throughput/memory performance on your hardware.



**Steps to configure an alternate embedding provider**

1. If using OpenAI embeddings, first obtain an API key, then update the `.env` file with the auth and base URL environment variables for the service as indicated in the [Supported LLM Services](#supported-llm-services) table. Otherwise, proceed to step 2.

2. Update the `rag_embedding` section of the config file as described above.



    - Example HuggingFace embedding configuration:

    ```

        "rag_embedding": {

          "_type": "huggingface",

          "model_name": "intfloat/e5-large-v2",

          "encode_kwargs": {

            "batch_size": 128

          }

        }

    ```

    - Example OpenAI embedding configuration:

    ```

        "rag_embedding": {

          "_type": "openai",

          "model_name": "text-embedding-3-small",

          "encode_kwargs": {

            "max_retries": 5,

            "chunk_size": 256

          }

        }

    ```

    - For HuggingFace embeddings, all parameters from LangChain's [HuggingFaceEmbeddings](https://api.python.langchain.com/en/latest/embeddings/langchain_huggingface.embeddings.huggingface.HuggingFaceEmbeddings.html) class are supported. However, for OpenAI models, only a subset of parameters are supported. The full set of available parameters can be found in the config definitions [here](./src/cve/data_models/config.py?#L182-203). Any non-supported parameters provided in the configuration will be ignored.



The current pipeline uses [FAISS](https://ai.meta.com/tools/faiss/) to create the vector databases. Interested users can customize the source code to use other vector databases such as [cuVS](https://rapids.ai/cuvs/).



### Customizing the LLM models



The configuration file also allows customizing the LLM model and parameters for each component of the workflow, as well which LLM service is used when invoking the model.



In any configuration file, locate the `engine` section to see the current settings. For example, in the `from_manual.json` configuration file, the following snippet defines the LLM used for the checklist model:

```

    "checklist_model": {

      "service": {

        "_type": "nvfoundation"

      },

      "model_name": "meta/llama-3.1-70b-instruct",

      "temperature": 0,

      "max_tokens": 2000,

      "top_p": 0.01

    }

```

* `service._type`: specifies the LLM service to use. Refer to the [Supported LLM Services](#supported-llm-services) table for available options.

* `model_name`: specifies the model name within the LLM service. Refer to the service's API documentation to determine the available models.

* `temperature`, `max_tokens`, `top_p`, ...: specifies the model parameters. Note that by default, the config supports only a subset of parameters provided by each LLM service. The available parameters can be found in the configuration object's definition [here](./src/cve/data_models/config.py?#L122-127]). Any non-supported parameters provided in the configuration will be ignored.



#### Supported LLM services

| Name                                                                                                         | `_type`        | Auth Env Var(s)               | Base URL Env Var(s)                                                             | Proxy Server Route |

| ------------------------------------------------------------------------------------------------------------ | -------------- | ----------------------------- | ------------------------------------------------------------------------------- | ------------------ |

| [NVIDIA Inference Microservices (NIMs)](http://build.nvidia.com/) (Default)                                  | `nvfoundation` | `NVIDIA_API_KEY`              | `NVIDIA_API_BASE`                                                               | `/nim_llm/v1`      |

| [NVIDIA GPU Cloud (NGC)](https://docs.nvidia.com/ngc/gpu-cloud/ngc-user-guide/index.html#generating-api-key) | `nemo`         | `NGC_API_KEY`
`NGC_ORG_ID` | `NGC_API_BASE`                                                                  | `/nemo/v1`         |

| [OpenAI](https://platform.openai.com/docs/api-reference/authentication)                                      | `openai`       | `OPENAI_API_KEY`              | `OPENAI_API_BASE` (used by `langchain`)
`OPENAI_BASE_URL` (used by `openai`) | `/openai/v1`       |



**Steps to configure an LLM model**

1. Obtain an API key and any other required auth info for the selected service.

2. Update the `.env` file with the auth and base URL environment variables for the service as indicated in the [Supported LLM Services](#supported-llm-services) table.

3. Update the config file as described above. For example, if you want to use OpenAI's `gpt-4o` model for checklist generation, update the above json object in the config file to:

```

    "checklist_model": {

      "service": {

        "_type": "openai"

      },

    "model_name": "gpt-4o",

      "temperature": 0,

      "top_p": 0.01,

      "seed": 0,

      "max_retries": 5

    },

```

Please note that the prompts have been tuned to work best with the Llama 3.1 70B NIM and that when using other LLM models it may be necessary to adjust the prompting.



### Customizing the Output



Currently, there are 3 types of outputs supported by the pipeline:

- **File output**: The output data is written to a file in JSON format.

- **HTTP output**: The output data is posted to an HTTP endpoint.

- **Print output**: The output data is printed to the console.



To customize the output, modify the configuration file accordingly. In any configuration file, locate the `output` section to see the output destination used by the pipeline. For example, in the configuration file `configs/from_manual.json`, the following snippet defines the output destination as a single json file and individual markdown files per CVE-ID:

```

    "output": {

      "_type": "file",

      "file_path": ".tmp/output.json",

      "markdown_dir": ".tmp/vulnerability_markdown_reports"

    }

```



To post the output to an HTTP endpoint, update the JSON object in the config file as follows, replacing the domain, port, and endpoint with the desired destination (note the trailing slash in the "url" field). The output will be sent as JSON data.



```

    "output": {

      "type": "http",

      "url": "http://:/",

      "endpoint": ""

    }

```



To print the output without saving to a file, update the JSON object in the config file to:

```

    "output": {

      "_type": "print"

    }

```



Additional output options will be added in the future.



### Integrating self-hosted LLM NIM



The worklow is configured by default to use an NVIDIA-hosted LLM NIM for which an NVIDIA API key (`NVIDIA_API_KEY`) is required. The workflow can also be used with a self-hosted LLM NIM. You can start [here](https://build.nvidia.com/meta/llama-3_1-70b-instruct?integrate_nim=true&self_hosted_api=true) for more information on how to pull and run the NIM locally.



Once NIM is deployed, update your pipeline configuration (e.g. `from_manual.json`) to now use the your NIM. For every component under `engine` that uses `nvfoundation`, update to now use `openai` and model name of your NIM (i.e. meta/llama-3.1-70b-instruct). For example:

```

"agent": {

  "model": {

    "model_name": "meta/llama-3.1-70b-instruct",

    "service": {

      "_type": "openai"

    },

    "max_tokens": 2000

    },

    "verbose": true

}

```



If using nginx cache, update `openai_upstream` in [nginx_cache.conf](./nginx/nginx_cache.conf) to point to your NIM URL:

```

set $openai_upstream http://llm-nim:8000;

```



Here `llm-nim` is the configured service name for the NIM if using helm chart of docker compose. Otherwise, set to actual host name or IP address of your NIM.



Now set `OPENAI_BASE_URL` environment variable to NIM or nginx URL depending on your configuration. The `OPENAI_API_KEY` must also be set but only to prevent a check error (not for authentication). This can be set to any string.



## Troubleshooting



Several common issues can arise when running the pipeline. Here are some common issues and their solutions.



### Git LFS issues



If you encounter issues with Git LFS, ensure that you have Git LFS installed and that it is enabled for the repository. You can check if Git LFS is enabled by running the following command:

```bash

git lfs install

```



Verifying that all files are being tracked by Git LFS can be done by running the following command:

```bash

git lfs ls-files

```

Files which are missing will show a `-` next to their name. To ensure all LFS files have been pulled correctly, you can run the following command:

```bash

git lfs fetch --all

git lfs checkout *

```



### Container build issues



When building containers for self-hosted NIMs, certain issues may occur. Below are common troubleshooting steps to help resolve them.



#### Device error



If you encounter an error resembling the following during the [container build process for self-hosted NIMs](#using-self-hosted-nims):



```bash

nvidia-container-cli: device error: {n}: unknown device: unknown

```



This error typically indicates that the container is attempting to access GPUs that are either unavailable or non-existent on the host. To resolve this, verify the GPU count specified in the [docker-compose.nim.yml](./docker-compose.nim.yml) configuration file:

- Navigate to the `deploy.resources.reservations.devices` section and check the count parameter.

- Set the environment variable `NIM_LLM_GPU_COUNT` to the actual number of GPUs available on the host machine before building the container. Note that the default value is set to 4.



This adjustment ensures the container accurately matches the available GPU resources, preventing access errors during deployment.



#### Deploy.Resources.Reservations.devices error



If you encounter an error resembling the following during the [container build process for self-hosted NIMs](#using-self-hosted-nims) process:

```

1 error(s) decoding:



* error decoding 'Deploy.Resources.Reservations.devices[0]': invalid string value for 'count' (the only value allowed is 'all')

```



This is likely caused by an [outdated Docker Compose version](https://github.com/docker/compose/issues/11097). Please upgrade Docker Compose to at least `v2.21.0`.



### NGINX caching server



Because the workflow makes such heavy use of the caching server to speed up API requests, it is important to ensure that the server is running correctly. If you encounter issues with the caching server, you can reset the cache.



#### Resetting the entire cache



To reset the entire cache, you can run the following command:

```bash

docker compose down -v

```

This will delete all the volumes associated with the containers, including the cache.



#### Resetting just the LLM cache or the services cache



If you want to reset just the LLM cache or the services cache, you can run the following commands:

```bash

docker compose down



# To remove the LLM cache

docker volume rm ${COMPOSE_PROJECT_NAME:-morpheus_vuln_analysis}_llm-cache



# To remove the services cache

docker volume rm ${COMPOSE_PROJECT_NAME:-morpheus_vuln_analysis}_service-cache

```



#### Vector databases

We've integrated VDB and embedding creation directly into the pipeline with caching included for expediency. However, in a production environment, it's better to use a separately managed VDB service.



NVIDIA offers optimized models and tools like NIMs ([build.nvidia.com/explore/retrieval](https://build.nvidia.com/explore/retrieval)) and cuVS ([github.com/rapidsai/cuvs](https://github.com/rapidsai/cuvs)).



### Service errors



#### National Vulnerability Database (NVD)

These typically resolve on their own. Please wait and try running the pipeline again later. Example errors:



404

```

Error requesting [1/10]: (Retry 0.1 sec) https://services.nvd.nist.gov/rest/json/cves/2.0: 404, message='', url=URL('https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2023-6709')

```



503

```

Error requesting [1/10]: (Retry 0.1 sec) https://services.nvd.nist.gov/rest/json/cves/2.0: 503, message='Service Unavailable', url=URL('https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2023-50447')

```



#### NVIDIA API Catalog / NVIDIA-hosted NIMs



429 errors can occur when your requests exceed the rate limit for the model. Try setting the `engine.agent.max_concurrency` to a low value such as 5 to reduce the rate of requests.

```

Exception: [429] Too Many Requests

```



### Running out of credits



If you run out of credits for the NVIDIA API Catalog, you will need to obtain more credits to continue using the API. Please contact your NVIDIA representative to get more credits added.



## Testing and validation

Test-driven development is essential for building reliable LLM-based agentic systems, especially when deploying or scaling them in production environments.



In our development process, we use the Morpheus public container as a case study. We perform security scans and collaborate with developers and security analysts to assess the exploitability of identified CVEs. Each CVE is labeled as either vulnerable or not vulnerable. For non-vulnerable CVEs, we provide a justification based on one of the ten VEX statuses. Team members document their investigative steps and findings to validate and compare results at different stages of the system.



We have collected labels for 38 CVEs, which serve several purposes:

- Human-generated checklists, findings, and summaries are used as ground truth during various stages of prompt engineering to refine LLM output.

- The justification status for each CVE is used as a label to measure end-to-end pipeline accuracy. Every time there is a change to the system, such as adding a new agent tool, modifying a prompt, or introducing an engineering optimization, we run the labeled dataset through the updated pipeline to detect performance regressions.



As a next step, we plan to integrate this process into our CI/CD pipeline to automate testing. While LLMs' non-deterministic nature makes it difficult to assert exact results for each test case, we can adopt a statistical approach, where we run the pipeline multiple times and ensure that the average accuracy stays within an acceptable range.



We recommend that teams looking to test or optimize their CVE analysis system curate a similar dataset for testing and validation. Note that in test-driven development, it's important that the model has not achieved perfect accuracy on the test set, as this may indicate overfitting or that the set lacks sufficient complexity to expose areas for improvement. The test set should be representative of the problem space, covering both scenarios where the model performs well and where further refinement is needed. Investing in a robust dataset ensures long-term reliability and drives continued performance improvements.

## Cite

Please consider citing [our paper](https://ceur-ws.org/Vol-3920/paper09.pdf) when using this code in a project. You can use the citation BibTeX:

```

@inproceedings{zemicheal2024llm,

  title={LLM agents for vulnerability identification and verification of CVEs},

  author={ZeMicheal, Tadesse and Chen, Hsin and Davis, Shawn and Allen, Rachel and Demoret, Michael and Song, Ashley},

  booktitle={Proceedings of the Conference on Applied Machine Learning in Information Security (CAMLIS 2024)},

  pages={161--173},

  year={2024},

  publisher={CEUR Workshop Proceedings},

  volume={3920},

  url={https://ceur-ws.org/Vol-3920/}

}

```

## License



By using this software or microservice, you are agreeing to the  [terms and conditions](https://www.nvidia.com/en-us/data-center/products/nvidia-ai-enterprise/eula/) of the license and acceptable use policy.



## Terms of Use



GOVERNING TERMS: The NIM container is governed by the [NVIDIA Software License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and [Product-Specific Terms for AI Products](https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/); and use of this model is governed by the [NVIDIA AI Foundation Models Community License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-ai-foundation-models-community-license-agreement/).



ADDITIONAL Terms: [Meta Llama 3.1 Community License](https://www.llama.com/llama3_1/license/), Built with Meta Llama 3.1.