• Table of Contents
• Overview
• Software components
• Target audience
• Prerequisites
• Hardware requirements
• API definition
• Use case description
  • How it works
  • Key components
  • NIM microservices
• Getting started
  • Install system requirements
  • Obtain API keys
  • Set up the workflow repository
  • Set up the environment file
  • Authenticate Docker with NGC
  • Build the Docker containers (optional)
  • Start the Docker containers
    • Using NVIDIA-hosted NIMs
    • Using self-hosted NIMs
• Running the workflow
  • From the quick start user guide notebook
  • From the command line
    • Configuration files
    • Example command: from_manual.json
      • Reviewing the output
    • Example command: from_http.json
  • Command line interface (CLI) reference
    • Overriding options on the command line
      • Example 1: Override max_retries
      • Example 2: Override input type
      • Example 3: Combine multiple overrides
    • Full list of options
  • Configuration file reference
  • NGINX caching server
• Customizing the Workflow
  • Customizing the input
  • Customizing the embedding model
  • Customizing the LLM models
    • Supported LLM services
  • Customizing the Output
  • Integrating self-hosted LLM NIM
• Troubleshooting
  • Git LFS issues
  • Container build issues
    • Device error
    • Deploy.Resources.Reservations.devices error
  • NGINX caching server
    • Resetting the entire cache
    • Resetting just the LLM cache or the services cache
    • Vector databases
  • Service outages
    • National Vulnerability Database (NVD)
  • Running out of credits
• Testing and validation
• License
• Terms of Use

This repository is what powers the build experience, showcasing vulnerability analysis for container security using NVIDIA NIM microservices and NVIDIA Morpheus.

The NVIDIA AI Blueprint demonstrates accelerated analysis on common vulnerabilities and exposures (CVE) at an enterprise scale, reducing mitigation from days and hours to just seconds. While traditional methods require substantial manual effort to pinpoint solutions for vulnerabilities, these technologies enable quick, automatic, and actionable CVE risk analysis using large language models (LLMs) and retrieval-augmented generation (RAG). With this blueprint, security analysts can expedite the process of determining whether a software package includes exploitable and vulnerable components using LLMs and event-driven RAG triggered by the creation of a new software package or the detection of a CVE.

The following are used by this blueprint:

• NIM of meta/llama-3.1-70b-instruct
• NIM of nvidia/nv-embedqa-e5-v5
• NVIDIA Morpheus Cybersecurity AI SDK

This blueprint is for:

• Security analysts and IT engineers: People analyzing vulnerabilities and ensuring the security of containerized environments.
• AI practitioners in cybersecurity: People applying AI to enhance cybersecurity, particularly those interested in using the Morpheus SDK and NIMs for faster vulnerability detection and analysis.

• NVAIE developer licence
• API keys for vulnerability databases, search engines, and LLM model service(s).
  • Details can be found in this later section: Obtain API keys

Below are the hardware requirements for each component of the vulnerability analysis pipeline.

The overall hardware requirements depend on selected pipeline configuration. At a minimum, the hardware requirements for pipeline operation must be met. The LLM NIM and Embedding NIM hardware requirements only need to be met if self-hosting these components. See Using self-hosted NIMs, Customizing the LLM models and Customizing the embedding model sections for more information.

• (Required) Pipeline operation: 1x L40 GPU or similar recommended
• (Optional) LLM NIM: Meta Llama 3.1 70B Instruct Support Matrix
  • For improved paralleled performance, we recommend 8x or more H100s for LLM inference.
  • The pipeline can share the GPU with the LLM NIM, but it is recommended to have a separate GPU for the LLM NIM for optimal performance.
• (Optional) Embedding NIM: NV-EmbedQA-E5-v5 Support Matrix
  • The pipeline can share the GPU with the Embedding NIM, but it is recommended to have a separate GPU for the Embedding NIM for optimal performance.

OpenAPI Specification

Determining the impact of a documented CVE on a specific project or container is a labor-intensive and manual task, especially as the rate of new reports into the CVE database accelerates. This process involves the collection, comprehension, and synthesis of various pieces of information to ascertain whether immediate remediation is necessary upon the identification of a new CVE.

Current challenges in CVE analysis:

• Information collection: The process involves significant manual labor to collect and synthesize relevant information.
• Decision complexity: Decisions on whether to update a library impacted by a CVE often hinge on various considerations, including:
  • Scan false positives: Occasionally, vulnerability scans may incorrectly flag a library as vulnerable, leading to a false alarm.
  • Mitigating factors: In some cases, existing safeguards within the environment may reduce or negate the risk posed by a CVE.
  • Lack of required environments or dependencies: For an exploit to succeed, specific conditions must be met. The absence of these necessary elements can render a vulnerability irrelevant.
• Manual documentation: Once an analyst has determined the library is not affected, a Vulnerability Exploitability eXchange (VEX) document must be created to standardize and distribute the results.

The efficiency of this process can be significantly enhanced through the deployment of an automated LLM agent pipeline, leveraging generative AI to improve vulnerability defense while decreasing the load on security teams.

The workflow operates using a Plan-and-Execute-style LLM pipeline for CVE impact analysis. The process begins with an LLM planner that generates a context-sensitive task checklist. This checklist is then executed by an LLM agent equipped with Retrieval-Augmented Generation (RAG) capabilities. The gathered information and the agent's findings are subsequently summarized and categorized by additional LLM nodes to provide a final verdict.

[!TIP] The pipeline is adaptable, supporting various LLM services that conform to the LLMService interface, including OpenAI, NeMo, or local execution with llama-cpp-python.

The detailed architecture consists of the following components:

• Security scan result: The workflow begins by inputting the identified CVEs from a container security scan as input. This can be generated from a container image scanner of your choosing such as Anchore.

• PreProcessing: All the below actions are encapsulated by multiple Morpheus preprocessing pipeline stages to prepare the data for use with the LLM engine. (See src/cve/pipeline/input.py.)

  • Code repository and documentation: The blueprint pulls code repositories and documentation provided by the user. These repositories are processed through an embedding model, and the resulting embeddings are stored in vector databases (VDBs) for the agent's reference.
    • Vector database: Various vector databases can be used for the embedding. We currently utilize FAISS for the VDB because it does not require an external service and is simple to use. Any vector store can be used, such as NVIDIA cuVS, which would provide accelerated indexing and search.
    • Lexical search: As an alternative, a lexical search is available for use cases where creating an embedding is impractical due to a large number of source files in the target container.
  • Software Bill of Materials (SBOM): The provided SBOM document is processed into a software-ingestible format for the agent's reference. SBOMs can be generated for any container using the open-source tool Syft.
  • Web vulnerability intel: The system collects detailed information about each CVE through web scraping and data retrieval from various public security databases, including GHSA, Redhat, Ubuntu, and NIST CVE records, as well as tailored threat intelligence feeds.

• Core LLM engine: (See src/cve/pipeline/engine.py.)

  • Checklist generation: Leveraging the gathered information about each vulnerability, the checklist generation node creates a tailored, context-sensitive task checklist designed to guide the impact analysis. (See src/cve/nodes/cve_checklist_node.py.)

  • Task agent: At the core of the process is an LLM agent iterating through each item in the checklist. For each item, the agent answers the question using a set of tools which provide information about the target container. The tools tap into various data sources (web intel, vector DB, search etc.), retrieving relevant information to address each checklist item. The loop continues until the agent resolves each checklist item satisfactorily. (See src/cve/nodes/cve_langchain_agent_node.py.)

  • Summarization: Once the agent has compiled findings for each checklist item, these results are condensed by the summarization node into a concise, human-readable paragraph. (See src/cve/nodes/cve_summary_node.py.)

  • Justification Assignment: Given the summary, the justification status categorization node then assigns a resulting VEX (Vulnerability Exploitability eXchange) status to the CVE. We provided a set of predefined categories for the model to choose from. (See src/cve/nodes/cve_justification_node.py.) If the CVE is deemed exploitable, the reasoning category is "vulnerable." If it is not exploitable, there are 10 different reasoning categories to explain why the vulnerability is not exploitable in the given environment:

    • false_positive
    • code_not_present
    • code_not_reachable
    • requires_configuration
    • requires_dependency
    • requires_environment
    • protected_by_compiler
    • protected_at_runtime
    • protected_by_perimeter
    • protected_by_mitigating_control

• Output: At the end of the pipeline run, an output file including all the gathered and generated information is prepared for security analysts for a final review. (See src/cve/pipeline/output.py.)

[!WARNING] All output should be vetted by a security analyst before being used in a cybersecurity application.

The Morpheus SDK can utilize various embedding model and LLM endpoints, and is optimized to use NVIDIA NIM microservices (NIMs). NIMs are pre-built containers for the latest AI models that provide industry-standard APIs and optimized inference for the given model and hardware. Using NIMs enables easy deployment and scaling for self-hosted model inference.

The current default embedding NIM model is nv-embedqa-e5-v5, which was selected to balance speed and overall pipeline accuracy. The current default LLM model is the llama-3.1-70b-instruct NIM, with specifically tailored prompt engineering and edge case handling. Other models are able to be substituted for either the embedding or LLM model, such as smaller, fine-tuned NIM LLM models or other external LLM inference services. Subsequent updates will provide more details about fine-tuning and data flywheel techniques.

[!NOTE] The LangChain library is employed to deploy all LLM agents within a Morpheus pipeline, streamlining efficiency and reducing the need for duplicative efforts.

[!TIP] Routinely checked validation datasets are critical to ensuring proper and consistent outputs. Learn more about our test-driven development approach in the section on testing and validation.

• git
• git-lfs
• Since the workflow uses an NVIDIA Morpheus pipeline, the Morpheus requirements also need to be installed.

To run the pipeline you need to obtain API keys for the following APIs. These will be needed in a later step to Set up the environment file.

• Required API Keys: These APIs are required by the pipeline to retrieve vulnerability information from databases, perform online searches, and execute LLM queries.

  • GitHub Security Advisory (GHSA) Database
    • Follow these instructions to create a personal access token. No repository access or permissions are required for this API.
    • This will be used in the GHSA_API_KEY environment variable.
  • National Vulnerability Database (NVD)
    • Follow these instructions to create an API key.
    • This will be used in the NVD_API_KEY environment variable.
  • SerpApi
    • Go to https://serpapi.com/ and create a SerpApi account. Once signed in, navigate to Your Account > Api Key.
    • This will be used in the SERPAPI_API_KEY environment variable.
  • NVIDIA Inference Microservices (NIM)
    • There are two possible methods to generate an API key for NIM:
      • Sign in to the NVIDIA Build portal with your email.
        • Click on any model, then click "Get API Key", and finally click "Generate Key".
      • Sign in to the NVIDIA NGC portal with your email.
        • Select your organization from the dropdown menu after logging in. You must select an organization which has NVIDIA AI Enterprise (NVAIE) enabled.
        • Click on your account in the top right, select "Setup" from the dropdown.
        • Click the "Generate Personal Key" option and then the "+ Generate Personal Key" button to create your API key.
    • This will be used in the NVIDIA_API_KEY environment variable.

The workflow can be configured to use other LLM services as well, see the Customizing the LLM models section for more info.

Clone the repository and set an environment variable for the path to the repository root.

export REPO_ROOT=$(git rev-parse --show-toplevel)

All commands are run from the repository root unless otherwise specified.

First we need to create an .env file in the REPO_ROOT, and add the API keys you created in the earlier Obtain API keys step.

cd $REPO_ROOT
cat <<EOF > .env
GHSA_API_KEY="your GitHub personal access token"
NVD_API_KEY="your National Vulnerability Database API key"
NVIDIA_API_KEY="your NVIDIA Inference Microservices API key"
SERPAPI_API_KEY="your SerpApi API key"
EOF

These variables need to be exported to the environment:

export $(cat .env | xargs)

In order to pull images required by the workflow from NGC, you must first authenticate Docker with NGC. You can use same the NVIDIA API Key obtained in the Obtain API keys section (saved as NVIDIA_API_KEY in the .env file).

echo "${NVIDIA_API_KEY}" | docker login nvcr.io -u '$oauthtoken' --password-stdin

If no customizations were made to the source code, you can proceed to Starting the Docker containers, where the docker compose up step will automatically pull the pre-built Blueprint Docker container from NGC: nvcr.io/nvidia/morpheus/morpheus-vuln-analysis:24.10.

If any customizations are made to the source code, we will need to build the container from source using the following command:

cd $REPO_ROOT

# Build the morpheus-vuln-analysis container
docker compose build morpheus-vuln-analysis

There are two supported configurations for starting the Docker containers. Both configurations utilize docker compose to start the service:

1. NVIDIA-hosted NIMs: The workflow is run with all computation being performed by NIMs hosted in NVIDIA GPU Cloud. This is the default configuration and is recommended for most users getting started with the workflow.
   1. When using NVIDIA-hosted NIMs, only the docker-compose.yml configuration file is required.
2. Self-hosted NIMs: The workflow is run using self-hosted LLM NIM services. This configuration is more advanced and requires additional setup to run the NIM services locally.
   1. When using self-hosted NIMs, both the docker-compose.yml and docker-compose.nim.yml configuration files are required.

These two configurations are illustrated by the following diagram:

Before beginning, ensure that the environment variables are set correctly. Both configurations require the same environment variables to be set. More information on setting these variables can be found in the Obtain API keys section.

[!TIP] The container binds to port 8080 by default. If you encounter a port collision error (e.g. Bind for 0.0.0.0:8080 failed: port is already allocated), you can set the environment variable NGINX_HOST_HTTP_PORT to specify a custom port before launching docker compose. For example:

export NGINX_HOST_HTTP_PORT=8081

#... docker compose commands...

When running the workflow in this configuration, only the morpheus-vuln-analysis service needs to be started since we will utilize NIMs hosted by NVIDIA. The morpheus-vuln-analysis container can be started using the following command:

cd ${REPO_ROOT}
docker compose up -d

The command above starts the container in the background using the detached mode, -d. We can confirm the container is running via the following command:

docker compose ps

Next, we need to attach to the morpheus-vuln-analysis container to access the environment where the workflow command line tool and dependencies are installed.

docker compose exec -it morpheus-vuln-analysis bash

Continue to the Running the workflow section to run the workflow.

To run the workflow using self-hosted NIMs, we use a second docker compose configuration file, docker-compose.nim.yml, which adds the self-hosted NIM services to the workflow. Utilizing a second configuration file allows for easy switching between the two configurations while keeping the base configuration file the same.

[!NOTE] The self-hosted NIM services require additional GPU resources to run. With this configuration, the LLM NIM, embedding model NIM, and the morpheus-vuln-analysis service will all be launched on the same machine. Ensure that you have the necessary hardware requirements for all three services before proceeding (multiple services can share the same GPU).

To use multiple configuration files, we need to specify both configuration files when running the docker compose command. You will need to specify both configuration files for every docker compose command. For example:

docker compose -f docker-compose.yml -f docker-compose.nim.yml [NORMAL DOCKER COMPOSE COMMAND]

For example, to start the morpheus-vuln-analysis service with the self-hosted NIMs, you would run:

cd ${REPO_ROOT}
docker compose -f docker-compose.yml -f docker-compose.nim.yml up -d

Next, we need to attach to the morpheus-vuln-analysis container to access the environment where the workflow command line tool and dependencies are installed.

docker compose -f docker-compose.yml -f docker-compose.nim.yml exec -it morpheus-vuln-analysis bash

Continue to the Running the workflow section to run the workflow.

Once the services have been started, the workflow can be run using either the Quick start user guide notebook for an interactive step-by-step process, or directly from the command line.

To run the workflow in an interactive notebook, connect to the Jupyter notebook at http://localhost:8000/lab. Once connected, navigate to the notebook located at quick_start/quick_start_guide.ipynb and follow the instructions.

[!TIP] If you are running the workflow on a remote machine, you can forward the port to your local machine using SSH. For example, to forward port 8000 from the remote machine to your local machine, you can run the following command from your local machine:

ssh -L 8000:127.0.0.1:8000 <remote_host_name>

The vulnerability analysis workflow is designed to be run using the command line tool installed within the morpheus-vuln-analysis container. This section describes how get started using the command line tool. For more detailed information about the command line interface, see the Command line interface (CLI) reference section.

The pipeline settings are controlled using configuration files. These are JSON files that define various pipeline settings, such as the input data, the LLM models used, and the output format. Several example configuration files are located in the configs/ folder. A brief description of each configuration file is as follows:

• from_manual.json: This configuration file starts the pipeline using manually provided data. All pipeline inputs are manually specified directly in the configuration file.
• from_file.json: This configuration file starts the pipeline using data fetched from a file. The pipeline fetches the input data from a file and processes it. This is very similar to from_manual.json, but the input data is read from a file instead of being directly specified in the config file.
• from_http.json: This configuration file starts an HTTP server to turn the workflow into a microservice. The pipeline fetches the input data from an HTTP source and processes it. To trigger the pipeline, you can send a POST request to the /scan endpoint with the input data in the request body. The pipeline will process the input data.

There are two main modalities that the pipeline can be run in. When using the from_file.json or from_manual.json configuration files, the pipeline will process the input data, then it will shut down after it is completed. This modality is suitable for rapid iteration during testing and development. When using from_http.json, the pipeline is turned into a microservice which will run indefinitely, which is suitable for using in production.

For a breakdown of the configuration file and available options, see the Configuration file reference section. To customize the configuration files for your use case, see Customizing the workflow.

The workflow pipeline can be started using the following command:

python src/main.py --log_level DEBUG \
  cve pipeline --config_file=${CONFIG_FILE}

In the command, ${CONFIG_FILE} is the path to the configuration file you want to use. For example, to run the pipeline with the from_manual.json configuration file, you would run:

python src/main.py --log_level DEBUG \
  cve pipeline --config_file=configs/from_manual.json

When the pipeline runs to completion, you should see logs similar to the following:

Message elapsed time: 28.240849 sec
Vulnerability 'GHSA-3f63-hfp8-52jq' affected status: FALSE. Label: code_not_reachable
Vulnerability 'CVE-2023-50782' affected status: FALSE. Label: requires_configuration
Vulnerability 'CVE-2023-36632' affected status: FALSE. Label: code_not_present
Vulnerability 'CVE-2023-43804' affected status: TRUE. Label: vulnerable
Vulnerability 'GHSA-cxfr-5q3r-2rc2' affected status: TRUE. Label: vulnerable
Vulnerability 'GHSA-554w-xh4j-8w64' affected status: TRUE. Label: vulnerable
Vulnerability 'GHSA-3ww4-gg4f-jr7f' affected status: FALSE. Label: requires_configuration
Vulnerability 'CVE-2023-31147' affected status: FALSE. Label: code_not_present
Source[Complete]: 7 messages [00:14,  2.02s/ messages]
LLM[Complete]: 7 messages [00:42,  6.05s/ messages]
====Pipeline Complete====
Total time: 45.61 sec
Pipeline runtime: 42.69 sec

[!WARNING] The output you receive from the pipeline may not be identical as the output in the example above. The output may vary due to the non-deterministic nature of the LLM models.

The full pipeline JSON output is stashed by default at .tmp/output.json. The output JSON includes the following top level fields:

• input: contains the inputs that were provided to the pipeline, such as the container and repo source information, the list of vulnerabilities to scan, etc.
• info: contains additional information collected by the pipeline for decision making. This includes paths to the generated VDB files, intelligence from various vulnerability databases, the list of SBOM packages, and any vulnerable dependencies that were identified.
• output: contains the output from the core LLM Engine, including the generated checklist, analysis summary, and justification assignment.

In addition to the raw JSON output, you can also view a Markdown-formatted report for each CVE in the .tmp/vulnerability_markdown_reports directory. This view is helpful for human analysts reviewing the results.

[!TIP] To return detailed steps taken by the LLM agent in the output, set return_intermediate_steps to true in the configuration file. This can be helpful for explaining the output, and for troubleshooting unexpected results.

Similarly, to run the pipeline with the from_http.json configuration file, you would run:

python src/main.py --log_level DEBUG cve pipeline --config_file=configs/from_http.json

This command starts an HTTP server that listens on port 26466 and runs the workflow indefinitely, waiting for incoming data to process. This is useful if you want to trigger the workflow on demand via HTTP requests.

Once the server is running, you can send a POST request to the /scan endpoint with the input parameters in the request body. The pipeline will process the input data and return the output in the terminal and the given output path in the config file.

Here's an example using curl to send a POST request. From a new terminal outside of the container, go to the root of the cloned git repository, and run:

curl -X POST http://localhost:26466/scan -d @data/input_messages/morpheus:24.03-runtime.json

In this command:

• http://localhost:26466/scan is the URL of the server and endpoint.
• The -d option specifies the data file being sent in the request body. In this case, it's pointing to the input file morpheus:24.03-runtime.json under the data/input_messages/ directory. You can refer to this file as an example of the expected data format.
  • Since it uses a relative path, it's important to run the curl command from the root of the git repository. Alternatively, you can modify the relative path in the command to directly reference the example json file.

Note that the results of the pipeline are not returned to the curl request. After processing the request, the server will save the results to the output path specified in the configuration file. The server will also display log and summary results from the workflow as it's running. Additional submissions to the server will append the results to the specified output file.

The top level entrypoint to each of the LLM example pipelines is src/main.py. The main entrypoint is a CLI tool with built-in documentation using the --help command. For example, to see what commands are available, you can run:

(morpheus) root@58145366033a:/workspace# python src/main.py --help
Usage: morpheus_llm [OPTIONS] COMMAND [ARGS]...

  Main entrypoint for the Vulnerability Analysis for Container Security

Options:
  --log_level [CRITICAL|FATAL|ERROR|WARN|WARNING|INFO|DEBUG]
                                  Specify the logging level to use.  [default:
                                  INFO]
  --use_cpp BOOLEAN               Whether or not to use C++ node and message
                                  types or to prefer python. Only use as a
                                  last resort if bugs are encountered
                                  [default: True]
  --version                       Show the version and exit.
  --help                          Show this message and exit.

Commands:
  cve  Run the Vulnerability Analysis for Container Security pipeline

It's common to want to override some options in the configuration file on the command line. This is useful to use some common options but tweak the input/output of the pipeline or to change a single setting without modifying the original config file.

For example, to override the max_retries option in the configuration file, you can run:

python src/main.py --log_level=DEBUG \
  cve pipeline --config_file=configs/from_manual.json \
    config \
      general --max_retries=3

This will run the pipeline with the from_manual.json configuration file, but with the max_retries option set to 3.

It's also possible to change the input type. For example, to use a different input message with the from_file.json configuration file, you can run:

python src/main.py --log_level=DEBUG \
  cve pipeline --config_file=configs/from_file.json \
    config \
      input-file --file=data/input_messages/morpheus:24.03-runtime.json

It's possible to combine multiple overrides in a single command. For example, to run the pipeline with the from_manual.json configuration file, but with the max_retries option set to 3, the input message set to morpheus:24.03-runtime.json, and the output destinations set to .tmp/output_morpheus.json and .tmp/morpheus_reports, you can run:

python src/main.py --log_level=DEBUG \
  cve pipeline --config_file=configs/from_manual.json \
    config \
      general --max_retries=3 \
      input-file --file=data/input_messages/morpheus:24.03-runtime.json \
      output-file --file_path=.tmp/output_morpheus.json --markdown_dir=.tmp/morpheus_reports

For the full list of possible options, use the --help options from the CLI.

The configuration defines how the workflow operates, including model settings, input sources, and output options.

1. Schema
   • $schema: Specifies the schema for validating the configuration file. This ensures the correct structure and data types are used throughout the config file.
2. LLM engine configuration (engine): The engine section configures various models for the LLM nodes.
   • LLM processing nodes: agent, checklist_model, justification_model, summary_model
     • model_name: The name of the LLM model used by the node.
     • prompt: Manually set the prompt for the specific model in the configuration. The prompt can either be passed in as a string of text or as a path to a text file containing the desired prompting.
     • service: Specifies the service for running the LLM inference. (Set to nvfoundation if using NIM.)
     • max_tokens: Defines the maximum number of tokens that can be generated in one output step.
     • temperature: Controls randomness in the output. A lower temperature produces more deterministic results.
     • top_p: Limits the diversity of token sampling based on cumulative probability.
     • Settings specific to the agent node:
       • verbose: Toggles detailed logging for the agent during execution.
       • return_intermediate_steps: Controls whether to return intermediate steps taken by the agent, and include them in the output file. Helpful for troubleshooting agent responses.
       • return_source_documents: Controls whether to return source documents from the VDB tools, and include them in the intermediate steps output. Helpful for identifying the source files used in agent responses.
         • Note: enabling this will also include source documents in the agent's memory and increase the agent's prompt length.
   • Embedding model for generating VDB for RAG: rag_embedding
     • _type: Defines the source of the model used for generating embeddings (e.g., nim, huggingface, openai).
     • Other model-dependent parameters, such as model/model_name, api_key, truncate, or encode_kwargs: see the embedding model customization section below for more details.
3. General configuration: The general section contains settings that influence the workflow's general behavior, including cache settings, batch sizes, and retry policies.
   • cache_dir: The directory where the node's cache should be stored. If None, caching is not used.
   • base_vdb_dir: The directory used for storing vector database files.
   • base_git_dir: The directory for storing pulled git repositories used for code analysis.
   • max_retries: Sets the number of retry attempts for failed operations.
   • model_max_batch_size: Specifies the maximum number of messages to send to the model for inference in a single batch.
   • pipeline_batch_size: Determines the number of messages per batch for the pipeline.
   • use_uvloop: Toggles the use of uvloop, an optimized event loop for improved performance.
   • code_search_tool: Enables or disables the use of the code search tool.
4. Input configuration: The input section defines how and where the input data (container images and vulnerabilities) is sourced.
   • _type: Defines the input type
     • manual: input data is provided manually in the config file
     • http: input data is provided through http POST call pointing to an input source file
     • file: input data is fetched from a provided source file
   • message: Contains details about the input image and its associated vulnerabilities. Required only for the manual input type.
     • image: Specifies the container image to be analyzed.
       • name: Specifies the name of the container image.
       • tag: Specifies the tag of the container image.
       • source_info: Specifies the sources (e.g., git repositories) used to retrieve code or documentation for analysis.
         • include: Specifies the file patterns to be included in the analysis.
         • exclude: Specifies the file patterns to exclude from analysis.
       • sbom_info: Specifies the Software Bill of Materials (SBOM) file to be analyzed.
     • scan: Provides a list of vulnerabilities (by ID) to be analyzed for the input image.
5. Output configuration: The output section defines where the output results of the workflow should be stored and how they should be managed.
   • _type: Specifies the output type. Use file to write the results to a file.
   • file_path: Defines the path to the file where the output will be saved.
   • markdown_dir: Defines the path to the directory where the output will be saved in individual navigable markdown files per CVE-ID.
   • overwrite: Indicates whether the output file should be overwritten when the pipeline starts if it already exists. Will throw an error if set to False and the file already exists. Note that the overwrite behavior only occurs on pipeline initialization. For pipelines started in HTTP mode, each new request will append the existing file until the pipeline is restarted.

The docker compose file includes an nginx-cache proxy server container that enables caching for API requests made by the workflow. It is highly recommend to route API requests through the proxy server to reduce API calls for duplicate requests and improve workflow speed. This is especially useful when running the pipeline multiple times with the same configuration (e.g., for debugging) and can help keep costs down when using paid APIs.

The NGINX proxy server is started by default when running the morpheus-vuln-analysis service. However, it can be started separately using the following command:

cd ${REPO_ROOT}
docker compose up --detach nginx-cache

To use the proxy server for API calls in the workflow, you can set environment variables for each base URL used by the workflow to point to http://localhost:${NGINX_HOST_HTTP_PORT}/. These are set automatically when running the morpheus-vuln-analysis service, but can be set manually in the .env file as follows:

CVE_DETAILS_BASE_URL="http://localhost:8080/cve-details"
CWE_DETAILS_BASE_URL="http://localhost:8080/cwe-details"
DEPSDEV_BASE_URL="http://localhost:8080/depsdev"
FIRST_BASE_URL="http://localhost:8080/first"
GHSA_BASE_URL="http://localhost:8080/ghsa"
NGC_API_BASE="http://localhost:8080/nemo/v1"
NIM_EMBED_BASE_URL="http://localhost:8080/nim_embed/v1"
NVD_BASE_URL="http://localhost:8080/nvd"
NVIDIA_API_BASE="http://localhost:8080/nim_llm/v1"
OPENAI_API_BASE="http://localhost:8080/openai/v1"
OPENAI_BASE_URL="http://localhost:8080/openai/v1"
RHSA_BASE_URL="http://localhost:8080/rhsa"
SERPAPI_BASE_URL="http://localhost:8080/serpapi"
UBUNTU_BASE_URL="http://localhost:8080/ubuntu"

The primary method for customizing the workflow is to generate a new configuration file with new options. The configuration file defines the pipeline settings, such as the input data, the LLM models used, and the output format. The configuration file is a JSON file that can be modified to suit your needs.

Currently, there are 3 types of input sources supported by the pipeline:

• Manual input: The input data is directly specified in the configuration file.
• File input: The input data is read from a file.
• HTTP input: The input data is fetched from an HTTP source.

To customize the input, modify the configuration file accordingly. In any configuration file, locate the input section to see the input source used by the pipeline. For example, in the configuration file configs/from_manual.json, the following snippet defines the input source as manual:

    "input": {
      "_type": "manual",
      "message": {
        ...Contents of the input message...
      }
    }

To use a file as the input source, update the JSON object in the config file to:

    "input": {
      "_type": "file",
      "file": "data/input_messages/morpheus:23.11-runtime.json"
    }

To use an HTTP source as the input, update the JSON object in the config file to:

    "input": {
      "_type": "http",
      "address": "127.0.0.1",
      "endpoint": "/scan",
      "http_method": "POST",
      "port": 26466
    }

Vector databases are used by the agent to fetch relevant information for impact analysis investigations. The embedding model used to vectorize your documents can significantly affect the agent's performance. The default embedding model used by the pipeline is the NIM nvidia/nv-embedqa-e5-v5 model, but you can experiment with different embedding models of your choice.

To test a custom embedding model, modify the configuration file in the engine.rag_embedding section. For example, in the from_manual.json configuration file, the following snippet defines the settings for the default embedding model:

    "rag_embedding": {
      "_type": "nim",
      "model": "nvidia/nv-embedqa-e5-v5",
      "truncate": "END",
      "max_batch_size": 128
    },

• rag_embedding._type: specifies the embedding provider. The current supported options are nim, huggingface and openai.
• model_name: specifies the model name for the embedding provider. Refer to the embedding provider's documentation to determine the available models.
• truncate: specifies how inputs longer than the maximum token length of the model are handled. Passing START discards the start of the input. END discards the end of the input. In both cases, input is discarded until the remaining input is exactly the maximum input token length for the model. If NONE is selected, when the input exceeds the maximum input token length an error will be returned.
• max_batch_size: specifies the batch size to use when generating embeddings. We recommend setting this to 128 (default) or lower when using the cloud-hosted embedding NIM. When using a local NIM, this value can be tuned based on throughput/memory performance on your hardware.

Steps to configure an alternate embedding provider

1. If using OpenAI embeddings, first obtain an API key, then update the .env file with the auth and base URL environment variables for the service as indicated in the Supported LLM Services table. Otherwise, proceed to step 2.

2. Update the rag_embedding section of the config file as described above.

   • Example HuggingFace embedding configuration:

       "rag_embedding": {
         "_type": "huggingface",
         "model_name": "intfloat/e5-large-v2",
         "encode_kwargs": {
           "batch_size": 128
         }
       }
   

   • Example OpenAI embedding configuration:

       "rag_embedding": {
         "_type": "openai",
         "model_name": "text-embedding-3-small",
         "encode_kwargs": {
           "max_retries": 5,
           "chunk_size": 256
         }
       }
   

   • For HuggingFace embeddings, all parameters from LangChain's HuggingFaceEmbeddings class are supported. However, for OpenAI models, only a subset of parameters are supported. The full set of available parameters can be found in the config definitions here. Any non-supported parameters provided in the configuration will be ignored.

The current pipeline uses FAISS to create the vector databases. Interested users can customize the source code to use other vector databases such as cuVS.

The configuration file also allows customizing the LLM model and parameters for each component of the workflow, as well which LLM service is used when invoking the model.

In any configuration file, locate the engine section to see the current settings. For example, in the from_manual.json configuration file, the following snippet defines the LLM used for the checklist model:

    "checklist_model": {
      "service": {
        "_type": "nvfoundation"
      },
      "model_name": "meta/llama-3.1-70b-instruct",
      "temperature": 0,
      "max_tokens": 2000,
      "top_p": 0.01
    }

• service._type: specifies the LLM service to use. Refer to the Supported LLM Services table for available options.
• model_name: specifies the model name within the LLM service. Refer to the service's API documentation to determine the available models.
• temperature, max_tokens, top_p, ...: specifies the model parameters. Note that by default, the config supports only a subset of parameters provided by each LLM service. The available parameters can be found in the configuration object's definition here. Any non-supported parameters provided in the configuration will be ignored.

Steps to configure an LLM model

1. Obtain an API key and any other required auth info for the selected service.
2. Update the .env file with the auth and base URL environment variables for the service as indicated in the Supported LLM Services table.
3. Update the config file as described above. For example, if you want to use OpenAI's gpt-4o model for checklist generation, update the above json object in the config file to:

    "checklist_model": {
      "service": {
        "_type": "openai"
      },
    "model_name": "gpt-4o",
      "temperature": 0,
      "top_p": 0.01,
      "seed": 0,
      "max_retries": 5
    },

Please note that the prompts have been tuned to work best with the Llama 3.1 70B NIM and that when using other LLM models it may be necessary to adjust the prompting.

Currently, there are 3 types of outputs supported by the pipeline:

• File output: The output data is written to a file in JSON format.
• HTTP output: The output data is posted to an HTTP endpoint.
• Print output: The output data is printed to the console.

To customize the output, modify the configuration file accordingly. In any configuration file, locate the output section to see the output destination used by the pipeline. For example, in the configuration file configs/from_manual.json, the following snippet defines the output destination as a single json file and individual markdown files per CVE-ID:

    "output": {
      "_type": "file",
      "file_path": ".tmp/output.json",
      "markdown_dir": ".tmp/vulnerability_markdown_reports"
    }

To post the output to an HTTP endpoint, update the JSON object in the config file as follows, replacing the domain, port, and endpoint with the desired destination (note the trailing slash in the "url" field). The output will be sent as JSON data.

    "output": {
      "type": "http",
      "url": "http://<domain>:<port>/",
      "endpoint": "<endpoint>"
    }

To print the output without saving to a file, update the JSON object in the config file to:

    "output": {
      "_type": "print"
    }

Additional output options will be added in the future.

The worklow is configured by default to use an NVIDIA-hosted LLM NIM for which an NVIDIA API key (NVIDIA_API_KEY) is required. The workflow can also be used with a self-hosted LLM NIM. You can start here for more information on how to pull and run the NIM locally.

Once NIM is deployed, update your pipeline configuration (e.g. from_manual.json) to now use the your NIM. For every component under engine that uses nvfoundation, update to now use openai and model name of your NIM (i.e. meta/llama-3.1-70b-instruct). For example:

"agent": {
  "model": {
    "model_name": "meta/llama-3.1-70b-instruct",
    "service": {
      "_type": "openai"
    },
    "max_tokens": 2000
    },
    "verbose": true
}

If using nginx cache, update openai_upstream in nginx_cache.conf to point to your NIM URL:

set $openai_upstream http://llm-nim:8000;

Here llm-nim is the configured service name for the NIM if using helm chart of docker compose. Otherwise, set to actual host name or IP address of your NIM.

Now set OPENAI_BASE_URL environment variable to NIM or nginx URL depending on your configuration. The OPENAI_API_KEY must also be set but only to prevent a check error (not for authentication). This can be set to any string.

Several common issues can arise when running the pipeline. Here are some common issues and their solutions.

If you encounter issues with Git LFS, ensure that you have Git LFS installed and that it is enabled for the repository. You can check if Git LFS is enabled by running the following command:

git lfs install

Verifying that all files are being tracked by Git LFS can be done by running the following command:

git lfs ls-files

Files which are missing will show a - next to their name. To ensure all LFS files have been pulled correctly, you can run the following command:

git lfs fetch --all
git lfs checkout *

When building containers for self-hosted NIMs, certain issues may occur. Below are common troubleshooting steps to help resolve them.

If you encounter an error resembling the following during the container build process for self-hosted NIMs:

nvidia-container-cli: device error: {n}: unknown device: unknown

This error typically indicates that the container is attempting to access GPUs that are either unavailable or non-existent on the host. To resolve this, verify the GPU count specified in the docker-compose.nim.yml configuration file:

• Navigate to the deploy.resources.reservations.devices section and check the count parameter.
• Set the environment variable NIM_LLM_GPU_COUNT to the actual number of GPUs available on the host machine before building the container. Note that the default value is set to 4.

This adjustment ensures the container accurately matches the available GPU resources, preventing access errors during deployment.

If you encounter an error resembling the following during the container build process for self-hosted NIMs process:

1 error(s) decoding:

* error decoding 'Deploy.Resources.Reservations.devices[0]': invalid string value for 'count' (the only value allowed is 'all')

This is likely caused by an outdated Docker Compose version. Please upgrade Docker Compose to at least v2.21.0.

Because the workflow makes such heavy use of the caching server to speed up API requests, it is important to ensure that the server is running correctly. If you encounter issues with the caching server, you can reset the cache.

To reset the entire cache, you can run the following command:

docker compose down -v

This will delete all the volumes associated with the containers, including the cache.

If you want to reset just the LLM cache or the services cache, you can run the following commands:

docker compose down

# To remove the LLM cache
docker volume rm ${COMPOSE_PROJECT_NAME:-morpheus_vuln_analysis}_llm-cache

# To remove the services cache
docker volume rm ${COMPOSE_PROJECT_NAME:-morpheus_vuln_analysis}_service-cache

We've integrated VDB and embedding creation directly into the pipeline with caching included for expediency. However, in a production environment, it's better to use a separately managed VDB service.

NVIDIA offers optimized models and tools like NIMs (build.nvidia.com/explore/retrieval) and cuVS (github.com/rapidsai/cuvs).

These typically resolve on their own. Please wait and try running the pipeline again later. Example errors:

404

Error requesting [1/10]: (Retry 0.1 sec) https://services.nvd.nist.gov/rest/json/cves/2.0: 404, message='', url=URL('https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2023-6709')

503

Error requesting [1/10]: (Retry 0.1 sec) https://services.nvd.nist.gov/rest/json/cves/2.0: 503, message='Service Unavailable', url=URL('https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2023-50447')

If you run out of credits for the NVIDIA API Catalog, you will need to obtain more credits to continue using the API. Please contact your NVIDIA representative to get more credits added.

Test-driven development is essential for building reliable LLM-based agentic systems, especially when deploying or scaling them in production environments.

In our development process, we use the Morpheus public container as a case study. We perform security scans and collaborate with developers and security analysts to assess the exploitability of identified CVEs. Each CVE is labeled as either vulnerable or not vulnerable. For non-vulnerable CVEs, we provide a justification based on one of the ten VEX statuses. Team members document their investigative steps and findings to validate and compare results at different stages of the system.

We have collected labels for 38 CVEs, which serve several purposes:

• Human-generated checklists, findings, and summaries are used as ground truth during various stages of prompt engineering to refine LLM output.
• The justification status for each CVE is used as a label to measure end-to-end pipeline accuracy. Every time there is a change to the system, such as adding a new agent tool, modifying a prompt, or introducing an engineering optimization, we run the labeled dataset through the updated pipeline to detect performance regressions.

As a next step, we plan to integrate this process into our CI/CD pipeline to automate testing. While LLMs' non-deterministic nature makes it difficult to assert exact results for each test case, we can adopt a statistical approach, where we run the pipeline multiple times and ensure that the average accuracy stays within an acceptable range.

We recommend that teams looking to test or optimize their CVE analysis system curate a similar dataset for testing and validation. Note that in test-driven development, it's important that the model has not achieved perfect accuracy on the test set, as this may indicate overfitting or that the set lacks sufficient complexity to expose areas for improvement. The test set should be representative of the problem space, covering both scenarios where the model performs well and where further refinement is needed. Investing in a robust dataset ensures long-term reliability and drives continued performance improvements.

By using this software or microservice, you are agreeing to the terms and conditions of the license and acceptable use policy.

GOVERNING TERMS: The NIM container is governed by the NVIDIA Software License Agreement and Product-Specific Terms for AI Products; and use of this model is governed by the NVIDIA AI Foundation Models Community License Agreement.

ADDITIONAL Terms: Meta Llama 3.1 Community License, Built with Meta Llama 3.1.