Vulnerability database and package search for sources such as Linux, OSV, NVD, GitHub and npm.

AppThreat, updated 🕥 2023-03-01 07:33:17

Introduction

This repo is a vulnerability database and package search for sources such as Aqua Security vuln-list, OSV, NVD, GitHub, and NPM. Vulnerability data are downloaded from the sources and stored in a custom file based storage with indexes to allow offline access and quick searches.

Vulnerability Data sources

  • Linux vuln-list (Forked from AquaSecurity)
  • OSV
  • NVD
  • GitHub
  • NPM

Linux distros

  • AlmaLinux
  • Debian
  • Alpine
  • Amazon Linux
  • Arch Linux
  • RHEL/CentOS
  • Rocky Linux
  • Ubuntu
  • OpenSUSE/SLES
  • Photon

Installation

bash pip install appthreat-vulnerability-db

Usage

This package is ideal as a library for managing vulnerabilities. This is used by dep-scan, a free open-source dependency audit tool. However, there is a limited cli capability available with few features to test this tool directly.

Download pre-built database

Use the ORAS cli to download a pre-built database containing all application and OS vulnerabilities.

export VDB_HOME=$HOME/vdb oras pull ghcr.io/appthreat/vdb:v5 -o $VDB_HOME

Cache vulnerability data

Cache application vulnerabilities

bash vdb --cache

Typical size of this database is over 1.1 GB.

Cache application and OS vulnerabilities

bash vdb --cache-os

Note the size of the database with OS vulnerabilities is over 3.1 GB.

Cache from just OSV

bash vdb --cache --only-osv

It is possible to customise the cache behaviour by increasing the historic data period to cache by setting the following environment variables.

  • NVD_START_YEAR - Default: 2018. Supports upto 2002
  • GITHUB_PAGE_COUNT - Default: 2. Supports upto 20

Periodic sync

To periodically sync the latest vulnerabilities and update the database cache.

bash vdb --sync

Basic search

It is possible to perform simple search using the cli.

```bash vdb --search android:8.0

vdb --search google:android:8.0

vdb --search android:8.0,simplesamlphp:1.14.11 ```

Syntax is package:version,package:version or vendor : package : version (Without space)

Issues

Spend more time on batch size for storage

opened on 2023-03-04 06:58:18 by prabhu

v5 gave the needed performance boost for searches. However, one of the things I am not happy with is the hardcoded batch size while storing a group of records.

https://github.com/AppThreat/vulnerability-db/blob/master/vdb/lib/storage.py#L9

I picked this number out of thin air. Bad hotel internet means I could never experiment with various batch sizes to measure the impact on storage vs. search performance.

Announce: Proper database support

opened on 2022-12-02 12:33:37 by prabhu

After adding support for OS vulnerabilities, it is clear that this project's simple file-based search and indexing approach is no longer enough to support large containers, OS and cloud scans. Instead of going the traditional RDBMS route, I am working on a replacement project to use graph db.

Will update this thread with progress.

Support for package metadata

opened on 2021-02-16 09:31:59 by prabhu

While working on a new risk scoring feature for dep-scan, I realized the need for a database for package metadata to prevent querying npm and pypi datasources for each invocation.

This perhaps will be a separate file with its own index to prevent the vulnerability database from becoming large. Separate flag will be added to perform package metadata fetching.

Releases

Release v5.0.1 2023-03-01 07:33:15

Release v5.0.0 2023-03-01 06:36:28

Release v4.3.1 2023-01-22 00:04:00

Release v4.3.0 2022-12-18 23:27:39

Release v4.2.1 2022-12-18 10:06:38

Release v4.2.0 2022-12-17 23:41:44

vulnerability-detection cve vulnerability-database nvd cli sca