exein-io/kepler

NIST-based CVE lookup store and API powered by Rust.

kepler's Issues

add kepler support for jar files

Casual import error

Sometimes I get the following errors with import_nist commands:

[2023-07-10T20:30:44Z INFO  domain_db::cve_sources] downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2009.json.gz to /data/nvdcve-1.1-2009.json.gz ...
[2023-07-10T20:30:44Z INFO  domain_db::cve_sources::nist] extracting /data/nvdcve-1.1-2009.json.gz to /data/nvdcve-1.1-2009.json ...
Error: could not extract /data/nvdcve-1.1-2009.json.gz

Caused by:
    invalid gzip header
[2023-07-10T20:30:44Z INFO  domain_db::cve_sources] downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2010.json.gz to /data/nvdcve-1.1-2010.json.gz ...
[2023-07-10T20:30:45Z INFO  domain_db::cve_sources::nist] extracting /data/nvdcve-1.1-2010.json.gz to /data/nvdcve-1.1-2010.json ...
Error: could not extract /data/nvdcve-1.1-2010.json.gz

Caused by:
    invalid gzip header
[2023-07-10T20:30:45Z INFO  domain_db::cve_sources] downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2011.json.gz to /data/nvdcve-1.1-2011.json.gz ...
[2023-07-10T20:30:45Z INFO  domain_db::cve_sources::nist] extracting /data/nvdcve-1.1-2011.json.gz to /data/nvdcve-1.1-2011.json ...
Error: could not extract /data/nvdcve-1.1-2011.json.gz

Caused by:
    invalid gzip header

If I try to launch the import again it finds the previous downloaded files and output this instead:

[2023-07-10T20:43:43Z INFO  domain_db::cve_sources::nist] found /data/nvdcve-1.1-2020.json
[2023-07-10T20:43:43Z INFO  domain_db::cve_sources::nist] reading /data/nvdcve-1.1-2020.json ...
Error: failed to parse cve file from /data/nvdcve-1.1-2020.json

Caused by:
    EOF while parsing a value at line 1 column 0
[2023-07-10T20:43:43Z INFO  domain_db::cve_sources::nist] found /data/nvdcve-1.1-2021.json
[2023-07-10T20:43:43Z INFO  domain_db::cve_sources::nist] reading /data/nvdcve-1.1-2021.json ...
Error: failed to parse cve file from /data/nvdcve-1.1-2021.json

Caused by:
    EOF while parsing a value at line 1 column 0
[2023-07-10T20:43:43Z INFO  domain_db::cve_sources::nist] found /data/nvdcve-1.1-2022.json
[2023-07-10T20:43:43Z INFO  domain_db::cve_sources::nist] reading /data/nvdcve-1.1-2022.json ...
Error: failed to parse cve file from /data/nvdcve-1.1-2022.json

Caused by:
    EOF while parsing a value at line 1 column 0

The workaround is to delete each "corrupted" data file and start the relative year import again

Refactoring the updater to remove ofelia dependency

Currently running Kepler in a cloud environment is quite difficult due to needing two separate services (kepler & ofelia) to share the same filesystem (ofelia calling kepler binary directly). Since the scope of ofelia is fairly small, we should consider removing it entirely and embedding the updater within kepler itself as a single service.

better configuration system

it would be good to have a real configuration system, for example config-rs

Improve data source importing

Currently the NIST importing functionality is too slow, often taking many hours to import the dataset. Taking a look into the codebase it looks like where spawning multiple database transactions in order to import a single entry:

https://github.com/Exein-io/kepler/blob/558afe222b3c21c72a66d26ea1e93695d2c3751c/kepler/src/main.rs#L146-L187

Since a lot of these entries are completely independent of each other we should batch insert them into the database in a single transaction (even packing 1000s of CVEs at a time).

INSERT INTO cves (columns)
VALUES
    (cve_1),
    (cve_2),
    ...
    (cve_n)
RETURNING * 

Which will result in a single BEGIN/COMMIT per chunk rather than multiple per-CVE. The relational properties are still held within the transaction itself.