curium-rocks/mitre-siphon

Syncs the mitre CVE JSON dumps to a PostgreSQL DB using Quartz and Kafka and exposes a rest API for running full text searches against the DB.

mitre-siphon's Issues

Include documentation generation in CI

use the output of the gradle javadoc task to push the html pages to something like readthedocs to include documentation as part of the CI chain.

Add ingress/load balancer

Evaluate options for simple load balancer/ingress that can bind to 80 and 443 and round robin to multiple mitre-siphon containers. Traefik is likely a good option, once selected update docker-compose files and helm templates if helm is present at time of merge.

Add CI

Projects needs CI definition that tests, builds and distributes the application to docker hub

Add Readme

Project needs a readme that instructs on how to build and run the application

Helm chart

Add a helm chart for deploying into a kubernetes cluster.

Should use sub charts for kafka and postgresql.

kafka and postgresql should use PVCs, may need to include a host path storage provisioner.

Container Sizing

The current resource allocations do not always provide enough resources to enter a ready state within the startup probe time, either increase resources or probe thresholds.

Further test full-text searching

tsvector PostgreSQL functionality is used for search matches, further evaluation should be done around

• Fuzzy matching from tri-grams
• Sorting

CVE Search 500 error

When searching with a search term that has multiple words (in a single term) the API returns a 500. Either handle this or reject with 400.

Setup DB Schema and Initial Migration

Setup DB Schema. Should handle the following

• Use JPA and Hibernate DAOs
• Have separate DTOs from DAOs used on REST API
• Handle seeding initial database state

Add ArgoCD project for a continously running test env

As a developer I want a environment that gets updated when the main branch is updated and is continuously running to shake out any issues that don't show up in smoke tests. I do not want to allow a direct connection from a GitHub runner to this environment.

Replace com.github.eirnym.js2p with new implementation

For a reason my plugin has changed the id to jsonschema2dataclass and has way better gradle support. Could you please apply new plugin?

Setup authentication on API

Configure authentication for API. Leverage something simple to start with such as http basic or digest.

Credentials should be configurable.

Add Coverage Badge

Add unit test code coverage badge

🧚🤖 Pixeebot Activity Dashboard

👋 This dashboard summarizes my activity on the repository, including available improvement opportunities.

Recommendations

Last analysis: May 07 | Next scheduled analysis: May 14

Open

✅ Nice work, you're all caught up!

Available

✅ Nothing yet, but I'm continuing to monitor your PRs.

Metrics

What would you like to see here? Let us know!

Resources

📚 Quick links
Pixee Docs | Codemodder by Pixee

🧰 Tools I work with
Sonar, CodeQL, Semgrep

🚀 Pixee CLI
The power of my codemods in your local development environment. Learn more

💬 Reach out
Feedback | Support

---

❤️ Follow, share, and engage with Pixee: GitHub | LinkedIn | Slack

Randomly generate secret for DB

Randomly generate any secrets needed for service connections at install time instead of using secrets in values.

Handle deprecation of mitre json CVE apis.

Mitre is deprecating the JSON apis.

https://medium.com/@cve_program/legacy-cve-download-formats-will-be-phased-out-beginning-january-1-2024-13de552c9029

Evaluate options.

Helm Instructions

The top level readme does not currently include instructions on how to install via helm, add and cleanup any dated info.

Remove usage of env in PR

Remove CI GitHub env object.

Set active spring profile automatically in CI

Tests currently rely on postgres hostname, this should be adjusted to automatically use spring active profiles to swap out properties when running in CI environment and change the default profile hostname for postgres back to localhost

Setup Scaffolding

Setup scaffolding for application

This includes:

• Gradle build
• Docker build (ability to spit out container and run it and get a primitive interface)
• UI Build
• docker-compose files for spinning up dependencies
• Test runners

Needs to occur before attaching CI actions

Cleanup code smells

As a developer I would like to clean up smells identified by sonar and keep a clean score card: https://sonarcloud.io/project/issues?resolved=false&types=CODE_SMELL&id=curium-rocks_mitre-siphon

NVD Job Exception

After getting things cleaned up and updated this exception is now being thrown in NVD job.

2022-12-11T21:16:59.514Z ERROR 1 --- [eduler_Worker-1] org.quartz.core.ErrorLogger              : Job (DEFAULT.NVDCVEComplete threw an exception.

org.quartz.SchedulerException: Job threw an unhandled exception.
	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) ~[quartz-2.3.2.jar!/:na]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ~[quartz-2.3.2.jar!/:na]
Caused by: java.lang.NullPointerException: Cannot invoke "javax.persistence.EntityManager.unwrap(java.lang.Class)" because "this.entityManager" is null

Push container builds to docker hub

Add automatic push of container build to docker hub. Use git hub actions to accomplish this.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

docker-compose

docker-compose.service-deps.yml

• confluentinc/cp-kafka 7.6.1
• confluentinc/cp-zookeeper 7.6.1

docker-compose.test.yml

• confluentinc/cp-kafka 7.6.1
• confluentinc/cp-zookeeper 7.6.1

docker-compose.yml

• confluentinc/cp-kafka 7.6.1
• confluentinc/cp-zookeeper 7.6.1

dockerfile

Dockerfile

test.Dockerfile

github-actions

.github/workflows/ci.yml

• actions/checkout v4
• actions/setup-java v4
• actions/setup-node v4
• gradle/gradle-build-action v3
• actions/upload-artifact v4
• codecov/codecov-action v4
• confluentinc/cp-kafka 7.6.1
• confluentinc/cp-zookeeper 7.6.1

.github/workflows/publish.yml

gradle

settings.gradle

build.gradle

• org.springframework.boot 3.2.5
• io.spring.dependency-management 1.1.5
• com.palantir.git-version 3.0.0
• org.flywaydb.flyway 10.13.0
• com.github.eirnym.js2p 1.0
• com.diffplug.spotless 6.25.0
• org.sonarqube 5.0.0.4638
• org.springdoc:springdoc-openapi 2.5.0
• org.springdoc:springdoc-openapi-starter-webmvc-ui 2.5.0
• org.mitre.taxii:taxii 1.1.0.1
• org.mitre:stix 1.2.0.2
• javax.persistence:javax.persistence-api 2.2
• jakarta.validation:jakarta.validation-api 3.1.0
• com.squareup.okhttp3:okhttp 4.12.0
• com.squareup.okhttp3:mockwebserver 4.12.0
• org.awaitility:awaitility 4.2.1

gradle-wrapper

gradle/wrapper/gradle-wrapper.properties

• gradle 8.7

helm-values

helm/mitre-siphon/values.yaml

helmv3

helm/mitre-siphon/Chart.yaml

• postgresql 15.4.0
• kafka 28.3.0

---

• Check this box to trigger a request for Renovate to run again on this repository

Move gradle dependencies out to versions

Currently leveraging latest version in a few places instead of pinning versions. Ideally should move versions out to a variable and define once.

Add Mitre CVE Scraping

Add background task that executes at set intervals and size regardless of replica count and scrapes mitre cve information.

Background task should:

• be stateless to execute on any active container.
• push capture data into kafka to allow for resilient and distributed processing

Version api path

Api does not have a versioning mechanism, IE api/v1/ api/v2.

Add mitre CVE data archiver

Add an actor on the kafka mitre CVE data topic that normalizes the mitre CVE data and persists it into the service data store.

It should be:

• stateless except that it activates a kafka subscription and saves data into the data store
• operate in a high level consumer group fashion to distribute the load across container instances.

404 on Swagger UI

After getting everything cleaned up/updated the swagger UI page is throwing a 404 error when bringing up /swagger-ui.html when running in kubernetes.

Add simple UI

Create a simple UI that displays the data available on the REST api

Expose mitre cve data over a REST API

Expose mitre cvedata over a REST API

API should be:

• protected via authentication
• follow REST principals