jordan-wright/ossmalware

ossmalware's Issues

Move to Google Cloud

Right now, this system runs on AWS which was fine for the initial experiment. As this project moves under the umbrella of the OpenSSF, we'll want to move it to GCP to better provide funding and support.

This involves both updating the service components to support GCP (ref #8), as well as updating our Terraform configs.

Add Worker Bootstrapping to Terraform

Right now EC2 instances are manually provisioned instead of using Terraform. This is from my inexperience with Terraform, but I'd like to take the time to change this.

I'm also ok if we hold off on this until we move to a workload based solution.

Make Processing More Resilient

For some reason, hosts would occasionally have defunct sysdig processes laying around. And more generally I saw the occasional package processing weirdness result in a major stalling of the workers.

If we want this to be a long term solution, we should try to clean up the processing workflow as much as possible to fail gracefully.

Move from EC2 to a container-based workload

Right now we need to maintain EC2 worker instances for processing which isn't ideal since weird errors that occur during processing can propagate to cause the entire host to stall.

Ideally, we would move to a workload based solution like Fargate so that each package is installed in a totally isolated environment. Most seem to support SYS_PTRACE which is required for sysdig to work. It'll just be a matter of figuring out how to make it work.

License?

Hey,

Any chance you could add a license to this repo? Many employers (including mine) only allow us to contribute to repos with an allowed license.

Automatically Bootstrap Workers

While we're still using persistent hosts (EC2 instances) as workers instead of something like Fargate, we should make the setup of workers easy.

Right now, users have to manually run setup.sh and start.sh. Ideally we could bootstrap both of these from the EC2 user data so new hosts are automatically setup and start running.

[Python] Create lambda for polling Python RSS feed.

To handle continuous monitoring for PyPI, we should create a lambda that runs on an interval to request PyPI's RSS feed and process new packages by pushing them up to the queue (currently SQS).

how to run the project

how to run the project, are there more specific steps we can follow, thanks~

Make Terraform Configurable

Right now the S3 and SQS names are hard coded. We should make these proper variables.

I'm not a terraform expert so I'm not sure what else we can do to make this more robust, so let me know if you have ideas!