homebrew/brew-pip-audit

:clipboard: Bulk auditing Python dependencies in Homebrew with pip-audit

brew-pip-audit: Bulk auditing Python dependencies in Homebrew with osv-scanner

Homebrew is a popular package manager for macOS. Many of the projects it packages are written in Python. In order to ensure reproducible builds, Homebrew precisely pins the version of each Python package a Homebrew formula depends on.

osv-scanner is a tool for checking a project's dependencies against vulnerability databases in order to determine if there are any known vulnerabilities.

This project takes all of the Python packages depended on by Homebrew formulas and runs them through osv-scanner. It then takes those audit results and uses them to submit patches to Homebrew.

This project previously used pip-audit, instead of osv-scanner, hence the name.

The repo

The following things can be found in this repository:

• formula2requirements.rb: Extracts the Python dependencies from Homebrew and writes them out in the requirements.txt format.
• pip-audit-bulk: Runs osv-scanner over a directory of requirements.txt files.
• generate-prs.rb: Automatically generates PRs against Homebrew/homebrew-core for formulae with vulnerable dependencies.
• requirements/: The extracted requirements.txt file for each Homebrew formula.
• audits/: The result of osv-scanner for each Homebrew formula. There will only be a file present if vulnerabilities were found.

requirements/ and audits/ are automatically refreshed on a daily basis by Github Actions.

Contributing

This repository is automated, but the automation isn't perfect. You can help out by:

• Looking at the skipped file, and trying to figure out why a particular dependency's audit was skipped.
• Looking at the incoming PRs against Homebrew/homebrew-core, and helping debug ones that fail.
• Improving the performance of our automation (it's currently very slow).
• Looking at the action logs for the PR automation, and helping debug/fix formulae and dependencies that can't be auto-updated.

brew-pip-audit's Issues

If a dependency doesn't have any vulnerabilities, remove it from the results

Right now we include the entire dependency tree in each formula's pip-audit JSON summary, which makes it hard to visually scan for vulnerabilities. Instead, we should reduce the JSON output down to just those dependencies that actually have vulnerabilities.

Consider switching back to `pip-audit`

We switched to osv-scanner because of a significant performance regression in pip-audit. That regression has now been fixed with the 2.6.0 release, so we should consider switching back.

Low priority since the performance of osv-scanner is probably good enough, and the data quality should be similar.

Auto-label our PRs to homebrew-core

We should auto-label our PRs as python, pip-audit, and maybe a few other things.

Actions job can't push to main due to protected branch

https://github.com/Homebrew/brew-pip-audit/actions/runs/3581386277/jobs/6024385118

Remove existing audit results if all vulnerabilities fixed

It used to be the case that a previous .audit.json file was removed if all vulnerabilities were fixed. That seems to have gotten lost in the re-write. It should be restored.

PR automation: follow-ups

Some things we should do:

• Bump the revision as well, so that auto-bumped formulae are automatically pushed to users who brew upgrade (#44);
• Issue PRs in alphabetical order, so that we have a rough idea of how much progress the automation has made;
• Look into throttling the PR automation when triggered periodically to avoid spamming the homebrew-core maintainers.
• Look into replacing pipgrip within Homebrew for dep upgrades, since it uses Poetry's resolver and is slow/does different resolutions than pip
  • Homebrew/brew#15307
• Don't send a PR if it doesn't actually fix a vulnerability. This happens when version constraints don't allow upgrading the vulnerable dep, but there are other packages to be updated (example: Homebrew/homebrew-core#122902).
• Include the list of fixed vulnerabilities in the PR message.

Debug all the formula for which `update_python_resources!` fails

• alot
• ansible
• bbot
• buku
• certbot
• certsync
• charm-tools
• libplacebo
• litani
• magic-wormhole
• mavsdk
• mvt
• onlykey-agent
• openai-whisper
• pdfalyzer
• pocsuite3
• psutils
• scoutsuite
• sickchill
• slither-analyzer
• snapcraft
• terminator
• tern
• textract
• torchvision
• trezor-agent

Figure out a way to update only the vulnerable deps

We currently bump all resources just to get at a single vulnerable dependency, which (1) produces large diffs and (2) introduces risks of breakage, both in CI and in built bottles.

We should really only bump the vulnerable dep. Maybe we can do that by using constraints files?

Burn down skips in generate-prs.rb

https://github.com/Homebrew/brew-pip-audit/blob/main/generate-prs.rb#L12

Some of these are due to packaging issues in the upstreams, and some are due to our own limitations.

Failing to generate any PRs

https://github.com/Homebrew/brew-pip-audit/actions/runs/7985826380

@woodruffw at one point we talked about adding more debug info when this happens

Attempt to auto-send PRs to brew

Ideally for any formula that we identify vulnerabilities in we'd:

• Bump dependencies (brew update-python-resources)
• Verify that this fixes vulnerabilities
• Bump revision
• Send a PR to homebrew

This would significantly reduce the burdens of keeping the ecosystem secure