cisagov/hash-http-content

Python package to generate a hash from the content of an HTTP URL.

hash-http-content

This is a Python library to retrieve the contents of a given URL via HTTP (or HTTPS) and hash the processed contents.

Content processing

If an encoding is detected, this package will convert content into the UTF-8 encoding before proceeding.

Additional content processing is currently implemented for the following types of content:

• HTML
• JSON

HTML

HTML content is processed by leveraging the pyppeteer package to execute any JavaScript on a retrieved page. The result is then parsed by Beautiful Soup to reduce the content to the human visible portions of a page.

JSON

JSON content is processed by using the json library that is part of the Python standard library. It is read in and then output in a deterministic manner to adjust for any styling differences between content.

Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

License

This project is in the worldwide public domain.

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

hash-http-content's Issues

Explore switching from `pyppeteer` to `playwright-python`

💡 Summary

We should consider switching the browser interaction package we use from pyppeteer to playwright-python.

Motivation and context

According to the pyppeteer README:

Attention: This repo is unmaintained and has been outside of minor changes for a long time. Please consider playwright-python as an alternative.

Reliance on unmaintained packages can result in undesirable situations such as breaking behavior that is unlikely to be fixed and feature stagnation.

Implementation notes

As this is an entirely different tool functionality will have to be verified to ensure a similar experience.

Acceptance criteria

How do we know when this work is done?

• Switching packages is found to be viable.
• We switch to using [python-playwright].
• We update cisagov/vdp-scanner-docker to use the new release.

Update mypy Workaround if It Is No Longer Needed

💡 Summary

We currently work around a perceived issue with type hinting with mypy in:

hash-http-content/src/hash_http_content/hasher.py

Lines 33 to 44 in 36d4e52

	# mypy relies on typeshed (https://github.com/python/typeshed) for
	# stdlib type hinting, but it does not have the correct type hints for
	# hashlib.new(). The PR I submitted to fix them
	# (https://github.com/python/typeshed/pull/4973) was approved, but I
	# am not sure if mypy will still have issues with the usage of this
	# keyword in non Python 3.9 (when the usedforsecurity kwarg was added)
	# environments. I believe the earliest I can test this will be in mypy
	# v0.900, and I have made
	# https://github.com/cisagov/hash-http-content/issues/3 to document
	# the status of this workaround.
	# hasher = hashlib.new(hash_algorithm, usedforsecurity=False)
	hasher = getattr(hashlib, "new")(hash_algorithm, usedforsecurity=False)

After a new version of mypy with type hint updates (possibly v0.900) is released, we should see if this workaround is still necessary to pass linting.

Motivation and context

Workarounds should only be used as long as there is something to work around.

Implementation notes

The preferred usage is commented out, so testing is simple as switching between the current and preferred usage.

Acceptance criteria

• mypy hook passes with the preferred usage in place
• Code is updated

or

• mypy hook still fails with the preferred usage in place
• Comment is updated to reflect the necessity of the workaround