python.invalid_lockfile_behavior = "ignore" or "warn" does not ignore sha256 mismatches for stale dependencies

[sriramv2]

python.invalid_lockfile_behavior = "ignore" or "warn" does not ignore sha256 mismatches for stale dependencies

[AlexTereshenkov]

Sorry to hear that! Would you please be able to share a minimal repro, either as a GitHub repo or at least source code for a couple of files that would be able to demonstrate the problem (pants.toml, a couple of BUILD files, and a lockfile with some dependency and the steps to modify it imitating the "volatility" you refer to). This would help addressing the issue a lot.

I am also wondering what do you mean by "volatility" of a dependency? I appreciate every built of a Python native extension may result in a different (bit-by-bit) .so file, but I am not sure if this is what you mean and how this dependency becomes part of the lockfile? The reason I ask is because the PyPI packages will stay the same between the builds (as you pull them). Sorry if I am missing some extra context.

Many thanks.

[sriramv2]

Hello Alex, thanks for getting back. Here is some more information:

By volatility, I mean a PyPI dependency that changes multiple times a day without any change in version. The dependency whl has .so files, some enum definitions, protobuf generated code, Python source code. One of the .so files has a timestamp field that is generated at compile time, and this field causes the whl to always generate a different hash. In development branches, this strategy of overwriting versions is followed by the providers of the whl. We are the consumers, and the lockfile keeps complaining due to the changing hash. That's the reason for wanting to set invalid_lockfile_behavior = "warn" or even "ignore" potentially.

To reproduce this the easy way, manually change the computed hash in the lockfile, and run a goal on a target that resolves to this lockfile.

In pants.toml:
[python]
enable_resolves = true
invalid_lockfile_behavior = "warn"

[python.resolves]
default = "devops/pantsbuild/lockfiles/python_default.lock"

In the BUILD file of the target to run:
defaults(all=dict(resolve="default"))

python_sources(
name="lib.target",
sources=[
"*.py",
"test/**",
],
dependencies=[
":pyproject",
":setup",
],
)

Run:
pants generate-lockfiles ::

Then run:
pants run target.py

Manually change the hash of the dependency.

Then run
pants run target.py

Expected behavior is that this would warn that the dependency is stale, or that there is a hash mismatch but still run the target.

Actual behavior is that the command errors out with a sha256 hash mismatch.

Do let me know if you need any more information. Happy to clarify further.

[jsirois]

@sriramv2 your understanding of https://www.pantsbuild.org/2.21/reference/subsystems/python#invalid_lockfile_behavior is incorrect. That option tells Pants what to do when your lock no longer matches its input requirements: i.e.: when you update requirements.txt (say) and forget to re-generate the lock. In your case the error comes from Pex and it is very deliberate: your input requirements have not changed, but the artifacts have! This is, in general, alarming and the very definition of an attack in most cases. The whole point of a (Pex) lock is a byte-wise reproducible dependency set. Your arrangement violates this. Perhaps you can get the wheel producer to stop doing what it's currently doing, which is horrible. The producer should either be attaching a local version (https://packaging.python.org/en/latest/specifications/version-specifiers/#local-version-segments) if it can't manage a reproducible build or it should be using https://reproducible-builds.org/docs/source-date-epoch/ to fix timestamps. I can pre-emptively say Pex has no motivation to change it's strictness here. Without it, the lock is not, in fact a lock. It's a lock "except for ...".

[sriramv2]

Thank you for explaining what the ignore flag is supposed to ignore- that makes sense. That makes this ticket invalid.

The behavior of the culprit whl indeed violates this agreement. I'll check out local versions. Thanks a bunch!