Comments (4)
Sorry to hear that! Would you please be able to share a minimal repro, either as a GitHub repo or at least source code for a couple of files that would be able to demonstrate the problem (pants.toml
, a couple of BUILD files, and a lockfile with some dependency and the steps to modify it imitating the "volatility" you refer to). This would help addressing the issue a lot.
I am also wondering what do you mean by "volatility" of a dependency? I appreciate every built of a Python native extension may result in a different (bit-by-bit) .so
file, but I am not sure if this is what you mean and how this dependency becomes part of the lockfile? The reason I ask is because the PyPI packages will stay the same between the builds (as you pull them). Sorry if I am missing some extra context.
Many thanks.
from pants.
Hello Alex, thanks for getting back. Here is some more information:
By volatility, I mean a PyPI dependency that changes multiple times a day without any change in version. The dependency whl has .so files, some enum definitions, protobuf generated code, Python source code. One of the .so files has a timestamp field that is generated at compile time, and this field causes the whl to always generate a different hash. In development branches, this strategy of overwriting versions is followed by the providers of the whl. We are the consumers, and the lockfile keeps complaining due to the changing hash. That's the reason for wanting to set invalid_lockfile_behavior = "warn" or even "ignore" potentially.
To reproduce this the easy way, manually change the computed hash in the lockfile, and run a goal on a target that resolves to this lockfile.
In pants.toml:
[python]
enable_resolves = true
invalid_lockfile_behavior = "warn"
[python.resolves]
default = "devops/pantsbuild/lockfiles/python_default.lock"
In the BUILD file of the target to run:
defaults(all=dict(resolve="default"))
python_sources(
name="lib.target",
sources=[
"*.py",
"test/**",
],
dependencies=[
":pyproject",
":setup",
],
)
Run:
pants generate-lockfiles ::
Then run:
pants run target.py
Manually change the hash of the dependency.
Then run
pants run target.py
Expected behavior is that this would warn that the dependency is stale, or that there is a hash mismatch but still run the target.
Actual behavior is that the command errors out with a sha256 hash mismatch.
Do let me know if you need any more information. Happy to clarify further.
from pants.
@sriramv2 your understanding of https://www.pantsbuild.org/2.21/reference/subsystems/python#invalid_lockfile_behavior is incorrect. That option tells Pants what to do when your lock no longer matches its input requirements: i.e.: when you update requirements.txt (say) and forget to re-generate the lock. In your case the error comes from Pex and it is very deliberate: your input requirements have not changed, but the artifacts have! This is, in general, alarming and the very definition of an attack in most cases. The whole point of a (Pex) lock is a byte-wise reproducible dependency set. Your arrangement violates this. Perhaps you can get the wheel producer to stop doing what it's currently doing, which is horrible. The producer should either be attaching a local version (https://packaging.python.org/en/latest/specifications/version-specifiers/#local-version-segments) if it can't manage a reproducible build or it should be using https://reproducible-builds.org/docs/source-date-epoch/ to fix timestamps. I can pre-emptively say Pex has no motivation to change it's strictness here. Without it, the lock is not, in fact a lock. It's a lock "except for ...".
from pants.
Thank you for explaining what the ignore flag is supposed to ignore- that makes sense. That makes this ticket invalid.
The behavior of the culprit whl indeed violates this agreement. I'll check out local versions. Thanks a bunch!
from pants.
Related Issues (20)
- Support exporting downloaded external tools HOT 1
- method to align install_requires of python_distribution with locked depdencies
- scala reproducibility is broken somwhere between 2.14.2 to 2.20.2 HOT 1
- Hard native parser errors cause 2.22 to fail to start, ignores `native_option_validation=warning`
- Rust options parser fails to parse 1 as float, different to legacy parser HOT 2
- Link to slack does not work HOT 4
- Using poetry as a build tool with python_distribution does not work HOT 3
- docker buildx bake support HOT 1
- pants 2.21: XML files for unit tests executed with pytest are no longer generated HOT 1
- poetry_requirement defined by relative path generates a lockfile with absolute path HOT 8
- pants fails to download golang module from private repo due to missing git binary HOT 2
- Better shell completions HOT 2
- Conflict between docformatter --check and --in-place options when using Pants HOT 5
- Cannot upgrade Pants from 2.19 due to `ModuleNotFoundError` for `pex`. HOT 1
- Conflicts between architecture of image present on system when building PEX with `docker_environment`
- Docker containers used by a docker_environment are not cleaned when using `--no-pantsd` HOT 1
- macOS 12 GitHub-hosted runners will stop working from 2024-12-03
- native options parser mismatch with quoted strings in env vars. HOT 3
- Consider using executable_search_path in go backend
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pants.