[v2.20.0] Lockfile generation with multiple indexes fails due to mismatching hashes

[seifertm]

[v2.20.0] Lockfile generation with multiple indexes fails due to mismatching hashes

[cburroughs]

I'm a little confused on two fronts here:

First you have interpreter_constraints = ["==3.11.*"] but the quoted hash mismatches look like Python 3.9?

Second, when I run this example I also get hash mismatches with Pants 2.16 through 2.19.

[seifertm]

I can reproduce the same hash mismatch with the following PEX call (with Pex v2.3.1, line breaks added for clarity):

$ PEX_VERBOSE=3 pex3 lock create --style universal --pypi --index https://download.pytorch.org/whl/cpu --interpreter-constraint "==3.11.*" -o lock.json --indent 2 --pip-version 24.0 "torch==2.2.1,!=2.2.1+cpu"

[…]

pex: Resolving interpreters :: Indexing downloads :: Downloading FileArtifact(
  url=ArtifactURL(
    raw_url='https://files.pythonhosted.org/packages/d7/23/e4d34a78adc26ea0415fbe78fb2f940bac2c0b637db34f4fa0538a57f832/torch-2.2.1-cp39-none-macosx_11_0_arm64.whl',
    download_url='https://files.pythonhosted.org/packages/d7/23/e4d34a78adc26ea0415fbe78fb2f940bac2c0b637db34f4fa0538a57f832/torch-2.2.1-cp39-none-macosx_11_0_arm64.whl',
    normalized_url='https://files.pythonhosted.org/packages/d7/23/e4d34a78adc26ea0415fbe78fb2f940bac2c0b637db34f4fa0538a57f832/torch-2.2.1-cp39-none-macosx_11_0_arm64.whl',
    scheme='https',
    path='/packages/d7/23/e4d34a78adc26ea0415fbe78fb2f940bac2c0b637db34f4fa0538a57f832/torch-2.2.1-cp39-none-macosx_11_0_arm64.whl',
    fragment_parameters={},
    fingerprints=()
  ),
  fingerprint=Fingerprint(
    algorithm='sha256',
    hash='26bd2272ec46fc62dcf7d24b2fb284d44fcb7be9d529ebf336b9860350d674ed'
  ),
  verified=False,
Expected sha256 hash of 26bd2272ec46fc62dcf7d24b2fb284d44fcb7be9d529ebf336b9860350d674ed when downloading torch but hashed to b90669b162984e302fbd05c9c270ef796e467903944ecefa7457babe9611607e.

When I leave out both the --pip-version and --interpreter-constraint options from the above command, the lockfile is generated successfully.

[seifertm]

I tried the different pip versions listed in Pex's --pip-version option. The lockfile creation was unsuccessful with all of them.

[seifertm]

The reproducer has just one resolve, but two indexes. In reality, I work with multiple resolves for PyTorch, one that includes the CPU only version, and another one that includes the CUDA version.

In case someone else runs into the same issue: I work around the described problem by generating the lockfile for the CPU resolve, disabling the second package index in pants.toml, generating the CUDA resolve, and resetting pants.toml to its original state.