System site-packages still not reused upon install

[sanmai-NL]

System site-packages still not reused upon install

[frostming]

This was resolved by the introduction of --update-reuse-installed strategy.

[sanmai-NL]

Additionally, locking doesn't work.

+ pdm lock --group :all --refresh
STATUS: Re-calculating hashes...
Changes are written to pdm.lock.
+ pdm install --fail-fast --no-editable --no-isolation --production --verbose
STATUS: Resolving packages from lockfile...
Traceback (most recent call last):
  File "/home/resources/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 290, in main
    return Core().main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 208, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 203, in main
    self.handle(project, options)
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 157, in handle
    command.handle(project, options)
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/commands/install.py", line 100, in handle
    actions.do_sync(
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 198, in do_sync
    candidates = resolve_candidates_from_lockfile(project, requirements)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 141, in resolve_candidates_from_lockfile
    mapping, *_ = resolve(
                  ^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/resolver/core.py", line 36, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/resolvelib/resolvers.py", line 399, in resolve
    raise ResolutionImpossible(e.criterion.information)
resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=NamedRequirement(name='cryptography', marker=None, extras=set(), specifier=<SpecifierSet('>=41.0.3')>, editable=False, prerelease=False), parent=None)]

Yet if I let pdm install ... do the locking, then a subsequent pdm install --dev ... can't find the dev group.

[sanmai-NL]

As for a solution requirement: I would want the lockfile to only cover dependencies that aren't sourced from the system site-packages. The system site-packages are effectively locked through the OS package manager, whereas the other dependencies aren't if I forego locking during build.

[sanmai-NL]

Another interesting case that can't be worked around by using --no-lockfile:

...
ruamel-yaml-clib 0.2.7 [ Not required ]
...
pdm.termui: ======== Starting round 61 ========
pdm.termui:   Adding requirement ruamel-yaml-clib>=0.2.7; platform_python_implementation == "CPython" and python_version < "3.13"(from ruamel-yaml 0.17.34)
STATUS: Resolving: new pin ruamel-yaml 0.17.34
pdm.termui: Pinning: ruamel-yaml 0.17.34
...
 ✖ Install ruamel-yaml-clib 0.2.8 failed
pdm.termui: Error occurs: 
Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/datapipeline_ff/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 283, in install_candidate
    self.manager.install(can)
  File "/home/datapipeline_ff/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/installers/manager.py", line 33, in install
    installer(str(prepared.build()), self.environment, prepared.direct_url())
                  ^^^^^^^^^^^^^^^^
  File "/home/datapipeline_ff/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 381, in build
    self.obtain(allow_all=False)
  File "/home/datapipeline_ff/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 422, in obtain
    raise CandidateNotFound(
pdm.exceptions.CandidateNotFound: No candidate is found for `ruamel-yaml-clib` that matches the environment or hashes
...

It seems platform_python_implementation somehow restricts package resolution to exclude system site-packages.

[sanmai-NL]

I don't know if you understand the trickery in https://sourceforge.net/p/ruamel-yaml/code/ci/default/tree/setup.py, @frostming ... I'm out of my depth.

[frostming]

The dependencies will always be resolved whether they are included by system-site-package or not. Even when writing lockfile is not required(--no-lock). This step is mandatory to make sure the dependencies are compatible with each other. The system site packages Can only skip the installation of packages if its version matches exactly with the one in the lockfile.

[sanmai-NL]

For context, does Poetry have the same behavior?

[sanmai-NL]

@frostming I fail to wrap my head around how to reuse system site-packages with PDM. If I remove/disregard the lockfile and my dependency has a dependency with a lower bound version range, then the latest version will be collected from the package index, whereas my system site-package version is lower but still compatible with the lower bound version range. How to avoid this issue?