[24.1] self signed certificates are no longer loaded

[rasmus91]

[24.1] self signed certificates are no longer loaded

[notatallshaw]

I've always wanted to know how pip's vendoring process worked, so I've created two branches to test if requests is the issue:

Branch with latest version of requests: pip install git+https://github.com/notatallshaw/[email protected]
Branch with version of requests pip 24.0 had: pip install git+https://github.com/notatallshaw/[email protected]

Can someone please try their workflow that works on 24.0 but breaks on 24.1 on both of these branches and report the results for both please (may need to use --no-cache-dir and --force to make sure there's a network connetion). It would also be apprecative if you could give exact steps of your workflow.

[sweetlilmre]

As per my tests, yes.
When you have the 2.31.0 branch sorted I will test

[sweetlilmre]

@pfmoore first off apologies, this was in no way intended as a slight toward the pip or request maintainers.
In fact the handling of this issue has been top notch with quick responses and the option of neophytes like myself to offer some value from clear directions and things to test.

The debacle I was referring to is the general certificate handling case in the python ecosystem. truststore is a massive leap forward in terms of an improvement, similar to the intent behind pip-system-certs etc.

Unfortunately I live in a corporate environment with SSL inspection. This causes havoc all the time with new starts and even folks that have come back to python after a while, or had to reinstall for whatever reason. REQUESTS_CA_BUNDLE is always quoted as the solution but works in only some cases (seems to work for python and pip, but with certifi we inevitably end up having to patch the CA bundle, pip-system-certs used to work but has stopped recently).

truststore or something like that, baked in would be a huge win for python as a whole and debacle was the word that (perhaps a little unfortunately) I chose to express my frustration with the state of play.

Again apologies, and thank you to you, and all involved in this work.

[sweetlilmre]

I can confirm similar behaviour on windows.
For 24.1 pip fails with certificate errors for a self-signed cert and only works by pointing REQUESTS_CA_BUNDLE to a bundle containing the self-signed CA.
24.0 requires --use-feature truststore but works as expected. I tested all non-beta versions from 23 onwards.

[ichard26]

Hey all, apologies for the disruption. Pip by default does not use the system certificate store. It is an optional feature enabled by the integration of truststore which did get bumped in 24.1. cc @sethmlarson

[sethmlarson]

@rasmus91 @sweetlilmre Just to confirm, you're running into this issue without using the optional "Truststore" feature or while you're using it (ie --with-feature=truststore)? Have to clarify because it's not clear from your posted reproduction steps.

[rasmus91]

Not using that option, no. fre. 21. jun. 2024 16.06 skrev Seth Michael Larson ***@***.***

…

: @sweetlilmre <https://github.com/sweetlilmre> Just to confirm, you're running into this issue *without* using the optional "Truststore" feature or while you're using it (ie --with-feature=truststore)? Have to clarify because it's not clear from your posted reproduction steps. — Reply to this email directly, view it on GitHub <#12779 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARQ7C5RHVTYOACO42J24XDZIQXNXAVCNFSM6AAAAABJVQC7RSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBSHAZDGNRWHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[sweetlilmre]

@sethmlarson, while using it:

• versions 23.0 to 24.0 using --with-feature=truststore work (these are the versions I tested).
• version 24.1 using --with-feature=truststore does not work and I need to have REQUESTS_CA_BUNDLE defined and pointing to a bundle containing the required CA.

Sorry for the ambiguity.
How it ever worked for @rasmus91 without the feature enabled surprises me.

[rasmus91]

Yes, it did until 24.1. i dont understand why it wouldn't when the CA cert is installed system wide?

[sethmlarson]

So I'm actually suspecting that a defect in Requests is the source of the issues here, pip uses Requests 2.32.0 but there have been 3 bugfix releases since that release to Requests with some relating to certificates.

[ichard26]

Looking at the requests changelog, I strongly suspect that psf/requests#6655 is implicated. psf/requests#6716 which landed in requests v2.32.3 restores the ability to specify custom SSLContexts in sub-classes of HTTPAdapter which pip does have.

pip/src/pip/_internal/network/session.py

Line 290 in 2753c77

class HTTPAdapter(_SSLContextAdapterMixin, _BaseHTTPAdapter):

Please try @notatallshaw's branches to help us determine whether requests is the root problem and upgrading requests fixes the bug. Thanks!

[sweetlilmre]

Let me preface this with: I am no python / pip expert, in fact I probably don't know what I'm doing.
That said I ran the following tests:

1. Create 2 directories, 2.32.3 and 2.31.0
2. Create a venv in both
3. Install the appropriate branch of pip in each
4. Test

Results:

2.32.3:

pip --version
pip 24.0 from C:\source\piptest\2.32.3\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable
Install:
python.exe -m pip install git+https://github.com/notatallshaw/[email protected]

pip --version
pip 24.2.dev0 from C:\source\piptest\2.32.3\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check:
- pip install pip-system-certs FAILS (SSL cert errors)
- pip --use-feature truststore install pip-system-certs SUCCEEDS

2.31.0

pip --version
pip 24.0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable
Install:
python.exe -m pip install git+https://github.com/notatallshaw/[email protected]

pip --version
pip 24.2.dev0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check:
pip install pip-system-certs FAILS (traceback)

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\source\piptest\2.31.0\.venv\Scripts\pip.exe\__main__.py", line 7, in <module>
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\cli\main.py", line 78, in main
    command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\commands\__init__.py", line 114, in create_command
    module = importlib.import_module(module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\importlib\__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\commands\install.py", line 15, in <module>
    from pip._internal.cli.req_command import (
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\cli\req_command.py", line 18, in <module>
    from pip._internal.index.collector import LinkCollector
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\index\collector.py", line 31, in <module>
    from pip._vendor import requests
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\exceptions.py", line 9, in <module>
    from .compat import JSONDecodeError as CompatJSONDecodeError
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\compat.py", line 10, in <module>
    import chardet
ModuleNotFoundError: No module named 'chardet'

[notatallshaw]

Thanks for running the tests, I'll fix the issue with the 2.31.0 branch later today and confirm.

Am I correct in understanding though that the 2.32.3 branch now succeeds where pip 24.1 fails?

[notatallshaw]

I fixed the 2.31.0 branch if you want to reinstall it and try, though it's a little less important seeing that 2.32.3 works, it at least prooves definitively it was a requests regression and unrelated to truststore.

[sweetlilmre]

I reran the 2.31.0 branch for completeness sake.

2.31.0

pip --version
pip 24.0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable
Install:
python.exe -m pip install git+https://github.com/notatallshaw/[email protected]

pip --version
pip 24.2.dev0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check:
pip install pip-system-certs FAILS (SSL)

pip --use-feature truststore install pip-system-certs SUCCEEDS

[sweetlilmre]

As a final comment on this issue, --use-feature truststore as a default and not a feature would be great. Doesn't fix the wider certifi / requests certificate handling debacle, but certainly helps for bootstrapping python, especially for new users.

[pfmoore]

Doesn't fix the wider certifi / requests certificate handling debacle

While I'm sure you didn't mean to offend, "debacle" is a bit strong here. We published a beta with the new requests release, and didn't get any feedback on this issue. The pip maintainers themselves don't have access to environments that need this feature, and so we weren't able to trigger the issue ourselves. If you have any suggestions (or better still a PR!) for how we could test this within our test suite then that would be appreciated - although it's arguable that this is something that should be covered in the requests test suite, so maybe proposing something at their end would also be useful.

We try to avoid regressions like this, but there's a lot of pip functionality that's only used by particular parts of our user base, and often those users don't (for perfectly good reasons) get involved in testing and proving new releases for us. Negative feedback tends to reduce our motivation for taking risks in those areas, which in turn slows the pace of developments and improvements for those users. It's difficult to find a good balance.

[notatallshaw]

FWIW, back in my large corporation days I had to teach people to use px-proxy: https://github.com/genotrance/px?tab=readme-ov-file#installation

Which I found as the most consistent workaround to Python http ecosystem vs. enterprise security. Although I didn't get to try truststore, I'm now at a relatively small company where I don't face super esoteric security protocols