[4.1.1] ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process

[hswong3i]

[4.1.1] ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process

[alex]

I'm not familiar with how ceph is implemented, is there an obvious reason why it'd be initializing modules multiple times per process? Does it use subinterpreters?

[hswong3i]

From my point of view, case should as similar as pyca/cryptography#9016 (comment).

BTW now I could combine both Ceph 18.2.1 + python3-cryptography 41.0.7 + python3-bcrypt 4.0.1 without error; the error only happen when running with bcrypt >= 4.1.0.

Therefore I guess bcrypt should reference some handling from cryptography between 41.0.1..41.0.7 for preventing multiple initialization?

[alex]

Hmm, so the problem requires using both cryptography 41.0.7 and bcrypt 4.1.1? This is very confusing, and I can't imagine what cross-package interaction would be.

…

On Tue, Dec 12, 2023 at 9:24 AM Wong Hoi Sing Edison < ***@***.***> wrote: From my point of view, case should as similar pyca/cryptography#9016 (comment) <pyca/cryptography#9016 (comment)> . BTW now I could combine both Ceph 18.2.1 + python3-cryptography 41.0.7 + python3-bcrypt 4.0.1 without error; the error only happen when running with bcrypt >= 4.1.0. Therefore I guess bcrypt should reference some handling from cryptography between 41.0.1..41.0.7 for preventing multiple initialization? — Reply to this email directly, view it on GitHub <#694 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBBWCEFUH7YKSX35JLLYJBSKDAVCNFSM6AAAAABAQ2VAX2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJSGEZTOMBYGA> . You are receiving this because you commented.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[hswong3i]

The case with Ceph 18.2.1 are:

• OK: cryptography 40.0.2 + bcrypt 4.0.1
• OK: cryptography 41.0.7 + bcrypt 4.0.1
• FAILED: cryptography 40.0.2 + bcrypt 4.1.1
• FAILED: cryptography 41.0.7 + bcrypt 4.1.1

Original report for Ceph 18.2.1 + cryptography 41.0.x could be found from:

• https://tracker.ceph.com/issues/63529
• bazaah/aur-ceph#20
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055212

BTW as above cross check shown, now cryptography no longer become the root cause, but coming from bcrypt ;-(

[alex]

Is it straightforward for me to reproduce this locally, or do I need a full ceph installation?

[hswong3i]

I have a Vagrant Box for Ceph 18.2: https://app.vagrantup.com/alvistack/boxes/ceph-18.2

It is now rebuilding with my latest functional combination (i.e. cryptography 41.0.7 + bcrypt 4.0.1): https://gitlab.com/alvistack/vagrant-ceph/-/pipelines/1103864469

If you are using VirtualBox (see https://github.com/alvistack/vagrant-ceph#quick-start):

# Initialize Vagrant
cat > Vagrantfile <<-EOF
Vagrant.configure('2') do |config|
  config.vm.hostname = 'ceph-18.2'
  config.vm.box = 'alvistack/ceph-18.2'

  config.vm.provider :virtualbox do |virtualbox|
    config.vm.disk :disk, name: 'sdb', size: '10GB'
    virtualbox.cpus = 2
    virtualbox.customize ['modifyvm', :id, '--cpu-profile', 'host']
    virtualbox.customize ['modifyvm', :id, '--nested-hw-virt', 'on']
    virtualbox.memory = 8192
  end
end
EOF

# Start the virtual machine
export VAGRANT_EXPERIMENTAL='1'
vagrant up

# SSH into this machine
vagrant ssh

Once the box is up and running:

• vagrant ssh then sudo su - as root
• Check with ceph -s, now should showing HEALTH_OK
• Manually upgrade the preinstalled bcrypt 4.0.1 under /usr/lib/python3/dist-packages/bcrypt into 4.1.1
• Reboot the box
• Once login as root again and check with ceph -s, now it should show HEALTH_WARN 9 mgr modules have failed dependencies
• With journalctl -xef -u ceph-mgr@* you should able to get the error log as show above

[alex]

If I built a custom wheel, would you be able to test that out and see if you can reproduce?

[hswong3i]

If I built a custom wheel, would you be able to test that out and see if you can reproduce?

Or you may provide your forked GitHub branch, so I could keep rebuilding and testing with it in my local dev env?

[alex]

#695 -- you can either download a wheel from the wheel-builder job, or you can build yourself from that branch.

[hswong3i]

@alex my quick check and report:

• OK: cryptography 40.0.2 + bcrypt 4.0.1
• OK: cryptography 41.0.7 + bcrypt 4.0.1
• OK: cryptography 41.0.7 (patched with py_limited_api="auto" and pyo3 = { version = "0.20", features = ["abi3"] }) + bcrypt 4.0.1
• FAILED: cryptography 40.0.2 + bcrypt 4.1.1
• FAILED: cryptography 41.0.7 + bcrypt 4.1.1
• FAILED: cryptography 41.0.7 (patched with py_limited_api="auto" and pyo3 = { version = "0.20", features = ["abi3"] }) + bcrypt 4.0.1 (patched with py_limited_api="auto" and pyo3 = { version = "0.20.0", features = ["abi3"] })

For the last case, the new error message is now:

Dec 13 06:41:15 node12 ceph-mgr[17264]: 2023-12-13T06:41:15.339+0000 7f3d7e793280 -1 mgr[py] Module status has missing NOTIFY_TYPES member
Dec 13 06:41:15 node12 ceph-mgr[17264]: 2023-12-13T06:41:15.479+0000 7f3d7e793280 -1 mgr[py] Module not found: 'mgr_module'
Dec 13 06:41:15 node12 ceph-mgr[17264]: 2023-12-13T06:41:15.479+0000 7f3d7e793280 -1 mgr[py] Traceback (most recent call last):
Dec 13 06:41:15 node12 ceph-mgr[17264]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 13 06:41:15 node12 ceph-mgr[17264]:     from mgr_util import profile_method
Dec 13 06:41:15 node12 ceph-mgr[17264]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 13 06:41:15 node12 ceph-mgr[17264]:     import bcrypt
Dec 13 06:41:15 node12 ceph-mgr[17264]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 13 06:41:15 node12 ceph-mgr[17264]:     from ._bcrypt import (
Dec 13 06:41:15 node12 ceph-mgr[17264]: ImportError: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576

See PyO3/pyo3#576

[alex]

Hmm, on the one hand, if ceph really is using subinterpreters, there's nothing we can do.

On the other hand, the fact that this error is showing up only with certain version combinations suggestions there's a deeper bug here. I'm afraid I don't have a suggestion other than that we really need a minimal reproducer here, something smaller than all of ceph.