Instantiating multiprocessing.SimpleQueue() from an imported package fails unless SimpleQueue() is instantiated inside Twisted application first

[michelcrypt4d4mus]

Instantiating multiprocessing.SimpleQueue() from an imported package fails unless SimpleQueue() is instantiated inside Twisted application first

[adiroiban]

Thanks for the report.

I don't see how this is related to twisted.

I don't see any twisted code in the traceback.

From what I can see, this is about mac-notifications and that project don't use Twisted.

Can you please share an self-contained example that replicate this error

---

Note that if you are using threads in your Twisted code, those thread should be handled only via Twisted helper API. You should not mix Twited code and threading code without using the Twisted helping API.

---

I am closing this.
If you think this is a bug with twisted please add a complete example that can be used to reproduce this bug and then we can look into this,

Without being able to reproduce this error, it is very hard to fix this.

If you have a fix for this, please submit a pull request and I we can reopen this ticket.

Regards

[michelcrypt4d4mus]

As I said in the bug report all I did was call macos_notifications from opencanary. OpenCanary is a twisted application; here's the .tac file. the bug arises in instantiation just a few lines in when SimpleQueue() constructor is called. i don't have more of the stack trace at hand right now.

Note that if you are using threads in your Twisted code, those thread should be handled only via Twisted helper API. You should not mix Twited code and threading code without using the Twisted helping API.

thanks, i suspect it is related to this, but the code never gets to the thread instantiation. it dies on literally the first line of __init__() in one of the startup classes.

[glyph]

I think there might be more to discuss here.

The fact that this is related to fds_to_keep suggests that Twisted is manipulating some global file descriptor state that it is fighting with Multiprocessing over, and we may have some work to do there.

I am not reopening right away because this really needs a clear reproducer to be actionable enough to be more than a vague "uhhh, maybe, global state, handwave handwave processes, file descriptors" magnet for anxiety or FUD :).

But if someone can do that so we can figure out what, if anything, we are having issues with, that would be great.

[michelcrypt4d4mus]

i think i can create a reproducible example relatively easily; i will try to do so.

[michelcrypt4d4mus]

The fact that this is related to fds_to_keep suggests that Twisted is manipulating some global file descriptor state that it is fighting with Multiprocessing over, and we may have some work to do there.

and fwiw that was my general impression as well.

[michelcrypt4d4mus]

ps i did find a very old issue over in the klein repo where someone seemed to have run into something similar - they were trying to use a package that did its own multiprocessing with a twistd server and getting fds_to_keep issues.

[michelcrypt4d4mus]

Easily reproducible on macOS:

1. Checkout my branch of opencanary: https://github.com/michelcrypt4d4mus/opencanary/tree/twistd_fds_to_keep_reproduce
2. run build_scripts/build_opencanary.sh
3. activate virtual env
4. run twistd -noy bin/opencanary.tac
5. ssh localhost from another window

[michelcrypt4d4mus]

I updated the issue description with the full stack trace instead of just the fragment i had put initially.

[adiroiban]

I am reopening this.

I don't have a macOS VM at hand to try to reproduce this... but I guess that there is enought information here to reproduce it.

It would be awesome to have this error reproduced via GitHub Actions