Unable to use on Windows? NotImplementedError of _make_subprocess_transport for SelectorEventLoop

[scrapy-plugins]

Unable to use on Windows? NotImplementedError of _make_subprocess_transport for SelectorEventLoop

[yadalik]

Hi, with WSL2, at least wsl2 on Ubuntu 20.04 LTS works quite well. It does not need any specific settings (like setting up event_loop_path or ASYNCIO_EVENT_LOOP), except changing start of links to any files from windows-specific drive_letter:\ to /mnt/drive_letter/.
Also, there will be self-explanatory errors on first start about how to install playwright and its browser binaries.

Those links can help if choose to use wsl2 instead of installing Linux:

1. https://docs.microsoft.com/en-us/windows/wsl/install-win10 - guide on installing wsl2 from Microsoft.
2. https://www.jetbrains.com/help/pycharm/using-wsl-as-a-remote-interpreter.html - guide for PyCharm on debugging and starting with wsl2 (only on pro-version or 30-day evaluation).
3. https://code.visualstudio.com/docs/remote/wsl-tutorial - guide for Visual Studio Code (free).

[elacuesta]

Indeed, it seems to me like it's currently not possible to run scrapy-playwright on Windows.

In the link to the Python docs that you posted, it says that

On Windows, the default event loop ProactorEventLoop supports subprocesses, whereas SelectorEventLoop does not

Playwright does make use of subprocesses, but the only available Twisted reactor based on asyncio event loops is AsyncioSelectorReactor. Here is where Twisted checks the event loop, and raises one of the exceptions you are getting.

Regarding the Windows Subsystem for Linux, I have zero experience with that, and I don't have access to a Windows machine to try it. If you do go that way, I'd very much appreciate you coming back and reporting your findings.

[BruceLee569]

Indeed, it seems to me like it's currently not possible to run scrapy-playwright on Windows.

In the link to the Python docs that you posted, it says that

On Windows, the default event loop ProactorEventLoop supports subprocesses, whereas SelectorEventLoop does not

Playwright does make use of subprocesses, but the only available Twisted reactor based on asyncio event loops is AsyncioSelectorReactor. Here is where Twisted checks the event loop, and raises one of the exceptions you are getting.

Regarding the Windows Subsystem for Linux, I have zero experience with that, and I don't have access to a Windows machine to try it. If you do go that way, I'd very much appreciate you coming back and reporting your findings.

Hi, I need to run both scrapy and playwright on a Windows computer, I don't know much about event loops, but still want to ask you after consulting the only explanation of ProactorEventLoop in the official Python document and twisted's own implementation of windows iocpreactor, is it impossible for the two to run compatible? And this is the only related PR I found in twisted project.

[elacuesta]

AFAICT there hasn't been any development on this, my original comment still applies.
Playwright needs subprocess, SelectorEventLoop does not support subprocess on Windows (I don't know what are the technical reasons behind this), and the asyncio twisted reactor is based on SelectorEventLoop.

[BruceLee569]

@elacuesta Ok, thank you very much for your reply! BTW, the official document of Playwright is also mentioned here.

[elacuesta]

That's good to know, I'll add a link to that on the README. Thanks!

[elacuesta]

Updated the readme: 1c5f96e