Fix multi-threading issue

[dedsecinside]

Fix multi-threading issue

[dblclik]

I’d like to tackle this—thinking multiprocessing.Pool.apply_async() will help nicely here, but we’ll have to fetch the results from async as they come in, can’t recall offhand if handling this with a for left open for streaming data will work or not 🤷‍♂️

[KingAkeem]

I'm working on this currently but my solution deals with using Golang. If you're able to write something in Python that has similar performance then that would be fine by me.

[dblclik]

@KingAkeem - Golang would be easier for this for sure, but I think there may be a way to do it in Python.

How were you planning on tying this project to a Golang based multithreaded URL analyzer?

[KingAkeem]

I'm going to try just using Go without any attempts at optimization since one of our bottlenecks is I/O, I think simply using Go to perform the request and print the status while provide a significant speed increase. After that, I'll run each request in a separate goroutine and take advantage of Go's built-in concurrency.

[KingAkeem]

I have a somewhat working prototype currently. You can check our this PR: #174 and this repo contains the Golang code that's being used as a plugin https://github.com/KingAkeem/display_status

[dblclik]

Sounds good, I'll un-assign myself from this issue. I'll take a look at the PR too 👍

[KingAkeem]

Sounds good to me 👍