huge number of threads spin up for lot's of messages processing.

[deepbrook]

huge number of threads spin up for lot's of messages processing.

[berentn]

Hi caliloo,

I addressed this issue today by using a scheduler so the timer threads are no longer needed. This greatly reduces the cpu load and increases the message processing speed. It is a very rough implementation and needs some refactoring and formal testing. But so far it seems to work.

However, I'm new to using git(hub), so I'm not sure how to share it with others (or anything else you're supposed to do on github). But you can find the updated file here:
https://github.com/berentn/Pysher/blob/master/pysher/connection.py

[deepbrook]

Hey @berentn ,
check the faq - I'd love to merge your changes!
https://help.github.com/articles/creating-a-pull-request/

[caliloo]

Hi I've tested the proposed merge.

While I do not see huge numbers of threads anymore which is a good thing, I d'ont see a significant drop in CPU usage (I'm going from 85/95% to 75/85% , I was expecting a drop in excess of 50-80% to get to 45/15% after the change ...)

Cheers

[berentn]

@nlsdfnbch
Thanks, I gave it a try :)

@caliloo
I also get an absolute reduction of about 10% on my CPU. However, that is a reduction of ~40% in my case. From my profiler I see that stopping the timers is now ~4 times faster and starting the timer is ~40 times faster.

Maybe your callback functions are using a lot of resources? Did you check what happens to your CPU load when you use a simple callback function which discards the messages?

[caliloo]

Yep have tried the function that discards messages, it has next to no effect on what I observe, maybe a drop by 1-2% of the numbers I see. As if all the load was generated by pysher rather than my callback.

In my normal case, I have actually moved my processing (which currently resumes itself to disk IO and not much else) to another process, the callback only packs stuff into a string and posts it to a queue, which is little load afaik.

What profiler do you use to figure which thread does what ? I'm willing to give it a try.

[berentn]

I constructed a test / benchmark file to compare the the timeout scheduling methods using cprofile and pstat. I'm not knowledgeable enough to create usable profiles / stats when running the pusherclient itself.

[caliloo]

ok, too bad, I was hoping to profile the pusherclient ...

[caliloo]

ok, I think I finally cracked this one. After some profiling, I figured that the webscoket module does dumb utf8 validation (byte by byte !!) unless you have wsaccel installed. So one pip intall wsccel later, it looks much better.

To resume my process was taking about 100-110% cpu (it was crashing regularly for lack of cycles and falling behind): the cpu contributors were :

• ssl about 10%
• dumb utf8 validation about 60% (vs wsaccel which is maybe 5-10%)
• crazy threads spin up 15% (vs sched 0%)
• the rest 10-15 %

so with ssl, wsaccel, sheduler instead of threads, my process shows about 25-30% cpu usage, that a 4x division of cpu usage which is nice :)
I'd definitely merge the timer shceduler patch if possible, it's much neater.

[deepbrook]

I've uploaded a new preview version (0.6.0b1) - Since tests are currently broken, it would be great if you guys could give that release a spin and see if anything breaks your current code.

[caliloo]

Thanks,
I'm giving it a try right now, it has started and run properly for the last 15 minutes, I'll give it a few days, see how it behaves and report back (I don't expect big stuff to show up, as most of the connection.py hasn't changed from berentn proposal, but pusher.py has seen some churn, so hey :) )

Could you add thread names to the various threads that are spun up ? I found that it helped with debugging.

connection.py:72:

    self.name = "PysherEventLoop"

connection.py:197:

    if not self.timeout_scheduler_thread:
        self.timeout_scheduler_thread = Thread(target=self.timeout_scheduler.run, daemon=True, name="PysherScheduler")
        self.timeout_scheduler_thread.start()

    elif not self.timeout_scheduler_thread.is_alive():
        self.timeout_scheduler_thread = Thread(target=self.timeout_scheduler.run, daemon=True, name="PysherScheduler")
        self.timeout_scheduler_thread.start()

(I may be wrong, but there may be one other thread flying around, havn't looked into it further, as this served my purpose)

[caliloo]

Have checked my logs for the past 3 days. Couldn't see anything that was a problem. This is good from my pov.