Add check on status code to long-polling

[nbyrnes-acv]

Add check on status code to long-polling

[robertmrk]

Hi @nbyrnes-acv

Thanks for the suggestion.
The only problem is that I never sent enough platform events to hit this limitation. So I'm not sure how Salesforce reports this problem. I suspect they're returning a response that can't be interpreted by the library as a proper message, but I'm not sure.
Can you tell me what kind of error message or exception you're getting at the moment?

[nbyrnes-acv]

That is what I am working on determining now... My status check in long_polling does not seem to be catching the error. I can replicate the condition I am trying to catch once or twice a day, but I am now thinking the post in _send_final_payload may not be the correct place, or, sfdc is not sending the error as expected according to their docs.... is that session.post where the actual messages from SFDC are received?

Unfortunately, there is no error message. What happens is that SFDC just stops sending events, and the long_polling just times out, and re-connects forever. I don't know that I've hit the limit until try to reconnect to SFDC, at which point I get a subscribe failed error message which tells me: '403::Organization total events daily limit exceeded'

It seems to me that I need to alert someone should this ever happen in production, rather than just silently sitting there....

[nbyrnes-acv]

It seems that SFDC will only emit the API limit error status on subscribe, not on the POST.... So, We need to find an indirect means of tracking this and quitting before SFDC stops sending us messages..... Thanks for engaging.

[robertmrk]

I don't know that I've hit the limit until try to reconnect to SFDC, at which point I get a subscribe failed error message which tells me: '403::Organization total events daily limit exceeded

There are two possibilities.
Either SF indeed stops responding to connect requests which causes the client to time out, or the error is signaled in some form, but the client fails to consume the response with the error message for some reason.

I'm not entirely sure that this is not a bug in the client's implementation. Unfortunately it's very tricky to test this, since once the daily event delivery limit is reached, I will get just a single chance to observe this, because on the next connection attempt everything behaves correctly.

I'll try to reproduce this and log all responses from SF to see exactly how it behaves.

[robertmrk]

I was able to reproduce the problem by exhausting my daily platform event limit and for some reason I didn't get any errors, nor did the client time out. SF just stopped delivering new events.

This behavior makes sense, in the case when a client is subscribed to multiple different event types, and if the platform event limit is reached, then SF will continue to deliver the remaining event types, like events from push topics for example. But in your case, if you're only interested in platform events, this is problematic.

I'll give it one more try tomorrow.

[nbyrnes-acv]

It occurred to me that perhaps there could be a configurable option that enables a behavior that would allow detecting SF's stopping of sending events. Specifically, what I am thinking of is a counter that would, after some number of long_polling timeouts with no messages received, try re-subscribing. The re-subscription will emit an error in the event of exceeded limits, and it should just continue as normal if they are not. This would increase the timeliness of "out of event" detection, and facilitate error handling and communicating.... thoughts?

[robertmrk]

Hi @nbyrnes-acv

I ran my test application a few more times, but I never got any response that would indicate that the platform event limit has been reached. The client continued the communication with the server, but the server stopped delivering messages.

Specifically, what I am thinking of is a counter that would, after some number of long_polling timeouts with no messages received, try re-subscribing.

This is exactly what I wanted to recommend to you.
This can be either implemented in client code, or as the part of the library. But if I chose the latter solution, then I have to make this a universal solution and take into account all the other types of limitations as well. Since this problem has nothing to do with the protocol itself, and it's SF specific, it'll be implemented in aiosfstream.