Unit tests don't all pass, due to HTTP 429 throttling

[azure]

Unit tests don't all pass, due to HTTP 429 throttling

[rnagpal]

Hi Carson,

Thanks for reporting this issue.

Running these tests all at the same time might result in 429s as you see but I'm surprised that adding the delay of 15 secs between the tests is still causing the throttle. In the second case, are the same set of test cases failing? Probably, each of those tests are making several requests too soon causing the throttle within the same test case. Most of the collections in the tests should be S1 which support 250 RUs/second and I'd guess that would be sufficient for most of the test cases.

Performance tier is tied to a collection, so as long as you are creating new test collections there shouldn't be any impact of the other load(if any) on the same database.

We probably would need to add retry policy around 429s for these test cases, so that always succeed(even if they take time).

I'll add this to our backlog but feel free to contribute to it if you wish.

Regards,
Rajesh

[carsonyl]

Hi Rajesh. Thanks for the insight. I believe that the failures with the 15 second delay are a subset of the failures with no delay -- I didn't look too closely at the results because without any delay, about half the tests fail.

It would be great to have a retry policy for all the methods in pydocumentdb. I am open to contributing, but I'm not yet familiar enough with DocumentDB. This post mentions how the .NET API has built-in retry for 429 responses, and I see signs of this being implemented in some parts of pydocumentdb too. I may file another issue about this tomorrow, because I have some suspicions about whether it's working correctly.

[rnagpal]

The 2 slightly different versions of the error messages you were seeing for throttle scenarios was due to the fact that sometimes throttle is detected at the gateway level and Gateway sends RequestTooLargerException(the second one that you are seeing). The first one is coming directly from the server and hence has a slightly different message but since both of them have status code 429, they are due to the same reason.

It will be a while until we implement the throttle retry policy for all documentdb APIs for Python SDK but you can take a look at http://stackoverflow.com/questions/29275447/exception-microsoft-azure-documents-requestratetoolargeexception-while-querying as an example how that's implement in .NET SDK, so you may have to do similar thing for Python. In short, you catch the exception, check if it's status code is 429, get the value of "RetryAfterInMilliseconds" property from the response header and wait until that many ms, and resubmit the request. You can loop for say 3 times before giving up.

Meanwhile, we should be able to modify the tests so that each one of them doesn't creates a new collection every time. We can have one or two test collections created which are shared by most of the tests for creating resources within that collection and cleaning them up at the end. Collection creates are the only one which are causing problems in this case, so if we can eliminate that we should be good. We may be able to get to this approach in our next release but if you are comfortable enough to make these changes, feel free to do so and we will accept it as a PR.

Regards,
Rajesh

[carsonyl]

For now, I'm experimenting with the REST API without using pydocumentdb. I'll try implementing automatic retry based on your link - thanks.

[rnagpal]

@rbcarson : This issue has been fixed with 1.9.0 release. You should be able to run the tests without any custom retry logic.