c:geo not refresh the geocache due to limits introduced to OCPL endpoints

[pawelko-g]

c:geo not refresh the geocache due to limits introduced to OCPL endpoints

[deg-pl]

There are two limits:

• "single page" limit (number of views of a single page/single OKAPI endpoint calls per second): 150/s
• "entire site" limit: 200 views/s

The first limiter did not work properly and took the default value (5) instead of the one I set.

[deg-pl]

It isn't OC PL code related problem, but site/user related.

[mzylowski]

@deg-pl I'm so surprised by your answer from above, and also I just read the Facebook discussion with @pawelko-g about this issue. This is so technically incorrect from so many points of view that I'm shocked you're still pushing for this approach.

The rate limits you introduced to the OC server are an undocumented breaking change for OKAPI. @pawelko-g was using c:geo before your changes, but now you blame him for making too much traffic to the OCPL server.
He is just an end-user of the opencaching site, and he sends his requests via the C: Geo app that he downloaded from Google Play. Why are you expecting the end user to stick to those limits just by using the app?

The 'actor' responsible for the violation of those rate limits is not pawelko, but the application (in this ticket it is c:geo, but there are a lot of others app affected by your change now). If you @deg-pl want to still keep these rate limits, you need to at least communicate about them to all OKAPI developers and at least add info about them to OKAPI documentation. However, the effort that needs to be made on the apps developer side to make possible use of opencaching data is so extreamlly huge (moving calling okapi to separate threads, some sleeps to avoid your limits etc), that I believe they need some time before those rate limits will be introduced. For now, you should do the rollback of those changes.

My proposal to mitigate the issue:

• Exclude OKAPI endpoints from those rate-limiting (long term solution)

Short-term solutions if you still want to keep those ridiculous limits:

• Add documentation for OKAPI description of your breaking change.
• Communicate about the changes to developers (send e-mail to all OKAPI consumers)
• Create a new endpoint to OKAPI or add some more info to the existing one (i.e https://opencaching.pl/okapi/services/apisrv/stats.html) to allow developers to get the allowed number of requests. Maybe @wrygiel has an idea?
• Exclude our biggest customers from those limits for now, like c:geo, to give them time to apply needed code changes.

As long as OCPL end users are banned due to those limits, this is a high-priority issue caused by a misconfiguration of the OCPL server.

P.S: my other personal applications that using OKAPI are also broken now :(
P.S2, EDIT: How small those limits are that c:geo is not able even to download 8 geocaches?

[mzylowski]

There is reproduction from my side

Now you will take responsibility for your changes?

[stefopl]

A quick test:
I'm using the map to download caches in my area.
I have 200 caches to download offline, in my case it looked like this:

• 48 and failure in c:geo

When downloading, I can select the option in c:geo to do nothing with the downloaded caches and continue downloading more.

• 12 and failure
• 4 and failure
• 22 and failure

When refreshing 200 caches, I don’t have this option, so it always ends in an error.

I believe that at least 200 caches should be downloadable without a limit. When traveling to a new area for events with new geopaths, one typically needs to download around this number of caches.

The last event I attended was on 17.05.2024, and I didn't have any problems with downloading caches at that time.

[deg-pl]

@pawelko-g is the only user I know who has such a problem with c:geo.
If the problem is actually related to limiting the number of queries, it is still not a problem related to the OC PL code.

Added later:
The current limit is 200/second and this causes a soft ban of only 2 seconds
The limit is the result of a DOS attacks on the server and has been in effect for many months.

[mzylowski]

@pawelko-g is the only user who reported this issue. The other users don't know what to do, and blame c:geo for hanging the update procedure.

[deg-pl]

@pawelko-g started a discussion on Facebook in the OC group, but no one else confirmed the problem.

[deg-pl]

@mzylowski Please provide steps to reproduce the problem

[mzylowski]

1. Download GPX for GeoPath (50 caches will be enough for sure) -> https://opencaching.pl/powerTrail.php?ptAction=showSerie&ptrail=912
2. Import downloaded GPX to c:geo
3. Refresh list of geocaches

'Odwiez wszystkie' menu option

[deg-pl]

Thanks @mzylowski.
Now I can reproduce problem. This makes it much easier to find a solution.

Server logs indicate that, in addition to OKAPI queries, images are being downloaded to caches. The logs also indicate that the real limits are lower than those I configured (both the number of queries and the time unit in which it is tracked).

[deg-pl]

@pawelko-g @mzylowski @stefopl Could you please check again?
Now the limiter seems to behave in accordance with the assumed configuration.

[stefopl]

@deg-pl
I intentionally chose more. Download completed successfully 264/264.
Thanks

[mzylowski]

After your intervention I was able to successfully refresh entire 'Rowerowe Pomorze' geopath (the biggest GP in our service)

Thanks!

How the limits are set currently?

[pawelko-g]

@deg-pl @mzylowski @stefopl Thank you for your help. It also works fine for me now.

[andrixnet]

There are two limits:

* "single page" limit (number of views of a single page/single OKAPI endpoint calls per second): 150/s

* "entire site" limit: 200 views/s

The first limiter did not work properly and took the default value (5) instead of the one I set.

Where/how are these limits configurable per OC site?

[deg-pl]

@andrixnet These limits are not OC PL code related. We use "evasive" as additional Apache2 module.