stscoundrel/dictionary-revalidator

Kotlin / Spring Boot API for triggering ISR revalidations in online dictionaries

dictionary-revalidator's Issues

Dockerize for render.com usage

Support custom batch sizes

Currently batch sizes are defined per dictionary. While fine for most cases, changing batch sizes requires code change & redeploy of API.

Lets allow override in controller endpoint. like optional batchSize=200, which would then be fed to the revalidate method. Should still default to original config values when not provided.

Decrease batch sizes (again)

Something seems to have changed in Vercel, as more and more batches are getting timeout. Past baches of up to 250 used to be matter of few seconds, now can take more than 10 (timeout).

As this is not exactly time critical, decrease the batch sizes even further. For example, 100 seems to work fine for Old Swedish.

Revalidator output: mention which dictionary is being processed

Revalidator instances should already have access to this info via DictionaryType enum

Controller: guard with api key

Revalidator batch size: set per dictionary

Currently uses same 250 entries as batch size for all dictionaries. However, the time it takes to revalidate the entries can vary quite a bit based on how large the dataset to be processed is. For example, Old Icelandic dictionary is quite fast and has few & coincise entries, whereas Old Swedish dictionary is both larger and generally has larger entries.

Add batch size to the per dictionary setup. Preliminary values could be:

Old Norse: 250
Old Icelandic: 300
Old Swedish: 200
Old Norwegian 200

Test cases

The script does its thing, but might as well add simple test cases

Add number of times to retry

Currently always runs retries only once. It could be done until there is nothing to retry or certain limit is met.

Add healthcheck endpoint

Render has automatic support for health monitoring. Add simple /health/ endpoint that just returns 200

Support start & endpoint as optional params

Occasionally one would like to run certain subset of entries to validate. For example, should the process have crashed around entry 10 000 of 20 000, one would only want to revalidate entries 10 000 -> 20 000 in the following request, ignoring the first 10 000.

• Add as optional params to the controller
• Feed to revalidator. Use if provided.