A better way to sort by difficulty

[quinnlas]

A better way to sort by difficulty

[jzohrab]

@quinnlas - a belated thanks for this issue which dropped off my radar. Good analysis too, it's hard to find the right metric for something like this, especially since it combined a few dimensions (X unique items that can repeat Y times).

If the next 5 pages had 1000 words, with 2 unknown words repeated 10 times each, say, and the rest of the text consisting of 100 repeated words: Unique unknown / total over 5 pages = 0.002. Unique unknown / unique total over 5 pages = ~ 0.02

I think that currently the code does something like actually renders the pages and then does its calculation based on the statuses of the words it sees. This is important for languages that are character-based, because the characters get combined into multi-word terms. But your logic should still be fine.