Text page can't handle text written in scripto continua (i.e. text without whitespace as word divider)

[vazkiimods]

Text page can't handle text written in scripto continua (i.e. text without whitespace as word divider)

[stanhebben]

还没注意中文😀. I can take care of this, but what would be the best way to determine if text is scriptio continua?

We could do so based on the characters and recognize character ranges. Maybe we can use Character.isIdeographic() for that, which can check for Chinese, Japanese, Korean and Vietnamese, but would that handle all cases of scriptio continua?

Alternatively, it could be done according to the book's language, but then foreign language snippets inside such books would not be formatted properly.

[3TUSK]

Off-topic: that BookTextParser is more like a BookTextTokener to me.

Character.isIdeographic might work - but...

1. Modern Vietnamese uses a romanized script. So the issue described does not apply to Vietnamese. One less thing to consider, which is good.
2. Modern Korean uses hangul - usually it uses whitespace to separate words; otherwise there is no readability at all. Another thing crossed out.
3. Within the set of languages that Vanilla Minecraft supports - the only case that Character.isIdeographic may not work is probably Thai. After all, modern languages rarely use scriptio continua. Unfortunately, I have zero knowledge on how Thai language actually works, and more worse - I am not sure if vanilla Minecraft can handle the rendering of Thai scripts...

I personally believe that this issue must be solved via boundary analysis (for example java.text.BreakIterator can do that; Mojang also uses icu4j, but I can't get that work).
Few months ago, I wrote this because I feel that vanilla isn't doing line wrapping correctly either, and I also wrote the explanation on what I did and why I did so. I was thinking adapting my work into BookTextParsr, but I soon realized that command handling makes the situation even trickier...
At least, I hope that can provide some hints.

[stanhebben]

Yeah, command handling makes it difficult to implement a solution with BreakIterator, since text is expanded on the fly (and styles need to be applied to words) so we can't simply feed the input text to a BreakIterator. Even a custom CharacterIterator may be difficult to implement, but I'm thinking about it.

I don't understand your reply concerning Hangul; is it crossed out because Korean does use spaces; or because Hangul isn't assumed Ideographic by Character.isIdeographic?

[stanhebben]

I think I may have a solution in mind with the BreakIterator; if processing of commands and positioning of text is performed in separate steps: command handlers can first generate a list of annotated spans. Once these spans are determined, I can have a custom character iterator iterate over these spans, splitting them into lines and performing positioning. Since the BreakIterator doesn't insert or delete characters, I can look up the spans in the original list using the character indices, applying styles appropriately.

[3TUSK]

I don't understand your reply concerning Hangul; is it crossed out because Korean does use spaces; or because Hangul isn't assumed Ideographic by Character.isIdeographic?

You can safely ignore my comments regarding hangul - all I want to say is that "Modern Korean does not have this issue because of the use of whitespace". Apologize for the confusion.

[stanhebben]

A fix for this has been implemented and should be available in the next release.

[3TUSK]

For future reference: fixed by #17.