android target - Lexer$TokeniserException - Lexer.java can't parse tokens following '<<' or '[[' -

[kyperbelt]

android target - Lexer$TokeniserException - Lexer.java can't parse tokens following '<<' or '[[' -

[jojomickymack]

I figured it out! In addition to using the regex as-is, which works fine for matching lines of Text, I created an additional Regex and Matcher which replaces the \G with ^ and uses a substring of the line being tokenized instead of the whole thing.

Regex myRegex = new Regex(rule.regex.pattern.toString().replace("\\G", "^"));
Matcher myMatch = myRegex.match(line.substring(column_number));

Using this new matcher works for directives inside of << and [[, but has issues matching regular lines of text - so in the block with if (rule.type == TokenType.Text), the original matcher is used, and for any other ruletype, token_text = myMatch.group();

My understanding of the actual issue is that on Android's Java 1.6, java.util.regex doesn't do what it's supposed to with the \G symbol.

Here is a fixed version of the Lexer.java file - the edits are on line 289, 293, and 332.

That yarnjava repo I set up so I could pull in the fixed version of yarngdx using jitpack - that might be a good thing for you to take advantage of here btw - all you'd have to do is create a 'release' (otherwise SHAPSHOT is the only option). Jitpack makes it so you can use 'implementation 'com.github.jojomickymack:yarnjava:v1.0' in your core build.gradle file instead of using the jar or java files as described in your readme.md - it's just easier to include your library that way.

In the meantime I'm really happy I can use yarn and keep building for Android!!! Thanks!

[kyperbelt]

@jojomickymack thanks for the fix! I couldnt test on android myself but it makes sense. I put in the changes and tested that desktop target still functions 👍

Feel free to do pull requests as any help is greatly appreciated :)

as for jitpack - i did try it out and it worked until i tried using it with a few other projects that had a different version of lwjgl and i wasnt able to only include the core module. Maybe I just didnt see a way of doing this but ill definitely take a look into that.

Ill close this issue for now.

[jojomickymack]

@kyperbelt
Nice - looks like it's not 100% yet - my fault for not highlighting this other change I'd made to get text to parse correctly - I forgot to note that additional line was changed.

line 326 of Lexar.java

So at 0c3c2f0, for android targets the lexar is parsing all non-text lines, but it chokes on text - I believe it's because previously, match.find was getting called on this line, and now it's not. I have to admit, I don't understand why match.find(column_number) does not work for android.

When I change match.find(column_number); to match.find(0); everything works.

Thanks for working on this - I'm not that great with regex and I know this is better left in your hands!

[jojomickymack]

it's working when I build for android now - thanks!