Setting non-string values on Tokens

[lark-parser]

Setting non-string values on Tokens

[erezsh]

Hello!

Yes, Token.value is defined as string, for performance reasons. It's defined here: https://github.com/lark-parser/lark_cython/blob/master/lark_cython/lark_cython.pyx#L20

But nothing's stopping you from converting them to lark.Tokens, like so:

    return lark.Token.new_borrow_pos(t.type, float(t.value), t)

[geographika]

Thanks @erezsh, and thanks for this project!
This approach works fine for all the transformer functions that convert to any types that aren't str.
Does this approach however negate any performance boosts from using lark_cython? The full test suite went from 1min20 to 1min50 (a very rough benchmark).
I'll continue to play around and look at failing tests.

[erezsh]

This shouldn't have any direct effect on lark-cython's performance. But it's possible that you are creating a lot of Token instances, and that's taking a lot of time. (I do remember mappyfiles having a lot of ints in them)