lark-parser/lark_cython

Cython plugin for Lark, reimplementing the LALR parser & lexer for better performance

lark_cython's Issues

Cython standalone version?

What is your question?

Is there a way to compile a standalone cython module?

how to install lark_cython on windows

Describe the bug

install lark_cython on windows

  lark_cython.obj : error LNK2001: unresolved external symbol __imp__PyClassMethod_New
  lark_cython.obj : error LNK2001: unresolved external symbol __imp__PyList_Type
  lark_cython.obj : error LNK2001: unresolved external symbol __imp__PyDict_GetItemString
...

To Reproduce

pip install lark-cython

What do i need to install if i want to install on Windows

Setting non-string values on Tokens

Token values in lark_cython are typed as str.

In my Transformer I'm changing the token values to their correct types, for example int:

    def int(self, t):
        v = t[0]
        v.value = int(v.value)
        # return full Token rather than an int as Token properties are required later
        return v

This throws the following error in lark_cython:

lark_cython\lark_cython.pyx:20: in lark_cython.lark_cython.Token.value.__set__  TypeError: Expected unicode, got int

I return the full token in the transformer, rather than simply an int value, as later logic (for error handling etc.) takes advantage of the token properties:

line, column = key_token.line, key_token.column

Do token values have to be strings in lark_cython?
If so, is there any approach which would allow token values to be converted to support int, float etc.?

Tests fail with the lastet Lark (unexpected keyword argument strict).

With the latest version of Lark, lark-cython tests throw the following errors:

FAILED tests/test_basic.py::test_minimal - TypeError: __init__() got an unexpected keyword argument 'strict'
FAILED tests/test_basic.py::test_lark_meta_propagation - TypeError: __init__() got an unexpected keyword argument 'strict'
FAILED tests/test_basic.py::test_no_placeholders - TypeError: __init__() got an unexpected keyword argument 'strict'
FAILED tests/test_basic.py::test_start - TypeError: __init__() got an unexpected keyword argument 'strict'
FAILED tests/test_basic.py::test_lexer_callbacks - TypeError: __init__() got an unexpected keyword argument 'strict'

One example test failure is:

    def test_lexer_callbacks():
    	comments = []
    
>   	parser = Lark("""
    	    start: INT*
    
    	    COMMENT: /#.*/
    
    	    %import common (INT, WS)
    	    %ignore COMMENT
    	    %ignore WS
    	""", parser="lalr", _plugins=lark_cython.plugins, lexer_callbacks={'COMMENT': comments.append})

tests/test_basic.py:60: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/lark/lark.py:438: in __init__
    self.parser = self._build_parser()
../../.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/lark/lark.py:484: in _build_parser
    return _construct_parsing_frontend(
../../.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/lark/parser_frontends.py:253: in _construct_parsing_frontend
    return ParsingFrontend(lexer_conf, parser_conf, options)
../../.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/lark/parser_frontends.py:58: in __init__
    self.parser = create_parser(lexer_conf, parser_conf, options)
../../.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/lark/parser_frontends.py:157: in create_lalr_parser
    return cls(parser_conf, debug=debug, strict=strict)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   def __init__(self, parser_conf, debug=False):
E   TypeError: __init__() got an unexpected keyword argument 'strict'

lark_cython/lark_cython.pyx:570: TypeError

I think this means that lark-cython needs to add the "strict" option. Not sure what else will need to happen.

Next release, when?

Hi 👋🏽 ,
is there any info about next release? I'd like to use the cython plugins with the meta propagation fixed. Thanks

Meta is always empty while using cython lark

This is my parser:

    lark_kwargs = {
        "parser": "lalr",
        "lexer": "contextual",
        "propagate_positions": True,
        "_plugins": lark_cython.plugins,
    }
    lang_parser = Lark(lang_def, start="start", **lark_kwargs)

I'd like to propagate the line positions, without cython-lark it works fine, but once i enable the plugins the meta value is always empty.

Do you have any suggestion on what's wrong ?

Resolve differences from Lark

Suggestion

I mentioned the differences from Lark, is there any way to resolve?
Describe alternatives you've considered
Deploy from lark without any changes