erezsh/plyplus

a friendly yet powerful LR-parser written in Python

👋 Welcome to my Github profile! Have a look at my open-source projects:

• Lark, an innovative parsing toolkit, that's grown to be the most popular parser in the Python ecosystem!
• Reladiff, a tool for high-performance diffing of large datasets across databases.
• Preql, an ambitious new programming language that compiles to SQL. (inactive, but I plan to resume working on it some day!)
• Runtype, a collection of run-time type utilities including validation and multiple dispatch.
• Many more!

👩‍💻 My stack is Python for everything, and Svelte+Typescript for webdev. I also write in C++, Rust, Julia, and many others.

🌎 I'm living the life of a digital nomad; I spend a lot of my time in Amsterdam.

👅 I speak Hebrew, English and pocito de Spanish.

💞️ I’m looking to collaborate! Let me know if you have a cool project in mind, or want to help with one of mine. I'm especially interested in programming languages, the semantic-web, machine learning, static analysis, and recommendation systems.

👷‍♂️ I am a freelancer, and may be available for hire for cool and interesting projects, either open-source and proprietry.

🔗 Would you like to know more? Visit my website.

plyplus's Issues

Is it possible to parse Python grammar using recursive descent?

I'm sorry of this looks like a random offtopic question. Indeed, it mostly is. But this question drills my mind for a while, and google is not helpful with it at all - it returns hits for parsers written in Python, not parsers for Python grammar.

So, seeing it in README:

It supports post-tokenizing code, so it's capable of parsing python (it comes with a ready-to-use python parser).

, I wonder, would you have an idea if it's possible to parse Python with a recursive descent parser, without explicit AST construction?

Why I'm asking: in https://github.com/micropython/micropython/, AST construction accounts for a large part of a limit on maximum file size parsed. While using AST allows for cool things like compile-time optimizations, it would be nice to have a memory-efficient alternative parser too.

Thanks in advance for any hints!

how to walk the AST

can you tell me how can i walk the AST like in the ast module that comes with python

Support for unicode tokens in grammars

I'd like to be able to use unicode in tokens within my grammar. Currently, '←'. This isn't possible currently because the grammar parser tries to decode this as ascii.

What is your opinion on unicode in grammars? I'm happy to work on implementing it, but wanted to check if you had any strong opposition to this.

Error trying to use `cache_grammar`

Initializing a grammar object with the flag cache_grammar the following type error in Python 3.6:

Traceback (most recent call last):
  File "main.py", line 8, in <module>
    grammar = Grammar(f.read(), auto_filter_tokens=False, cache_grammar=True)
  File "/lib/python3.5/site-packages/plyplus/plyplus.py", line 550, in __init__
    plyplus_cache_filename = PLYPLUS_DIR + '/%s-%s-%s.plyplus' % (tab_filename, hashlib.sha256(grammar).hexdigest(), __version__)
TypeError: Unicode-objects must be encoded before hashing

Looks like this could be solved by simply adding a .encode() after grammar (at line 550).
But then pickle will complain about being unable to pickle the grammar (at line 558).

json parser barf

Just looking at this for the first time. I ran examples/json.py on some examples from http://json.org/example.html and it choked pretty hard on this one:

{"web-app": {
  "servlet": [   
    {
      "servlet-name": "cofaxCDS",
      "servlet-class": "org.cofax.cds.CDSServlet",
      "init-param": {
        "configGlossary:installationAt": "Philadelphia, PA",
        "configGlossary:adminEmail": "[email protected]",
        "configGlossary:poweredBy": "Cofax",
        "configGlossary:poweredByIcon": "/images/cofax.gif",
        "configGlossary:staticPath": "/content/static",
        "templateProcessorClass": "org.cofax.WysiwygTemplate",
        "templateLoaderClass": "org.cofax.FilesTemplateLoader",
        "templatePath": "templates",
        "templateOverridePath": "",
        "defaultListTemplate": "listTemplate.htm",
        "defaultFileTemplate": "articleTemplate.htm",
        "useJSP": false,
        "jspListTemplate": "listTemplate.jsp",
        "jspFileTemplate": "articleTemplate.jsp",
        "cachePackageTagsTrack": 200,
        "cachePackageTagsStore": 200,
        "cachePackageTagsRefresh": 60,
        "cacheTemplatesTrack": 100,
        "cacheTemplatesStore": 50,
        "cacheTemplatesRefresh": 15,
        "cachePagesTrack": 200,
        "cachePagesStore": 100,
        "cachePagesRefresh": 10,
        "cachePagesDirtyRead": 10,
        "searchEngineListTemplate": "forSearchEnginesList.htm",
        "searchEngineFileTemplate": "forSearchEngines.htm",
        "searchEngineRobotsDb": "WEB-INF/robots.db",
        "useDataStore": true,
        "dataStoreClass": "org.cofax.SqlDataStore",
        "redirectionClass": "org.cofax.SqlRedirection",
        "dataStoreName": "cofax",
        "dataStoreDriver": "com.microsoft.jdbc.sqlserver.SQLServerDriver",
        "dataStoreUrl": "jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon",
        "dataStoreUser": "sa",
        "dataStorePassword": "dataStoreTestQuery",
        "dataStoreTestQuery": "SET NOCOUNT ON;select test='test';",
        "dataStoreLogFile": "/usr/local/tomcat/logs/datastore.log",
        "dataStoreInitConns": 10,
        "dataStoreMaxConns": 100,
        "dataStoreConnUsageLimit": 100,
        "dataStoreLogLevel": "debug",
        "maxUrlLength": 500}},
    {
      "servlet-name": "cofaxEmail",
      "servlet-class": "org.cofax.cds.EmailServlet",
      "init-param": {
      "mailHost": "mail1",
      "mailHostOverride": "mail2"}},
    {
      "servlet-name": "cofaxAdmin",
      "servlet-class": "org.cofax.cds.AdminServlet"},

    {
      "servlet-name": "fileServlet",
      "servlet-class": "org.cofax.cds.FileServlet"},
    {
      "servlet-name": "cofaxTools",
      "servlet-class": "org.cofax.cms.CofaxToolsServlet",
      "init-param": {
        "templatePath": "toolstemplates/",
        "log": 1,
        "logLocation": "/usr/local/tomcat/logs/CofaxTools.log",
        "logMaxSize": "",
        "dataLog": 1,
        "dataLogLocation": "/usr/local/tomcat/logs/dataLog.log",
        "dataLogMaxSize": "",
        "removePageCache": "/content/admin/remove?cache=pages&id=",
        "removeTemplateCache": "/content/admin/remove?cache=templates&id=",
        "fileTransferFolder": "/usr/local/tomcat/webapps/content/fileTransferFolder",
        "lookInContext": 1,
        "adminGroupID": 4,
        "betaServer": true}}],
  "servlet-mapping": {
    "cofaxCDS": "/",
    "cofaxEmail": "/cofaxutil/aemail/*",
    "cofaxAdmin": "/admin/*",
    "fileServlet": "/static/*",
    "cofaxTools": "/tools/*"},

  "taglib": {
    "taglib-uri": "cofax.tld",
    "taglib-location": "/WEB-INF/tlds/cofax.tld"}}}

The error message made me think that it doesn't like / so I replaced those with _ and then I got this uninteligible error:

Traceback (most recent call last):
  File "json2.py", line 65, in <module>
    result = json_parse(json)
  File "json2.py", line 35, in json_parse
    return JSON_Transformer().transform(json_grammar.parse(json_string))
  File "/home/m/.local/lib/python2.7/site-packages/plyplus/plyplus.py", line 556, in parse
    return self._grammar.parse(text)
  File "/home/m/.local/lib/python2.7/site-packages/plyplus/plyplus.py", line 637, in parse
    raise ParseError('\n'.join(self.errors))
plyplus.plyplus.ParseError: Syntax error in input at '0' (type _ANON_1) line 1 col 769
Syntax error in input at ',' (type _ANON_10) line 1 col 2175
Syntax error in input at ',' (type _ANON_10) line 1 col 2271
Syntax error in input at ',' (type _ANON_10) line 1 col 2369
Syntax error in input at ']' (type _ANON_6) line 1 col 3074
Syntax error in input at ',' (type _ANON_10) line 1 col 3253
Syntax error in input at '}' (type _ANON_4) line 1 col 3351
Could not create parse tree!

Cool project by the way.

Problem with subgrammars

Python 3.4.1 with plyplus 0.6.1.
I'm trying to use https://github.com/erezsh/plyplus/blob/master/plyplus/grammars/config.g grammar and ini:

[GRAPHICS]
width=640
height=480

[MAP]
x_location=0
y_location=0
width=48
height=48
path=
autoload=1
vertical_shift=100

[MINIMAP]
size=128

[AMBIENT_OCCLUSION]
use_ambient_occlusion=yes
relative_tile_width=8.000000
ray_cast_distance=3

while it goes without errors it don`t build the correct tree.

>>> t.select('section > start > name *')
STreeCollection[]

With graphviz(but with pydotplus 2.0.2, normal don't wont to work on newest python) i see that the main grammar was opened but not the subs.

Also i found that in https://github.com/erezsh/plyplus/blob/master/README.md you open your file with
t = g.parse(file(r"C:\Python2...npymath.ini").read()) and not with open(file_name).read(). This is intentional and i miss some other lib, or it's typo?

plyplus imports ply indirectly at setup time

Hello Erez,

Plyplus setups with error on machine without ply installed:
https://gist.github.com/aurzenligl/365ae75ce86ea18ff715#file-plyplus_setup_error-txt

In setup.py, line:

import plyplus

which imports ply indirectly, conflicts with:

install_requires = ['ply'],

which installs ply later, at setup time.

Consider applying this patch, which allows ply import to fail during import time, thus installing ply and plyplus successfully:
https://gist.github.com/aurzenligl/365ae75ce86ea18ff715#file-plyplus_fix-diff

Best Regards,
Krzysztof

Not ANTLR compatible but uses .g extension.

.g is ANTLR's extension and plyplus uses it. It should not be used because it may confuse users and programs. pyplus's syntax is not ANTLR compatible.

NameError: global name 'unicode' is not defined

File "/plyplus/plyplus.py", line 588, in rule
if not isinstance(rule_name, (str,unicode)):
NameError: global name 'unicode' is not defined

Too permissive rules for list in Python grammar

In the Python grammar (plyplus/test/python.g) there are the following rules for lists:

list : LBRACK (list_inner|comprehension)? RBRACK ;
...
list_inner  : expr | expr COMMA (expr (COMMA)? )* ;

It seems to me this permits items that are not separated by commas. E.g. [1, 2 3], which does not seem to be a legal Python list. Is it intentional that the grammar matches a wider set than just legal lists?

This only occurred to me because I was writing a plyplus grammar that needed support for something similar to Python lists and my own rule does not look like this. It instead reads:

list: '\[' (item (',' item)* ','?)? '\]';

operator precendece

How can I implement operator precendece here? (*, /, +, -)

Grammar rule ordering problem (PEBKAC?)

This is quite likely user error, but I can't figure out why my parser fails when I simply reorder 2 of its rules:

# success:
print Grammar("""
  @start: field value;

  value: 'bar';
  field: '\w+';

  WS: '[ \t]+' (%ignore);
""").parse('foo bar')

# fail (swapped field/value declaration order):
print Grammar("""
  @start: field value;

  field: '\w+';
  value: 'bar';

  WS: '[ \t]+' (%ignore);
""").parse('foo bar')

The latter grammar crashes with:

  File "grammar_test.py", line 21, in <module>
    """).parse('foo bar')
  File "/Users/evzijst/work/venv/bbql/lib/python2.7/site-packages/plyplus/plyplus.py", line 575, in parse
    return self._grammar.parse(text)
  File "/Users/evzijst/work/venv/bbql/lib/python2.7/site-packages/plyplus/plyplus.py", line 716, in parse
    raise ParseError('\n'.join(self.errors))
plyplus.plyplus.ParseError: Syntax error in input at 'bar' (type _ANON_0) line 1 col 5
Could not create parse tree!

N.B.
PlyPlus==0.6.2
ply==3.6

Missing real License file

The MIT license file cited in Readme doesn't point to anything real now.

I guess that the simpler fix is using the one of lark-parser and adding it to tgz next release on pypi.

Glitch when there are undefined rules

The script:

from plyplus import Grammar

grammar = Grammar("""
start: foo;
""")

generates the following traceback:

Traceback (most recent call last):
  File "err.py", line 5, in <module>
    """)
  File "/tmp/e/lib/python3.6/site-packages/plyplus/plyplus.py", line 560, in __init__
    self._grammar = self._create_grammar(grammar, source, tab_filename, options)
  File "/tmp/e/lib/python3.6/site-packages/plyplus/plyplus.py", line 569, in _create_grammar
    return _Grammar(grammar_tree, source, tab_filename, options)
  File "/tmp/e/lib/python3.6/site-packages/plyplus/plyplus.py", line 626, in __init__
    GrammarVerifier().verify(grammar_tree)
  File "/tmp/e/lib/python3.6/site-packages/plyplus/plyplus.py", line 620, in verify
    raise ParseError("Undefined rules: %s" % undefined_rules)
plyplus.common.ParseError: U
n
d
e
f
i
n
e
d
 
r
u
l
e
s
:
 
{
'
f
o
o
'
}

The two exceptions raised near line 620 of plyplus.py should probably be:

if undefined_tokens:
    raise ParseError(["Undefined tokens: %s" % undefined_tokens])
if undefined_rules:
    raise ParseError(["Undefined rules: %s" % undefined_rules])

STree tail equivalence incorrect

Since tail is a list, to properly check equivalence the lists must be sorted

Error parsing \forall

text to parse is "\forall cinc"

Code is:

import plyplus

path = "Latex/sample_mip3.tex"
with open(path) as f:
    content = f.read()

    grammer = plyplus.Grammar(r"""

    @start: quant ;
    quant : '\\forall' string ;
    string : '[A-Za-z][A-Za-z0-9]*' ;
    WS: '[ \t\n]+' (%ignore) (%newline);


    """)
    print(grammer.parse(content))

Fails with Error:
plyplus.common.TokenizeError: Illegal character in input: '\forall cinc
', line: 1, error

however if i change \forall to \a everything is fine. It seems to have a problem with the string forall? What is going there?

Function fails if package installed as binary egg

This line (

plyplus/plyplus/grammars/__init__.py

Line 5 in 5a4c8cb

return _open( os.path.join(os.path.dirname(__file__), grammar_name) )

) fails if package is installed as binary egg.

======================================================================
ERROR: test_get_user_with_filter (tests.test_views.ViewTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/paul/Code/python/15five/django_scim/django_scim/tests/test_views.py", line 13, in test_get_user_with_filter
    url = reverse('scim:user') + '?filter=userName eq ""'
  File "/Users/paul/.pyenv/versions/scim/lib/python2.7/site-packages/django/core/urlresolvers.py", line 549, in reverse
    app_list = resolver.app_dict[ns]
  File "/Users/paul/.pyenv/versions/scim/lib/python2.7/site-packages/django/core/urlresolvers.py", line 351, in app_dict
    self._populate()
  File "/Users/paul/.pyenv/versions/scim/lib/python2.7/site-packages/django/core/urlresolvers.py", line 284, in _populate
    for pattern in reversed(self.url_patterns):
  File "/Users/paul/.pyenv/versions/scim/lib/python2.7/site-packages/django/core/urlresolvers.py", line 401, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/Users/paul/.pyenv/versions/scim/lib/python2.7/site-packages/django/core/urlresolvers.py", line 395, in urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)
  File "/Users/paul/.pyenv/versions/2.7.11/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/paul/Code/python/15five/django_scim/django_scim/tests/urls.py", line 3, in <module>
    from django_scim.urls import urlpatterns as scim_urls
  File "/Users/paul/Code/python/15five/django_scim/django_scim/urls.py", line 4, in <module>
    from django_scim.views import SearchView, UserView
  File "/Users/paul/Code/python/15five/django_scim/django_scim/views.py", line 10, in <module>
    from django_scim.filter import SCIMFilterTransformer
  File "/Users/paul/Code/python/15five/django_scim/django_scim/filter.py", line 4, in <module>
    from plyplus import Grammar, STransformer, PlyplusException
  File "/Users/paul/Code/python/15five/django_scim/.eggs/PlyPlus-0.6.2-py2.7.egg/plyplus/__init__.py", line 22, in <module>
  File "/Users/paul/Code/python/15five/django_scim/.eggs/PlyPlus-0.6.2-py2.7.egg/plyplus/selector.py", line 227, in <module>
  File "/Users/paul/Code/python/15five/django_scim/.eggs/PlyPlus-0.6.2-py2.7.egg/plyplus/grammars/__init__.py", line 5, in open
    return _open( os.path.join(os.path.dirname(__file__), grammar_name) )
IOError: [Errno 20] Not a directory: '/Users/paul/Code/python/15five/django_scim/.eggs/PlyPlus-0.6.2-py2.7.egg/plyplus/grammars/selector.g'

Can't use regular expressions

The following script is running without problem

import plyplus

grammar = plyplus.Grammar("""
start: a another_a;

a: 'a';
another_a: 'a';
""")

print(grammar.parse('aa'))

After a slight modification (replace another_a: 'a'; with another_a: '[a]';), the script can't parse the target anymore:

import plyplus

grammar = plyplus.Grammar("""
start: a another_a;

a: 'a';
another_a: '[a]';
""")

print(grammar.parse('aa'))

If I didn't completely failed at understanding the grammar notation, this slight modification should be also runnable.

Environment

• Linux Mint 18
• PlyPlus Version: 0.7.5
• Installed via pip

Problem with hard coded path to temporary files, when starting two application instances from one code base.

Consider two instances of application: 'development' and 'production'. Each on different project version (different commit). Now plyplus uses one hard coded location to store temporary files for both applications. Now when touching .wsgi following error occur:

Unable to create '/tmp/plyplus/plyplus_grammar_parsetab.py'
[Errno 13] Permission denied: '/tmp/plyplus/plyplus_grammar_parsetab.py'
Unable to create '/tmp/plyplus/parsetab_selector_g.py'
[Errno 13] Permission denied: '/tmp/plyplus/parsetab_selector_g.py'

which forcing to use production version temporary files (older version, used by users) with development version (newer, under development version).

What is correct grammar for parsing linear equations?

Hi. I'm use follow grammar for every part of equation:

list_parser = Grammar("""
start: ( a | b )+;

// terminals
number: '\d+';
identifier: 'x';
add: '\+' | '-';
mul: '\*';
div: '/';

// non-terminals
@factor: identifier | number;
a: add? factor ((mul | div) factor)*;
b: add? number (div number)*;
""")

It's correct for follow input:
print list_parser.parse('-5-2*x').pretty()

start
  b
    add	-
    number	5
  a
    add	-
    number	2
    mul	*
    identifier	x

But follow:
print list_parser.parse('-5-2/3*x').pretty()
gives an error:

Traceback (most recent call last):
  File ".../LALR.py", line 37, in <module>
    print list_parser.parse('-5-2/3*x').pretty()
  File "...\python27\lib\site-packages\plyplus\plyplus.py", line 575, in parse
    return self._grammar.parse(text)
  File "...\python27\lib\site-packages\plyplus\plyplus.py", line 700, in parse
    tree = self.engine.parse(text)
  File "...\python27\lib\site-packages\plyplus\engine_ply.py", line 105, in parse
    raise ParseError(self.errors)
plyplus.common.ParseError: Syntax error in input at '*' (type _ANON_4) line 1 col 7

I think in my rule a: add? factor ((mul | div) factor)*;, the ((mul | div) factor)* must match /3 at first and *x after.
What is problem here? Or, please, show me true way to give coefficients a and b form input string.

README.md example is incorrect

The calc example does not work. You need to wrap "calc.g" w/ an open. If you don't, you get:

>>> plyplus.Grammar("calc.g").parse("(1 + 2) * -3")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jmg/work/scobreak/p/lib/python2.7/site-packages/plyplus/plyplus.py", line 560, in __init__
    self._grammar = self._create_grammar(grammar, source, tab_filename, options)
  File "/Users/jmg/work/scobreak/p/lib/python2.7/site-packages/plyplus/plyplus.py", line 565, in _create_grammar
    grammar_tree = grammar_parser.parse(grammar)
  File "/Users/jmg/work/scobreak/p/lib/python2.7/site-packages/plyplus/grammar_parser.py", line 156, in parse
    return _parser.parse(text, lexer=lexer, debug=debug)
  File "/Users/jmg/work/scobreak/p/lib/python2.7/site-packages/ply/yacc.py", line 333, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/Users/jmg/work/scobreak/p/lib/python2.7/site-packages/ply/yacc.py", line 1063, in parseopt_notrack
    lookahead = get_token()     # Get the next token
  File "/Users/jmg/work/scobreak/p/lib/python2.7/site-packages/ply/lex.py", line 386, in token
    newtok = self.lexerrorf(tok)
  File "/Users/jmg/work/scobreak/p/lib/python2.7/site-packages/plyplus/grammar_lexer.py", line 57, in t_error
    raise Exception("Illegal character in grammar: %r in %r" % (t.value[0], t.value[:10] ))
Exception: Illegal character in grammar: '.' in '.g'

if instead you do:

start(mul(add(number(u'1'), add_symbol(u'+'), number(u'2')), mul_symbol(u'*'), neg(number(u'3'))))

you can see that it works.

Utils module not found

In pearley.py, the utils module cannot be imported. The error is the following:

File "/Users/sallai/PycharmProjects/foyer/foyer/smarts.py", line 2, in
import plyplus
File "/Users/sallai/anaconda/envs/foyer3/lib/python3.5/site-packages/plyplus/init.py", line 20, in
from .plyplus import Grammar
File "/Users/sallai/anaconda/envs/foyer3/lib/python3.5/site-packages/plyplus/plyplus.py", line 24, in
from .engine_pearley import Engine_Pearley
File "/Users/sallai/anaconda/envs/foyer3/lib/python3.5/site-packages/plyplus/engine_pearley.py", line 6, in
from . import pearley
File "/Users/sallai/anaconda/envs/foyer3/lib/python3.5/site-packages/plyplus/pearley.py", line 1, in
from utils import classify
ImportError: No module named 'utils'

I'm suspecting that modifying pearley.py to do "from .utils import classify" rather than "from utils import classify" should solve the problem.

setup.py tries to import plyplus before it's installed.

You can't have the setup.py for a package import itself. It just won't work. This causes the package to not be installable.

Tests are not py3k compatible

There are remaining print statements.