Tolerant parser with option to recover from unexpected token error

[nikic]

Tolerant parser with option to recover from unexpected token error

[pscheit]

class WebforgeCreateTestForClassCommand(sublime_plugin.WindowCommand):
    def run(self):
        view = self.window.active_view()
        classString = view.substr(view.find("^(?:abstract\s*|final\s*)*(?:class|interface)\s+(\w+)\s+", 0, sublime.IGNORECASE))
        print "searching page"
        print classString
        className = re.match(r"(?:abstract\s*|final\s*)?(?:class|interface)\s+(\w+)\s+", classString, re.I).group(1)

        print "searching ns"
        nsString = view.substr(view.find("namespace\s+(.*?);", 0, sublime.IGNORECASE))
        print nsString
        namespace = re.match(r"namespace\s+(.*?);", nsString).group(1)

        fqn = namespace+"\\"+className

if using with parser try to cut the string of php code before the syntax error appears. Or cut it before the line the syntax error appears. But for rself-made sublime plugins (with some conventions made) regexp would suffer

[deweller]

Thanks. The regex is a good idea for a workaround if I can't solve it with PHP-Parser. I'd like to use PHP-Parser if I can. Any ideas on how to modify or extend PHP-Parser to provide fault-tolerant parsing?

[nikic]

It's possible to make the parser reentrant, but it's not particularly simple (would require some changes to the parsing algorithm and the inclusion of error-recovery information in the grammar). I'll look into this, but can't guarantee it'll be anytime soon.

[deweller]

I'd be willing to contribute the code if you are open to a pull request. It would be a good excuse for me to get my head around how the parsing code works.

[nikic]

Yes, sure :) Just went through the commit history to find the old code for error recovery: b153bfa By now some of the surrounding code may have changed, but it should still work with some small modifications.

The docs for bison error recovery tokens can be found here: http://dinosaur.compilertools.net/bison/bison_9.html I guess one would have to add error rules in some key places like statements (error ';') and blocks (error '}').

The most complicated part is probably figuring out how to best report the errors that occur, e.g. one could introduce Error nodes, though not sure if that's a good idea. Alternatively just put all the errors in some array.

[deweller]

Thanks for the links above. That will help me to get started.

I would expect that the current behavior of throwing an error on a parse failure should stay the same. But perhaps we add an optional parser that reports errors. Perhaps we call it PHPParser_Parser_ErrorReporter. And maybe this version of the parser can recover from errors and report a list of all errors found.

[deweller]

I got as far as restoring the old commented-out test code. Here is my commit:
deweller@b0816a2

It is not obvious to me how $yyerrflag would ever be set to anything but 0. But maybe that is where bison's recovery tokens are required.

[nikic]

I just added an initial implementation in the errorRecovery branch: https://github.com/nikic/PHP-Parser/tree/errorRecovery

A test script:

$parser = new PHPParser_Parser(new PHPParser_Lexer, ['errorCallback' => function() {
    var_dump(func_get_args());
}]);
$nodeDumper = new PHPParser_NodeDumper;

$code = <<<'CODE'
<?php

while (true) {
    doSomething()
    echo 'Hello world';
}
CODE;

echo $nodeDumper->dump($parser->parse($code));

This will output:

array(2) {
  [0]=>
  string(31) "Syntax error, unexpected T_ECHO"
  [1]=>
  array(1) {
    ["startLine"]=>
    int(5)
  }
}
array(
    0: Stmt_While(
        cond: Expr_ConstFetch(
            name: Name(
                parts: array(
                    0: true
                )
            )
        )
        stmts: array(
            0: Stmt_Echo(
                exprs: array(
                    0: Scalar_String(
                        value: Hello world
                    )
                )
            )
        )
    )
)

So you get an error because of the missing semicolon after the call, but the next statement is parsed correctly.

Will need some additional tinkering though, some things that have to be figured out:

• How to report the errors. Right now it's using an error callback with the error message and the attributes of the token. That's probably not the most convenient way to do it.
• What to do when the error can not be recovered. Right now I throw an exception, but in the recovery-mode it might make more sense to just return false.
• How to handle errors that are creates by exceptions thrown during parsing (e.g. invalid modifiers). Right now this will still throw the exception and not be recovered.

[deweller]

This is great.

I went a different route (for now) in solving the problem that required the error recovery.

But I suspect I might need to come back to this later. When I do, I'll test it out.

[nikic]

Closing this in favor of #170 which links to a more up-to-date implementation.