zaach / jison-lex Goto Github PK

generates lexical analyzers. used by jison.

JavaScript 100.00%

jison-lex's Introduction

jison-lex

A lexical analyzer generator used by jison. It takes a lexical grammar definition (either in JSON or Bison's lexical grammar format) and outputs a JavaScript lexer.

install

npm install jison-lex -g

usage

Usage: jison-lex [file] [options]

file     file containing a lexical grammar

Options:
   -o FILE, --outfile FILE       Filename and base module name of the generated parser
   -t TYPE, --module-type TYPE   The type of module to generate (commonjs, js)
   --version                     print version and exit

programatic usage

var JisonLex = require('jison-lex');

var grammar = {
  rules: [
    ["x", "return 'X';" ],
    ["y", "return 'Y';" ],
    ["$", "return 'EOF';" ]
  ]
};

// or load from a file
// var grammar = fs.readFileSync('mylexer.l', 'utf8');

// generate source
var lexerSource = JisonLex.generate(grammar);

// or create a parser in memory
var lexer = new JisonLex(grammar);
lexer.setInput('xyxxy');
lexer.lex();
// => 'X'
lexer.lex();
// => 'Y'

## license
MIT

jison-lex's People

Contributors

Stargazers

Watchers

jison-lex's Issues

Tests fail with Node ≥ 10

Hello,
using Node 10.5, I've these 2 errors when launching test:

    test more()
      ⚡ SyntaxError: Unexpected token (
          at new RegExpLexer (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/regexp-lexer.js:124:22)                                                 
          at exports.test more() (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/tests/regexplexer.js:322:17)                                        
          at test (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:32:20)                                                   
          at next (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:70:72)                                                   
          at done (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:29:7)                                                    
          at test (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:44:21)                                                   
          at next (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:70:72)                                                   
          at done (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:29:7)                                                    
          at test (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:44:21)                                                   
          at next (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:70:72)                                                   
    test defined token returns
      ✓ passed
    test module generator from constructor
      ✓ passed
    test module generator
      ✓ passed
    test generator with more complex lexer
      ⚡ SyntaxError: Unexpected token (
          at new RegExpLexer (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/regexp-lexer.js:124:22)                                                 
          at exports.test generator with more complex lexer (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/tests/regexplexer.js:413:18)             
          at test (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:32:20)                                                   
          at next (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:70:72)                                                   
          at done (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:29:7)                                                    
          at test (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:44:21)                                                   
          at next (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:70:72)                                                   
          at done (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:29:7)                                                    
          at test (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:44:21)                                                   
          at next (/home/xavier/dev/debian/src/pkg-js/packages/node-jison-lex/node_modules/test/test.js:70:72)

Lexer should pass character position to parseError

The Jison parser provides useful location information to its parseError function - the lexer currently does not provide any location information to parseError at all.

https://github.com/zaach/jison/blob/5e13d8563c306c66cc00b9fe22ff6da74617e792/lib/jison.js#L1037-L1043

While implementing TryGhost/GQL#4 the only solution I could find to get the location of the error was by parsing it out of the error message.

Problem with unput in IE

I'm having problems using unput("\n") in IE:

jison-lex/regexp-lexer.js

Line 185 in 25afbec

unput: function (ch) {

The problem seems to be that the following line returns different values in IE and Chrome:

var lines = "\n".split(/(?:\r\n?|\n)/g);
// Chrome: lines == ["", ""]
// IE9: lines == []

This means that lines[0] on the line below fails.

Provide function regex

Could be possible require a external library that export a function that evaluate if the current term is a token?

Example:

WORD             require('lodash.words')

Invalid deletes - noticed when in strict mode

There is the following code:

var backup;
....
if (this.options.backtrack_lexer) {
    delete backup;
}

The delete here is does nothing, and in strict mode flags up as an error.

Unicode support

It would be nice to be able to use for example a Unicode-aware version of \w.

This would mean that jison-lex needs to support more than caseless aka /i as an option.

I wanted to try out a few things today with jison. I was looking through the code and it wasn't immediately obvious how to call jison-lex from scripts. While working out the lex section of my jison file I thought it would be good if I could just use jison-lex directly from a script, that way I could write a test in jasmine-node and do something like jasmine-node spec/ --autotest while I worked out what I was doing.

Here's something I wrote after reading through the source code. I'm posting it here in case you might want to use it in the readme.

var fs = require('fs');
var jisonLex = require('jison-lex');

var grammar = fs.readFileSync('./comments.jison-lex', 'utf8');
var input = fs.readFileSync('./comments.txt', 'utf8');

var lexer = new jisonLex(grammar, input);

function saveLexerToFile () {
    var mymodule = lexer.generate({
        moduleType : 'commonjs',
        moduleName : 'comments'
    });
    fs.writeFileSync("./comments.lex.js", mymodule);
}

// grabs tokens from whatever lexer is available.
function getResults () {
    var result = lexer.lex();
    console.log(result);
    if(result !== 'EOF') {
        setTimeout(getResults);
    }
}


// replaces lexer instance with the lexer stored in the file.
function runLexerFile () {
    lexer = require("./comments.lex.js").lexer;
    lexer.setInput(input);
    getResults();
}


getResults();

Unexpected token < while building from stateful lex file

I'm just trying to build a lexer from documentation example:

%s expect

%%
expect-floats        this.begin('expect');

<expect>[0-9]+"."[0-9]+      {
            console.log( "found a float, = " + yytext );
            }
<expect>\n           %{
            /* that's the end of the line, so
             * we need another "expect-number"
             * before we'll recognize any more
             * numbers
             */
            this.begin('INITIAL');
            %}

[0-9]+      console.log( "found an integer, = " + yytext );

"."         console.log( "found a dot" );

and get case 0:expect-floats this.begin('expect')

Ok, I deleted this line, but get:

undefined:5
case 0:<expect>[0-9]+"."[0-9]+      {
SyntaxError: Unexpected token <

Parse error when using arrow function in rules

It will parse error:

let grammar = {
    lex: {
        rules: [
            ['\\s+', ''],
            ['\\d+', () => 'NUMBER'],
            ['\\+', () => '+'],
            ['$', () => 'EOF'],
        ]
    },
    operators: [
        ['left', '+']
    ],
    bnf: {
        'es': [
            ['e EOF', 'return $1']
        ],
        'e': [
            ['e + e', '$$ = $1 + $3'],
            ['NUMBER', '$$ = Number(yytext)']
        ]
    }
}

while this is ok:

let grammar = {
    lex: {
        rules: [
            ['\\s+', ''],
            ['\\d+', function () { return 'NUMBER'}],
            ['\\+', function () { return '+' }],
            ['$',  function () { return 'EOF' }],
        ]
    },
    operators: [
        ['left', '+']
    ],
    bnf: {
        'es': [
            ['e EOF', 'return $1']
        ],
        'e': [
            ['e + e', '$$ = $1 + $3'],
            ['NUMBER', '$$ = Number(yytext)']
        ]
    }
}

Create LICENSE file

Would you please create a LICENSE or LICENSE.md file with your copyright information and the text of the MIT license? The MIT license specifically states that the license text must accompany the source code.

Not able to use INITIAL and an exclusive state

I have this line which causes an error:

<INITIAL,MULTILINE_COMMENT>"/*"                    this.begin("MUTLILINE_COMMMENT");

where MULTILINE_COMMENT is successfully defined with %x MULTILINE_COMMENT

/usr/local/lib/node_modules/jison/node_modules/jison-lex/regexp-lexer.js:42
                startConditions[conditions[k]].rules.push(i);
                                              ^

TypeError: Cannot read property 'rules' of undefined
    at prepareRules (/usr/local/lib/node_modules/jison/node_modules/jison-lex/regexp-lexer.js:42:47)
    at Object.buildActions (/usr/local/lib/node_modules/jison/node_modules/jison-lex/regexp-lexer.js:112:18)
    at processGrammar (/usr/local/lib/node_modules/jison/node_modules/jison-lex/regexp-lexer.js:484:39)
    at RegExpLexer (/usr/local/lib/node_modules/jison/node_modules/jison-lex/regexp-lexer.js:122:16)
    at o.constructor.Jison_Generator [as constructor] (/usr/local/lib/node_modules/jison/lib/jison.js:111:22)
    at o.constructor.(anonymous function) [as constructor] (/usr/local/lib/node_modules/jison/lib/util/typal.js:23:28)
    at new o.constructor (/usr/local/lib/node_modules/jison/lib/util/typal.js:77:70)
    at new Jison_Generator (/usr/local/lib/node_modules/jison/lib/jison.js:1910:20)
    at Object.generateParserString (/usr/local/lib/node_modules/jison/lib/cli.js:153:21)
    at processGrammar (/usr/local/lib/node_modules/jison/lib/cli.js:72:22)

startConditions error

I'm using latest jison version for generating a parser and i've got this error :

...\node_modules\jison\node_modules\jison-lex\regexp-lexer.js:44
                  startConditions[conditions[k]].rules.push(i);
                                                ^
TypeError: Cannot read property 'rules' of undefined
    at prepareRules (...\node_modules\jison\node_modules\jison-lex\regexp-lexer.js:44:49)
    at Object.buildActions (...\node_modules\jison\node_modules\jison-lex\regexp-lexer.js:115:18)
    at Object.RegExpLexer (...\node_modules\jison\node_modules\jison-lex\regexp-lexer.js:141:39)
    at Jison_Generator (...\node_modules\jison\lib\jison.js:111:22)
    at (anonymous function) (...\node_modules\jison\lib\util\typal.js:23:28)
    at new o.constructor (...\node_modules\jison\lib\util\typal.js:77:70)
    at Object.Jison_Generator [as Generator] (...\node_modules\jison\lib\jison.js:1740:20)
    at new Parser (...\node_modules\jison\lib\jison.js:1745:25)

To reproduce this issue you can use this syntax file :

/* BUG : TypeError: Cannot read property 'rules' of undefined */

%lex

%x INITIAL

%%

<INITIAL>"{" %{
  this.begin('BRACKET');
  console.log('{');
%}
<BRACKET>"}" %{
  this.popState();
  console.log('}');
%}

/lex

%start start

%%

start: [^];

I've tried to change regexp-lexer.js line 43 with the following, but I'm not sure :

  if (startConditions.hasOwnProperty(conditions[k])) {
    startConditions[conditions[k]].rules.push(i);
  }

generate 'optimal' lexers, i.e. the Need For Speed

Can't generate a lexer for CSS String

I'm trying to implement CSS parser using Jison project. I found CSS2.1 definition here, but the lexer keeps failing and I don't know why. Here's a test case.

https://gist.github.com/beatak/0b8b29fbddc0a28ddad2

I'm new to this area and it might be me doing totally wrong, but if you could comment how to solve this (like "you're doing it wrong, read Bison intro"), that will be super appreciated. Thanks!