praveenmunagapati/lexertl

lexertl: The Modular Lexical Analyser Generator

lexertl: The Modular Lexical Analyser Generator

lexertl is a header-only library for writing lexical analysers. With lexertl you can:

• Build lexical analysers at runtime
• Scan Unicode and ASCII input
• Scan from files or memory
• Generate C++ code or even write your own code generator

Construct a Lexer and Tokenise input

#include "lexertl/generator.hpp"
#include "lexertl/lookup.hpp"
#include <iostream>

int main()
{
    lexertl::rules rules;
    lexertl::state_machine sm;

    rules.push("[0-9]+", 1);
    rules.push("[a-z]+", 2);
    lexertl::generator::build(rules, sm);

    std::string input("abc012Ad3e4");
    lexertl::smatch results(input.begin(), input.end());

    // Read ahead
    lexertl::lookup(sm, results);

    while (results.id != 0)
    {
        std::cout << "Id: " << results.id << ", Token: '" <<
            results.str () << "'\n";
        lexertl::lookup(sm, results);
    }

    return 0;
}

The same thing using lexertl::iterator:

#include "lexertl/generator.hpp"
#include "lexertl/iterator.hpp"
#include <iostream>

int main()
{
    lexertl::rules rules;
    lexertl::state_machine sm;

    rules.push("[0-9]+", 1);
    rules.push("[a-z]+", 2);
    lexertl::generator::build(rules, sm);

    std::string input("abc012Ad3e4");
    lexertl::siterator iter(input.begin(), input.end(), sm);
    lexertl::siterator end;

    for (; iter != end; ++iter)
    {
        std::cout << "Id: " << iter->id << ", Token: '" <<
            iter->str() << "'\n";
    }

    return 0;
}

More examples and documentation

See http://www.benhanson.net/lexertl.html for full documentation and more usage examples.