adrian-thurston/colm

The Colm Programming Language

colm's Issues

[colm] the * and + operators should use left-recursion

Original motivation for right recursion was convenient lisp-like list deconstruction into head and tail. There is a place for right recursion, however, it makes more sense now to be consistent with the parsing algorithm and use left-recursion.

[colm] bring back string concatenations in match/cons/string top level

These were removed because of the ambiguity with bare sends. However, it's quite annoying to no longer be able to concat strings. Any kind of multi-line pattern/concat/string will no longer work. Also, we have enabled concat for accumulation, but not for pattern/concat/string, which is an apparent inconsistency.

[colm] ref output in function call introduces leaks

working on a new code generator and as soon as ref output in a function call is introduces there are leaks

[colm] translate block with no send causes crash

[colm] regex subexpression capture

This may be finished. Need to investigate.

[colm] using stdin.pull in a token transformation results in IN_HALT

can be shown with the commitbt.lm test case. Change input.pull to stdin.pull.

[colm] parse accumulator crashes if required to backtrack

[colm] integer expressings not complete

[colm] optimization of reference chains

I think this is done already. Need to verify.

[colm] remove ? * + ( ) and | from allowed chars in position 2+ of literal tokens

Want to promote usage of `,? and other forms in productions. Currently only ] is disallowed in position 2+ of literals. Expand this to ? * + ( ) and |.

Note that () and | are not currently used, but want to add these in the future.

[colm] sub patterns and sub constructs

maybe use ( ) for these

type( construct ) for sub-construct
Name: type ( pattern ) for sub-pattern

[colm] some syntactic sugar for unamed grammar defs used inline ( | )

Would be convenient to allow a grammar def to be defined inside a production. For exampe

[`var id ( form1 | form2 | form3 )+ `;]

This would cause a grammar def for the three forms to be created, using a unique name, with the same techniques employed for * + ?.

[colm] string manipulation library needs expansion

[colm] some colm parse errors take a really long time to be known

[colm] rudimentary pattern matching during parsing

Needed so we can require that IDs have certain values instead of forcing a keyword.

colm scanner failure: longer pattern interferes with matching of shorter pattern

Test case showing scanner failure has been added to the test suite. Removing the '//' comment pattern will allow the test to pass. The comment prevents the single '/' token from matching properly. Failure is somewhere in the code that reverts to a previously matched pattern when a longer pattern fails. In ragel this is achieved by consuting the 'act' variable.

test/colm.d/scan1.lm

[colm] eradicate the use of replItemList

The replacement, accumulator, and string concat sections of the grammar all use this global (to Parser). This is problematic in the face of recursion. It should be replaced with code that passes the list up through the productions.

[colm] add literal token syntax that allows ] ? * and +

The more we exclude from ` literals the more we are forced to using named tokens. Would be nice to keep these as literals using another syntax.

Maybe l'>*'

[colm] do we really need the literal keyword?

maybe we can eliminate it and just use 'token'

[colm] mutually recursive functions are not possible

This needs to be fixed.

[colm] parse accumulator forces token boundaries on each call

Not possible for a token to span calls to the fragment parser. This imposes a restriction.

Might be done.

[colm] embedding a colm program into a host C/C++ program

This is a desirable feature. Requires returning a tree providing some means of walking the returned tree.

compilation question

Nothing urgent, but now that colm, ragel are all in a single repo, figured I'd try to give it a build.

Which branch and which tool should I be using?

master appears to be development activities.

The README states that it is autoconf-based, but after autogen, configure make, it doesn't find the "version.h" file. Tried again with cmake instead (out of source build), but that fails with not finding <colm/map.h>.

[colm] switch on production names, resulting in errors for unhandled cases

Would be very useful to be able to switch on production names and get warned when a case is added that isn't handled in the switches that cover the production.
comment: Have the switch syntax working, need to check for covering all cases.

[colm] function for stripping ignore tokens

[colm] go grammar results in huge output file

There are two factors leading to huge output files in go.

1. using utf8 directly in the lexer <- affects more
2. patterns for inserting semis

Would be nice to have an additional pass on the input before going to the parser that allows transformations. The same technique could be used both for transformation from utf8 to unicode and for inserting semis. it would be a chain either before or after lexing.

[colm] need a syntax for constructing tokens with a str

Currently using parse, which is just way too much overhead for just a token.

[colm] need to disable commit in recursive strucutures

Running into a case in statement commit where a sub-expression is causing a full commit too early.

This is occurring in closure expressions in the rust grammar

fn from_str() {
    fello();
    parse_i64( |x| -> a { if b { c } } )
}

[colm] one output only per run of the program

Generating multiple outputs from multiple inputs makes writing makefiles more difficult. Move to a model where we run colm (or some other program) multiple times, once for each output file.

Could either

1. roll up multiple outputs of colm into some pack (tarball?) then use separate rules to unpack them.
2. invoke colm multiple times ... the most costly thing (generating a parser) should only happen once.

Probably 1 above is the most simple approach. Colm runs once, but is careful to pack into multiple output files. An external program, or some run of colm unpacks them one by one.

[colm] syntactic sugar for * + ? operators should be done in colm

If we add another bootstrap pass it is quite possible to express these in colm.

Same goes for proposed ( | ) operators in grammar defs

[colm] Really need ?expr in pattern matches.

Pain doing it after the pattern.

[colm] print_xml_ac fails when used at end of commitbt.lm

[colm] export the colm namespace hierarchy using C++ namespaces

In cases where there is a type with the same name as a namespace, augment the type name with the prefix "t_".

[colm] eliminate the circular dependency between toklen and tokend

There are times we set tokend from toklen and vice-versa. Really toklen needs to exist to track the size of partial token matches across buffer blocks, but it ended up getting used as the token length indicator in pdarun. Really this is what tokend is for (as is done in ragel). Need to tease these two vars apart. One: rename toklen and use it only for tracking match length coming from previous buffers. Two: eliminate use of toklen outside of the scanner. Use tokend instead as indication of match length.

[colm] segfault while compiling a colm program does not get reported as failure

If not noticed as the cases rush by then the failure is never reported in the final tally. Need to investigate if the same is true in ragel tests

Best place to solve this would be in the runtests program, then solved once for all kinds of tests.

merge ragel into colm repository

The first order of business here on github is to unify the Colm and Ragel repositories. This is a long-term strategic decision.

My next major goal with these suite of tools is to bring the language-agnostic features of Ragel to Colm. Colm is currently based on C++. It generates C++ and allows you to write a reducer for your languages in C++. I would like to extend this to additional languages in the same way Ragel 7.0 has been extended to support multiple languages. This will entail quite a bit of code sharing, particularly the intermediate language and tools.

Another reason to merge is that Ragel and Colm already share much state machine building and generation code, however the code has been cloned. I would like to eliminate these clones for the sake of code quality. This include the scanner portion of Colm, but also the LALR(1) table construction. Both of these can use libfsm.

I'm probably going to merge Ragel into the Colm repository, considering that between Ragel and Colm, Colm is really the thing that comes first. Ragel now depends on Colm. I understand this could create confusion at first, considering that at this time, most people are interested in Ragel. However, the opposite doesn't make much sense to me.

[colm] can't match against a terminal

need this

String concat hello_world_ext.lm fail

Hello !
Just cloned colm-suite and going through the documentation and testing the "hello_world_ext.lm" we get this error:

colm "hello_world_ext.lm"
hello_world_ext.lm:1:26: hello_world_ext.lm: parse error: hello_world_ext.lm:1:26: parse error

Cheers !

[colm] set iterators constant when iteration requires creation of a temporary

It is currently possible to iterate a global, however a temporary is made and the and the temp is modified. The user then just sees no effect. Need to set the iterator as constant so an error is generated.

[colm] iterating and modifying a struct field does not leave field changed

The following modification does not work. From genf

    Packet->PacketDef = PacketDef
                
    Offset: int = 0
    for FD: pkt_field_def in Packet->PacketDef {
        switch FD.pkt_field_type
        case [`bool] {
            FD.Offset = Offset
            Offset = Offset + 1
        }           
        case [`long] {
            FD.Offset = Offset
            Offset = Offset + 8
        }
        case [`string] {
            FD.Offset = Offset
            Offset = Offset + 4
        }   
        case [list_type] {
            FD.Offset = Offset
            Offset = Offset + 4
        }   
    }

CLANG 9.01 warnings

Hello !
Trying to compile colm on an Android device with Termux we have some errors and warnings:

gcc -DHAVE_CONFIG_H -I. -I../src  -I../aapl  -Iinclude  -Wall -DPREFIX='"/data/data/com.termux/files/home/local/colm"' -Wall -g -MT gen/bootstrap2-parse2.o -MD -MP -MF gen/.deps/bootstrap2-parse2.Tpo -c -o gen/bootstrap2-parse2.o `test -f 'gen/parse2.c' || echo './'`gen/parse2.c
mv -f gen/.deps/bootstrap2-parse2.Tpo gen/.deps/bootstrap2-parse2.Po
In file included from loadboot2.cc:2:
./loadfinal.cc:1682:8: warning: comparison of two values with different enumeration types in switch statement ('enum prod_name' and 'string_el::prod_name') [-Wenum-compare-switch]
                case string_el::Sq: {
                     ^~~~~~~~~~~~~
./loadfinal.cc:2712:8: warning: comparison of two values with different enumeration types in switch statement ('enum prod_name' and 'struct_item::prod_name') [-Wenum-compare-switch]
                case struct_item::InHost:
                     ^~~~~~~~~~~~~~~~~~~
./loadfinal.cc:2793:8: warning: comparison of two values with different enumeration types in switch statement ('enum prod_name' and 'struct_item::prod_name') [-Wenum-compare-switch]
                case struct_item::InHost:
                     ^~~~~~~~~~~~~~~~~~~
./loadfinal.cc:2769:8: warning: comparison of two values with different enumeration types in switch statement ('enum prod_name' and 'root_item::prod_name') [-Wenum-compare-switch]
                case root_item::IgnoreCollector:
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~
mv -f gen/.deps/bootstrap2-if2.Tpo gen/.deps/bootstrap2-if2.Po
4 warnings generated.

/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../src -I../colm  -I../aapl -I../colm/include -DBINDIR='"/data/data/com.termux/files/home/local/colm/bin"'   -Wall -g -MT libragel_la-longest.lo -MD -MP -MF .deps/libragel_la-longest.Tpo -c -o libragel_la-longest.lo `test -f 'longest.cc' || echo './'`longest.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../src -I../colm -I../aapl -I../colm/include -DBINDIR=\"/data/data/com.termux/files/home/local/colm/bin\" -Wall -g -MT libragel_la-longest.lo -MD -MP -MF .deps/libragel_la-longest.Tpo -c longest.cc  -fPIC -DPIC -o .libs/libragel_la-longest.o
parsetree.cc:494:21: warning: variables 'm' and 'numMachines' used in loop condition not modified in loop body [-Wfor-loop-analysis]
                        for ( int m = 0; m < numMachines; )
                                         ^   ~~~~~~~~~~~
mv -f .deps/ragel-main.Tpo .deps/ragel-main.Po

/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../src -I../colm  -I../aapl   -Wall -g -MT libfsm_la-fsmcond.lo -MD -MP -MF .deps/libfsm_la-fsmcond.Tpo -c -o libfsm_la-fsmcond.lo `test -f 'fsmcond.cc' || echo './'`fsmcond.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../src -I../colm -I../aapl -Wall -g -MT libfsm_la-fsmcond.lo -MD -MP -MF .deps/libfsm_la-fsmcond.Tpo -c fsmcond.cc  -fPIC -DPIC -o .libs/libfsm_la-fsmcond.o
fsmap.cc:144:19: warning: 'this' pointer cannot be null in well-defined C++ code; pointer may be assumed to always convert to true [-Wundefined-bool-conversion]
        afterOpMinimize( this );
        ~~~~~~~~~~~~~~~  ^~~~
fsmap.cc:266:19: warning: 'this' pointer cannot be null in well-defined C++ code; pointer may be assumed to always convert to true [-Wundefined-bool-conversion]
        afterOpMinimize( this );
        ~~~~~~~~~~~~~~~  ^~~~
fsmap.cc:548:19: warning: 'this' pointer cannot be null in well-defined C++ code; pointer may be assumed to always convert to true [-Wundefined-bool-conversion]
        afterOpMinimize( this );
        ~~~~~~~~~~~~~~~  ^~~~
fsmap.cc:603:19: warning: 'this' pointer cannot be null in well-defined C++ code; pointer may be assumed to always convert to true [-Wundefined-bool-conversion]
        afterOpMinimize( this );
        ~~~~~~~~~~~~~~~  ^~~~
fsmap.cc:660:19: warning: 'this' pointer cannot be null in well-defined C++ code; pointer may be assumed to always convert to true [-Wundefined-bool-conversion]
        afterOpMinimize( this );
        ~~~~~~~~~~~~~~~  ^~~~
fsmap.cc:716:19: warning: 'this' pointer cannot be null in well-defined C++ code; pointer may be assumed to always convert to true [-Wundefined-bool-conversion]
        afterOpMinimize( this );
        ~~~~~~~~~~~~~~~  ^~~~
fsmap.cc:965:8: warning: comparison of two values with different enumeration types in switch statement ('ValPairIter<PiList<CondAp>, PiList<CondAp> >::UserState' and 'ValPairIter<CondAp, CondAp>::UserState') [-Wenum-compare-switch]
                case ValPairIter<CondAp>::RangeOverlap: {
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fsmap.cc:959:8: warning: comparison of two values with different enumeration types in switch statement ('ValPairIter<PiList<CondAp>, PiList<CondAp> >::UserState' and 'ValPairIter<CondAp, CondAp>::UserState') [-Wenum-compare-switch]
                case ValPairIter<CondAp>::RangeInS2: {
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fsmap.cc:953:8: warning: comparison of two values with different enumeration types in switch statement ('ValPairIter<PiList<CondAp>, PiList<CondAp> >::UserState' and 'ValPairIter<CondAp, CondAp>::UserState') [-Wenum-compare-switch]
                case ValPairIter<CondAp>::RangeInS1: {
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fsmap.cc:1010:9: warning: comparison of two values with different enumeration types in switch statement ('ValPairIter<PiList<CondAp>, PiList<CondAp> >::UserState' and 'ValPairIter<CondAp, CondAp>::UserState') [-Wenum-compare-switch]
                        case ValPairIter<CondAp>::RangeOverlap: {
                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fsmap.cc:1004:9: warning: comparison of two values with different enumeration types in switch statement ('ValPairIter<PiList<CondAp>, PiList<CondAp> >::UserState' and 'ValPairIter<CondAp, CondAp>::UserState') [-Wenum-compare-switch]
                        case ValPairIter<CondAp>::RangeInS2: {
                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fsmap.cc:998:9: warning: comparison of two values with different enumeration types in switch statement ('ValPairIter<PiList<CondAp>, PiList<CondAp> >::UserState' and 'ValPairIter<CondAp, CondAp>::UserState') [-Wenum-compare-switch]
                        case ValPairIter<CondAp>::RangeInS1: {
                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fsmap.cc:1178:19: warning: 'this' pointer cannot be null in well-defined C++ code; pointer may be assumed to always convert to true [-Wundefined-bool-conversion]
        afterOpMinimize( this );
        ~~~~~~~~~~~~~~~  ^~~~
13 warnings generated.

libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../src -I../colm -I../aapl -Wall -g -MT libfsm_la-gendata.lo -MD -MP -MF .deps/libfsm_la-gendata.Tpo -c gendata.cc -o libfsm_la-gendata.o >/dev/null 2>&1
In file included from allocgen.cc:33:
In file included from ./bingoto.h:26:
In file included from ./binary.h:27:
./codegen.h:156:1: warning: 'CodeGen' defined as a class here but previously declared as a struct; this is valid, but may result in linker errors under the Microsoft C++ ABI [-Wmismatched-tags]
class CodeGen : public CodeGenData
^
./codegen.h:84:1: note: did you mean class here?
struct CodeGen;
^~~~~~
class
./codegen.h:170:9: warning: class 'TableArray' was previously declared as a struct; this is valid, but may result in linker errors under the Microsoft C++ ABI [-Wmismatched-tags]
        friend class TableArray;
               ^
./codegen.h:86:8: note: previous use is here
struct TableArray
       ^
./codegen.h:170:9: note: did you mean struct here?
        friend class TableArray;
               ^~~~~
               struct
In file included from allocgen.cc:41:
./switchvar.h:23:9: warning: 'RAGEL_SWITCHVAR_H' is used as a header guard here, followed by #define of a different macro [-Wheader-guard]
#ifndef RAGEL_SWITCHVAR_H
        ^~~~~~~~~~~~~~~~~
./switchvar.h:24:9: note: 'RAGEL_BINVAR_H' is defined here; did you mean 'RAGEL_SWITCHVAR_H'?
#define RAGEL_BINVAR_H
        ^~~~~~~~~~~~~~
        RAGEL_SWITCHVAR_H
In file included from allocgen.cc:44:
./ipgoto.h:106:7: warning: 'IpGoto::NFA_PUSH' hides overloaded virtual function [-Woverloaded-virtual]
        void NFA_PUSH( RedStateAp *state );
             ^
./codegen.h:447:15: note: hidden overloaded virtual function 'CodeGen::NFA_PUSH' declared here: type mismatch at 1st parameter ('std::string' (aka 'basic_string<char, char_traits<char>, allocator<char> >') vs 'RedStateAp *')
        virtual void NFA_PUSH( std::string );
                     ^
In file included from allocgen.cc:45:
./asm.h:63:1: warning: 'AsmCodeGen' defined as a class here but previously declared as a struct; this is valid, but may result in linker errors under the Microsoft C++ ABI [-Wmismatched-tags]
class AsmCodeGen : public CodeGenData
^
./asm.h:53:1: note: did you mean class here?
struct AsmCodeGen;
^~~~~~
class
In file included from codegen.cc:23:
./codegen.h:156:1: warning: 'CodeGen' defined as a class here but previously declared as a struct; this is valid, but may result in linker errors under the Microsoft C++ ABI [-Wmismatched-tags]
class CodeGen : public CodeGenData
^
./codegen.h:84:1: note: did you mean class here?
struct CodeGen;
^~~~~~
class
./codegen.h:170:9: warning: class 'TableArray' was previously declared as a struct; this is valid, but may result in linker errors under the Microsoft C++ ABI [-Wmismatched-tags]
        friend class TableArray;
               ^
./codegen.h:86:8: note: previous use is here
struct TableArray
       ^
./codegen.h:170:9: note: did you mean struct here?
        friend class TableArray;
               ^~~~~
               struct
mv -f .deps/libfsm_la-redfsm.Tpo .deps/libfsm_la-redfsm.Plo
5 warnings generated.

libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../src -I../colm -I../aapl -Wall -g -MT libfsm_la-tabgoto.lo -MD -MP -MF .deps/libfsm_la-tabgoto.Tpo -c tabgoto.cc  -fPIC -DPIC -o .libs/libfsm_la-tabgoto.o
In file included from binvar.cc:23:
In file included from ./binvar.h:26:
In file included from ./binary.h:27:
./codegen.h:156:1: warning: 'CodeGen' defined as a class here but previously declared as a struct; this is valid, but may result in linker errors under the Microsoft C++ ABI [-Wmismatched-tags]
class CodeGen : public CodeGenData
^
./codegen.h:84:1: note: did you mean class here?
struct CodeGen;
^~~~~~
class
./codegen.h:170:9: warning: class 'TableArray' was previously declared as a struct; this is valid, but may result in linker errors under the Microsoft C++ ABI [-Wmismatched-tags]
        friend class TableArray;
               ^
./codegen.h:86:8: note: previous use is here
struct TableArray
       ^
./codegen.h:170:9: note: did you mean struct here?
        friend class TableArray;
               ^~~~~
               struct
binvar.cc:95:97: error: no viable conversion from 'Variable' to 'std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >'
                        "       " << ckeys << " = " << OFFSET( ARR_REF( condKeys ), ARR_REF( transOffsets ) + "[" + string(trans) + "]" ) << ";\n"
                                                                                                                           ^~~~~
/data/data/com.termux/files/usr/bin/../include/c++/v1/string:800:5: note: candidate constructor not viable: no known conversion from 'Variable' to 'const std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > &' for 1st argument
    basic_string(const basic_string& __str);
    ^
/data/data/com.termux/files/usr/bin/../include/c++/v1/string:805:5: note: candidate constructor not viable: no known conversion from 'Variable' to 'std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > &&' for 1st argument
    basic_string(basic_string&& __str)
    ^
/data/data/com.termux/files/usr/bin/../include/c++/v1/string:820:5: note: candidate constructor not viable: no known conversion from 'Variable' to 'const char *' for 1st argument
    basic_string(const _CharT* __s) {
    ^
/data/data/com.termux/files/usr/bin/../include/c++/v1/string:874:5: note: candidate constructor not viable: no known conversion from 'Variable' to 'initializer_list<char>' for 1st argument
    basic_string(initializer_list<_CharT> __il);
    ^
./codegen.h:61:2: note: candidate function
        operator const std::string() { isReferenced = true; return name; }
        ^
/data/data/com.termux/files/usr/bin/../include/c++/v1/string:805:33: note: passing argument to parameter '__str' here
    basic_string(basic_string&& __str)
                                ^
binvar.cc:104:57: error: no viable conversion from 'Variable' to 'std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >'
                        COND_EXEC( ARR_REF( transCondSpaces ) + "[" + string(trans) + "]" );
                                                                             ^~~~~
/data/data/com.termux/files/usr/bin/../include/c++/v1/string:800:5: note: candidate constructor not viable: no known conversion from 'Variable' to 'const std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > &' for 1st argument
    basic_string(const basic_string& __str);
    ^
/data/data/com.termux/files/usr/bin/../include/c++/v1/string:805:5: note: candidate constructor not viable: no known conversion from 'Variable' to 'std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > &&' for 1st argument
    basic_string(basic_string&& __str)
    ^
/data/data/com.termux/files/usr/bin/../include/c++/v1/string:820:5: note: candidate constructor not viable: no known conversion from 'Variable' to 'const char *' for 1st argument
    basic_string(const _CharT* __s) {
    ^
/data/data/com.termux/files/usr/bin/../include/c++/v1/string:874:5: note: candidate constructor not viable: no known conversion from 'Variable' to 'initializer_list<char>' for 1st argument
    basic_string(initializer_list<_CharT> __il);
    ^
./codegen.h:61:2: note: candidate function
        operator const std::string() { isReferenced = true; return name; }
        ^
/data/data/com.termux/files/usr/bin/../include/c++/v1/string:805:33: note: passing argument to parameter '__str' here
    basic_string(basic_string&& __str)
                                ^
2 warnings and 2 errors generated.

make[3]: Entering directory '/data/data/com.termux/files/home/dev/colm-suite-dad/ragel'
g++ -DHAVE_CONFIG_H -I. -I../src -I../colm  -I../aapl -I../colm/include   -Wall -g -MT ragel-main.o -MD -MP -MF .deps/ragel-main.Tpo -c -o ragel-main.o `test -f 'main.cc' || echo './'`main.cc
gcc -DHAVE_CONFIG_H -I. -I../src -I../colm  -I../aapl -I../colm/include   -Wall -g -MT ragel-parse.o -MD -MP -MF .deps/ragel-parse.Tpo -c -o ragel-parse.o `test -f 'parse.c' || echo './'`parse.c
g++ -DHAVE_CONFIG_H -I. -I../src -I../colm  -I../aapl -I../colm/include   -Wall -g -MT ragel-rlreduce.o -MD -MP -MF .deps/ragel-rlreduce.Tpo -c -o ragel-rlreduce.o `test -f 'rlreduce.cc' || echo './'`rlreduce.cc
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../src -I../colm  -I../aapl -I../colm/include -DBINDIR='"/data/data/com.termux/files/home/local/colm/bin"'   -Wall -g -MT libragel_la-parsetree.lo -MD -MP -MF .deps/libragel_la-parsetree.Tpo -c -o libragel_la-parsetree.lo `test -f 'parsetree.cc' || echo './'`parsetree.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../src -I../colm -I../aapl -I../colm/include -DBINDIR=\"/data/data/com.termux/files/home/local/colm/bin\" -Wall -g -MT libragel_la-parsetree.lo -MD -MP -MF .deps/libragel_la-parsetree.Tpo -c parsetree.cc  -fPIC -DPIC -o .libs/libragel_la-parsetree.o
parse.c:7973:20: warning: result of comparison of constant -17 with expression of type 'char' is always false [-Wtautological-constant-out-of-range-compare]
        if ( (*pdaRun->p) == -17 )
             ~~~~~~~~~~~~ ^  ~~~
parse.c:7980:20: warning: result of comparison of constant -69 with expression of type 'char' is always false [-Wtautological-constant-out-of-range-compare]
        if ( (*pdaRun->p) == -69 )
             ~~~~~~~~~~~~ ^  ~~~
parse.c:7987:20: warning: result of comparison of constant -65 with expression of type 'char' is always false [-Wtautological-constant-out-of-range-compare]
        if ( (*pdaRun->p) == -65 )
             ~~~~~~~~~~~~ ^  ~~~
parse.c:8001:20: warning: result of comparison of constant -17 with expression of type 'char' is always false [-Wtautological-constant-out-of-range-compare]
        if ( (*pdaRun->p) == -17 )
             ~~~~~~~~~~~~ ^  ~~~
parse.c:8008:20: warning: result of comparison of constant -69 with expression of type 'char' is always false [-Wtautological-constant-out-of-range-compare]
        if ( (*pdaRun->p) == -69 )
             ~~~~~~~~~~~~ ^  ~~~
parse.c:8015:20: warning: result of comparison of constant -65 with expression of type 'char' is always false [-Wtautological-constant-out-of-range-compare]
        if ( (*pdaRun->p) == -65 )
             ~~~~~~~~~~~~ ^  ~~~
6 warnings generated.

libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../src -I../colm -I../aapl -Wall -g -MT libfsm_la-gendata.lo -MD -MP -MF .deps/libfsm_la-gendata.Tpo -c gendata.cc -o libfsm_la-gendata.o >/dev/null 2>&1
In file included from allocgen.cc:44:
./ipgoto.h:106:7: warning: 'IpGoto::NFA_PUSH' hides overloaded virtual function [-Woverloaded-virtual]
        void NFA_PUSH( RedStateAp *state );
             ^
./codegen.h:447:15: note: hidden overloaded virtual function 'CodeGen::NFA_PUSH' declared here: type mismatch at 1st parameter ('std::string' (aka 'basic_string<char, char_traits<char>, allocator<char> >') vs 'RedStateAp *')
        virtual void NFA_PUSH( std::string );
                     ^
1 warning generated.

libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../src -I../colm -I../aapl -Wall -g -MT libfsm_la-ipgoto.lo -MD -MP -MF .deps/libfsm_la-ipgoto.Tpo -c ipgoto.cc -o libfsm_la-ipgoto.o >/dev/null 2>&1
In file included from dot.cc:24:
./dot.h:39:15: warning: 'GraphvizDotGenOrig::writeStatement' hides overloaded virtual function [-Woverloaded-virtual]
        virtual void writeStatement( InputLoc &, int, std::string * );
                     ^
./gendata.h:425:15: note: hidden overloaded virtual function 'CodeGenData::writeStatement' declared here: different number of parameters (5 vs 3)
        virtual void writeStatement( InputLoc &loc, int nargs,
                     ^
1 warning generated.

[colm] garbage collector needs work

[colm] need a list concatenation operator

List1 . List2 should put List2 on end of List1 (requires same type, or list1 == list2*).

[colm] segfault while parsing a constructor

Segfault while parsing constructor. Removing the trailing " or removing the optional name both eliminate the segfault.

lex
    token DEF / 'def' /
    token id / ( 'a' .. 'z' ) + /
    token SQOPEN /'['/
    token SQCLOSE /']'/
    token COLON /':'/
    ignore / ( '\n' | ' ' )+ /
end

def opt_prod_el_name
    [id COLON] 
|   []

def prod_el
    [opt_prod_el_name id]

def prod
    [SQOPEN prod_el SQCLOSE]

def cfl_def
    [DEF id prod]

cons Def: cfl_def 
    "def x \[x\]"

[colm] might be able to make idempotent for loops automatic by checking if the root was copied

Before an idempotent transformation we could take a copy of the root, then (shallow) compare the copy against the result. Need further investigation into possible reasons why it would not work.

[colm] single literal strings cannot be concatenated

Single literal strings cannot be concatenated because they are not LitPat tokens. They are Literals, and parsed differently in productions and patterns. But they are available in code. This inconsistency is somewhat confusing.

[colm] syntax highlighting deficiency

The parse keyword isn't highlighted properly.

Output <<
        "#ifndef _[f( parse something(Tree) )]

[colm] tree walk list iterator assumes right recursion, want both left and right

The + and * operators are currently implemented with right recursion. The tree walker (generated C++ interface), used to load colm programs, assumes right recursion and can keep only a pointer to do this. We want to be able to support both left and right recursion so this needs to change to a stack.

[colm] alphabet is signed by default, preventing ranges that cross signedness boundary

Colm assumes a signed alphabet, which requires regular language defintions to also be signed. In this mode we cannot specify character ranges that cross the signedness boundary. For example, 0x00 .. 0x80 cannot be used. Single chars work, they are translated to signed, and since the input is also interpreted as signed, we have no problem.

[colm] put position information in the Head struct

This may be a good place for it, rather than in all trees. Heads represent strings, which are the entities that have the position information.