db47h/ragel

Go driver for ragel scanners

ragel's Issues

Allow customization of the buffer size

The arbitrary 32KiB buffer size can be problematic in some cases. It's not unseen to have very large comments in source code and for successful scanning, they need to fit in the buffer.

This can be done with no incompatible changes to the API via functional options or a public variable.

The buffer size could start with a reasonable default, like min(requestedSize, 32768) and expand dynamically up to requestedSize. The upper limit would be math.MaxInt.

Generate utf8.rl from Go's unicode package

No need to have ruby installed and consistent unicode version in all parts of the application.

Scanner.Next should return an actual error

The idea is to distinguish lexing errors from I/O errors.

On the other hand, if an I/O error occurs, s.abort is set to true, so any subsequent calls to next will return an EOF token.

Clients that need to distinguish error types may consider the built-in Error token as fatal and implement their own soft error token in their ragel file.

Something possibly more versatile is to have any returned error token added to a list of errors (with the actual error interface) and add ErrorCount and Errors method to Scanner.

Concurrent scanner

A concurrent version that can be applied on top of cdb0e77 is available here: https://gist.github.com/db47h/51579601ceb620aaf1427857e37da691

BenchmarkNext_largeishFile reports 6 ms/op in concurrent mode vs. 1.17 ms/op with a FIFO on an AMD Bulldozer CPU. On an Intel Skylake CPU, the performance drop is less dramatic at 2.53 ms/op vs. 1.22 ms/op.

The implementation will stay non-concurrent until I see a proper use case where the parser gets busy enough to offset the performance drop caused by the IPC (if that's even possible).

Eventhough the AMD Bulldozer architecture is pathetic in terms of IPC, I am loath to implement optimizations that only marginally benefit higher end hardware while severely penalizing the others.

Add stack handling

The current implementation does allow fcall out-of-the-box. Need to add stack and top.

Time for a v3 and rethink SaveVars/GetVars?

EDIT: stack handling can be implemented via a custom ragel.Interface

Readme.md example WCHAR vs UTF8 imported machine

word.rl:27:19: graph lookup of "uletter" failed

had to change line 22: from

include WCHAR "utf8.rl";

...to...

include UTF8 "utf8.rl";

Possibly wrong behavior when failing to match alternatives at EOF

suppose the following grammar:

'/*' { s.Emit(ts, slashStar, string(data[ts:te])) };
'//' { s.Emit(ts, slashSlash, string(data[ts:te])) };

where a single / has no meaning.

Trying to scan / will likely fail with an unexpected error (i.e. a very unhelpful error).

Same for '/ //': we'll get a 1:2: invalid character ' '. ragel's built-in error handling might be useful here.

Error "token too long" when buffer size == input size and single token in input

This sample code will fail with a token too long error (input buffer overflow):

input := "42"
l := lexer.New("", strings.NewReader(input), ragel.BufferSize(len(input)))
tok, err := l.Next()

Workaround: add 1 to buffer size.

Fix: Do an EOF check before complaining.

scanning `"...` with no terminating double quote fails to report an error

it simply returns EOF. This should be handled as part of the buffer overrun check.

Not in buffer overrun check, but in

ragel/scanner.go

Lines 305 to 310 in 9735ac3

	if err != nil {
	s.abort = true
	if err != io.EOF {
	(*State)(s).Errorf(0, true, err.Error())
	}
	}

where we need to check that if the current state != start state, then something is missing.