Large generated code size, long compilation time

[dashbitco]

Large generated code size, long compilation time

[bruce]

Thanks @michalmuskala. After a few changes to commonly used combinators, I'm already at < 1MB. Closing this issue while I continue to improve things. I'll try to put in a PR later to improve the docs regarding trade-offs on inlining vs defparsec.

[michalmuskala]

Taking a quick look, one thing that jumps at me is the use of skip_ignored. This is a complex combinator that is applied repeatedly in many places - this means that in each place, the entire combinator has to be compiled. Assigning a combinator to a variable and using it multiple times means it will be compiled multiple times and inlined at each use - which can be expensive for large combinators.

[bruce]

@michalmuskala What's the right way to do something like that? Use defparsec/2 and parsec/1? Should I just do that with all the grammar entities? For some reason that felt like it would create more code rather than less (but, I've been very lax in my reading of the actual implementation).

[bruce]

Definitely on the right track; removing skip_ignored entirely, down to 8.7M. Going to try using defparsec instead now (I do need to skip those chars, of course). Will report that as well.

[bruce]

9.8M adding back in skip_ignored, but converting it to defparsec/2 and parsec/1. I'll try this with the rest of the grammar and see how things change.

(FWIW, having small combinators to test externally and use as utilities is a win anyway.)

[michalmuskala]

Yes, using defparsec and parsec for commonly repeating things should decrease the code size - the parser will be compiled just once. It will be probably slightly less performant, but that needs measurements anyway.

[josevalim]

This looks great @bruce! Probably the biggest grammar written with NimbleParsec so far. I would love to see how it compares speed-wise with the xrl+yrc parser, especially because NimbleParsec doesn't require converting to a list, which should make it use less memory too.

Regarding compilation times, I have played a bit with your fork and it seems the time to compile .xrl+.yrl to beam versus nimble_parsec to beam is roughly the same right now. The .beam size of a nimble_parsec is a bit bigger though (250k vs 330k), so you did a great work on abstracting the repeated pieces with defparsec+parsec.