wbsoft/parce

:deciduous_tree: Python lexer that can remember tokens and state and only reparse changed parts of a text document

parce's Issues

Use async/await

It would be nice if asyncio and the new async/await features of Python could be used. Still the work should not be done in the main thread but a background thread (because parce needs to be usable for GUI applications that have the GUI in the main thread), and signals or events could still be used when a job is done. But it would be nice if that was handled by awaitables. Interaction with parceqt and/or other GUIs is important.

This thread is created to collect (my own or others) thinking about this subject and determine a way to go.

GPL for use as a library?

Have you considered a GPL exception for usage as a library in a larger application?

In python 3.9 `'text' in context` or `<lexicon> in context` no longer works

It seems Python 3.9 changed the semantics of __contains__.

A Token can be tested for equality with a string, in that case the Token's text is tested for equality with the string.

Likewise, a Context can be compared with a Lexicon, in which case the Context's lexicon is test for equality with the specified Lexicon.

This used to work for the in operator of lists and tuples, "text" in context or some_lexicon in context used to work, because __eq__ was called on the items of the sequence. But starting with Python 3.9, it seems the in operator calls __eq__ the other way around, not on the sequence's item, but on the item that's searched for in the sequence.

Although we could keep stuff like:

if token == "text":
    do_something()

we can't use the in operator anymore, as it calls "text".__eq__(token) instead of token.__eq__(text).
This also has consequences for the query.__call__() method, which currently depends on the pre 3.9 behaviour.

Better integration of Transformer with Document

A Document is neatly integrated with a tree builder which updates the parce tree on every change.
This should also be the case for transforming jobs.

Without cluttering Document further (it is already too cluttered!) there should be a clean possibility to either/or:

• connect a transformer with a Document
• know if a transform is busy or not
• get the transformed object for a Document
• await the now being transformed object
• get a callback if a transform is ready
• etc