Query doesn't work correctly when using read_callable

[MrPrezident]

Query doesn't work correctly when using read_callable

[MrPrezident]

Actually no need to do the tree edit. I don't know why I included that in the example. Here is a simpler example showing an incorrect match:

# Initialize test query (match identifier with all caps)
query_text = '((identifier) @test (#match? @test "^[A-Z]$"))'
python_query = python_language.query(query_text)

# Original Source Code
orig_src  = b'def myfunc():\n    pass\n'

def read_callable_bytes(byte_offset, point):
    byte = orig_src[byte_offset:byte_offset + 1]
    return byte

# Parse original tree and query
orig_tree = parser.parse(read_callable_bytes)
orig_captures = python_query.captures(orig_tree.root_node)

# Print results
print(f"orig_sexp:{orig_tree.root_node.sexp()}")
print(f"orig_text:{orig_tree.root_node.text}")
print(f"orig_captures:{orig_captures}")

Output:

orig_sexp:(module (function_definition name: (identifier) parameters: (parameters) body: (block (pass_statement))))
orig_text:None
orig_captures:[(<Node type=identifier, start_point=(0, 4), end_point=(0, 10)>, 'test')]

[amaanq]

In this case - a read callable does not have the ability to index/remember the source input, so a query match predicate can't execute on that

[MrPrezident]

Is there a fundamental reason why read callable is not used for query? It seems inefficient to have to pass in the entire source code on every keystroke.

[MrPrezident]

Also, can you point me to where the code is for handling the match predicate?

[amaanq]

Is there a fundamental reason why read callable is not used for query? It seems inefficient to have to pass in the entire source code on every keystroke.

You cannot get the text at a specific point for the match to execute unlike using a buffer, sure, some read callbacks might be able to, but it's not a guarantee hence why it's omitted

Also, can you point me to where the code is for handling the match predicate?

py-tree-sitter/tree_sitter/binding.c

Line 1883 in da6716b

// if there is no source, ignore the text predicates

[MrPrezident]

@amaanq let me know what you think of this PR. I tested this out and it works great for me. I don't see why this wouldn't work for any type of read callback.

[MrPrezident]

@amaanq any love or feedback on this PR?

[amaanq]

Hey @MrPrezident I believe upstream (Rust/Wasm) doesn't keep text from sources that use a "read_callable" as well, so I'd like to implement it there first before here - I'm also a bit concerned about the performance cost about keeping the text in certain cases when conditionally switching the input based on external programmatic sources (e.g. a test in Rust will run forever until a timeout is reached - wouldn't that be expensive to keep? https://github.com/tree-sitter/tree-sitter/blob/6019b7c84cd5a7f580d57bb981315afd9afec826/cli/src/tests/parser_test.rs#L676

[MrPrezident]

I believe upstream (Rust/Wasm) doesn't keep text from sources that use a "read_callable" as well, so I'd like to implement it there first before here

Hmm.. ok. I can try to look into that.

I'm also a bit concerned about the performance cost about keeping the text in certain cases when conditionally switching the input based on external programmatic sources

but it's not keeping the text, it's just keeping the callback, or am I missing something?

[amaanq]

Well it collects the bytes from the callable right, that's what I meant sorry for not clarifying, like in that timeout test it's possible that it might accumulate a lot of bytes before the timeout is hit.

Maybe a flag in core to enable keeping the source text when using a callable would be nice

[MrPrezident]

Yeah but that test you are talking about is doing parsing, not querying. I didn't make any changes that would affect what happens to the callback during parsing.