👀 Editor View

[codeeditapp]

👀 Editor View

[mattmassicotte]

I got interested, so I played around with SPM-izing tree-sitter directly. I was inspired by how Simon did it with Runestone. Note that tree-sitter produces some warnings during its build, so these spill over to the project when used via SPM. This should be fixable, but I haven't yet investigated.

I was also able to do this for a single parser. But that should be enough of a proof-of-concept that it can be done for all of them. I don't have anything public for this yet.

Here's the fork of tree-sitter that supports SPM directly:
https://github.com/mattmassicotte/tree-sitter/tree/feature/swift-package

And here's the fork of SwiftTreeSitter that depends on it, and not on a pre-built binary:
https://github.com/ChimeHQ/SwiftTreeSitter/tree/feature/no-xcframework

So, this is a complete built-from-source tree-sitter library. And, it should now be possible to select languages as desired, and just include them as SPM dependencies. I think I like this approach better, so I'm going to start moving towards this.

But, you should absolutely feel ok deciding to take control whichever parts you need to. This is your project and you need to proceed in the way that makes the most sense.

[mattmassicotte]

@MarcoCarnevali Hello! It looks like you've already found something you want to use, but I thought I'd let you know what we're up to just in case.

My app (Chime) does highlighting (and indentation) using tree-sitter. We've open sourced a low-level library for working with it in Swift.

However, Chime has a huge amount of infrastructure on top of that for its actual implementation. We were actually in the process of pulling out another layer in the system now. A few others have asked about this before, so it was something we wanted to do anyways.

If any of this sounds interesting, let me know and we can discuss more!

[MarcoCarnevali]

I'm actively working on a custom solution built over Highlightr.js!

[mchakravarty]

I am the developer of CodeEditorView that @austincondiff mentioned above. One of the fancier aspects of CodeEditorView is that it already supports an Xcode-like minimap. It also has Xcode-inspired inline message reporting, but that still needs some more polishing.

Syntax highlighting, as somebody mentioned, is necessarily always a two stage endeavour. There is simple highlighting that the editor view can do right away and then more sophisticated (especially, semantic) highlighting that necessarily needs (asynchronous) support from a language-aware framework, such as Xcode's SourceKit or via LSP. For that first step, CodeEditorView doesn't use an external framework to do the initial highlighting (such as Highlightr or TreeSitter); instead, I have written a simple parsing framework on the basis of NSRegularExpression. That has the advantage that it is very fast and doesn't add any build-time or runtime dependencies.

Anyway, I'll happily contribute to this discussion here if that is perceived to be of value.

[lukepistrol]

Current State of the Editor

I just want to publicly share what the current state of development of our core feature - the editor - is.

Current Implementation

Currently we have a temporary solution in place which uses Hightlightr for syntax highlighting. Highlightr is based on highlight.js - a javascript based parser.

Syntax Highlighting

Our plan is to base our syntax highlighting on tree-sitter - an incremental parser written in C. Due to its incremental parsing it is very efficient when processing changes in the syntax tree. And since it is written in C instead of Javascript it is extremely fast and low cost on performance.

We already looked into existing Swift wrappers for tree-sitter like SwiftTreeSitter/Neon and based on some proof-of-concept code it is working pretty nicely.

Existing Libraries vs. Our Own Implementation

SwiftTreeSitter is pretty enhanced already. The only drawback is, that it is based on a closed-source framework tree-sitter-xcframework which we have no control over.

Since the editor is the most crucial part of our app we decided to not rely on dependencies (especially closed-source ones). We will write our own implementation of a tree-sitter wrapper which will definitely take more time initially but will pay off later when we need to enhance features and also when implementing our plugin ecosystem.

We know that this definitely isn't the easy path to go but it's the most future proof way we can think of.

I also want to note that Simon just launched an awesome text editor for iOS & iPadOS - Runestone. He open-sourced the backbone of Runestone as well, and we might take some inspiration from his implementation of tree-sitter too.

Also I want to thank @mattmassicotte for helping me understand the basics of tree-sitter which is quite a complex topic.

Resources:

• Highlightr: https://github.com/raspu/Highlightr
• highlight.js: https://highlightjs.org
• tree-sitter: https://tree-sitter.github.io/tree-sitter/
• SwiftTreeSitter: https://github.com/ChimeHQ/SwiftTreeSitter
• Neon: https://github.com/ChimeHQ/Neon
• tree-sitter-xcframework: https://github.com/krzyzanowskim/tree-sitter-xcframework
• Runestone App: https://runestone.app
• Runestone Repo: https://github.com/simonbs/runestone

[mattmassicotte]

I'm not stunned to see that the xcframework was what tripped you up. It's of course open source, but is pre-compiled. SPM provides limited options here, and this was the only one we could find that offered drop-in functionality.

But is that really the only issue? This seems completely solvable. And whatever solution you come up with for the actual binary tree-sitter dependencies may work better for others. I'd like to find a way to manage the binary dependencies without relying on a pre-compiled binary too!

How do you plan on actually building/bundling those pieces?

[lukepistrol]

I'll dump some resources to consider here:

• https://github.com/mchakravarty/CodeEditorView
• https://github.com/dwarvesf/CodeViewer
• https://github.com/ZeeZide/CodeEditor
• https://github.com/twostraws/Sourceful
• https://github.com/louisdh/savannakit

[mattmassicotte]

I'm not familiar with highlight.js, but it looks like a great tool!

I've taken many different paths along my syntax highlighting journey. Right now, Chime uses a hybrid tree-sitter + LSP semantic tokens.

Tree-sitter's support for incremental parsing gives it really good performance characteristics. However, Chime has to wrap it up with a bunch of extra stuff to make it both lazy (only coloring what the user sees) and asynchronous (computing coloring in the background, if needed). Plus, it supports generalized syntax-tree queries, which we use for structure highlighting (matching if-{-}) and precise indentation calculations. Tree-sitter is awesome, but very complex.

LSP also has a syntax highlighting system called semantic tokens. Glueing together both of these is tricky. But, it's desirable because semantic tokens can provide better information, but is slower and always asynchronous. Tree-sitter isn't as precise, but is faster and can be used synchronously if needed. It also works well in the face of syntax errors, which is the common case during editing.

I was just in the process of pulling out some more of the tree sitter system we use, it will be here: TreeSitterClient . But, I'll need some more time to finish that. And, even when done, that's just perf/convenience around the system, not a full highlighting implementation...

[mattmassicotte]

Just a quick update: after some thought, we decided that it just made more sense to build a project around working with syntax as a whole. The core tree-sitter abstraction is now available. https://github.com/ChimeHQ/Neon

[mchakravarty]

@austincondiff wrote,

Not via LSP, what does that imply for new languages wanting support in our editor? The idea of the LSP is effortless support. Would that not increase the barrier to entry?

It does imply that the editor needs basic awareness (= configuration) for every supported language. That is quite standard AFAIK and I don't see how to get around it.

The issue is simply the following. There are some things where you want an immediate per keystroke response. For example, if you comment out a line, you want syntax highlighting to reflect that immediately without any delay. (That might sound simple, but in the presence of nested comments, multiline strings, etc. it is quite involved.)

On the other hand, there are things that might deserve highlighting, but that do require a sophisticated language-dependent analysis of the code, such as whether an identifier is locally bound or globally, or whether an identifier refers to a variable or a function. (In C even something like whether an identifier is a typedef or variable, which requires knowing all included headers including macro expansion.) All this requires the kind of static code analysis that the frontend of a compiler performs.

I can't see how you can have both, the immediate response on a per keystroke basis and the sophisticated code analysis-based highlighting, in one system. It requires two: (1) fast, synchronous syntax analysis in the editor itself and (2) slow, asynchronous code analysis via LSP, SourceKit, or a similar system. You can see that in Xcode, how some highlighting is immediate and some of it happens with a delay.

[mchakravarty]

@austincondiff wrote,

so we need to support things like code completion, hover suggestions, decorations like color swatches next to hex values, etc. Is this something that we can easily control and extend with CodeEditorView?

These are features that still need to be implemented in CodeEditorView. Code completion is of course crucial. I am less sure about hover suggestions — I guess, I am partial to the way Xcode behaves. However, it is certainly something to discuss as an optional feature.

Important, from an architectural POV, is that all this is a lot of data, which is also frequently changing, that needs to be fed into the editor view and it needs to lend itself to fast lookups. Moreover, completions need lookups on prefixes and/or close matches. Hence, this needs some thinking about appropriate data structures.

[lukepistrol]

Basic implementation of tree-sitter using SwiftTreeSitter and STTextView is now available on feature/new-editor branch.

Contributions are welcome on the editor package CodeEditTextView! Documentation is available here.

Thanks to @mattmassicotte for creating SwiftTreeSitter and for helping me with questions.
Also big thanks to Marcin Krzyzanowski for creating STTextView as an alternative to NSTextView.

Note that STTextView is still in active development and if some features are missing you might consider contributing there as well.

[austincondiff]

I've spoken to the developer of CodeEditorView and added mchakravarty/CodeEditorView#43 and mchakravarty/CodeEditorView#44. I also left some comments on mchakravarty/CodeEditorView#3.

[austincondiff]

Let's reopen because this editor is just a start. I assume based on PR comments that it will not support #46 among other things like diff, code completion, messages, validation, etc. all of which are being worked on or supported by https://github.com/mchakravarty/CodeEditorView.

[MarcoCarnevali]

@mattmassicotte I'm currently using a wrapper around highlightjs but happy to change it to something better! I would definitely love to discuss more on that, seems interesting!

[MarcoCarnevali]

That's definitely something we need, the current system it's not really scalable and good performances wise. What do you think would the timeline be like?

[mattmassicotte]

Pulling out stuff like this is always a little easier said that done, haha. I'm making progress, but I have some higher-priority stuff going on this week. I'm hoping by next week!

However, I do want to stress that this is not a drop-in highlighting library. It just makes it much easier to connect a text storage system (like one backed by NSTextStorage) to tree-sitter in a performant way. The incremental stuff will just work, and could be helpful right from the beginning. But, background processing is necessary for low-latency with large documents. That is much more complex, and requires maintaining a stable and thread-safe view of your text across user edits. I do this a few different ways, but BufferingTextStorage from TextStory may be interesting here.

Progressive highlighting (multiple passes of varying latency/quality) also really helps for performance and user perception, but is an entirely different thing.

Do you have any highlighting performance/quality targets for your first release?

[MarcoCarnevali]

Do you have any highlighting performance/quality targets for your first release?

Good question! We've recently merged a new PR with a custom editor built around highlghtr, I would say the performances are good/usable for now so anything that matches that would be very good! But we would definitely need something more performant in the future

[MarcoCarnevali]

Just a quick update: after some thought, we decided that it just made more sense to build a project around working with syntax as a whole. The core tree-sitter abstraction is now available. https://github.com/ChimeHQ/Neon

That's awesome, I'll definitely try this soon, is there an usage example? and btw our current solution uses a NSTextStorage to have a performant real-time highlighting

[mattmassicotte]

I took a bit of time to study Highlightr and highlight.js, just to see what was up. I'm really glad to hear it's working out!

If you do want to start looking for higher performance/correctness in the future, I'm lurking in the discord, and generally available. I would love to help out! And in the mean time, I'm going to continue, here and there, pulling stuff out of Chime into Neon. The exercise has already been helpful. And, as I do, I'll put a little more effort into explaining how to use it.

[austincondiff]

FYI, I just pinned this issue and updated the description to track other issues.

[austincondiff]

There is simple highlighting that the editor view can do right away and then more sophisticated (especially, semantic) highlighting that necessarily needs (asynchronous) support from a language-aware framework, such as Xcode's SourceKit or via LSP.

@mchakravarty Not via LSP, what does that imply for new languages wanting support in our editor? The idea of the LSP is effortless support. Would that not increase the barrier to entry?

[austincondiff]

In talking about this with @MarcoCarnevali who has put in place our current temporary editor solution, it sounds like we are waiting until we have a clearer understanding regarding extension architecture (#76, #180).

@mchakravarty so we need to support things like code completion, hover suggestions, decorations like color swatches next to hex values, etc. Is this something that we can easily control and extend with CodeEditorView?

If so, I personally think we should start building this out now so we will be ready for extensions when it is ready. Thoughts?

[austincondiff]

Thanks a lot @mattmassicotte! This is great! With this in mind, do we want to follow a similar path for our editor @lukepistrol?

[s0me0ne-coder]

Consider adding these issues here
#701
#699

[austincondiff]

@s0me0ne-coder Those two don't really pertain to the editor view itself.

[matthijseikelenboom]

Closing because we have mostly implemented this