Painfully slow on 16MB (minified) JSON

[corneliusroemer]

Painfully slow on 16MB (minified) JSON

[ynqa]

I suppose some async/await could help with not freezing the TUI
The biggest usability problem for large files is that keystrokes are delayed

@corneliusroemer You're absolutely right; there's a potential to degrade user experience.

jnv utilizes the mechanisms from promkit, where I'm considering implementing a system that can handle keystrokes and data structure editing concurrently using async/await at promkit (I've tried this before, but it's been a bit challenging and is currently pending).

[ynqa]

@corneliusroemer Thanks for your reporting! 👍

Could you share the actual JSON, if possible? I'd like to test it myself and understand the current data size limits.

[ynqa]

@corneliusroemer fx and jless are indeed famous for their ability to display content very quickly.

On the other hand, jnv relies on the processing speed of libjq, so it can take time to handle very large JSON files. For example, you can see how long it takes by running cat test.json | jq.

I believe there can be a division of use cases: utilize fx or jless when you want to get an overview of the large JSON, and use jnv when you want to try out complex queries with jq for the mid-size JSON.

fx and jless do allow for field lookup with .path.to.field and have some simple DSL query capabilities, but I understand that they cannot handle complex queries like jq can.

• https://fx.wtf/advanced-usage
• PaulJuliusMartinez/jless#70

[corneliusroemer]

Sure, here we go: test.json

I've meanwhile found fx to be very good for such large JSON files, don't even notice any slowdown at all.

[corneliusroemer]

Thanks for the detailed reply. I suppose some async/await could help with not freezing the TUI - but the added complexity may well not be worth it. The tool would still be usable with the 16MB file if it simply didn't show output until say 3s after the last keystroke.

The biggest usability problem for large files is that keystrokes are delayed.

[rocurley]

Hey, I wrote a similar project a while ago: Jex. While the UI looks a lot better here, I didn't have this problem. I believe the difference is that I parsed the JSON once at load and then used the parsed structure from there on out.

Looking at your code, you already parse the output to json so you're in a pretty good spot there. Even if you don't think it's worth it to switch your rendering code to use JQ's internal representation, I bet you could convert from JQ's representation to yours a lot faster if you didn't go through a string representation between the two.

[rockwotj]

I would recommend Jaq, it's native rust and you can won't have to convert each time, and was built to fix some speed cases with many filters compared to jq.

https://docs.rs/jaq-interpret/latest/jaq_interpret/

[ynqa]

@rockwotj Regarding the replacement with jaq, I have created #24, so please provide your opinions/reactions there 🙏