Full-text search

[atomicdata-dev]

Full-text search

[AlexMikhalev]

@joepio may be https://endler.dev/2019/tinysearch/ is a better approach?
What is the type of data we are searching for?

[joepio]

Thanks for sharing, @AlexMikhalev! I think tinysearch might be a bit too static for this usecase - the index should be updated very frequently (every time a user creates a commit).

I'm currently considering Tantivy, a lucene alternative, as it can be fully embedded and seems really fast.

[jonassmedegaard]

qualities that I appreciate in web-based search engines are a) support for offline search (which also means less round-trips and less tracking while online), and b) support for OpenSearch

For a) I generally like lunr - and seem to recall that there's a crate integrating that.

[jonassmedegaard]

yes, not the original Lunr but a derivative: https://crates.io/crates/elasticlunr-rs

[joepio]

That one's also statically indexed, which is not a good option for atomic-server, I'm afraid. I need incremental indexing, so new commits are also searchable as users update stuff.

[AlexMikhalev]

Sonic is also statically re-building fst trees - they don't support incremental indexing either, but sonic hides it from the user.

[joepio]

Hmm, interesting. Maybe if indexing is fast enough (takes <60secs for 1gb atomic data) then we could do it max once every minute.

[AlexMikhalev]

qualities that I appreciate in web-based search engines are a) support for offline search (which also means less round-trips and less tracking while online), and b) support for OpenSearch

For a) I generally like lunr - and seem to recall that there's a crate integrating that.

Do you see atomic server deployed on server or "at the edge" - at users device? if it's on users device it's allready "offline first".

[jonassmedegaard]

• on an untrusted system (i.e. a remote server) I would prefer efficient network traffic and an option to do client-side search (i.e. hand me a lookup table and JavaScript code to traverse it, allowing me to only "reveal" the hits I want to get, not the queries leading me to the hits)
• on a separate trusted system (i.e. a local server) I might prefer efficient network traffic or efficient processing, depending on network bandwidth versus capacity of the server (might be a corporate rack server or a 32-bit ARM device)
• directly on a user-facing system (i.e. a laptop or a phone) I would prefer efficient processing (think battery-powered) and direct APIs (no JavaScript or web browsing but socket-based more direct connections

[joepio]

I made quite a bunch of progress in one night, and it's actually working quite well! I'm really amazed that it took so little time to implement. Hats off to tantivy for creating this awesome library.

[joepio]

Allrighty, tests are passing, time to release a new version! It's pretty decent, but there's still room for improvement. See #210 for some of the things that can be improved upon.