A Json searcher support fast search on defined entities
OS: *Debian (Docker)
sbt clean test
sbt clean run
Please be aware, the docker image will be built from scratch, which means the building time for the first run would be super long....
after the first run, two external volume would be created to cache all the loaded dependencies to make the next run much faster;
./auto/test
./auto/run
________________________________ ______________________________________________________
| Console | | Json-Searcher-Core |
| | | __________________________________ |
| atStart | |---------|-----> |- materialise denormolised-data | |
| atInstruction | | | |- preload indices | |
| atSearchableFields | | | |__________________________________| |
| atStartSearch -------------|--------| | | |
| atEnterSearchTerm --->| | |--|-->A In memory SearchStore <-------| |
| atsEnterSearchValue |<---| | | | | |
| atOperateSearch <---| | | | | |
| | | | | | |
| |_____________SearchTerm________| | | |
| | | | |
| stdout <---------------|----SearchResults-|----------------| |
|_______________________________| |_____________________________________________________|
- Searhable Data should be denormolised for fast search
- Data should be indexed by the available search key
- Searchable Data should be materialised into memory before SearchOperation
- Console should have finite states and each state should know its previous restatable state
- Exceptions should not break the user experiences and should have instructive notifications
Although diskIO by streaming in-memory-denormolised data into multiple partitions in disk by hash key for a given search key could guarantee the memory would not be explored given a big enough disk volume;
This appraoch would slow down the search dramactically even with HDFS support;
The ideal way to optimise this large volume data scenario would be applying distributed computing in a cluster with mutiple machines; each instance could be working on a partition of searchkey and later on assemble the result to serve back to the searcher; It is a well-known map-reduce application senario;
-
Organisation:
All Fields are required
-
Tickets:
Optional Fields:
[type, description, assignee_id, organization_id, due_at]
-
User:
Optional Fields:
[alias, verified, locale, timezone, email, phone, organization_id]
- on user granularity:
UserSearchResult { user: User, org: Option[Organisation], assigned_tickets: List[Ticket], submitted_tickets: List[Ticket] }
- on ticket granularity:
TicketSearchResult { ticket: Ticket, org: Option[Organisation], assignee: Option[user], submitter: Option[user] }
- on organisation granularity:
OrganisationSearchResult { org: Organisation, users: Array[User], tickets: Array[Ticket] }