okamiio/nomure

A proof-of-concept graph Database on top of FoundationDB

nomure's Issues

Implement full text search engine

Sadly I'm don't know too much about string index/searching/match algorithms so I need to rely on another solutions for it, since is an important part of any database

Tantivy looks really promising since works like lucene so is plugable, I need to explore the futher posibilities

The FDB record layer implements a full text search indexing technique whichs I think I can use it natively, so I can have all in a single place

Features/Roadmap

Features/RoadMap

An issue describing short and long term features as well the roadmap

Short term goals

• Property Values

  • Primitive types List
  • I18N Strings (for this we need string indexing in order have sense, otherwise is just a property with and @ and the language at the end, so far theres no solutions for this on the erlang/elixir ecosystem, I don't know too much about the topic either, so futher discussions/PRs are welcome)
  • Enums
  • Dictionary (Use a node)
  • Datetime
  • Time
  • Date

• Index

  • Datetime
  • Enum
  • Time
  • Date
  • List
  • String (Exact match)
  • String (Hash match)
  • String (Full text search)
  • Integer
  • Float
  • Dictionary (Use a node)

• Node implementation

  • Update
  • Delete
  • One to One relationships! (User <--> UserConfig)
  • Set data
  • Get data
  • Index property edges
  • Inverse Node support (relation.uid, "edge_name", node.uid) = node_relation_edge.uid

• Edge implementation

  • Index property edges
  • Update
  • Delete
  • Set data
  • Get data

• Query support

  • where

    • Datetime storage { year, month, day, seconds }
    • > KeySelector.first_greater_than(key)
    • < KeySelector.last_less_than(key)
    • == get_by_function(property, value, function = Function.equal)
    • >= KeySelector.first_greater_or_equal(key)
    • <= KeySelector.last_less_or_equal(key)
    • and
    • or

  • pagination support (cursor based)

    • after Transaction.get_range(after_cursor, end, limit)
    • before Transaction.get_range(before_cursor, end, limit, reverse=True)
    • limit
    • first Transaction.get_range(start, end, limit=first)
    • count

  • sorting

    • order_asc index query
    • order_des index query

  • reverse node

  • functions

    • min atomic
    • max atomic

• Ztandart data compression

  Compress long sized strings in order to save space, unnoticeable performance impact

Long term goals

• Types check!

• More tests

Maybe

• Network Protocol implementation

  • GraphQL (standard query, mutation language) [this uses the connection
    features of GraphQL to make it easier to query graph data]
  • Opengraph (complex query for things like recommendation engines etc)

• query optimization/planning, what most databases does.

Ecto Support

Well as the title says, could be great having ecto support, the problem with Ecto is that it works with the SQL mindset which here does not make too much sense overall

Minimize the use of keys related to index operations

Right now any data type index operation could face a duplicate key prefix which is quite inefficient

Let's take for example 2 record

user_a = %{name: "Artorias", deaths: 50}
user_b = %{name: "Chosen Undeath", deaths: 50}

the index result of the deaths record would looks like

("user", "deaths", 50, user_a_node_uid) = ''
("user", "deaths", 50, user_b_node_uid) = ''

As you can see we have repeated the prefix "node_name", "deaths", 50 which is quite long by itself, containing bit_string data as node_name, property_name and property_value, so this proposal is to change it this way

("user", "deaths", 50, random_id) = [user_a_node_uid, user_b_node_uid]

we use the random_id in order to "extend" the index, since FDB has a limitation on value size, so when it reach this size we split the index in order to add more uids into the result of the index

This way a single key prefix can contain a quite big amount of node uids

I don't think it will have any repercution, normally on any index you need to query them all in order to bring any query result

Schema definition on the database side (Strongly Typed!)

Right now Nomure is somewhat schema less, I do think adding types is a must for any database, tho is not an easy task and need a lot of coordination in order of making in runetime schema updates etc... in conjunction with FDB