cgrand/enlivez

enlivez's Issues

Better derivatives

According to #6, result sets are not real sets they have a. Key. However this information is not leveraged by derivatives: a change is conveyed as a retraction and an addition of tuples sharing the same key. Reporting just the addition is enough because key unicity implies that this addition is an upsetting.

Likewise retraction just need to specify the key.

This should simplify derivatives.

Day #11

Event handlers are working.

Now, on to component state!

Day #1

Working on the working prototype. The goal is to get the lib in the hand of devs asap even if the implementation is utterly naive and inefficient.

Some realizations of the day: component identity, query maps in arguments and subscriptions.

Component identity

A component is fully identified by the arguments that were passed to it (if they change then the component is replaced, not mutated) and its mount point (relative to its parent).
However one must not panick at these words: when a component represents a todo-list item, its argument is the list item entity id which is stable. Changing the item title would not replace the component, just refresh it (or some subcomponents).

Query maps as arguments

When using query maps as a kind of destructuring (for example {[title status] ::item/attrs}) in arguments there's the risk the resulting query may return 0 to many rows where we expect only one.

Too many rows: this can be prevented by analysis of the query and attributes cardinality.

For zero rows returned (eg because the entity as a missing field and no default has been provided), this has to be dealt with at runtime. Great, but how? Through a "default default value" (nil being a candidate)? Warning? Exception?

Subscriptions

So in our model, the database is not available as a first class value (except in transaction functions).
One has to subscribe to a (rel-returning) query. If the query returns N columns then the subscribed handler receives a relation with N+1 columns: the last one is a boolean, true for addition, false for retractation.

I was uneasy with using d/listen! to implement it because you have to take Javascript "thread model" in account to be sure that there's no lost transaction between the initial query (used to bootstrap the live-query) and the first delta. (There are 2 atoms: one for the listeners, one for the db...)
While it can be worked through, I opted for a simpler and more general solution that works on the JVM too. This solution is to have only one listener (created at db init), and all subscriptions are reified in the db itself!
Thus a live query kicks off the moment its subscription is transacted!

;; Datasource is only accessible through subscription

(def ^:private conn
  (doto (d/create-conn {::child {:db/valueType :db.type/ref
                                 :db/isComponent true
                                 :db/cardinality :db.cardinality/many}})
    (d/listen! ::meta-subscriber
      (fn [{:keys [tx-data db-after]}]
        (doseq [[eid q f] (d/q '[:find ?eid ?q ?f :where [?eid ::live-query ?q] [?eid ::handler ?f]] db-after)]
          (f (d/q q db-after)))))))

(defn subscription
  "Returns transaction.
   Upon successful transaction, the "
  "Subscription is immediate: upon subscription f receives a (positive only) delta representing the current state."
  [q f]
  [{::live-query q
    ::handler
    (let [prev-rows (atom #{})]
      (fn [rows]
        (let [prev-rows @prev-rows
              added (reduce disj rows prev-rows)
              retracted (reduce disj prev-rows rows)
              delta (-> #{}
               (into (map #(conj % true)) added)
               (into (map #(conj % false)) retracted))]
          (when (not= #{} delta)
            (f delta)))
        (reset! prev-rows rows)))}])

Day #7

Thinking out loud about what happens at runtime (todo list example).

1. A new item is added to the database.
2. It triggers the LQ for the with, this LQ emits a path [0] [] [12345] ["title" false]: [0] is the key in the root fragment (the one with the <ul>) to identify its first and only child: the with. Then [] is additionals values (always empty fro fragments) then a key anew [12345] this time its the eid of the new item, it's the identifier of the iteration, and then additional values.
   1. The thing that bugs me at the moment is that terminal (foreign) templates don't have queries and thus need support from the fragment and with to render: the with must pass its "additional values" to the fragment which must pass them to its terminals and implement logic of checking which values have changed!
   2. Let's imagine a world where we get 3 paths [0] [12345], [0] [12345] [0] ["title"] and [0] [12345] [1] [false] then, in this world, we have full selectivity and all the diffing logic is removed from fragments etc. We also don't have the keys/other vals split

Day #14

States, as always. Now it's datascript that is buggy:

(d/q
  '[:find ?a ?v ?s
    :in $x
    :where
    [$x ?a]
    [(vector ?a) ?v]
    [(str ?a) ?s]]
  [[1] [2]])
#{[1 [2] "1"] [2 [2] "2"]} ; should be #{[1 [1] "1"] [2 [2] "2"]}

Tables with dynamic columns

Deriving negation

You just swap added and retracted.

Unstub DOM

True dom, remove reagent/react

Collapsing empty sections

Imagine you have a header followed by a list and that you don't want to display the whole section where there's no entry in the list.

(ez/for {:db/id list :list/name name :list/entries _}
  [:div 
    [:h1 name]
    [:ul
      (ez/for {:db/id list :list/entries {:item/title title}}
        [:li title])]])

is the best one can actually do.
In the absence of rules the query must be duplicated (and this can be a maintenance woe if some filtering is going on).

How could this be avoided syntactically?

Day #10

It renders!

Both rendering and derivations are not the real thing (resp. using reagent and sets difference) BUT updates are local while we have a global model.

Plan for tomorrow: event handlers.

Recursive templates

Let's imagine displaying a tree.

(ez/deftemplate node [self]
  (ez/for {:db/id self [name] :node/attrs}
    [:div name
       (ez/for [[self :node/children child]]
         (node child))]))

Currently it fails as the system tries to get a tree of queries out of this and this tree ends with an infinite depth.

Obviously recursive templates may be dangerous in the same way ... is dangerous in datomic's pull. A recursion limit option on the template may be good.

Day #9

The proper clj/cljs split has begun. Updates work on a pseudo dom (that's going to be wrapped in reagent for a start).

Day #12 and #13

Working on state, root state works, nested states have conflicts.

I stumbled on a couple of CLJS bugs: https://clojure.atlassian.net/browse/CLJS-3092
and https://clojure.atlassian.net/browse/CLJS-3093

Why no public or ambient conn or db?

No db to enforce all reads to be declared (no ad hoc reads hidden in Clojure code) at the template level (but we can have non-dom components: IO and History API fro example).

No conn to enforce that the only way to alter state is through tx-data returning handlers.

tx function is the only place where random queries can occur.

TODOs

• constant fragments in loops
• less awkward predicates
• sorting
• proper shadowing (lexical scope)
• inclusion

Unique composite key on rows

Given a query how do I determine a subset of columns that are enough to uniquely identify a row?

`:state` maps shouldn't be query maps

Query maps make for awkward state maps for two reasons:
1/ A query may return 0, 1 or n results but for a state entity we expect strictly one component
2/ Defaults are defaults, if an attribute value is retracted then the default is back on; this is a counter intuitive behavior to me.

State maps should be more like entity maps, they should only use (in values) symbols which are part of the component's key, so that the initialization state is stable.

This would also allow for many-valued attributes to be set too.

Is `:state` needed at all?

Relational expressions

A relational expression is an expression. I use the word expression in contrast to statement (see https://en.wikipedia.org/wiki/Expression_(computer_science) for example). Expressions nest, statements don't.

In typical datalog, clauses don't nest. To me it's a loss in expressivity because it means that you must name a lot more of variables.

In Datalog a predicate has no direction: it has no output or input attributes, all attributes are full duplex so to say. However while a predicate may not technically have a direction, it generally has a semantical one: that's why by default the last argument is considered to be the return attribute.

Unstub derivatives

Currently derivatives are mocked by brute force diff

Day #6

Ok, so I have figured the 3 basic types of templates (I can think of at least one more but less essential):

with

It's for looping, it has one query, one child, an optional sort.

foreign

Foreign nodes are leaves of the template tree: that's where EZ stops, usually foreign nodes will render to attributes or text nodes, but they may return full blown DOM nodes (eg to include a React component).

Foreign nodes have no child, but vars they depend on and a function to render and update.
Foreign components have state.

fragments

Fragments are fill-in-the-blanks html. They have no query, they don't use vars on their own, they do have children (one for each blank).

Summary

type	with	foreign	fragment
query	yes	no	no
vars	no	yes	no
children	1	0	N

Day #2

What is the runtime model of the UI? A hierarchy of components.

When a delta arrives we get a long row, which can be split into stages (see rules in #4) , each stage mapping to a component into the hierarchy.
If there's no component then it should be instantiated.

Let's imagine (syntax with #5):

[:ul
 (with [[item :item/title]]
   [:li (:item/title item) " is " (if (:item/done item) "done" "to do")])]

Here I count 4 templates:

• outer with
• inner with
• title
• done

Which would mean 3 queries

• [[item :item/title]]
• inner with has no query and should never have one
• [[item :item/title] [(get-else item :item/title nil) title]]
• [[item :item/title] [(get-else item :item/done nil) done]]

The last two would need to be differentiated by adding an id (the mount point id for example): [[item :item/title] [(ground :mount_421) mnt] [(get-else item :item/done nil) done]]

Thus such code should work:

(defn add [{:as component :keys [n instantiate]} row]
  (if (seq row)
    (let [k (subvec row 0 n)]
      (update-in component [:children k]
        (fn [child]
          (add (or child (instantiate k) (subvec row n))))))
    component))

It's too functional/persistent for something which is linear.

aggregations

aggregations once implemented may suffer of the same query duplication issue as #36

Day #8

Small victory

=> (let [adom (atom nil)
         root (fragment-component [:ul :_]
                [[[1] (with-component 
                        (fragment-component [:li :_ "is" :_]
                          [[[1] (terminal-component identity)]
                           [[3] (terminal-component #(if % "done" "yet to do"))]]))]])
         root (root #(reset! adom %))]
     (doto root
       (ensure-path! [[0] [12345] [0] ["title"]])
       (ensure-path! [[0] [12345]])
       (ensure-path! [[0] [12345] [1] [false]]))
     @adom)
[:ul ([:li "title" "is" "yet to do"])]

Less awkward predicates

currently we have things like

(ez/for [[(pred expression)]] ...)

which is not great

Days #15 #16 #17

inclusion and sort!

Handlers and activations

I have a typical handler which is a relational expression

(ez/defhandler end-edition [selfx itemx]
  [[:db/add selfx :editing false]
   [:db/add itemx :title (:draft selfx)]])

The issue is that if it becomes a rule it needs to be passed an activation because its argument must be known before hand (because it's mostly doing interop/impure calls).

I'm starting to realize that these activations look suspiciously like supplementary predicates.

Allow expressions in sort key

as a way to handle changing sort orders

should it be done query side or component side? (eg like a template) query side seems better

how to handle asc/desc for non-numeric values?

Clean for-component

for-component is a mess of reifies now that sort is implemented

Day #4

Keysets (cont & end)

Ok after admitting that I'd better go the DNF route to compute a keyset (set of columns that are a key on the result set) I realized that if the query had a cycle (does it occur often in practice?) my algorithm would fail to return a proper key.

As usual to deal with cycles, at some point I have to switch from a graph of nodes to a graph of SCCs. It works, in the absence of cycle the keyset is minimal, with cycles the keyset may not be minimal (there may be redundancy but it no case non-key vars would be included).

Birth of a protocol

Here we go (there's nothing yet about instantiation)

(defprotocol Template
  (query [t]
    "returns the query for this template")
  (used-vars [t]
    "vars used in its own body (not by nested templates)")
  (children [t] "returns a map of ids (mount points) to templates"))

So, using the TODO example with 4 templates: the with, the inner with, the title and the status:

• only with returns a query: {[title done] :item/attrs}whose keyvars are [?id421] (the item eid); with has no used vars on its own but it has one child,
• inner with has no query, no used vars and two children,
• each children has no query, one used var each and no child.

(I wonder if "inner with" is there to stay.)

Death of a protocol

A protocol where all methods have no args is not a protocol: it's a map!

Let's start with a simple case: the foreign template -- that is the template which calls clojure code.

(defn foreign-template [vars f]
  {:vars vars
   :instance f})

The idea is to call f with values bound to vars.

Now, to with:

(defn with-template [q body]
  {:query q
   :children [body]})

So q is executed and for each returned row, children are instantiated. Each instance is identified by the key (derived from the query).

Main algorithm

Reminder: template is to component what class is to object.

1. Retrieve all queries from the template hierarchy
2. Instantiate the root component (technical root component no matching DOM)
3. Run queries derivatives, for each upsert row, traverse the component tree until matching (key-wise) component is found
   1. If found, update the component
   2. If not found, instantiate all missing components (using children template information from the last traversed component)
4. for each delta row, traverse the component tree until matching (key-wise) component is found and remove it (retract state entity too)

How many subscriptions?

(todo list example)

If each component has its own query then for a todo-list of N items we get at least N+1 subscriptions (1 for the list and N for each item).

Having 2 queries sounds better; is it even doable to have only 1 query?

Day #5

Todo: use keys (as per #11) in the fake derivatives

transact! in subscribers

subscribers being run on the listener it means they can recursively trigger themselves.
transactions should be postponed until the end of the meta listener. (.setTimeout 0) may well be enough.

Entity-like syntax

At some point when doing exploratory design, I played with entities. However they are too powerful, too dynamic: once they are passed to clojure code you can't know what they are going to be used for (and no, wrapping and tracking is not an option).

However the appeal was just to be able to write (:item/title item) in the template, not being forced to destructure upfront.

Hence the following idea: make (:attr eid) work in templates (and all variants).

lifting ors

Imagine a rule such as:

(defrule is-all-powerful [user] 
 (info user name group) (or (= "root" user) (= group "admin")) 

I want to lift the or out the rule to go to plain datalog.

My first idea was to lift ors as rules but then in this case we would have the following rules

xxx(name, group) :- eq(user, "root")
xxx(name, group) :- eq(group, "admin")

which violates the constraint that all variables appearing in the head must be grounded.

Implementing rules

Adding rules to EZ has two impacts: visible and not.

The user-visible impact is (defrule ...) (or something in this spirit) offering a way to factor code (and the experience so far proves that it's way due).

The invisible impact is that there's currently a LOT of queries which are run at each change and these queries have common parts (they can be organized as a tree and that's something I've started doing and this tree follows the structure). Using rules (well idb predicates) would allow a more natural way to structure the sharing (and also to get shared computation).

This would also allow recursive widgets (trees yay!).

Day #3

Spent too much time on #6 to figure out that the simplest solution was:

1. to normalize the query to DNF
2. compute the composite key of each branch
3. take the union of the keys
4. it's ok to include a column (var) in the key even if it was not part of the initial :find.

Focused on this subproblem, I almost forgot why it is important: a template query may retrieve data only useful for child templates (often queryless) and we don’t want to trigger a redraw of a component when only data for a su component has changed.