quoll/asami

A flexible graph store, written in Clojure

asami's Issues

Configuration setting to use RDF properties

Asami uses :a/first, :a/rest, and :a/nil for list structures. Introduce a setting where these can change to:
:rdf/first, :rdf/rest, :rdf/nil

statement IDs

All statements currently have IDs, however, these are not accessible through user interfaces. This issue is to introduce a query operation that can reference and bind statement IDs.

Basic graph patterns currently look like:
[subject predicate object]

where the predicate and object can be optional.

Add an optional transaction and :as values:
[subject predicate object tx-id :as id]

This ticket focuses on the statement IDs.

Implementation

There are 3 possibilities here:

1. [s p o] This is an unbound anonymous var for the ID.
2. [s p o :as ?id] where ?id is unbound
3. [s p o :as ?id] where ?id is bound

In (1) we have 2 options: allocating a var that we never see, or keeping the code as-is. We may want to inspect the options here, but I think that by handling the 3-wide vs. 4-wide patterns then we could end up with a lot of extra code.

In (2) we expand BGP resolution. This may need a whole new set of patterns to match against in the protocol. Depending on (1) then we're either going to make all the pattern resolutions 4, or else we're going to double the side of the graph API.

In (3) then look up the statement index keyed by ID. If bound columns do not match, return nothing. Otherwise, return bindings on any unbound columns.

New release to fix CLJS `Use of undeclared Var asami.entities/format`?

Hi Paula,

thanks for the lib.

I noticed that the latest release in Clojars does not include this commit which would fix CLJS warnings about format:

WARNING: Use of undeclared Var asami.entities/format at line 213 .cljs/.aot_cache/1.11.60/A8BA44C/asami/entities.cljc
WARNING: Use of undeclared Var asami.entities/format at line 221 .cljs/.aot_cache/1.11.60/A8BA44C/asami/entities.cljc

Can be reproduced in v2.3.2 with: clj -Sdeps '{:deps {org.clojure/clojurescript {:mvn/version "1.11.60"}} :paths ["src"]}' -M -m cljs.main --optimizations advanced -c asami.entities

---

Though it seems that commit brought a different warning into CLJS compilation. Running the same cljs.main command as above against main branch results in:

WARNING: Use of undeclared Var schema.core/split at line 18 /Users/joni.hamalainen/work/asami/src/asami/entities.cljc

Maybe you need to add #?(:cljs [clojure.string :as str]) and change the line 18 to use str/split ?

Thanks and happy new year!

access

Vars need to consider namespaces

asami/src/asami/projection.cljc

Line 60 in 1194563

(and (symbol? x) (boolean (#{\? \%} (first (name x))))))

Only looks at the symbol name. If a namespace on a symbol exists then this should be considered instead.
Also, look at aggregate binding selection and the schema test for cars in query.clj, as these also forget namespaces

Asami fails to retract :id sometimes

In some cases Asami fails to retract the :id attribute:

I do

(d/transact conn [
 [:db/retract :a/node-78464 :id [:address/id #uuid "ffffffff-ffff-ffff-ffff-000000000001"]]
 [:db/retract :a/node-78464 :address/id #uuid "ffffffff-ffff-ffff-ffff-000000000001"]
 [:db/retract :a/node-78464 :a/entity true]
 [:db/retract :a/node-78464 :address/street "X St"]])

yet the transact's returned :tx-data only have 3, not 4 datoms, missing the first one with :id :

(#datom [:a/node-78464 :address/id #uuid "ffffffff-ffff-ffff-ffff-000000000001" 2 false]
 #datom [:a/node-78464 :a/entity true 2 false]
 #datom [:a/node-78464 :address/street "X St" 2 false])

The following test demonstrates the issue:

(deftest delete-ident
  @(d/transact *conn* {:tx-data [{:id [:test/id "existing-ident"]
                                  :test/id "existing-ident" ; <----- comment out to "fix" the test
                                  }]})
  (let [eid (d/q '[:find ?e . :where [?e :id [:test/id "existing-ident"]]] (d/db *conn*))]
    (write/retract-entity *conn* [:test/id "existing-ident"])
    (is (= nil (d/entity (d/db *conn*) [:test/id "existing-ident"])) "The entity is no more")
    (is (= '() (d/q '[:find ?e ?a ?v :where [?e ?a ?v] :in $ ?e] (d/db *conn*) eid)))))

; where write/retract-entity is:
(defn retract-entity [conn id]
  (d/transact
    conn
    (d/q '[:find :db/retract ?e ?a ?v :where [?e ?a ?v] [?e :id ?id] :in $ ?id]
         (d/db conn) id)))

Notice that if I either change the :id to something else or I change either the key or the value of :test/id "existing-ident" then the test will pass. So there is some collusion between the two.

Metadata on Bindings

Context

Right now bindings are in the format:
^{:cols [String]} [[Object]]

Meaning that it's a seq of seqs of objects, with metadata that contains the names of the columns.

An example might be a binding of names and ages:

^{:cols ["?name" "?age"]} [["Alice" 22] ["Bob" 21]]

This pattern is then used in query resolution (conjunctive joins, disjunctive joins, filters, minus, etc).

Proposal

Extend the metadata to include column types: local or global
The above example would form a global type. This means that the data in the binding makes sense in a global context. Hence, the new representation would be:

^{:cols ["?name" "?age"] :types [:global :global]} [["Alice" 22] ["Bob" 21]]

A local example of the same thing is:

^{:cols ["?name" "?age"] :types [:local1 :local1]} [[-1927139965642932224 -9223372036854775786] [-2070970411939528704 -9223372036854775787]]

These ID values are what is stored in the triples.

NB: These examples are definitive because each of the bound values can be encapsulated into those IDs. (Values are encapsulated when they are negative). However, larger values (such as a long string) will be mapped to their data-pool IDs (which are positive numbers).

Desired Outcome

This will allow data to go through join operations without needing to be globalized, thereby speeding queries from storage significantly.

Describing a type for each column individually will allow queries to continue to join data from difference sources. This means that the benefit will only apply when joins are occurring for bindings with matching columns that have come from the same storage.

Concerns

Joins

Each type of "local" storage will need its own type value, since data from different data-pools will have different IDs, although encapsulation is universal across stores. The resulting comparison operations are:

left	right	join operation
global	global	left / right
local1	global	(globalize left)/ right
local1	local1	left / right
local1	local2	(globalize left) / (globalize right)
local1	local2	iff (local1 < 0) and (local2 < 0) then: left / right

Note that the in-memory store is always in globalized form.

Types for a Binding or for all Bindings

Incoming data always starts in local form. There may be edge cases (for instance, in filtering) where it would be more efficient to only convert a column for some bindings. However, this will introduce complexity that may slow the querying down. Globalizing already happens automatically, so unnecessary globalization is no slower than the current system.

Filters and Bindings

Both of these operations rely on global values from their source data. This requires globalization.

Mixed Operations

Because columns may be joined after filtering, it will be more efficient in many cases to have the local value still. So globalizing a column should result in a new column, not a replacement column.

Projection

The final projection operation currently looks for column names. Under this new approach projection must now:

• Look for the column by name.
• If the column is local, then record this and continue looking.
• If a global column is found:
  • Select the global column
• Else:
  • Map the column to global and select the new column.

cannot load data

Hi,
I have json file, 87MB .
I am not able to load it.
here is my code:

(ns my.kn
  (:require [asami.core :as d])
  (:require [clojure.java.io :as io])
  (:require [cheshire.core :refer :all]))

(defn import-data
  [json-file db-uri]
  (let [data (parsed-seq (io/reader json-file) true)]
    (d/transact (d/connect db-uri) {:tx-data data})))

(def db-uri "asami:local://my-data")  
(d/create-database db-uri)   
(import-data "my-file.json" db-uri)     

here is log:

; eval (current-form): (import-data "my-file ...
; (err) Execution error (ExceptionInfo) at asami.core/eval19083$transact$fn (core.cljc:278).
; (err) Transaction timeout

could you please give me a hint how to debug this issue?

Change to broad-node-type?

Checking for node type is not really helping out. Switch to broad node type.

https://github.com/quoll/asami/blob/169301f0a783a40a73ad730b0645cacba7ea571f/src/asami/index.cljc#L142C40-L142C40

reasoning in asami

Hello,
this is a fantastic project. Thank you so much for open source it!
I was wondering if there is an interest on adding some reasoning beginning with concept and entity satisfiability prior loading the data. Given the use of clojure, reasoning should be solved efficiently.
Any thoughts?
Many thanks in advance.

Create new `get-pattern-vars` fn for SPARQL MINUS

The SPARQL MINUS operation requires that outer vars that don't appear in patterns (EPVPatterns) should not be included in bindings. This means that the get-vars function should not be recursing into FILTER or BINDING. That will require a new operation, which is essentially the same as the existing one, but which returns nothing for filters.

asami/src/asami/query.cljc

Line 298 in 3609943

vars (reduce #(into %1 (map plain-var (get-vars %2))) #{} patterns)

Bug in `abs` defn

Hello, thanks for working on this cool project. I was trying to set up latest version "2.3.3" to play around with but got complaints about not finding symbol abs or something wrong with asami.durable.tree ns when loading the lib. Looking through the code to the best of my ability (as someone very new to Clojure), it seems that while abs is still being used here, it was removed in "v2.3.2" in this commit, thereby causing the issue?

Using version "2.3.1", I was able to load data just fine with some typical code based off the example.

Immutable API/db-wth support

Description

Asami is great for producing data-driven apps with arbitrary data sources, and it pairs well with other triple-based tools. I'm working on a library that uses Asami as its state atom. Currently, it uses the DB name to separate sessions/states. However, I would like to start being able to reuse earlier states and exploit Clojure's cheap persistent data structures for features such as incremental dataflow or pre-computing some expensive initial data set and then using the db value at this point as the initial db value.

Datomic and Datascript have the db-with function for this in-memory/speculative work.

Problem

Looking at the implementation of transactions in the repo, it's not too clear if such a thing exists or how one might roll their own. I've tried to bake a connection value based on the shape of the "atoms in an atom" connection, but it was a bit flakey and felt like a code smell. This also makes subscribing to updates non-trivial.

Suggested Approach

A separate functional API for in-memory graph values that leverages a db-with approach to transactions would provide a good balance between keeping the existing implementation and supporting this functionality. Such a thing may already exist if you know where to look, which, right now, I don't

I'm happy to be shown in the right direction to be able to implement this myself.