substance/data

A uniform interface for domain data (deprecated)

Substance.Data

No longer maintained. Use http://github.com/substance/substance instead.

Substance.Data is a data representation framework for Javascript. It's being developed in the context of Substance, an open publishing platform.

With Substance.Data you can:

• Model your domain data using a simple graph-based object model that can be serialized to JSON.
• Traverse your graph, including relationships using a simple API.
• Manipulate and query data on the client (browser) or on the server (Node.js) by using exactly the same API.

Features

• Data.Graph (A data abstraction for all kinds of linked data)
• Persistence (You can persist your data to a Data.Store)
• Operational Transformation (for incremental updates)
• Versioning (Every graph operation is remembered and can be reverted)

Install

Using the latest NPM release

$ npm install substance-data

Or clone from repository

$ git clone https://github.com/substance/data
$ cd data 
$ npm install

For running the testsuite, make sure you have mocha installed

sudo npm install -g mocha

Run the tests

npm test

data's Issues

Docs

Hey-hey!
Sorry, I see some improvements on data.js. Last time there was only one file and now... :) What should I do with docs? Can I continue? Maybe I should do it in master branch, so it'll be save after next spreading?
And let's decide something about docs style, ok?

Observing, pub/sub, or other events?

How do you observe model changes, so if the value of a property on a model changes, it triggers an event for (e.g.) DOM updates?

graph.fetch not able to read 'views'

Hi,

First, thanks on a great library. Having fun with it and it seems like it's just what I needed.

I'm having an issue with graph.fetch(). I am able to connect to couchdb and insert documents of different types based on my schema. But when I try to retrieve a document by doing this:

graph.fetch({
"type": "/type/user"
}, function(err, nodes) {
console.log(nodes);
});

I get:

Performing query:
{ type: '/type/user' }
TypeError: Cannot read property 'views' of undefined
at Array.0 (/Users/lgomez/node_modules/data/adapters/couch_adapter.js:169:20)
at /Users/lgomez/node_modules/data/lib/couch-client.js:176:24
at Array.forEach (native)
at /Users/lgomez/node_modules/data/lib/couch-client.js:171:22
at IncomingMessage. (/Users/lgomez/node_modules/data/lib/couch-client.js:76:25)
at IncomingMessage.emit (events.js:81:20)
at HTTPParser.onMessageComplete (http.js:133:23)
at Socket.ondata (http.js:1214:22)
at Socket._onReadable (net.js:684:27)
at IOWatcher.onReadable as callback

Am I doing something wrong here?

In my schema I have:

"/type/user": {
"type": "type",
"name": "User",
"properties": {
"name": {
"name": "Name",
"unique": false,
"type": "string",
"required": true
}
},
"indexes": {
"by_name": ["name"]
}
}

Thank you!

Add support for 'object' value types

This allows to store arbitrary JavaScript objects on as Properties of Data.Objects.

Spec

{
  "/type/user"{
    "_id": "/type/user"
    "type": "/type/type",
    "name": "User",
    "properties": {
      "settings": {
        "type": "object",
        "name": "User Settings"
      }
    }
  },

  "/user/john_doe": {
    "_id": "/user/john_doe",
    "settings": {
      background_color: "#555",
      font: "Helvetica"
    }
  }
}

Query Interface Specification

In order to specify and stabilize the Data.js query interface (expressed as query objects), here's a working draft discussing various options.

https://gist.github.com/1007780

Transactions: operations on a temporary graph clone.

In some situations it would be handy to apply operations on a temporary graph.
This would allow to create sequences/compounds of operations more easily without having any other side effects.
At the end, the operations would be applied to the real graph.

Allow the specification and verification of unique constraints

E.g. for Substance Documents, you shouldn't be able to store two documents with the same doc.creator + doc.name combination.

Dublication callback invoke

In examples/tasks.
When i synk graph with dirty nodes, throws exception:
Uncaught TypeError: Cannot read property '/project/test' of undefined - data.js:1363
In nowjs adapter in write function after now.write call, callback invoke twice: first with graph == undefined, second with normal graph. Why so?
Sorry for my english.

Documentation for fetch() is wrong

Hi,

Seems like the fetch() function is improperly documented.
The documentation states:
fetch graph.fetch(qry, callback)

When in fact the source code looks like this:
fetch: function(qry, options, callback)

Of course, if you call fetch() with the callback as the second parameter, an error is thrown.

Typo in data.js

I'm just perusing this project, so don't want to do a pull request, but I noticed that on line 221 in data.js, you attempt to call an intersect method on the Underscore object, but the method is actually intersection, not intersect.

Resolve any path

@oliver---- can we adjust Data.Graph.resolve so it works for any complex path?

E.g. I've linked a document entry with a user (by providing the user id). Now I'd like to be able to write:

var authorName = graph.resolve(["document_1", "creator", "name"]) 

instead of having to write:

var creatorId = graph.resolve(["document_1", "creator"]);
var authorName = graph.resolve([creatorId, "name"]);

Do not add empty groups in Data.Graph#group

add append/remove to modify graph node relations

I'm not sure but it, could look like

Append

var bart = graph.get('/person/bart'); // Data.Object
bart.append('locations', '/location/springfield');

Parameters of append:

• relation property name
• object _id
• (option with default false) boolean for adding back-link

Remove

bart.remove('locations', '/location/springfield');

Parameters of append:

• relation property name
• object _id
• (option with default false) boolean for removing back-link

Supporting map/reduce

Like:

graph.map("by_name_and_language", function(obj) {
  this.emit([obj.get('name'), obj.get("language")], obj);
});


graph.find("by_name_and_language", ["Michael", "German"]);

Dealing with deleted documents

Investigate why CouchClient#get returns no error object in case of a deleted document.

Consider removing Chronicle dependency

Couch db authorization

It's more question than issue.

How to specify username and password for the couchdb connection?

This should work?

graph.connect('couch', {
url: "https://USERNAME:PASSWORD@HOST/DB_NAME"
});

BTW,
what is the best way to submit question?

Allow more efficient queries (through a key property)

Dynamic queries with CouchDB have the drawback that for every possible query a new CouchDB view needs to be created (on the fly). In order to make use of the full power that CouchDB views provide we need to extend our query language and introduce a special key property:

Without keyed indexes:

{
  "type|=": "/type/user",
  "username": "michael"
}

{
  "type|=": "/type/user",
  "username": "michael"
}

=> two independent CouchDB views are created. For every user we query a new view is created -> Bad.

With keyed indexes:

{
  "type|=": "/type/user",
  "username": "michael",
  "key": "username"
}

{
  "type|=": "/type/user",
  "username": "michael",
  "key": "username"
}

Now that we know that username is the key of the CouchDB map function we just need one view that we can query using the value of username as a key.

Use a prefix for type index documents

Instead of

_/design/article

Use

'_/design/type:article'

There's a problem with regular expressions during validation on CouchDB side

See Substance "_design/user".

Make Data.Middleware asynchronous

Interface should be asynchronous. That way, a middleware can itself do asynchronous requests until calling next(). Cmp. visionmedia/express

Explicit Null values are not stored in the graph

Aggregates are wrong for non unique columns

If you have a repeated value in a Data.Collection, the aggregation produces incorrect values:

Test data, note we have two german countries with the same population:

var countries_data = {
"properties": {
"name": {"name": "Country Name", "type": "string", "unique": true },
"official_language": {"name": "Official Language", "type": "string", "unique": false },
"population": { "name": "Population", "type": "number", "unique": false }
},
"items": {
0: {
"name": "Austria",
"official_language": "German",
"population": 8356700,
},

  'usa': {
    "name": "United States of America",
    "official_language": "English",
    "population": 310955497,
  },
  'ger': {
    "name": "Germany",
    "official_language": "German",
    "population": 82062200,
  },
  'ger2': {
    "name": "Germany2",
    "official_language": "German",
    "population": 82062200,
  }
}

}

var countries = new Data.Collection(countries_data);
console.log('oh sweet, just made a collection:');
console.log(countries_data);

var population = countries.properties().get('population');
console.log('get returns a Data.Property: ');
console.log(population);

console.log('which supports aggregation: ');
var population_sum = population.aggregate(Data.Aggregators.SUM) // => Returns total population of all countries in the collection.
console.log(population_sum);

equals(population_sum, 310955497 + 82062200 + 8356700 + 82062200, 'We can aggregate a column');

This test fails, I think because Data.Property.aggregate is defined:

aggregate: function (fn) {
  return fn(this.values("values"));
},

this.values produces a hash where each key is the value considered, so in this case it has three rather than 4 values do to the repeated value.

Implement LocalStorageAdapter

Implement a LocalStorageAdapter so people can use modern browsers localstorage as a data store.

Value deregistration on object removal

When an object gets deleted, unregister all associated values.

Problem with recursive queries

For the query:

var qry = {
  "_id": id,
  "children": {
    "_recursive": true,
    "comments": {"_recursive": true}
  },
  "subjects": {},
  "entities": {},
  "creator": {}
};

Only comments of level1 children are returned.

Any place to see how versioning is working?

Came from chronicle project hoping to find the implementation (and better yet, examples) of how versioning with OT works, ideas?

On demand value registration (on a property basis)

Should make things way faster. Most of the times you don't need to access Data.Objects using a particular value. And when you do, you should be able to calculate those group memberships explicitly.

Strange character on line 1447 causes error.

Hello,

there's a strange character on line 1447 of data.js, right before null.

Firebug complains:

missing ; before statement
    nodes[o._id] = o._rev ||Â null; 

I noticed it running the test suite locally (file:///...) on Firefox 3.6.19.

Cheers,
Giacomo

Watch out for edge cases related to explicitly setting an object type property to null

Write a test case for this.

Use the version property of /type/comment as a reference.

Let Data.Graph.fetch yield a SortedHash containing the objects fetched from the DB.

graph.fetch({type: "/type/type"}, function(err, nodes) {
  nodes instanceof Data.Hash // => true
});

Syching with couchdb without requiring node.js

data.js looks amazing -- especially the offline caching and synchronization! I'd love to use it in a project I'm beginning. However, the requirement for two servers (node.js and couchdb) seems overkill. Especially for a small web-app given couchdb's ability to serve all resources (html, js, css, images) directly.

The ability to connect to a couchdb backend directly from the browser would be a huge benefit. It would also allow for much simpler use cases, reducing the number of required backend servers.

Are there plans to include this capability?

Indexes on array properties

For Lens I'd like to access all figure_refs for a given node. So I'd have to tell the index that it should take the nodeid which corresponds to the first element from path.

  "figure_refs_by_node": {
    "type": "figure_reference",
    "properties": ["path[0]"] // no idea how to specify that
  }

Desired usage:

doc.find('figur_refs_by_node', 'paragraph_12');

Consider supporting Data.Collections from arrays

Currently, in order to create a Data.Collection, you need to pass a list of key, value pairs like so:

var  data = { properties: {...}, items: {...} };

Since this is for tabular style data, you should consider allowing a spec of the form:

var data = { properties: {...}, items: [...] };

This almost works at the moment, except Data.Object presumes a node has a string id in order to do things like ( https://github.com/michael/data/blob/master/data.js#L969 ):

this.html_id = id.replace(/\//g, '_');

Is it really required that all ids are strings? If so, you could support it by using key.toString when you create the graph at https://github.com/michael/data/blob/master/data.js#L1555

Also, would is it not natural to expect a Data.Collection to support index based lookup, such as at?

Allow specifying query paths to fetch associated nodes in one go

graph.fetch({
  "type": "/type/document",
  "include": ["children"]
}, function(err, nodes) {
});

Add some debugging information if needed

Right now i'm keep getting "not_found" error when i call Data.Graph.fetch() method.

How am i supposed to see what actually is going on? What query is sent to CouchDB? It would be nice to request this kind of output in ie. .setAdapter() method

unable to export doc at http://substance.io/michael/data-js

I am unable to export the doc from http://substance.io/michael/data-js. When I choose any export formats the server just hangs and never actually serves the exported file. Also, access to the file seems to be generally sluggish.

I've tried it on Firefox 10 and Chrome with HTML, PDF, and RTF. All the same result. Thx.

Documentation problem, result of group does not support 'at'

In http://substance.io/michael/data-js/_section_6ddff355a5f41bd31d963e1a6459ca7f

This line fails:

languages.at(0).get('name'); // => "German"

because languages has no at method

Improved Validation Errors

By Charles Munat:

Rather than provide an error object with message and property name, why not return as the error object the property's type info itself, merging in a message and the type of failure? For example:

this.properties().each(function(property, key) {

var error = property.toJSON();

...

_.extend(error, { message: 'Property "Name" is required", reason: 'required' })

...

or

_.extend(error, { message: 'Invalid value for property "Date of birth"', reason: 'invalid' })

Then I can tell whether the error is because a required value is missing, or a validation regex failed, and I can access the property name, the validation regex, and the property's meta data and write my own message without having to go looking all over the graph for it. And maybe I'll store required and invalid messages in the meta data and just access them there.

Website doesn't work

The website link returns a 404.

Graph.set

From the example in readme:

graph.set('/person/bart', {
name: "Bart Simpson"
});

Source of Graph::set(id, properties):

var types = _.isArray(properties.type) ? properties.type : [properties.type];

In your example, properties = {name: "Bart Simpson"}, doesn't have a 'type' member, hence
'types' becomes [undefined] and the code fails later when it's trying to call a function on undefined.

Is there something I'm missing ?

Error when assigning a null value to a unique string property.

As happend with Substance's user.tan field when being reset.

Different adapters for different graphs?

It would be nice to associate different graphs in application with different adapters (and thus databases). Right now by design we have to explicitly change adapters before syncing or fetching.

Consider adopting JSON-LD syntax

Now Data.js follows CouchDB-esque data format with _id, type, etc. It might make sense to go to JSON-LD instead, as that is already a (in-progress) standard for JSON-encoding graph data:

http://json-ld.org/

api change proposal

change api from

graph.set("/person/bart", {
  type: "/type/person",
  attr1: val1,
  attr2: val2
})

to

graph.set({
  _id: "/person/bart",
  type: "/type/person",
  attr1: val1,
  attr2: val2
})

i believe this is easier to understand …

Additional event hooks

In order to promote the event-driven use of this framework, it would be nice to have finer-grained event hooks.

Suppose I'm making a system whereby graph updates are pushed from the server to the client, how would I be able to listen for additions/removals/changes to the graph?

One possibility would be to merge() them in with the dirty flag set, but this would not reflect the fact that the server (the authority over clean/dirty) has pushed them to the client and that they are therefore clean. This means that it would be difficult to detect the addition of clean nodes to the graph. Is this something that could be hacked together with the concept of conflicting nodes?

I really like the graph-oriented nature of this data framework and look forward to seeing 0.3!

Specify configuration object for type properties in schema

Currently for properties we only specify a type.

      "location": {
        "type": "location",
        "name": "Location",
        "properties": {
          "name": "string",
          "citizens": ["array", "person"]
        }
      }

Maybe we should change this in favor of an object where we can add more specs like a required property.

      "location": {
        "type": "location",
        "name": "Location",
        "properties": {
          "name": {
            "type": "string",
            "required": true
          },
          "citizens": {
            "type": ["array", "person"]
        }
      }

Or we just say such integrity checks should be done by the app.. not the schema. What do you think?

Data.Graph example does not work on v0.4.1

I was playing with data.js - the example of - var graph = new Data.Graph(schema); - did not work using v0.4.1, it gives me a type error: ""this.replace is not a function"

When I use v0.4.0 the Data.Graph function seems to work as shown on the example.

rename "unique" to something like "string" or "single_value"

»unique« has a very strong meaning to a lot of people when it comes to databases. As far as I understand it, it's meaning at data.js is that the value is not unique, but that it's a flat value like a string and not something like an array or a hash, correct?

To avoid confusion, I strongly recommend renaming this attribute.

backbone.js models and data.js graphs relation

Probably I'm missing something here, but I cannot understand the profound difference between a data.graph, and a backbone.model object.

Eventually, it could be useful to integrate the graphs with backbone and inherit its model features (such as bind on change etc.). Maybe it's already possibile. Isn't it?