kinto/kinto.js

An Offline-First JavaScript Client for Kinto.

Kinto.js

An Offline-First JavaScript client for Kinto.

Note: This library also includes a pure JS HTTP client for Kinto. You can learn more in the docs.

The idea is to persist data locally in the browser by default, then synchronizing them with the server explicitly when connectivity is guaranteed:

const kinto = new Kinto({ remote: "https://demo.kinto-storage.org/v1/" });
const posts = kinto.collection("posts");

// Create and store a new post in the browser local database
await posts.create({ title: "first post" });

// Publish all local data to the server, import remote changes
await posts.sync();

Documentation

• Installation
• Tutorial
• Api documentation
• Extending Kinto.js
• Upgrading Kinto.js
• Known limitations
• Contributing

kinto.js's Issues

Expose events for server backoff

Refs #68, #82. We should expose public events for server backoff, so users can subscribe to them to handle UI changes accordingly, eg:

kinto.on("server-backoff", seconds => {
  if (seconds > 0) 
    $("#sync-btn").disable();
  else 
    $("#sync-btn").enable();
});

Handle deprecation headers

As specified here http://cliquet.readthedocs.org/en/latest/api/deprecation.html, the client should be aware of Alert headers, and use them to effectively alert the users/developers

Improve error reporting when fetching changes failed.

   71            return {items: []};
   72          } else if (res.status >= 400) {
   73:           // TODO: attach better error reporting
   74            throw new Error("Fetching changes failed: HTTP " + res.status);
   75          } else {

We'd need to distinguish between possible 4xx and 5xx status codes so the end user knows more precisely what happened.

Set up a .babelrc

https://babeljs.io/docs/usage/babelrc/

This also might ease building dist files (concat + minification)

Error when server returns 400 error response

Cannot read property 'errno' of null

To reproduce:

var collection = kinto.collection("ZjE1ZjgxODMtOTcyNy00ZDgzLWE2MDEtOTUyMTFlYTIzMzQ5OnMzY3IzdA==-items");
collection.sync();

Server will respond:

GET https://kinto.dev.mozaws.net/v1/buckets/default/collections/ZjE1ZjgxODMtOTcyNy00ZDgzLWE2MDEtOTUyMTFlYTIzMzQ5OnMzY3IzdA==-items/records

Access-Control-Allow-Origin:*
Connection:keep-alive
Content-Length:167
Content-Type:application/json; charset=UTF-8
Date:Wed, 12 Aug 2015 07:24:54 GMT
Server:nginx/1.4.6 (Ubuntu)

{"errno":107,"message":"path: Invalid record id","code":400,"details":[{"location":"path","name":null,"description":"Invalid record id"}],"error":"Invalid parameters"}

Curiously, it looks like in kinto.js code response.json() is null.

Cached Kinto collections instances are not uniquely identified

This doesn't comply with setting the bucket name, so a collection sharing a common name with another one from another bucket will override it.

    if (!this._collections.hasOwnProperty(collName)) {
      this._collections[collName] = new Collection(bucket, collName, api, {
        events: this.events
      });
    }

We need to cache these using keys including the bucket name as well.

Error on synchronization with a HTTP 304 on the collection from the server

HTTP304 - SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data

Builds are marked green even if failures occured.

Sample build: https://travis-ci.org/mozilla-services/kinto.js/builds/71136831

…
  177 passing (13s)
  1 failing
  1) Integration tests "before each" hook:
     Error: timeout of 12500ms exceeded. Ensure the done() callback is being called in this test.
…
The command "npm test" exited with 0.

Handle backoff indicators

As specified here http://cliquet.readthedocs.org/en/latest/api/backoff.html

The clients should be aware of Backoff and Retry-After response headers, in order to give the server a breathe :)

Create a localStorage adapter.

Support opening different cliquetis instances in browser normal vs private modes

Maybe add fetch-only / push-only option in Collection#sync()

The way we use kinto.js for syncing browser history is:

data on remote FxSync
<--network-->
data exposed by syncto
<--kinto.js sync-->
local kinto.js collection
<--adapter-->
DataStore 'places' on the device

Since the device is already using the 'places' DataStore all over the place, we want to store all data twice, locally. Once in the kinto.js collection, and once in the 'places' DataStore. The 'adapter' compares the two, and decides how to import changes from the kinto.js collection into the DataStore, and export changes from the DataStore to the kinto.js collection.

Before this adapter runs, we want to make sure we have the latest data from remote, so we want to sync fetch-only.

After the adapter runs, we want to push out any changes it made in the kinto.js collection, so we want to sync push-only.

It's not so expensive to push zero changes in the pre-adapter sync and to pull (probably) zero changes in the post-adapter sync, but at the same time it would be nice to be able to just make the first sync fetch-only, and make the second sync push-only

Prevent concurrent conflicting transactions by incrementing a local revision number

Use case

• We have an edition form for a given record in state A, and we don't have submitted the form yet;
• Concurrently, a worker modifies this record and turns it into state B;
• We now submit the form, which overrides what's just been updated by the worker, which is the problem we're trying to solve here.

Flow

• Form loads existing record with local_rev==1 (eg. that value is contained in some hidden input field, CSRF-token protection style or any other way, eg in React component's props);
• Worker loads existing record with local_rev==1;
• Worker updates existing record, persistence layer bumps local_rev to 2;
• Form is submitted, data are sent to the local persistence layer adapter;
• Persistence layer knows that local_rev for that record is 2, but form has just sent local_rev==1: an error is raised.

Notes

• Do not synchronize the local revision number to the server;
• Each time we synchronize a remote record from the server to the local database, we reset its local revision number;
• Any synchronized remote records should bump matching local equivalent record's local_rev number.

We'd love hearing thoughts and feedback from @daleharvey, @leplatrem, @Natim, @michielbdejong

Host a demo instance on Github pages

Sync flow should stop when pushing changes fails

Current flow is:

    return this.db.getLastModified()
      .then(lastModified => this._lastModified = lastModified)
      .then(_ => this.pullChanges(result, options))
      .then(result => {
        if (!result.ok) {
          return result;
        } else {
          return this.pushChanges(result, options)
            .then(result => this.pullChanges(result, options));
        }
      });

We should probably resolve early when pushChanges result isn't ok as well. This is how the final flowchart should look like:

Install from github branch

With the current strategy (see #105), kinto cannot be installed from a github branch.

diff --git a/package.json b/package.json
index 1f15289..fe037bf 100644
--- a/package.json
+++ b/package.json
@@ -38,7 +38,7 @@
     "karma-webpack": "^1.7.0",
-    "kinto": "^1.0.0-rc.3",
+    "kinto": "git://github.com/mozilla-services/kinto.js.git#db-prefix",
     "mocha": "^2.2.5",

Module not found: Error: Cannot resolve module 'kinto'.

Implement conflict resolution helper

Suggested API:

collection.sync().then(res => {
  if (res.conflicts.length === 0)
    return;
  return Promise.all(res.conflicts.map(conflict => {
    const resolution = …
    return collection.resolve(conflict, resolution);
  })).then(res => collection.sync());
});

Collection#resolve should return a Promise being the result of a sucesful local update.

Note: Maybe we should add a new resolved entry to SyncResultObject to list these?

What are the work in progress and limitations ?

It would be relevant to explicit somewhere:

• the intended limitations (aka. naive approach)
  • no revision tree
  • no magic for conflict merge
  • server/client wins
  • ...
• the current limitations (aka. work in progress)
  • no local revisions
  • no lock for concurrent updates
  • ...
• ideas for the long-term vision (aka. fantasy)
  • Kinto futon
  • crypto layer
  • ...

Handle synchronization when the server was flushed.

The development server is flushed every day.

At first, we could think that the JS client would just reupload everything stored locally. Is it really the case?

Hints:

• If the server is flushed, the timestamps are reinitialized.
• If the server is flushed, the permissions will be too (off topic?)

Just close this issue if cliquetis behaves as expected :)

Add a simple functional test suite for the demo, using jsdom.

Perform the whole sync (pull + push) within a single transaction

So we can abort the whole import process on error, avoiding leaving the db in an inconsistent state.

Handle BATCH 409 responses

Handle 409 response when uniqueness rule is violated (eg. POST with existing id, unique field, etc.)

Expose conflicting record details on BATCH 412 responses

Refs mozilla-services/cliquet#369

Don't mention Cliquet in the repo description

This issue follows up on #42. I've noticed that Cliquet is still mentioned in the GitHub repo description. I obviously can't change it myself, hence this issue :)

Host static assets on a CDN

Hotlinking to Github static assets hosting doesn't work well with browsers having strong mimetype checks. Also that's probably unfair to Github, so we have to find a proper solution.

Add checks for unsupported Cliquet protocol version.

Goals

The client should:

• check that the protocol version defined by the remote server url matches the one it actually supports;
• check that for every HTTP request made to the server, the protocol version matches its own expectations.

Tasks

• Define and export a SUPPORTED_PROTOCOL_VERSION constant defining the protocol version the client actually supports; eg. export const SUPPORTED_PROTOCOL_VERSION = "v1";
• Ensure the version passed in the remote property matches that constant value; raise on mismatch with an appropriate error message;
• Request server endpoints using that SUPPORTED_PROTOCOL_VERSION value;
• On each request made, check for 404 status code with an errno set to VERSION_NOT_AVAILABLE; reject with an explicit message in such case;
• Update docs.

Validate passed uuids

This would apply to #get(), #update(), #delete() and #resolve(). We should throw when an id doesn't match the UUID format. Tests would have to be updated accordingly.

For #resolve(), we should also validate that the resolution object uuid matches the one from the conflict, and reject accordingly.

Persist per-collection lastModified value

We need to store the timestamp of latest synchronization for each collection. We can use some private collection to achieve that.

Handle BATCH request limit

If more than X results (default is 25 on server), we should send incremental requests to the server.

Related: mozilla-services/cliquet#318

Support unlimited batch requests

In Kinto, if the setting cliquet.batch_max_requests is set to None, then the server has no limit for batch requests.

Currently, Kinto.js expects a value for this setting.

Clarify if 'clearing the collection' affects only client, or client+server

Sometimes I will want to remove all the items from the collection on the server.
Other times (actually, probably only when debugging, or maybe when offline and offering the user something like git checkout -- . to discard unpushed changes), I only want to flush the cache, to make sync start from zero.
It is not clear to me which one of these two is done by http://kintojs.readthedocs.org/en/latest/api/#clearing-the-collection

Add an Authentication section to the docs

Even if really simple, it's especially important to document the FxA authentication process.

Refactor generic HTTP error handling.

Current situation is a mess, eg duplicated code and most importantly, we're losing stack traces when throwing new errors instead of propagating them.

Support passing default auth headers to Cliquetis constructor

Documentation says we can define common headers to be passed to the Cliquetis constructor, though that's not implemented just yet.

Implement local list filtering

Target API:

// Single filter
articles.list({
  filter: { unread: { $eq: true } }
}).then(console.log.bind(console));

// Combined field filtering
articles.list({
  filter: {
    added_on: { 
      $gt: 1432800794000, 
      $lt: 1432800795000 
    }
  }
}).then(console.log.bind(console));

// Multiple fields filtering (AND only)
articles.list({
  filter: { 
    unread: { $eq: true }, 
    added_on: { $gt: 1432800794000 } 
  }
}).then(console.log.bind(console));

First minimal batch of operators to support:

• $eq: Strict equality check (strings, numbers);
• $contains: Partial equality check (strings only);
• $gt, $gte: Greater than, greater than equal (strings, numbers);
• $lt, $lte: Lesser than, lesser than equal (strings, numbers).

Notes

• As we're using schemaless stores, we can't rely on indexed fields here (because we can't predict them);
• All queries are performed using the AND condition logic.

Move docs to RTD

http://readthedocs.org

Reflect record ids sent to BATCH in result object

Since responses come in the same order requests were sent, there should be a way to map original record ids to their respective BATCH response results.

Add lint/style check in Travis

Versions should be handled in the client

Currently, Cliquetis handles the version in the host url, with a prefix (e.g. http://localhost:8888/v1).

It's important to have a way to tell which version of the protocol is supported by Cliquetis, in order to detect potential problems due to version mismatches.

To do so, I believe a request to the / endpoint of the server should be made, to get the protocol version, and then check that the protocol is the same as the one of Cliquetis.

Don't store collection last_modified on sync() when conflicts have been encountered

This prevents calling sync again using an erroneous last_modified value.

Allow defining local indexes.

Filtering and ordering being currently achieved in-memory, the performance penalty can be important in case of large datasets.

We should allow users to create indexes on fields and rely on native IDB queries to speed things up.

Suggested API:

const db = new Kinto();
const tasks = db.collection("tasks");
tasks.ensureIndexed("label", "done");

Or even:

const tasks = db.collection("tasks", {
  indexes: ["label", "done"]
});

Thoughts? @Natim @leplatrem

Expose Collection.strategy.*

Maybe I'm looking in the wrong place, but I can't see how I can access the Collection.strategy.* constants from outside the library (I am currently working around this by using the literal strings from https://github.com/Kinto/kinto.js/blob/master/src/collection.js#L85-L87).

Fetch remote server settings.

Goals

The client should retrieve server settings from the /vX endpoint; for now, the only setting we need is the batch_limit one.

Tasks

• On each outgoing HTTP request, first ensure we already have retrieved the settings information;
• If we haven't yet, request that information from the /vX endpoint, then store the settings in a local Api property;
• On subsequent requests, reuse the cached settings property to avoid hammering the server for no reason.

Notes

• The implementation shouldn't impact existing documentation;
• Batch limit support should be done along #13, not here.

Add integration test for deprecation headers

Refs #82 which adds contextual test server configuration, so we now should add integration tests for deprecation headers.

Investigate local handling of permissions.

What should we provide as features/API for frontend developers regarding permissions in the client?

Thoughts? @leplatrem @Natim @almet

Provide a hook to interecept deprecation warnings.

Follow-up of #79.

We need to provide a way for the user to intercept deprecation warnings to they can handle them however they want.

It feels like we should rely on an event, eg.:

const kinto = new Kinto({remote: "http://deprecated.server/v0"});
kinto.on("deprecated", (message, url) => {
   // don't take this example too seriously please :)
  alert(message);
  window.open(url);
});

Be able to provide the Collection URL in order to get Kinto.js compatible with Cliquet specific app.

On my way to build Syncto and after reading the specification document for Kinto integration with Balrog, I think we should extends Kinto to let people choose their collection URL in case it doesn't follow the exact Kinto formalism.

• For the Readinglist server, the collection URL looks like: /v1/articles
• For CSP policy the collection URL may looks like: /1/preloaded-sts-pkp
• For Syncto, the collection URL may looks like: /v1/history (complete list of collection can be found here: https://docs.services.mozilla.com/storage/apis-1.5.html#collections)

The rest of the protocol is the one Cliquet uses.

The configuration could be something like:

  var db = new Kinto({remote: "https://syncto.dev.mozaws.net/v1"});
  var history = db.collection("history", url="/history", use_middlewares=[SyncCrypto]);

Or for the readinglist:

  var db = new Kinto({remote: "https://readinglist.services.mozilla.com/v1"});
  var history = db.collection("history", url="/articles");

Demo is broken.

Related Kinto/kinto#126

Accept Syncto base64 record_ids

Refs mozilla-services/syncto#6

Remote URL should have a trailing slash.

It allows to copy and paste the URL directly from the browser.