rethinkdb/horizon-docs

RethinkDB

What is RethinkDB?

• Open-source database for building realtime web applications
• NoSQL database that stores schemaless JSON documents
• Distributed database that is easy to scale
• High availability database with automatic failover and robust fault tolerance

RethinkDB is the first open-source scalable database built for realtime applications. It exposes a new database access model, in which the developer can tell the database to continuously push updated query results to applications without polling for changes. RethinkDB allows developers to build scalable realtime apps in a fraction of the time with less effort.

To learn more, check out rethinkdb.com.

Not sure what types of projects RethinkDB can help you build? Here are a few examples:

• Build a realtime liveblog with RethinkDB and PubNub
• Create a collaborative photo sharing whiteboard
• Build an IRC bot in Go with RethinkDB changefeeds
• Look at cats on Instagram in realtime
• Watch how Platzi uses RethinkDB to educate

Quickstart

For a thirty-second RethinkDB quickstart, check out rethinkdb.com/docs/quickstart.

Or, get started right away with our ten-minute guide in these languages:

• JavaScript
• Python
• Ruby
• Java

Besides our four official drivers, we also have many third-party drivers supported by the RethinkDB community. Here are a few of them:

• C#/.NET: RethinkDb.Driver, rethinkdb-net
• C++: librethinkdbxx
• Clojure: clj-rethinkdb
• Elixir: rethinkdb-elixir
• Go: GoRethink
• Haskell: haskell-rethinkdb
• PHP: php-rql
• Rust: reql
• Scala: rethink-scala

Looking to explore what else RethinkDB offers or the specifics of ReQL? Check out our RethinkDB docs and ReQL API.

Building

First install some dependencies. For example, on Ubuntu or Debian:

sudo apt-get install build-essential protobuf-compiler \
    # python \  # for older distros
    python3 python-is-python3 \
    libprotobuf-dev libcurl4-openssl-dev \
    libncurses5-dev libjemalloc-dev wget m4 g++ libssl-dev

Generally, you will need

• GCC or Clang
• Protocol Buffers
• jemalloc
• Ncurses
• Python 2 or Python 3
• libcurl
• libcrypto (OpenSSL)
• libssl-dev

Then, to build:

./configure --allow-fetch
# or run ./configure --allow-fetch CXX=clang++

make -j4
# or run make -j4 DEBUG=1

sudo make install
# or run ./build/debug_clang/rethinkdb

See WINDOWS.md and mk/README.md for build instructions for Windows and FreeBSD.

Need help?

A great place to start is rethinkdb.com/community. Here you can find out how to ask us questions, reach out to us, or report an issue. You'll be able to find all the places we frequent online and at which conference or meetups you might be able to meet us next.

If you need help right now, you can also find us on Slack, Twitter, or IRC at #rethinkdb on Freenode.

Contributing

RethinkDB was built by a dedicated team, but it wouldn't have been possible without the support and contributions of hundreds of people from all over the world. We could use your help too! Check out our contributing guidelines to get started.

Donors

• CNCF
• Digital Ocean provides infrastructure and servers needed for serving mission-critical sites like download.rethinkdb.com or update.rethinkdb.com
• Atlassian provides OSS license to be able to handle internal tickets like vulnerability issues
• Netlify OSS license to be able to migrate rethinkdb.com
• DNSimple provides DNS services for the RethinkDB project
• ZeroTier sponsored the development of per-table configurable write aggregation including the ability to set write delay to infinite to create a memory-only table (PR #6392)

Licensing

RethinkDB is licensed by the Linux Foundation under the open-source Apache 2.0 license. Portions of the software are licensed by Google and others and used with permission or subject to their respective license agreements.

Where's the changelog?

We keep a list of changes and feature explanations here.

horizon-docs's Issues

horizon client getting started guide

Migrated from rethinkdb/horizon#433 by @hnordt

In http://horizon.io/docs/getting-started/ the docs say:

However, if you’re using [Webpack][] or a similar build setup, or requesting the .js library at load time isn’t desirable, just add the client library as an NPM dependency (npm install @horizon/client).

In your application, you’ll need to include the Horizon client file, and specify the Horizon port number when initializing the connection.

<script src="localhost:8181/horizon/horizon.js"></script>

// Specify the host property for initializing the Horizon connection
const horizon = Horizon({host: 'localhost:8181'});

In your application, you’ll need to include the Horizon client file, and specify the Horizon port number when initializing the connection. shouldn't be In your application, you’ll need to import the Horizon client file, and specify the Horizon port number when initializing the connection.?

Also the code shouldn't be:

import Horizon from '@horizon/client';

// Specify the host property for initializing the Horizon connection
const horizon = Horizon({host: 'localhost:8181'});

?

Broken markdown links

In the Collection.fetch() section, there's a broken link to subscribe

fetch or watch may be iterated through with [subscribe](#subscribe) or forEach, or converted to an array with toArray.

Schema clarity

• We should specify a place for a schema.toml file to live, I vote .hz/schema.toml - http://horizon.io/docs/permissions/
• we should be explicit that the schema declarations do not go inside .hz/config.toml (I made this mistake, and don't think it would be unreasonable for others to assume the same)
• We should have an example showing how to move an application from local to a production environment which involves a schema migration .
  • For the mean time this will have to be without being able to specify schema.toml to hz serve

Multi-line ES6 template string not highlighting correctly

http://localhost:4000/docs-archive/readme

`Collection` API docs still suggest integer primary keys are special

For example here:

// the same as above, but using the integer shorthand for ID
messages.order("id").above(100).fetch();

As far as I know, the shorthand works for any type of primary key. It is distinguished from the long syntax by the fact that you're not passing an object/dictionary to above.

`defaultIfEmpty` example for `find`

This came up in rethinkdb/horizon#451.

If you do a collection.find(id).fetch().subscribe(...) and no matching document exists, neither an error nor any result will be generated.

Since this might be a common use case, we should add an example that uses defaultIfEmpty to handle missing rows. Something like this:

var messages = hz('messages');
messages.find(id).fetch().defaultIfEmpty().subscribe((msg) => {
  if (msg == null) {
    console.log("Message not found");
    return;
  }
  ...
});

See details at http://reactivex.io/rxjs/class/es6/Observable.js~Observable.html#instance-method-defaultIfEmpty

Add list of Observable methods available

We currently patch in the following Observable static methods:

• Observable.empty()
• Observable.of()
• Observable.from()
• Observable.merge()
• Observable.fromPromise()

And these operators (instance methods):

• observable.publishReplay()
• observable.scan()
• observable.filter()
• observable.map()
• observable.toArray()
• observable.do()
• observable.catch()
• observable.concatMap()
• observable.filter()
• observable.share()
• observable.mergeMap()
• observable.defaultIfEmpty()

subscribe isn't part of the Collection class, it's a method on Observables

Right now the docs indicate it's our method. We may want to link to the rxjs docs instead of documenting it directly. Alternately, if we do document it, to avoid people having to go to another site to learn how to use it (which is reasonable), we should rename:

• readFunction(result): a callback that receives a single document as the Collection is iterated through
• errorFunction(error): a callback that receives error information if an error occurs
• completedFunction(): a callback executed when the result set has been iterated through completely

They should be named next, error, and complete respectively. This is because you can also pass an object to subscribe:

.subscribe({
next: (x) => {},
error: (err) => {},
complete: () => {},
})

So the names need to line up

Write deployment documentation

https://github.com/rethinkdb/horizon-docs/blob/master/deploy.md is currently a stub

We don't need this for the 1.0 launch, but will need it when we publicly launch Horizon Cloud.

Proof-read documentation before release

Some of the examples and APIs might need to be updated.

Auth docs don't explain how to make use of the session / user information

https://github.com/rethinkdb/horizon-docs/blob/master/authentication.md doesn't explain how I can access the current user's session ID or user name.

@deontologician How do I access that information in the current API?

Complete Horizon API docs

https://github.com/rethinkdb/horizon-docs/blob/master/api/horizon.md is still full of stubs.

Describe how to run Horizon server without the `hz` CLI

We have examples for running a Horizon server by instantiating the Server class in node.js (without using the hz serve CLI), but we don't have any formal documentation of this.

http://horizon.io/docs/server/ only covers how to use the CLI to start the server.

Some of the server options are also slightly different from the CLI options (e.g. hostname and port are separate options): https://github.com/rethinkdb/horizon/blob/061f784fa6f8cd6b30ffcaf6b88e823e0e386b3f/server/src/schema/server_options.js#L5 .

Horizon server readme/docs shouldn't mention npm dependencies

I think we should only mention openssl, rethinkdb, and node.js since those aren't handled by the package.json and the user needs to install them

Configuration precedence isn't mentioned anywhere

People might be confused when they set config in the .toml file but then also set --dev which then override those things.

config file > environment variables > flags

https://github.com/rethinkdb/horizon/blob/next/cli/src/serve.js#L478-L482

Directory tree image

https://github.com/rethinkdb/horizon-docs/blob/master/images/hz-dirs.png

.hzconfig is now stored in .hz/config.toml
.hz will also later store config for a certain other project related to horizon and similarly named.

Document write conflict resolution

jlu on Hacker News asked:

Does it handle conflict resolution, ike when two people modified the same field at almost the same time?

I know that we use versions internally to detect concurrent modifications between reading a document and a call to replace.

I'm not sure if this also applies to upsert. I'm also not sure what error we emit on the write operation if there is a conflict.
@Tryneus can you clarify?

We need to document this and provide some guidance for how to deal with conflicts. I'm not sure where this would go best. Maybe just into the Collection API entries for the write operations? Or into the FAQ, and then we can link it from the API docs?

In general I think we should also provide error handlers for write operations more consistently throughout the example codes to illustrate that writes can fail (by using subscribe on the write terms).

RemoveAll example not working

Hello,

Maybe im doing this the wrong way, but i tried the example:

const hz = Horizon();
const messages = hz("messages");

// get all messages from Bob and Agatha...
var messageList = messages.findAll({from: "bob"}, {from: "agatha"}).fetch();

// ...and delete them
messages.removeAll(messageList);

I couldn't get this working since removeAll expects an object, and messageList seems to be just a snapshot? This is how I made it, but then again, i'm a n00b:

const hz = Horizon();
const messages = hz("messages");

// get all messages from Bob and Agatha...
messages.findAll({from: "bob"}, {from: "agatha"}).fetch().subscribe(messageList => {
     // ...and delete them
     messages.removeAll(messageList);
});

Document horizon constructor arguments

Right now it's in client/src/index.js:

• host: hostname of horizon server (defaults to window.location)
• secure: whether the server should use secure websockets (defaults to true)
• path: the reserved namespace path for horizon (defaults to "horizon")
• lazyWrites: whether to emit write operations lazily or immediately (defaults to false)
• authType: What kind of authentication to use (defaults to 'unauthenticated') options are token | anonymous | unauthenticated

Unauthenticated is where everyone essentially gets the same unauthenticated user
Anonymous is where you get an ad-hoc user, but if you clear your browser cache no one can ever log in as that user again

Token is what you want if you're authenticating with Twitch/Twitter/Github/Google/ (right now all OAuth of one form or another, but this may not always be true)

Lazy writes are something like:

var horizon = Horizon()
horizon('documents').insert({a: 'document'}) # insert is sent right here

var horizon = Horizon({lazyWrites: true})
var query = horizon('documents').insert({a: 'document'}) # nothing is sent to the server yet

query.forEach(insertedUUID => {
  console.log(`Document ${insertedUUID} was created`)
}) # .forEach or .subscribe causes the insert to be sent

Horizon API: constructor now accepts a JWT token

The Horizon object now accepts a JWT token, which makes it easier to integrate with non-browser-based authentication methods in environments like React Native.

You can also use a token from hz make-token and provide it directly to the Horizon object.

Schema toml format doesn't describe how to specify indexes

If you create a horizon app using --auto-create-collection and --auto-create-indexes, and then do hz get-schema on it, you will get stuff that looks like:

[collections.test_data]
indexes = ["a","a_b","a_c","a_id","abracadabra","b","field","id_$hz\\_v$","score","time","value_time"]

These index names have a particular format that tells horizon how to create the index when it's passed back to set-schema, but this isn't documented anywhere. We should document it so people are able to specify their indexes without needing to have horizon autogenerate them first.

@Tryneus can probably give us exact details on how the mapping works

Example: Horizon and React Native

We have a lot of users who are experimenting with Horizon and React Native (e.g. rethinkdb/horizon#255 and rethinkdb/horizon#477), and an open issue for building an example app: rethinkdb/horizon#285

We should have an article that introduces the example and describes using Horizon with React Native.

Added a new `access_control_allow_origin` option in config.toml

See rethinkdb/horizon#565

It directly sets this http header for statically served files:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS#Access-Control-Allow-Origin

Basically, what this does is let you specify which domains should be allowed to initial xhr requests that fetch the /auth_methods resource. The basic use case is you serve your javascript from a cdn, and the script on the cdn is allowed to fetch this resource exposed by the horizon server. If we add more http endpoints in the future, we'll also set the header on those endpoints.

Users and groups

In http://horizon.io/docs/users/:

horizon.users.find("D6B8E9D0-CD96-4C01-BFD6-2AF43141F2A7").fetch().subscribe(
    (user) => {
        user.data.name = "Bob";
        // store() ? isn't it replace()?
        // store() docs: "(...) must have id values that do not already exist
        // in the collection or an error will be raised"
        horizon.users.store(user); 
    }
);

Added .take to Observable methods

This came in in the most recent version http://reactivex.io/rxjs/class/es6/Observable.js~Observable.html#instance-method-take

Incorrect `auto_create_table` option in "Configuring Horizon"

As seen in rethinkdb/horizon#442, the auto_create_table option is actually auto_create_collection.

This should be updated in Configuring Horizon. We should verify other Horizon server options.

Getting started document doesn't give enough structure

Opened by @danielcompton in rethinkdb/horizon#402 :

The getting started document has a lot of good information, but doesn't tie all of the information together into an application. For example:

• In the "putting it all together" section, it doesn't say where the JS file should be saved.
• renderChats is mentioned once in the getting started document, but never defined AFAICT.
• There is no updated HTML page to render the chat messages into.

Lots of good stuff there, but not quite enough to get a working application on my part without expending more brain energy to fill in the gaps.

Quick start should mention minimum required Node version

https://github.com/rethinkdb/horizon-docs/blob/master/quickstart.md and https://github.com/rethinkdb/horizon/blob/next/GETTING-STARTED.md
currently don't state which node version is required for using Horizon.

At least Node 4.4 seems to be fine, but probably we can support from 4.0 up?

Also npm install -g horizon should probably be sudo npm install -g horizon.

Ensure docs don't refer to onConnected anywhere

Anywhere onConnected is mentioned, it should be replaced with onReady

Write user management documentation

https://github.com/rethinkdb/horizon-docs/blob/master/users.md is currently a stub.

It should explain the users pseudo table (when it gets implemented in rethinkdb/horizon#3 ), as well as group management.

It can link to the separate authentication and permission documents, so it doesn't need to cover those parts.

Related to #3

Basic Docker setup

We should have an article on using Horizon in a basic Docker setup (@dalanmiller is working on this). Ideally we'd include details on using LetsEncrypt for a basic, production-ready setup with SSL / TLS.

@dalanmiller: could you include an outline and details that should be covered?

Document permissions system

Incorrect mention of user/password in authentication docs

In the short summary at the top of the documentation for authentication, for Anonymous auth the sentence "Your application will need to prompt for username and password." is incorrect.

That sentence should probably just be removed, it's correct otherwise.

Permissions cookbook page

Since the permissions system is very wide open and flexible, it makes sense to have a permissions cookbook page that shows you how to implement common types of rules, like:

• giving an admin user permission to do any operation,
• creating a group to administrate another group
• giving a user ownership of a group (power to create and destroy it)

Write server documentation

https://github.com/rethinkdb/horizon-docs/blob/master/server.md is currently a stub.

I think it needs to contain information on server behavior (e.g. which network ports it provides) command line arguments, and the configuration file.

It's now possible to get the handshake value from the horizon object

We have a subobject for every horizon instance called utensils (not sure if documented). It now has a new attribute, handshake which will get you raw access to the handshake information.

Everything in .utensils is considered a low-level api, so we probably should have a special page for it or something, rather than documenting it with everything else. The utensils api is also intended to be for things that are a bit less stable or internal.

Example of handshake:

horizon.utensils.handshake.subscribe((handshake) => {
 console.log('UserId is:', handshake.id)
})

It has a few fields in it, but the only one we should document right now is id, which is the currently logged in userId. The other fields are unstable and subject to change.

Also note that subscribing to .handshake won't cause a connection to the server, it only returns once the handshake has occurred. To force a connection, you still need to use horizon.connect() or some other query

Add docs page for contributors

... with information on where to go, links to resources etc.

Document `hz make-token` and how to boot-strap a user

the hz make-token command is going to be added via rethinkdb/horizon#390

Current limitations document

We should add a document outlining some current limitations in the server, and certain operations (as well as temporary workarounds as features get added). For example:

• Reconnection logic
• Pagination
• User / password authentication
• GraphQL support

Other suggestions are welcome.

Document the `context` object in permissions.md

That section isn't there yet, because the contents of the object were still unclear.

Document how to extend a Horizon backend

Copying this comment from @sunjay rethinkdb/horizon#466 here:

The docs specify that Horizon can be extended to add custom backend functionality:

Horizon exposes a powerful API that allows writing sophisticated realtime applications. However, we carefully designed Horizon to provide an upgrade path in case your application outgrows the Horizon API. If you need to add backend functionality, instead of using the standalone Horizon server you can require Horizon in Node.js as a library, and mix and match Horizon services along with the rest of the Node.js ecosystem. Since Horizon is built on top of RethinkDB, your app will have full access to the underlying database, and you can write arbitrarily complex code without being limited by Horizon’s base APIs and tools.

However I haven't been able to find any documentation about how to do any of that. Can something about this be added?

My use case is some rules that need to be enforced (and hidden). I can't manage with simple validation functions. I want to add a custom endpoint on the backend (or something of that nature) to run my code. How would this look with horizon? Would this be a custom endpoint or would I be adding a method to the horizon object? The docs would hopefully answer these questions.

Horizon example: notifications

A simple notification scheme:

Users create documents in the 'notifications' collection.
Each document has a sender, receiver and read field
Permissions would be:

• collection('notifications').store({sender: context.id, recipient: any(), read: false})
• collection('notifications').findAll({ recipient: context.id }).anyRead()
• collection('notifications').update({ recipient: context.id, read: any() })

Users create a changefeed like hz('notifications').findAll({recipient: userId}).watch(), and can do store to send a notification to another user, and update to mark their own notifications as read.

Integrations needs to be written

There is a link to /docs/integrations but no integrations.md file. Someone who knows what projects Horizon is already integrated with needs to come up with a bulleted list for this.

fetch docs probably shouldn't mention forEach

Technically, you can call forEach on fetch, but it's not very useful since there will only ever be a single item (an array, or in the case of a .find query, an object). We should probably just remove mention of forEach from fetch. It's more useful for .watch where you'll actually get multiple items on the observable

Horizon libraries and tools

There are a number of Horizon tools that are popping up on GitHub. For example:

• lovli.js -- a boilerplate for developing react+redux applications with rethinkdb/horizo
• react-horizon -- React bindings for rethinkdb/horizon.
• react-hz -- React Horizon makes it easier to use your React application with horizon.io realtime backend.
• horizontal -- example of Vue.js and Horizon.
• elm-horizon -- WIP integration of Elm and Horizon.
• ember-cli-horizon -- An Ember service for managing a Horizon connection, and an adapter to request data from that connection.

There are probably a few more that I'm missing from this list.

We should have a list of example Horizon projects, and useful libraries and tools in the ecosystem.

Link to RxJS when we talk about Observables

We should link to the corresponding sections of http://rxjs.codeplex.com/wikipage?title=Observable&referringTitle=Documentation in a few places of our documentation. For example we currently have a description of the forEach method in https://github.com/rethinkdb/horizon-docs/blob/master/api/collection.md#collectionforeach-foreach , but don't point out that this is the standard Observable forEach.

Carefully check all uses of forEach in examples, it's no longer a synonym of .subscribe

Any calls that look like this:

foo.forEach(
 (item) => doSomething(item),
 (error) => handleError(error),
 () => onComplete()
)

Should be changed to use .subscribe() like:

foo.subscribe({
 next(item) {
  // do something for each item
 },
 error(err) {
   // do something with error
 },
 complete() {
  // do something on completion
 },
)

Now .forEach() should only take one function, the iteration function. If the example isn't explicitly trying to show error handling, it might make sense to leave the example using .forEach and just remove the second two functions.

Permissions expansion

Right now the permissions document is a good outline, but it should probably mention some practical things like

• query template rules are faster than javascript validator functions. Try to use them wherever possible
• query template rules can be combined with js validators, do this to have a "first pass" to rule out classes of validators so they don't have to run on every incoming query.

Write Tutorial

We currently have a quick start document https://github.com/rethinkdb/horizon-docs/blob/master/quickstart.md , but no real tutorial yet ( https://github.com/rethinkdb/horizon-docs/blob/master/tutorial/1-start.md )

Make horizon-docs independently buildable

Need jekyll build so I can enable this repo to be autodeployed for PRs.