couchbase/sync_gateway

Manages access and synchronization between Couchbase Lite and Couchbase Server

Sync Gateway

Sync Gateway is a horizontally scalable web server that securely manages the access control and synchronization of data between Couchbase Lite and Couchbase Server.

Download

Download Sync Gateway and other Couchbase packages for Linux, Windows and macOS at Couchbase Downloads.

Build from source

Pre-requisites

To build Sync Gateway from source, you must have the following installed:

• Go 1.19 or later.
• Building the Enterprise Edition requires access to private repos, and is intended for internal use only.

Install Go

See Installing Go

Build instructions

See the Extended Build Instructions to build with dependency pinning via the repo multi-repository tool.

Resources

• Sync Gateway Documentation
• Sync Gateway Issue Tracker
• Couchbase Products:
  • Sync Gateway
  • Lite
  • Mobile
  • Server
  • Developer SDKs
• Couchbase Downloads
• Couchbase Discussion Forum
• Couchbase Mobile Mailing List

License

Business Source License 1.1

sync_gateway's Issues

you should be able to write documents even if you can't read them

so that I can write a document to a channel that you can read, but I cannot. also so that users with an empty channels list can write documents.

it should be noted prominently that validation functions are crucial if you want to control write access, otherwise anyone can write any document

Ability to partial wildcard match user channels

It would be useful to be able to match user channels by wildcard.

e.g. I would like to add a set of channels to a user where the set of channels is dynamic.

I would like to add a channel entry on the use object like"

CHANNEL_PREFIX*

The user would be allowed to access:

CHANNEL_PREFIX1
CHANNEL_PREFIX2
.
.
.
CHANNEL_PREFIXN

Docs are no longer being assigned to channels via explicit property?

It appears that assigning channels to a doc via the explicit property route may have broken with a recent commit...

This used to work just fine with the Grocery Sync app ⤵

    // Create the new document's properties:
    NSDictionary *inDocument = [NSDictionary dictionaryWithObjectsAndKeys:text, @"text",
                                [NSArray arrayWithObjects:@"achannel", nil], @"channels",
                                [NSNumber numberWithBool:NO], @"check",
                                [TDJSON JSONObjectWithDate: [NSDate date]], @"created_at",
                                nil];

However, it no longer seems to work—this is what I see now in the sync_gateway log:

_Assigning doc "F21EE2FD-26C2-49AF-A288-1D9F74C1922F" to channels []_

Syncing as an "admin" user (with channels:["*"]) works just fine though.

GUEST user has many channels?

curl http://localhost:4985/sync_gateway/user/GUEST
{"disabled":true,"admin_channels":[],"all_channels":["ch-1C8744ED-2D04-43DA-A46F-97A4B613CC2B","ch-149EE60F-B8BF-4F4F-A25D-FCBB4124BC50","ch-E62ECB2D-16BB-4AE9-8DD9-3D2AFBE23F5F","ch-4EC06FC1-AA06-4337-95C9-78207446ECEC","ch-3A3A79AD-2FB3-468B-AC21-B1A50E8BB4D2"]}

Error on 'go get' with latest changes

FYI, since yesterday, the following error happens when running 'go get' on master.

> go get -u github.com/couchbaselabs/sync_gateway                                   
# github.com/couchbaselabs/sync_gateway/base
../lib/Go/src/github.com/couchbaselabs/sync_gateway/base/error.go:60: function ends without a return statement

Cannot do a first replication from CouchDB to sync_gateway

Environment

• Ubuntu Linux 10.04 32 bit
• go 1.0.3
• CouchDB 1.2.0 working on 172.16.1.8 on standard port 5984
• couchbase 2.0 community edition installed on a 3 computer cluster (including 172.16.1.8)

Done the following

• created sync_gateway bucket
• started main.go on 172.16.1.8 just with -verbose=true parameter line
• created a couchdb database called "apia" on 172.16.1.8 with 4 documents containing "channels" attribute
• tried to do a first replication from CouchDB to sync_gateway issuing the replicate comand to CouchDB instance on 172.16.1.8

teo@teo:~$ curl  -X POST "http://172.16.1.8:5984/_replicate" -d '{"source":"http://172.16.1.8:5984/apia","target":"http://localhost:4984/sync_gateway"}' -H "Content-Type: application/json"
{"error":"db_not_found","reason":"could not open http://localhost:4984/sync_gateway/"}
teo@teo:~$ curl -X GET http://172.16.1.8:4984/sync_gateway/
 {"db_name":"sync_gateway","doc_count":0,"update_seq":0}

Tried the same thing without and then with user "teo" defined on 172.16.1.8:4985, no change at all!
Any ideas?
Teo

continuous and longpoll not waiting

When I run something like this: curl -X GET 'http://localhost:4984/jndev/_changes?feed=continuous' -u "testuser:test1234"

It immediately returns to the command prompt without waiting.

Though the basecouch log shows:

2013/01/05 01:59:03 GET /jndev/_changes?feed=continuous
2013/01/05 01:59:03 waiting for a revision...

It seems like when I set feed=longpoll with a user that I've defined channels for, the curl command waits but even if I add new documents to an appropriate channel through couchbase the curl command just continues to wait. If I authenticate with a user that has channels=* then curl doesn't wait.

If I use feed=continuous, curl doesn't wait regardless of the user I authenticate with.

Though in all of these situations, the basecouch log says "waiting for a revision...".

push design doc via admin port

If we did it like that I wouldn't have to set up an admin account to push the channelmapper and validation functions.

attachments not pulling to TouchDB

It looks like TouchDB is doing this, and not getting the attachment back.

curl 'http://jchris:jchris@localhost:4984/sync_gateway/3685C224-0056-46E8-A234-DE121E441E14?rev=2-9f85dd7a2b0a5d8ba08097918181e95e&revs=true&attachments=true&atts_since=%5B%221-fee61c00c35908d65e79d30798363f49%22%5D' 

It should be getting the attachment body, but it's not. If I remove the atts since, I get the body inline in base64.

Here is the response to the request with att_since specified:

{"_attachments":{"photo.jpg":{"content_type":"image/jpg","digest":"sha1-SeNdYuBu6eswUmmB+B2C9qj0CPs=","length":13296,"revpos":2,"stub":true}},"_id":"3685C224-0056-46E8-A234-DE121E441E14","_rev":"2-9f85dd7a2b0a5d8ba08097918181e95e","_revisions":{"ids":["9f85dd7a2b0a5d8ba08097918181e95e","fee61c00c35908d65e79d30798363f49"],"start":2},"author_id":"jchris","created_at":"2013-01-22T19:48:21.839Z","seq":0,"text":"","thread_id":"cjf3bhbag199","type":"message","updated_at":"2013-01-22T19:48:21.839Z"}

Sync behavior when last doc removed from one of many sync channels

In auth/role.go, the following operation, prevents sync of any docs if any of the requested channels does not exist.

// Returns an HTTP 403 error if the Role is not allowed to access all the given channels.
// A nil Role means access control is disabled, so the function will return nil.
func authorizeAllChannels(princ Principal, channels ch.Set) error {
        var forbidden []string
        for channel, _ := range channels {
                if !princ.CanSeeChannel(channel) {
                        if forbidden == nil {
                                forbidden = make([]string, 0, len(channels))
                        }
                        forbidden = append(forbidden, channel)
                }
        }
        if forbidden != nil {
                return princ.UnauthError(fmt.Sprintf("You are not allowed to see channels %v", forbidden))
        }
        return nil
}

If a client sets up a pull repl with a list of valid channels, then a third-party removes the last doc from one of the channels, presumably the pull repl will no longer sync any documents?

How can a client detect this and update the channel list of the pull repl accordingly?

Support for custom Persona verifiers

Currently the persona Verifier service URL is hardcoded, supporting a custom verifier URL per DB in a .conf file would provide a flexible integration point for identity/CRM management systems.

These systems could implement features such as, email blacklists, whitelists, subscription validation e.t.c

A custom verifier would have to implement the same API as the default Mozilla service.

Only first if statement evaluated in sync function

I need to write a function that generates the channel map differently depending on the document content.

Here is the outline of the if else if

function(doc, oldDoc, userCtx, secObj) { if (doc.channel_id) { channel("ch-"+doc.channel_id); } else if (doc.other_id) { channel("ch-"+doc.other_id); }}

I have reversed the order of the if statements and only the first one ever evaluates, so only those documents that match the first test are mapped to channels.

Out of memory in go, after replicating one million records

I build a CouchDB database with 2 millions documents containing 8 attributes (2 to 10 chars long) and a "channel" attribute with 2 strings.
Tried to replicate it to sync_gateway, after a hour (CouchBase reports already 2.587.000 records in data bucket) the go program has crashed with the following message

...
2013/01/31 22:20:13     Assigning doc "c5e35237e98fc32e21b8be5142097c30" to channels []
2013/01/31 22:20:13     Assigning doc "c5e35237e98fc32e21b8be51420988ab" to channels []
2013/01/31 22:20:13     --> 201 
2013/01/31 22:20:40 HEAD /sync_gateway/
2013/01/31 22:20:42 GET /sync_gateway/
2013/01/31 22:20:42 GET /sync_gateway/_local/eff358b2433202e86fd73599c24b3d84
2013/01/31 22:20:42     --> 404 missing
2013/01/31 22:20:42 GET /sync_gateway/_local/784b528714c810ff80f22b6a202ae4f5
2013/01/31 22:20:42     --> 404 missing
2013/01/31 22:20:42 GET /sync_gateway/_changes?feed=normal&style=all_docs&since=0&heartbeat=10000
2013/01/31 22:20:47 GET /sync_gateway/
...
2013/01/31 22:23:39 GET /sync_gateway/
runtime: memory allocated by OS (0xb326a000) not in usable range [0x18a00000,0x98a00000)
throw: out of memory

goroutine 516 [running]:
reflect.unsafe_NewArray(0x18a7b000, 0x826fff4, 0x8d, 0x81eeb58, 0x176, ...)
    /usr/local/go/src/pkg/runtime/iface.c:702 +0x80
reflect.MakeSlice(0x18a7b000, 0x81eeb58, 0x5e, 0x8d, 0x81eeb58, ...)
    /usr/local/go/src/pkg/reflect/value.go:1650 +0x14a
encoding/json.(*decodeState).array(0x58b430e4, 0x81eeb58, 0x62ac6c84)
    /usr/local/go/src/pkg/encoding/json/decode.go:364 +0x471
encoding/json.(*decodeState).value(0x58b430e4, 0x81eeb58, 0x62ac6c84)
    /usr/local/go/src/pkg/encoding/json/decode.go:246 +0x1ba
encoding/json.(*decodeState).object(0x58b430e4, 0x826b5b0, 0x62ac6c80)
    /usr/local/go/src/pkg/encoding/json/decode.go:555 +0x946
encoding/json.(*decodeState).value(0x58b430e4, 0x81ec4e8, 0x62ac6c80)
    /usr/local/go/src/pkg/encoding/json/decode.go:249 +0x19f
encoding/json.(*decodeState).unmarshal(0x58b430e4, 0x81ec4e0, 0x62ac6c80, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/encoding/json/decode.go:136 +0x121
encoding/json.(*Decoder).Decode(0x58b430d0, 0x81ec4e0, 0x62ac6c80, 0x18aa4820, 0x74a37500, ...)
    /usr/local/go/src/pkg/encoding/json/stream.go:48 +0xff
github.com/couchbaselabs/go-couchbase.(*Bucket).ViewCustom(0x18a6e3c0, 0x82a306c, 0xc, 0x8298e6c, 0x8, ...)
    /home/teo/eclipse/workspace/bd2012/src/github.com/couchbaselabs/go-couchbase/views.go:105 +0x403
github.com/couchbaselabs/sync_gateway/db._func_001(0x1e0cd680, 0x1e0cd678, 0x1e0cd650, 0x1e0cd658, 0x1e0cd668, ...)
    /home/teo/eclipse/workspace/bd2012/src/github.com/couchbaselabs/sync_gateway/db/changes.go:97 +0x244
created by github.com/couchbaselabs/sync_gateway/db.(*Database).ChangesFeed
    /home/teo/eclipse/workspace/bd2012/src/github.com/couchbaselabs/sync_gateway/db/changes.go:163 +0x3d2

Running again the go program and trying to resume replication fails with the same message.

2013/02/01 00:12:17 GET /sync_gateway/
2013/02/01 00:12:19 GET /sync_gateway/_changes?feed=normal&style=all_docs&since=0&heartbeat=10000
2013/02/01 00:12:22 GET /sync_gateway/
2013/02/01 00:12:28 GET /sync_gateway/
2013/02/01 00:12:33 GET /sync_gateway/
runtime: memory allocated by OS (0xb3216000) not in usable range [0x18a00000,0x98a00000)
throw: out of memory

goroutine 12 [running]:
encoding/json.Unmarshal(0x98939d90, 0x66, 0x67, 0x81ec580, 0x9896f270, ...)
    /usr/local/go/src/pkg/encoding/json/decode.go:55 +0x29
github.com/couchbaselabs/sync_gateway/db.RevTree.UnmarshalJSON(0x98937d20, 0x98939d90, 0x66, 0x67, 0x0, ...)
    /home/teo/eclipse/workspace/bd2012/src/github.com/couchbaselabs/sync_gateway/db/revtree.go:77 +0x110
github.com/couchbaselabs/sync_gateway/db.(*RevTree).UnmarshalJSON(0x9896f1cc, 0x98939d90, 0x66, 0x67, 0x0, ...)
    /home/teo/eclipse/workspace/bd2012/src/github.com/couchbaselabs/sync_gateway/db/attachment.go:0 +0x74
encoding/json.(*decodeState).object(0x98873f80, 0x8281368, 0x9896f1cc)
    /usr/local/go/src/pkg/encoding/json/decode.go:415 +0xc7
encoding/json.(*decodeState).value(0x98873f80, 0x8281368, 0x9896f1cc)
    /usr/local/go/src/pkg/encoding/json/decode.go:249 +0x19f
encoding/json.(*decodeState).object(0x98873f80, 0x82772e0, 0x9896f1b0)
    /usr/local/go/src/pkg/encoding/json/decode.go:555 +0x946
encoding/json.(*decodeState).value(0x98873f80, 0x81ec560, 0x9896f1b0)
    /usr/local/go/src/pkg/encoding/json/decode.go:249 +0x19f
encoding/json.(*decodeState).unmarshal(0x98873f80, 0x81ec558, 0x9896f1b0, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/encoding/json/decode.go:136 +0x121
encoding/json.Unmarshal(0x98939d20, 0xd7, 0xd7, 0x81ec558, 0x9896f1b0, ...)
    /usr/local/go/src/pkg/encoding/json/decode.go:65 +0xcd
github.com/couchbaselabs/sync_gateway/db._func_001(0x18b20d78, 0x18b20d70, 0x18b20d48, 0x18b20d50, 0x18b20d60, ...)
    /home/teo/eclipse/workspace/bd2012/src/github.com/couchbaselabs/sync_gateway/db/changes.go:126 +0x85b
created by github.com/couchbaselabs/sync_gateway/db.(*Database).ChangesFeed
    /home/teo/eclipse/workspace/bd2012/src/github.com/couchbaselabs/sync_gateway/db/changes.go:163 +0x3d2

Teo

external sync functions

It should be possible to run the sync function via an external process that speaks to the server via stdio or sockets.

The config file would specify a process to launch, instead of a sync function body. Then the server would launch that process and send documents to it, and expect JSON responses (either a list of channels and access grants, or a validation error).

User roles (a la CouchDB)

It would be useful to be able to assign users to groups/roles, and then be able to grant roles access to channels. That would make access control more flexible and reduce the amount of work that access() calls need to do.

Admin API to list all users/roles

On the admin port, GET /db/users/ should return a JSON list of all user accounts — at least the names, and optimally other metadata so that an admin UI (see #60) doesn't have to make multiple requests to display a list of users.

Same goes for roles, of course.

Allow multiple BaseCouch instances to share a bucket (AKA "shared-nothing")

For scalability, we need to support any number of BaseCouch processes sharing the same Couchbase bucket.

Here is a design wiki for this

Currently this almost works; the exception is the _changes feed. It may need to wait for changes to the database, and the mechanism it uses for this is based on in-process Go notifications. So a _changes feed request being served by one process will wait forever without noticing a database change made by another process.

To fix this we need to hook into the Couchbase Server 'TAP' feed, or else some new mechanism TBD.

obsolete validation warning

no need to warn of lacking validation when sync function contains "throw"

Channel inspection API

For debugging and/or visualization purposes, it would be really useful to have some kind of local API to:

• get the list of 'active' channels,
• get the list of docs belonging to a given channel.

(Re)implement compaction

Ever since I changed the schema to map one data document to one Couchbase document, database compaction doesn't work. The issue is that the bodies of all the old revisions are stored inside the document body, down in the _sync property. This means that getting rid of them would mean updating every single data document in the bucket. Ugh.

Not sure what to do about this. To make compaction more efficient we'd need the old revision bodies to be stored as separate documents (either one per revision, or one doc that holds all the revisions for a data doc.) But that conflicts with our goal of not messing with the bucket too much.

_all_docs doesn't respect authorization

GET /db/_all_docs returns all documents, whether or not you have access to their channels. Instead it needs to filter by accessible channels, similar to _changes.

anyone with a valid browserid assertion should get an implied user

this will allow signup via sync. so that eg a background changes listener can see new wikis and then grant the associate email address access to those wikis' channels

Store doc attachments in CBFS

Storing doc attachments in the main bucket is inefficient, limits them to something like 40MB, and puts lots of extra documents in the bucket (which we want to avoid.)

We should instead store them externally, probably as a content-addressable store the way Couchbase Lite does. CBFS would be the best way to do this, since the files need to be accessible from anywhere in the cluster.

config files should accept multi-line strings

No one wants to edit JavaScript where all the newlines look like \n and quotes like "

Failure reported on second replication of a couchdb database after editing a single document

Got the new sources - installed and succeeded in make a first replication from a CouchDB database with 3 documents to the sync_gateway

teo@teo:~$ curl -X POST http://172.16.1.8:5984/_replicate \
   -d '{"source":"http://172.16.1.8:5984/apia","target":"http://172.16.1.8:4984/sync_gateway"}'  \
   -H "Content-Type: application/json"
{"ok":true,"no_changes":true,"session_id":"ced39a4fc582193438e008449fa587ae","source_last_seq":8,"replication_id_version":2,"history":[{"session_id":"ced39a4fc582193438e008449fa587ae","start_time":"Thu, 31 Jan 2013 11:50:18 GMT","end_time":"Thu, 31 Jan 2013 11:50:18 GMT","start_last_seq":0,"end_last_seq":8,"recorded_seq":8,"missing_checked":5,"missing_found":5,"docs_read":5,"docs_written":5,"doc_write_failures":0}]}

After editing a single document in the CouchDB database, I tried a second replication:

teo@teo:~$ curl -X POST http://172.16.1.8:5984/_replicate \
  -d '{"source":"http://172.16.1.8:5984/apia","target":"http://172.16.1.8:4984/sync_gateway"}' \
  -H "Content-Type: application/json"
{"error":"checkpoint_commit_failure","reason":"Error updating the target checkpoint document: conflict"}

Tried to GET the changed document from sync_gateway:

teo@teo:~$ curl -X GET http://172.16.1.8:4984/sync_gateway/2ab65b876a29ff310ae2882c4306c5b3
{   "_id":"2ab65b876a29ff310ae2882c4306c5b3",
    "_rev":"2-4cc34455eb88aff3fc30a1c2489050c3",
    "channels":["pndr","apia","sport"],
    "nume":"Ovidiu",
    "oras":"Mallnitz"}

I got the last version of the document ( with oras:"Mallnitz" ) , this is OK!
But what is the {"error":"checkpoint_commit_failure","reason":"Error updating the target checkpoint document: conflict"} error from the second replication?

Teo

admin port should mirror regular port but with addition of some _special APIs

Right now if you want to inspect a database, you need to have some credentials to access it on the public port. But that means you have to create an admin user and grant them access to all channels.

It would simplify things if you could access http://mygateway:4985/my_database/_changes as well as documents directly on the admin port, without supplying credentials.

I think this will require moving the /my_database/user and /my_database/role namespaces to /my_database/_user and /my_database/_role

TouchDB local_api push replication to base couch _bulk_docs warning

When replicating from a local_api TouchDB database to a basecouch instance I get the following error continuously (every 60 seconds)

My documents and attachments have been transferred, so I'm not sure what this document is.

I am seeing some issues with design documents which I posted in another thread, not sure if this is related.

21:52:41.621| Sync: TDPusher[http://192.168.0.13:4984/basecouch] RETRYING, to transfer missed revisions...
21:52:41.687| Sync: TDPusher[http://192.168.0.13:4984/basecouch]: Sending 1 revisions
21:52:41.709| WARNING*** : TDPusher[http://192.168.0.13:4984/basecouch]: _bulk_docs got an error: {
error = missing;
id = da39a3ee5e6b4b0d3255bfef95601890afd80709;
}
21:52:41.710| Sync: TDPusher[http://192.168.0.13:4984/basecouch]: Failed to xfer 1 revisions; will retry in 60 sec

Additionally this warning appears in the basecouch console every iteration of the above error

POST /basecouch/_bulk_docs
2012/12/25 04:31:49 WARNING: storeAttachments: no such parent rev "" to find map[digest:sha1-2jmj7l5rSw0yVb/vlWAYkK/YBwk= length:0 content_type:application/octet-stream revpos:1]

Finish implementation of _bulk_get REST API

I started working on an experimental _bulk_get REST API that will allow clients to greatly speed up their pull replications. Still need to finalize the design and add unit tests. (Of course we also need an implementation in CBL/TouchDB.)

No longer get info in console on document mapping/access

This is not a major, but prior to the work to sort out throws in the JS sync function every document that was passed to sync had a console entry listing the channels and access applied by the sync function.

So output used to look something like:

2013/03/06 14:05:02 Doc "EF218C65FE223DDD1304508C58BE09ACC7EEBC48" in channels ["EF218C65FE223DDD1304508C58BE09ACC7EEBC48"]
2013/03/06 14:05:02 Doc "EF218C65FE223DDD1304508C58BE09ACC7EEBC48" grants access: map[]

This no longer appears, the mapping is taking place (I make that statement based on what I see in the DB)

Here is the entire console output for three docs being sync'd:

18:20:32.407356 HTTP: GET /sync_gateway/_local/e1409ef09f09f0b88dd299f0d99e1416fb4230e2
18:20:32.408560 HTTP: --> 404 missing
18:20:32.412896 HTTP: GET /sync_gateway/_local/c873146004ce79ba6bc2885df5c6fd54f5b15665
18:20:32.414600 HTTP: --> 404 missing
18:20:32.438746 HTTP: POST /sync_gateway/_revs_diff
18:20:32.472574 HTTP: GET /sync_gateway/_changes?feed=normal&heartbeat=300000&style=all_docs&limit=100&filter=sync_gateway/bychannel&channels=*
18:20:32.500269 HTTP: PUT /sync_gateway/DF40608948B59C24E3387646325AE8F312BB9B4F?new_edits=false
18:20:32.503537 HTTP: POST /sync_gateway/_bulk_docs
18:20:32.546855 HTTP: GET /sync_gateway/_changes?feed=longpoll&heartbeat=300000&style=all_docs&limit=100&filter=sync_gateway/bychannel&channels=*
18:20:32.673665 HTTP: GET /sync_gateway/_changes?feed=longpoll&heartbeat=300000&style=all_docs&since=2&limit=100&filter=sync_gateway/bychannel&channels=*
18:20:32.676898 waiting for a revision...
18:20:34.407299 ...done waiting
18:20:34.454595 waiting for a revision...
18:20:37.544482 HTTP: PUT /sync_gateway/_local/c873146004ce79ba6bc2885df5c6fd54f5b15665
18:20:38.541616 HTTP: PUT /sync_gateway/_local/e1409ef09f09f0b88dd299f0d99e1416fb4230e2

Here are the _sync sections for the three docs taken from the DB after sync completed, NOTE: one doc is not mapped to a channel so sync is correct:

"_sync": {
"rev": "1-c5d5575826f21ead8749fed2746a7d85",
"sequence": 2,
"history": {
"revs": [
"1-c5d5575826f21ead8749fed2746a7d85"
],
"parents": [
-1
],
"bodies": [
""
]
},
"channels": {
"[email protected]": null
}
}

"_sync": {
"rev": "1-297d2fc5d3578b07d6c2f4bfe5506c45",
"sequence": 3,
"history": {
"revs": [
"1-297d2fc5d3578b07d6c2f4bfe5506c45"
],
"parents": [
-1
],
"bodies": [
""
]
},
"channels": {
"sha1-30BgiUi1nCTjOHZGMlro8xK7m08=": null
},
"access": {
"*": [
"sha1-30BgiUi1nCTjOHZGMlro8xK7m08="
]
}
}

"_sync": {
"rev": "1-4dbec45ad93f9ca502e512d4f36bb2a2",
"sequence": 1,
"history": {
"revs": [
"1-4dbec45ad93f9ca502e512d4f36bb2a2"
],
"parents": [
-1
],
"bodies": [
""
]
}
}

special logging when auth fails due to site mismatch

when I configure my gateway slightly differently than my app code (missing trailing slash...) it's not obvious what the problem is. I tend to blame everything else from Persona to my app code, before I find the real issue. It'd be nice to make this failure mode easier to detect.

pull with attachments to Apache CouchDB 1.2.0 fails

Only a handful of my documents were synced, and this crash in the logs indicates something is up with the attachments format:

working sync function results in rejected messages in console

Hi, after picking up the patch to improve writing JS errors to the console, a working sync function that calls the channel() operation results in mapped documents but the console no longer shows the information about the mapping and now displays messages for each document like:

Sync fn rejected: new={"_attachments":{"data":{"content_type":"application/octet-stream","digest":"sha1-30BgiUi1nCTjOHZGMlro8xK7m08=","length":1747625,"revpos":1,"stub":true}},"_id":"DF40608948B59C24E3387646325AE8F312BB9B4F","_rev":"1-297d2fc5d3578b07d6c2f4bfe5506c45","_revisions":{"ids":["297d2fc5d3578b07d6c2f4bfe5506c45"],"start":1}} old= -->

The document looks like this in the DB, which indicates the channel mapping worked:

{
"_attachments": {
"data": {
"content_type": "application/octet-stream",
"digest": "sha1-30BgiUi1nCTjOHZGMlro8xK7m08=",
"length": 1747625,
"revpos": 1,
"stub": true
}
},
"_sync": {
"rev": "1-297d2fc5d3578b07d6c2f4bfe5506c45",
"sequence": 3,
"history": {
"revs": [
"1-297d2fc5d3578b07d6c2f4bfe5506c45"
],
"parents": [
-1
],
"bodies": [
""
]
},
"channels": {
"sha1-30BgiUi1nCTjOHZGMlro8xK7m08=": null
}
}
}

continuous replication hangs on the first update

When I have a TouchDB running continuous sync from basecouch, I see a ton of requests to the _changes, like the longpoll is not waiting and so the client is issuing another request.

Then when there IS an update, the requests stop and I get output like this:

2012/12/18 15:32:53 GET /channelsync/_changes?feed=longpoll&heartbeat=300000&style=all_docs&since=137&limit=100&filter=channelsync/bychannel&channels=043527951231226325,28639595629647374,907225544564426
2012/12/18 15:32:53 GET /channelsync/_changes?feed=longpoll&heartbeat=300000&style=all_docs&since=137&limit=100&filter=channelsync/bychannel&channels=043527951231226325,28639595629647374,907225544564426
2012/12/18 15:32:53 GET /channelsync/_changes?feed=longpoll&heartbeat=300000&style=all_docs&since=137&limit=100&filter=channelsync/bychannel&channels=043527951231226325,28639595629647374,907225544564426
2012/12/18 15:32:53 GET /channelsync/_changes?feed=longpoll&heartbeat=300000&style=all_docs&since=137&limit=100&filter=channelsync/bychannel&channels=043527951231226325,28639595629647374,907225544564426
2012/12/18 15:32:53 GET /channelsync/_changes?feed=longpoll&heartbeat=300000&style=all_docs&since=137&limit=100&filter=channelsync/bychannel&channels=043527951231226325,28639595629647374,907225544564426
2012/12/18 15:32:53 POST /channelsync/_revs_diff
2012/12/18 15:32:53 POST /channelsync/_bulk_docs
2012/12/18 15:32:53 Assigning doc "testdoc" to channels []
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 GET /channelsync/_changes?feed=longpoll&heartbeat=300000&style=all_docs&since=137&limit=100&filter=channelsync/bychannel&channels=043527951231226325,28639595629647374,907225544564426
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 ...done waiting
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...
2012/12/18 15:32:53 waiting for a revision...

Document with attachment not being passed to sync function

I have seen this working but now when my iOS device syncs with sync_gateway the documents with attachments are not passed to the sync function, here is a console output from sync_gateway

2013/03/07 11:29:24 GET /sync_gateway/_local/e1409ef09f09f0b88dd299f0d99e1416fb4230e2
2013/03/07 11:29:24 GET /sync_gateway/_local/c873146004ce79ba6bc2885df5c6fd54f5b15665
2013/03/07 11:29:24 GET /sync_gateway/_changes?feed=normal&heartbeat=300000&style=all_docs&limit=100&filter=sync_gateway/bychannel&channels=*
2013/03/07 11:29:24 GET /sync_gateway/_changes?feed=longpoll&heartbeat=300000&style=all_docs&limit=100&filter=sync_gateway/bychannel&channels=*
2013/03/07 11:29:24 waiting for a revision...
2013/03/07 11:29:36 POST /sync_gateway/_revs_diff
2013/03/07 11:29:36 POST /sync_gateway/_bulk_docs
2013/03/07 11:29:36 POST /sync_gateway/_revs_diff
2013/03/07 11:29:36 JS: first if evaluated false
2013/03/07 11:29:36 JS: second if evaluated false
2013/03/07 11:29:36 Doc "E6170221157F793648E799025D7413DFEEB50A6E" grants access: map[]
2013/03/07 11:29:36 ...done waiting
2013/03/07 11:29:36 JS: first if evaluated true
2013/03/07 11:29:36 JS: second if evaluated false
2013/03/07 11:29:36 Doc "317585690691231914726A978C987EF45F0E55D5" grants access: map[]
2013/03/07 11:29:36 waiting for a revision...
2013/03/07 11:29:37 PUT /sync_gateway/DF40608948B59C24E3387646325AE8F312BB9B4F?new_edits=false
2013/03/07 11:29:39 ...done waiting
2013/03/07 11:29:39 GET /sync_gateway/_changes?feed=longpoll&heartbeat=300000&style=all_docs&since=2&limit=100&filter=sync_gateway/bychannel&channels=*
2013/03/07 11:29:39 waiting for a revision...
2013/03/07 11:29:42 PUT /sync_gateway/_local/c873146004ce79ba6bc2885df5c6fd54f5b15665

2013/03/07 11:29:45 PUT /sync_gateway/_local/e1409ef09f09f0b88dd299f0d99e1416fb4230e2

I am using a dummy sync function that does no mapping but logs whether tests for the existence of doc properties evaluate to true or false.

Two documents without attachments look like they get sync'd with POST /sync_gateway/_bulk_docs

And you can see the sync function being called

But the document that has an attachment looks like it gets sync'd with

PUT /sync_gateway/DF40608948B59C24E3387646325AE8F312BB9B4F?new_edits=false

And the sync function is not called.

The attachments property of the failing document is below:

"_attachments": {
"data": {
"content_type": "application/octet-stream",
"digest": "sha1-30BgiUi1nCTjOHZGMlro8xK7m08=",
"length": 1747625,
"revpos": 1,
"stub": true
}
}

Silent failure if sync function maps a document to a channel with an invalid name

I tried to map a document to the following channel name:

sha1-+92hjqzhtl7AxLfTQ1ATpIxEDtQ=

This contains characters that are not allowed in channel names.

The document does not get mapped to the channel, but the sync function fails silently with no error being written to the console.

basecouch default channel mapping not working

The default channel mapping (using channels property in documents) is not working with an out-of-the box installation.

I don't know the underlying cause but problem manifests in the getChannels operation of the document.go file. The db.channelMapper is not returning nil even though no mapper function has been set.

A naive workaround for initial testing is to always use the default mapper logic as shown below.

// Determines which channels a document body belongs to
func (db *Database) getChannels(body Body) (result []string) {
log.Printf("Entered getChannels")
//if db.channelMapper != nil {
// log.Printf("db.channelMapper is NOT null")
// jsonStr, _ := json.Marshal(body)
// result, _ = db.channelMapper.MapToChannels(string(jsonStr))

    //} else {
            // No ChannelMapper so by default use the "channels" property:
            log.Printf("No ChannelMapper so by default use the channels property")
            value, _ := body["channels"].([]interface{})
            if value == nil {
                    log.Printf("valus is nil")
                    return nil
            }
            result = make([]string, 0, len(value))
            for _, channel := range value {
                    channelStr, ok := channel.(string)
                    if ok && len(channelStr) > 0 {
                            log.Printf("\tAdding channel to doc %q", channelStr)
                            result = append(result, channelStr)
                    }
            }
    //}
    return

}

Add ability to block users from channels ('deny' mode)

it should be possible to manually block a given user from a channel, even if the sync function grants access to them by name or role.

Admin UI

the admin port :4985 should have an Admin UI for editing the sync function, browsing users, inspecting channels, etc.

Support direct writes to Couchbase Server bucket

We currently support direct read-only access to a Couchbase Server bucket, but we really need a transparent cluster info URL we can use in Smart Client SDK configuration. This should allow read and write access transparently to the client. It will still be possible to skip to direct read-only for high-volume access.
More info here: https://github.com/couchbaselabs/sync_gateway/wiki/Smart-Client-SDK-Access

Updated: Implementing the server API would be hard to implement and difficult to QA (against all of our SDKs.) What we're going to do instead is let clients write directly to the bucket, and then have the gateway detect those writes and add/update the _sync metadata to make them into bona fide revisions.

crash PUTing design doc with channelmapper function

This happens when setting a channelmapper function for the first time

2013/01/21 16:19:03     Assigning doc "_design/channels" to channels []
2013/01/21 16:19:03 http: panic serving 10.17.35.141:54022: runtime error: invalid memory address or nil pointer dereference
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:589 (0x4cd4f)
    _func_004: buf.Write(debug.Stack())
/usr/local/Cellar/go/1.0.3/src/pkg/runtime/proc.c:1443 (0x116a1)
    panic: reflect·call(d->fn, d->args, d->siz);
/usr/local/Cellar/go/1.0.3/src/pkg/runtime/runtime.c:128 (0x1216d)
    panicstring: runtime·panic(err);
/usr/local/Cellar/go/1.0.3/src/pkg/runtime/thread_darwin.c:418 (0x152bc)
    sigpanic: runtime·panicstring("invalid memory address or nil pointer dereference");
/Users/jchris/code/go/src/github.com/couchbaselabs/sync_gateway/channels/channelmapper.go:95 (0xb2e7b)
    com/couchbaselabs/sync_gateway/channels.(*ChannelMapper).SetFunction: return mapper.js.SetFunction(fnSource)
/Users/jchris/code/go/src/github.com/couchbaselabs/sync_gateway/db/document.go:338 (0x7f92d)
    com/couchbaselabs/sync_gateway/db.(*Database).updateDoc: if changed, _ := db.ChannelMapper.SetFunction(src); changed {
/Users/jchris/code/go/src/github.com/couchbaselabs/sync_gateway/db/document.go:200 (0x7edb4)
    com/couchbaselabs/sync_gateway/db.(*Database).Put: })
/Users/jchris/code/go/src/github.com/couchbaselabs/sync_gateway/rest/rest.go:125 (0x1da47)
    com/couchbaselabs/sync_gateway/rest.(*handler).handlePutDoc: newRev, err = h.db.Put(docid, body)
/Users/jchris/code/go/src/github.com/couchbaselabs/sync_gateway/rest/rest.go:180 (0x1e1fb)
    com/couchbaselabs/sync_gateway/rest.(*handler).handlePutDesignDoc: return h.handlePutDoc()
/Users/jchris/code/go/src/github.com/couchbaselabs/sync_gateway/rest/handler.go:93 (0x1af52)
    com/couchbaselabs/sync_gateway/rest.(*handler).invoke: return method(h) // Call the actual handler code
/Users/jchris/code/go/src/github.com/couchbaselabs/sync_gateway/rest/handler.go:61 (0x25217)
    com/couchbaselabs/sync_gateway/rest._func_002: err := h.invoke(method)
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:703 (0x40b8c)
    HandlerFunc.ServeHTTP: f(w, r)
/Users/jchris/code/go/src/github.com/gorilla/mux/mux.go:86 (0xa108b)
    com/gorilla/mux.(*Router).ServeHTTP: handler.ServeHTTP(w, req)
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:941 (0x419e0)
    (*ServeMux).ServeHTTP: mux.handler(r).ServeHTTP(w, r)
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:669 (0x4099f)
    (*conn).serve: handler.ServeHTTP(w, w.req)
/usr/local/Cellar/go/1.0.3/src/pkg/runtime/proc.c:271 (0xf7a7)
    goexit: runtime·goexit(void)

Rename BrowserID --> Persona

"Persona" is now the official name, "BrowserID" is obsolete/deprecated.

should convert passwords in config files to hashed form on launch

so we don't leave plaintext passwords on the filesystem

When an exception is thrown in the sync function the error message is not printed to the console

This is the actual issue as result of original issue #30

If an error is thrown in the sync function, either due to JS errors or explicitly by the function, nothing is written to the console and the sync function fails silently.

Errors should be caught by the JS runtime that executes the sync function and an ERROR line containing the thrown message should be written to the console.

Specifying more than one database when using a sync function in the config file breaks sync_gateway

It looks like the earlier issue that I thought was a problem with CouchChat-iOS/Persona auth may actually be an issue with sync_gateway.

From what I can tell, specifying multiple databases wherein none have sync functions in the config file works fine. Also, specifying only 1 database that has a sync function works fine. However, when I specify multiple databases, and 1 (or more) has a sync function, sync_gateway no longer replicates across any of the buckets.

undefined: SimplifyChannels

When building latest I'm getting:

./auth.go:61: undefined: SimplifyChannels
./auth.go:91: undefined: SimplifyChannels
./auth.go:144: undefined: ContainsChannel
./auth.go:145: undefined: ContainsChannel
./auth.go:172: undefined: ContainsChannel
./channelmapper.go:90: undefined: SimplifyChannels
./rest.go:357: undefined: SimplifyChannels

Seems to be related to changes made with this commit - https://github.com/couchbaselabs/basecouch/commit/3ca6b4f7f23993d839ca6860eb3cf198da7ff38e

push design doc via admin port

If we did it like that I wouldn't have to set up an admin account to push the channelmapper and validation functions.

util/main.go:12:8: import "basecouch": cannot find package

I was getting the following error when first trying to get basecouch to work:

util/main.go:12:8: import "basecouch": cannot find package

To get it to work, in main.go I changed

import "basecouch"

to

import "github.com/couchbaselabs/basecouch"

and everything worked perfectly then. Not sure if that's the proper approach with the Go package structure or not (today was my first time using Go), but thought I'd point it out.

shouldn't need to drop and retrigger replications to see new channels you are granted access to

right now you have to do a new sync with the new channel set to get up to date on the new channels. (or be notified about the new channels somehow, and set up a parallel new sync for just the new channels...)

changes feed timeout isn't reset when there is a new change

if you set the timeout to eg 5 seconds, and the changes feed is very active, it will still shut off at 5 seconds.

resetting the timer is challenging due to Golang scoping rules...

suport for external JS libs in sync function

To support anything other than simple property mapping in the sync function it would be useful to be able to use external JS libs.

In TouchDB it's possible to add JS libs as properties of the design document, then they can be referenced in map and filter functions.

Ideally sync_gateway would support one of the JS module loader frameworks so that the require(module); operator may be called at the start of the sync function.

As a side note, most libs either expect a full browser runtime or a node.js runtime so will probably need adapting for sync_gateway use, but once ported these could be made available in a git repo.