resgateio/resgate

A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.

resgate's Issues

Add logging

Issue

Similar logging capabilities that exists in NATS Server should be available in Resgate, to allow a unified way of logging activities in both servers.

Related discussion

#34

Queued event may be sent multiple times

Issue

If a subscription queues an events (change or add) which introduces a resource reference to a resource that is loaded but not yet sent, the event may be sent multiple times.

How to recreate

Assume we have the following resources:

• test.a - a simple model
• test.b - a simple model
• test.collection - an empty collection
• test.model - contains reference to test.a and test.b.

1. Client sends request: get.test.model
2. Resgate sends NATS request: get.test.model
3. Resgate sends NATS requests: get.test.a and get.test.b
4. Service responds to get.test.a (but not get.test.b), making test.a loaded but not sent to client.
5. Service sends event.test.collection.add event, adding test.b reference, causing test.collection to start queuing events.
6. Service sends event.test.collection.add event, adding test.a reference.
7. Service responds to get.test.b request.
8. Resgate faultily sends the event.test.collection.add event for test.a two times to client, corrupting the state of the client.

Impact

The bug has no security impact, and low impact on usage as it only affects a rare corner case. There are no reported occurrences in production or development environments.

In case the bug is encountered, the client will get a corrupted state not matching that of the service.

Examples

Issue

More examples are needed to make it easier to learn different concepts.

The examples to be added are:

• Hello world (Previous hello-world should be renamed to edit-model)
• Password authentication - Similar to jwt-authentication but with password login
• Client session - Similar to password-authentication but with client sessions.
• Pagination

Move repository to resgateio organisation

Issue

The resgate respository is currently hosted under jirenius. But with the launch of resgate.io and the stable release, it should be moved to the resgateio organisation.

Support for primitive items in collections

Issue

Collection currectly only supports model items.
In case you wish for a collection of string values, you currently have to wrap them in objects.

Example response to get.imgService.img.42.tags:

[
    "tagService.tag.10",
    "tagService.tag.12",
    "tagService.tag.15"
]

Example response to get.tagService.tag.10:

{
	"name": "nature"
}

Suggested solution

On a collection get request, the array should contain the same type of value format as used for model properties, instead of sending an array of rid strings. Model items should instead be a resource link object:

[
	"text",
	123,
	{"rid":"serviceName.model.42"}
]

Example response to get.imgService.img.42.tags:

[
    "nature",
    "forest",
    "morning"
]

This would also affect the event data structure of collection add/remove events.

Reaccess event discarded if received prior to get response

Issue

If a resource "reaccess" event is received from a service before the response to the "get" request, the event will be discarded without triggering another "access" request.

resourceSubscription.go:114
subscription.go:465

This may cause clients to be able to receive events on a resource for which access has been removed.

Solution

Pass "reaccess" events further to the Subcription that may queue it. The event should be handled as soon as the resource is loaded.

Allow WebSocket compression

Story

Resgate should have an option to enable WebSocket message compression.

This would effectively counter the bloated message size when receiving resources with multiple models containing the same key names.

Example
A collection of reference to models could look like this:

{
  "result": {
    "models": {
      "notes.note.1": {
        "id": 1,
        "message": "One"
      },
      "notes.note.10": {
        "id": 10,
        "message": "Ten"
      },
      "notes.note.2": {
        "id": 2,
        "message": "Two"
      },
      "notes.note.3": {
        "id": 3,
        "message": "Three"
      },
      "notes.note.4": {
        "id": 4,
        "message": "Four"
      },
      "notes.note.5": {
        "id": 5,
        "message": "Five"
      },
      "notes.note.6": {
        "id": 6,
        "message": "Six"
      },
      "notes.note.7": {
        "id": 7,
        "message": "Seven"
      },
      "notes.note.8": {
        "id": 8,
        "message": "Eight"
      },
      "notes.note.9": {
        "id": 9,
        "message": "Nine"
      }
    },
    "collections": {
      "notes.notes": [
        {
          "rid": "notes.note.1"
        },
        {
          "rid": "notes.note.2"
        },
        {
          "rid": "notes.note.3"
        },
        {
          "rid": "notes.note.4"
        },
        {
          "rid": "notes.note.5"
        },
        {
          "rid": "notes.note.6"
        },
        {
          "rid": "notes.note.7"
        },
        {
          "rid": "notes.note.8"
        },
        {
          "rid": "notes.note.9"
        },
        {
          "rid": "notes.note.10"
        }
      ]
    }
  },
  "id": 1
}

Due to the repetitiveness of the data, gzip compression would reduce this from 704 bytes to 202 bytes.

Implementation

If enabled in the configuration, the Gorilla WebSocket updater should have the EnableCompression flag set to true.

Since the Gorilla WebSocket Compression feature is still flagged as Experimental, Resgate should have compression disabled by default.

An option should be added to the configuration file. Since http compression might later be added, the option name should clearly indicate it is for the WebSocket:

wsCompression (boolean)
Flag telling if WebSocket per message compression (RFC 7692) is enabled.
Example: false

Resgate may stop on HTTP requests

Issue

If HTTP requests are sent with URL:s that are translated into invalid NATS subjects, NATS will disconnect Resgate, which in turn responds by stopping.

Most common is using trailing slash:

/api/service/model/

Which translates into the invalid subject:

service.model.

Solution

Tighten up the validation on URL's to only allow paths that translates to valid NATS subjects.

At the same time, to avoid http.ServerMux's MovedPermanently redirects caused by other invalid URLS, such as /api/server//model, the ServerMux should be replaced with a custom handler.

Security vulnerability in lodash

Issue

There is a security vulnerability in the npm package lodash, used by concurrently, used as a dev depencency by the examples.

The warning message can be show with the following command:

cd examples/hello-world
npm audit

Solution

Run npm update on the affected examples.

Delete event

Story

The services need a way to tell Resgate that a resource has been deleted, and that it should no longer be served from the cache.

Solution

Expand the RES protocol to include support for a delete event. Since the protocol already states that delete is a reserved event name, this change will not break backwards compatibility.

Any subsequent client request for the resource should be handled the same way as for any other resource.

Notes

• delete events are valid for all resource types
• when Resgate receives a delete event, it should delete the resource from the cache.
• if the resource is not a query resource, or if there are no more query subscriptions, the resource should eventually be unsubscribe to NATS.
• the event should be propagated further on to the client.
• the client should not do anything. The resource will remain in the client's cache until unsubscribed.

Update README documentation

Issue

With the launch of https://resgate.io , the README.md should be updated:

• Include Resgate logo
• Include link to Resgate.io
• Update architecture illustration
• Slim down to focus on usage and configuration
• Include Support Resgate section

Security

Could you point me to a few links in resgate, client and server JS code to show me the security architecture.

I want to try to I integrate Vault (https://www.vaultproject.io/) to make it really enterprise extensible.
At least to see what sensible opportunity there is to make it easy and powerful.

Vault has a Web GUI too written in golang which makes it easy to get the hang of:
https://github.com/Caiyeon/goldfish
I think that there is also a Web GUI built into Vault too, but i have not tried it.

Multiple responses to get request

Issue

In case the access request is delayed when getting a resource with resource references, it is possible that the client's get/subscribe request will get two responses.

This happens when linked resources are loaded before the subscription.OnReady(...) call is made from wsConn.go.

Example

12:02:10 [Main] Starting server
12:02:10 [Main] Connecting to messaging system
12:02:10 [NATS] <=S system
12:02:10 [NATS] <=S conn.bj5c08l8smgjbc0l8ajg
12:02:10 [bj5c08l8smgjbc0l8ajg] Connected
12:02:10 [bj5c08l8smgjbc0l8ajg] --> {"method":"subscribe.test.model.parent","id":1}
12:02:10 [NATS] <=S event.test.model.parent
12:02:10 [NATS] <== access.test.model.parent: {"token":null,"cid":"bj5c08l8smgjbc0l8ajg"}
12:02:10 [NATS] <== get.test.model.parent: {}
12:02:10 [NATS] ==> get.test.model.parent: {"result":{"model":{"name":"parent","child":{"rid":"test.model"}}}}
12:02:10 [NATS] <=S event.test.model
12:02:10 [NATS] <== get.test.model: {}
12:02:10 [NATS] ==> get.test.model: {"result":{"model":{"string":"foo","int":42,"bool":true,"null":null}}}
12:02:10 [NATS] ==> access.test.model.parent: {"result":{"get":true}}
12:02:10 [bj5c08l8smgjbc0l8ajg] <-- {"result":{"models":{"test.model":{"bool":true,"int":42,"null":null,"string":"foo"},"test.model.parent":{"child":{"rid":"test.model"},"name":"parent"}}},"id":1}
12:02:10 [bj5c08l8smgjbc0l8ajg] <-- {"result":{},"id":1}

go client

Hey @jirenius Congrats on getting onto the NATS blog :)
The article is well written.

I am wondering what happened to the go-client ? Cant find the repo for it.

Service response to call not chronological with events

Issue

If a service receives a request (eg. call.userService.user.42.set) which triggers an event (eg. event.userService.user.42.change) before sending the call response, the call response might arrive to the client prior to the event, even if they were sent from the service in the opposite order.

Cause

The issue is caused by call responses being passed directly from the message system (mq) goroutine to the connection (wsConn) queue, while all get-responses and events are passed from the mq goroutine to the eventSubscription's queue before being passed to the connection (wsConn) queue.

Solution

Call responses should be passed to the eventSubscription's queue to ensure all calls and events are handled by the same go routine. The eventSubscritpion worker will then directly passes on to the response to the connection (wsConn) queue.

Unable to resolve complex cyclic references

Issue

Resgate handles resources with simple path cyclic reference, such as:

a -> b
b -> a

It fails to resolve more complex resource references, such as:

a -> b,c
b -> d
c -> d
d -> a,b

This is due to d having two cyclic paths (d->a->d and d->b->d), while Resgate only handle single path cycle.

Whenever such multi-path cycles are encountered, the connection requesting the resource will never get a response for the request as the resources will never be considered done loading.

Solution

Resgate needs to be able to handle more than a single path when resolving cyclic references.

Discussion: Microservices interacting with RES over http and websockets

Proble:
The problem with NATS is that is does not support Multi-Tenancy.
Its been talked about being fixed and its unlikely to be.
It also does not support on the fly config.

SO i was thinking that because there is a Client API for Res Clients to use, why cant there also be one for Microservices to use over HTTP and websockets ?

Why ?

1. Enterprise patterns such as Security, metrics, logging, tracing etc can all be done at the Res Gateway level.
2. You can secure Microservices so that they can only access parts of the NATS namespace ( loosely using this but i guess you know what i mean ).

Access timeout error stored

Issue

If an access request returns an error other than system.accessDenied, such as system.timeout, the error will be stored in Resgate memory and used to deny access for any subsequent request.

Solution

Only system.accessDenied should be considered a valid error response. All other error responses should be seen as temporary, and should not be persisted.

Recreating the issue

1. Start book-collection example service
2. Load example client
3. Stop example service
4. Edit a book entry and click OK
5. After a Timeout error, start service
6. Edit the same book entry and click OK
7. Error is still Timeout

Log

$ resgate --config=config.debug.json
08:12:02 [Main] Starting server
08:12:02 [Main] Connecting to messaging system
08:12:02 [Main] Starting HTTP server
08:12:02 [Main] Listening on http://0.0.0.0:8080
08:12:15 [bia8ljt8smghgb2g5ol0] Connected
08:12:15 [bia8ljt8smghgb2g5ol0] --> {"id":1,"method":"subscribe.library.books"}
08:12:15 [NATS] <== access.library.books: {"token":null,"cid":"bia8ljt8smghgb2g5ol0"}
08:12:15 [NATS] <== get.library.books: {}
08:12:15 [NATS] ==> _INBOX.cerdnYTSkhu6JlTQDbK56F: {"result":{"get":true,"call":"*"}}
08:12:15 [NATS] ==> _INBOX.cerdnYTSkhu6JlTQDbK5B0: {"result":{"collection":[{"rid":"library.book.1"},{"rid":"library.book.2"},{"rid":"library.book.3"}]}}
08:12:15 [NATS] <== get.library.book.3: {}
08:12:15 [NATS] <== get.library.book.2: {}
08:12:15 [NATS] <== get.library.book.1: {}
08:12:15 [NATS] ==> _INBOX.cerdnYTSkhu6JlTQDbK5PH: {"result":{"model":{"id":3,"title":"Coraline","author":"Neil Gaiman"}}}
08:12:15 [NATS] ==> _INBOX.cerdnYTSkhu6JlTQDbK5Fl: {"result":{"model":{"id":2,"title":"Brave New World","author":"Aldous Huxley"}}}
08:12:15 [NATS] ==> _INBOX.cerdnYTSkhu6JlTQDbK5KW: {"result":{"model":{"id":1,"title":"Animal Farm","author":"George Orwell"}}}
08:12:15 [bia8ljt8smghgb2g5ol0] <-- {"result":{"models":{"library.book.1":{"author":"George Orwell","id":1,"title":"Animal Farm"},"library.book.2":{"author":"Aldous Huxley","id":2,"title":"Brave New World"},"library.book.3":{"author":"Neil Gaiman","id":3,"title":"Coraline"}},"collections":{"library.books":[{"rid":"library.book.1"},{"rid":"library.book.2"},{"rid":"library.book.3"}]}},"id":1}
08:12:30 [bia8ljt8smghgb2g5ol0] --> {"id":2,"method":"call.library.book.1.set","params":{"title":"Animal Farming","author":"George Orwell"}}
08:12:30 [NATS] <== access.library.book.1: {"token":null,"cid":"bia8ljt8smghgb2g5ol0"}
08:12:32 [NATS] x=> Request timeout
08:12:32 [bia8ljt8smghgb2g5ol0] <-- {"error":{"code":"system.timeout","message":"Request timeout"},"id":2}
08:12:37 [NATS] ==> system.reset: {"resources":["library.>"]}
08:12:37 [NATS] <== get.library.book.3: {}
08:12:37 [NATS] <== get.library.book.1: {}
08:12:37 [NATS] <== get.library.books: {}
08:12:37 [NATS] <== get.library.book.2: {}
08:12:37 [NATS] ==> _INBOX.cerdnYTSkhu6JlTQDbK5Yn: {"result":{"model":{"id":3,"title":"Coraline","author":"Neil Gaiman"}}}
08:12:37 [NATS] ==> _INBOX.cerdnYTSkhu6JlTQDbK5dY: {"result":{"model":{"id":1,"title":"Animal Farm","author":"George Orwell"}}}
08:12:37 [NATS] ==> _INBOX.cerdnYTSkhu6JlTQDbK5iJ: {"result":{"collection":[{"rid":"library.book.1"},{"rid":"library.book.2"},{"rid":"library.book.3"}]}}
08:12:37 [NATS] ==> _INBOX.cerdnYTSkhu6JlTQDbK5n4: {"result":{"model":{"id":2,"title":"Brave New World","author":"Aldous Huxley"}}}
08:12:42 [bia8ljt8smghgb2g5ol0] --> {"id":3,"method":"call.library.book.1.set","params":{"title":"Animal Farming","author":"George Orwell"}}
08:12:42 [bia8ljt8smghgb2g5ol0] <-- {"error":{"code":"system.timeout","message":"Request timeout"},"id":3}
08:12:46 [Main] Stopping server...
08:12:46 [Main] Disconnecting all ws connections...
08:12:46 [bia8ljt8smghgb2g5ol0] Disconnecting - Server is shutting down
08:12:46 [bia8ljt8smghgb2g5ol0] Disconnected: read tcp [::1]:8080->[::1]:54711: use of closed network connection
08:12:46 [Main] All ws connections gracefully closed
08:12:46 [Main] Stopping HTTP server...
08:12:46 [Main] HTTP server gracefully stopped
08:12:46 [Main] Closing the messaging system's client connection...
08:12:46 [Main] Message queue connection closed
08:12:46 [Main] http: Server closed
08:12:46 [Main] Stopping rescache workers...
08:12:46 [Main] rescache stopped
08:12:46 [Main] Service stopped

Server not stopping with correct signals

I setup resgate to run in tandem with some other processes as part of my setup.

Using go-cmd, and raised an issue here:

go-cmd/cmd#19 (comment)

The test code to reproduce I don't have accessible but the 20 line example code in go-cmd is the same and so very easy to try.

It is important that spawned servers inside NATS terminate correctly in order for them to be well managed by the process manager.
I have not tried resgate with systemd or launchd or windows service yet.
I intend to try that next

How Microservice interface with RES

Does the RES gateway support a typical Microservice interfacing with it over http ? As opposed to using the golang NATS client.

If yes, does the Microservice need to support websockets ? I presume a definite yes.

Additionally can a Microservice also use the golang NATS drivers instead. There are certain use cases where that prefer that.

Also if I have a Microservice that needs to provide web GUI to the browser can the RES gateway be used for it ? I presume yes. It would mean I don't need a sperate reverse proxy & the security is all applied unifoly at the RES gateway level. It also means that I only have to run a single docker of my web service that also provides the web rendering.

Lastly .. there is no lastly. That's alot of questions ..

Delete property on change event

Issue

RES protocol currently provides no way of deleting a model property once it is set.

Suggested solution

A special object, eg. {"action":"delete"}, could indicate deletion.

{
	"foo": "New or changed value",
	"bar": {"action":"delete"}
}

API encoding configuration

Story

When fetching resources using HTTP, the json representation of the data currently includes href's to the linked resources:

Below is an example from resgate-test-app's notesService.notes resource:

[
  {
    "href": "/api/notesService/note/1",
    "model": {
      "id": 1,
      "message": "One"
    }
  },
  {
    "href": "/api/notesService/note/2",
    "model": {
      "id": 2,
      "message": "Two"
    }
  }
]

In some cases, a more flat representation is wanted as opposed to the the HATEOAS style currently used.

[
  {
    "id": 1,
    "message": "One"
  },
  {
    "id": 2,
    "message": "Two"
  }
]

Solution

A new config option should be added:

apiEncoding (string)
Encoding used to represent web resources.

• "json" - JSON encoding with resource references included in the data (default)
• "jsonFlat" - JSON encoding with no resource references except for cyclic references.

Example: "json"

Notes

Cyclic references

By using the jsonFlat setting, there is no unambiguous way to represent cyclic references:

Example of the jsonFlat data representation of cyclic.a:

{
  "a": { "href": "cyclic.a" }
}

It is most likely a cyclic reference, but could also be that "a" links to a simple model "b" that just happens to have a single property, href, with the value "cyclic.a".

This is impossible to avoid with jsonFlat. Users just needs to be aware of the case.

Multiple encodings

The solution opens up for other encodings as well, such as xml.

Later on (not in this story), it might be possible to use file-endings in the URL to specify which encoding to use.

• /api/notesService/notes.json - Default json encoding with href
• /api/notesService/notes.flat.json - jsonFlat encoding
• /api/notesService/notes.xml - XML encoding

The setting set for apiEncoding will be used when no file ending is given.

Resource not sent on successive get request

Issue

In case a client get request is immediately followed by another client get or subscribe request, while the first request is still collecting resources, the response for the second request may not contain the resources that was included in the response to the first request.

Since any resource sent in a response to a client get request should not be considered subscribed, any successive get or subscribe request should resend those resources if needed.

How to recreate

Assume we have a resource, test.a, which has a resource reference to another resource, test.b.

1. Client sends request: get.test.a
2. Resgate sends NATS request: get.test.a
3. Client sends request: get.test.b
4. Resgate gets result from request 3)
5. Resgate sends NATS request: get.test.b
6. Resgate gets result from request 4)
7. Resgate responds to request 1), including both test.a and test.b
8. Resgate responds to request 3) with no resource data, wrongly assuming client already has test.a

Impact

The bug has no security impact, and low impact on usage as it only affects a get request corner case. In addition, get requests are currently not used by any known client. ResClient only uses subscribe requests to fetch data, and is therefor unaffected.

Crash on timed out resources

Issue

Resources that time out causes Resgate to crash.

Log

13:23:35 [Main] Starting server
13:23:35 [Main] Connecting to messaging system
13:23:35 [Main] Starting HTTP server
13:23:35 [Main] Listening on http://0.0.0.0:8080
13:23:39 [bj5d6et8smghhs6or5ng] Connected
13:23:39 [bj5d6et8smghhs6or5ng] --> {"id":15,"method":"subscribe.viewerService.collection"}
13:23:39 [NATS] <== access.viewerService.collection: {"token":null,"cid":"bj5d6et8smghhs6or5ng"}
13:23:39 [NATS] <== get.viewerService.collection: {}
13:23:41 [NATS] x=> Request timeout
13:23:41 [bj5d6et8smghhs6or5ng] <-- {"error":{"code":"system.timeout","message":"Request timeout"},"id":15}
13:23:41 [NATS] x=> Request timeout
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x69b8fb]

goroutine 50 [running]:
github.com/jirenius/resgate/server/rescache.(*ResourceSubscription).Unsubscribe(0x0, 0x7d7ec0, 0xc0001b6000)
        C:/dev/go/src/github.com/jirenius/resgate/server/rescache/resourceSubscription.go:119 +0x6b
github.com/jirenius/resgate/server.(*Subscription).Loaded.func1()
        C:/dev/go/src/github.com/jirenius/resgate/server/subscription.go:144 +0x16b
github.com/jirenius/resgate/server.(*wsConn).outputWorker(0xc0001a0000)
        C:/dev/go/src/github.com/jirenius/resgate/server/wsConn.go:531 +0x64
created by github.com/jirenius/resgate/server.(*Service).newWSConn
        C:/dev/go/src/github.com/jirenius/resgate/server/wsConn.go:66 +0x301

Note

Bug was introduced in #84

Reaccess on system.reset

Issue

When a system.reset event is received, the gateway will currently just reload the data, but will assume previous access requests to still be valid.

Suggested solution 1

Reaccess events should be created for each resource affected by the system.reset event, forcing each subscription to check with the service if it still has access.

Suggested solution 2

A separate event, system.reaccess, should be supported. The event should work in a similar way as system.reset, but would trigger reaccess instead of reloading.

Suggested solution 3

Another property, access, should be added to the system.reset event, which would be similar to the resources property, but for triggering reaccess.

Support NATS User Credentials

Issue

Resgate should be able to connect to NATS server using NATS user credential files.

Note

The scope only covers user credential files, and will not include TLS certificates.

Other methods, such as token and username/password authentication, are already supported as the url can contain username/password semantics. Eg.:

resgate --nats nats://user:[email protected]:4222

Implementation

A command-line option should be added:

• --creds - NATS User Credentials file.

An option should be added to the configuration file:

natsCreds (string)
NATS User Credentials file.
Example: "~/ngs.creds"

Scratch docker image

Issue

Resgate needs to be available as a scratch docker image.

Events might be missed while reconnecting to NATS.

##Issue
When resgate's connection to NATS server is lost, it will try to reconnect once to any other NATS server in the cluster. If a service sends an event during the reconnect, the resgate might miss it, resulting in a stale resource without resgate trying to recover from it.

##Solution
Change the way resgate connects to NATS.
Use nats client's:

func NoReconnect() Option

to prevent any reconnects.

Query event preceding get response not discarded

Issue

If a resource "query" event is received from a service before the response to the "get" request, the event will still be handled causing undefined behaviour.

Solution

Prevent event query requests to be sent if the resourceSubscription is not yet loaded
(resourceSubscription.state <= stateRequested)

eventSubscription.go:274

Inconsistent timeout units

Issue

Milliseconds is the timeout unit specified in the RES-service pre-response.

Seconds are used as the requestTimeout unit in Resgate config.

Solution

Change config requestTimeout units to milliseconds.

To ensure acceptable backwards compatibility, if the default timeout setting is >= 10, seconds will be assumed, and a deprecated warning message will show in the log.

Also, the default requestTimeout should be lowered to 3000 ms.

Bind to host address

Issue

Resgate needs to allow binding to host IP address.

Solution

A new configuration option to be added:

-i, --addr <host>     Bind to HOST address (default: 0.0.0.0)

For json configuration file:

// Bind to HOST address.
{
    "addr": "0.0.0.0"
}

A null value or omitted configuration should default to "0.0.0.0".
An empty string ("") would bind to both ipv4 and ipv6.

Logging

So far, all the logging stuff has been ignored.

The problem is that all the different golang log libraries don't share a common standard interface, so the solution would be to design a standard interface and then make an adapter for each one, similar to the way the SQL package has all the different database drivers, at the cost of not being able to use specific features to each logger package. That itself could be a big task. After a quick Google there seems to be an example which uses adapters approach here:
https://goa.design/implement/logging/

Query string normalization

Issue

When a query request is made, resgate have no way to determine if two non-identical query strings may represent the same data, and will have to fetch the data for both queries as well as responding twice to each query event on the resource.

Example of different queries for the same resource

The following resource IDs may represent the same data
bookService.books?genre=scifi&year=2017&page=1
bookService.books?year=2017&genre=scifi&page=01
bookService.books?year=2017&genre=scifi&nonsense=param&page=1

Suggested solution

A new property, query, should be added to the response of a get request. The property should contain the normalized version of the query. Normalization is defined by the service, but may be sorting the fields, removing fields not part of the query, and normalizing the field values.

If no fields are used by the resource, the property should be omitted indicating that the response is not a query resource, but the same as requesting the resource without query. This will make it easier for non-query resources as they don't have to bother if the request has a query field or not, while currently they have to specifically check if the query property on get requests, sending a system.notFound response if a query exists.

Example of query normalization

Request: get.bookService.books
Payload:

{
    "query": "year=2017&genre=Scifi&foo=bar&page=01"
}

Response:

{
    "result": {
        "collection": [ ... ],
        "query": "genre=scifi&page=1&year=2017"
    }
}

Status code 404 on system.methodNotFound

Issue

When using HTTP POST to make a call request through Resgate, if the response is a system.methodNotFound error code, Resgate would currently use the status code 400 Bad Request in the HTTP response.

A 404 Not Found status code would be more appropriate, since the call method is exposed as a unique URL that is not available.

why not use Redis as the storage?

I personally use Redis as the backend storage.

How to let the client listen to certain event based on the resource id without a cached resource?

Support for NGS

Issue

Add support to connect Resgate to NGS.

Indirectly subscribed resources not queuing while reconfirming access

Issue

When reaccess (Subscription.Reaccess()) is triggered, either through a token-, reaccess-, or system.reset event, any events for directly susbcribed resources will be queued until access has been reconfirmed.
However, events for indirectly subscribed resources will not be queued, but passed to the client unhindered. These events should also be queued until the directly subscribed resource's access is confirmed.

Rename gnatsd to nats-server

Issue

NATS server executable has, with version 2, been renamed from gnatsd to nats-server.

Links and references to gnatsd should be updated to reflect this change.

Add system.invalidQuery error response

Issue

The RES Protocol lacks any proper way to respond to requests where the provided query is invalid.

Service developers currently has to to use workarounds which includes replying with a custom error, or improperly use the existing system.invalidParams error code.

Solution

This can be solved by introducing a new pre-defined error:

Code	Message	Meaning
system.invalidQuery	Invalid query	The query is malformed or contains invalid values

The new error code would be similar to system.invalidParams, but differ in regards that system.invalidParams refers to the params of a call/auth method requests, whereas system.invalidQuery would refer to the query in any type of request.

Notes

HTTP status code

For HTTP requests, the system.invalidQuery error code would translate to a 400 Bad Request by Resgate. While using 422 Unprocessable Entity from the WebDAV extension might be more appropriate, the 400 status code is consistent with the status code Resgate uses for system.invalidParams error codes.

When to use

The system.invalidQuery should be sent when the query string cannot be parsed due to malformed content. It should also be sent if the parsed query contains values that are not allowed (eg. limit=1000 when limit must be <= 50), but the service may choose not to send an error, but to use a default value (eg. limit=50) in the normalized query. However, this behavior would be unclear to the consumer of the API, and is not recommended.

It error should not be sent if the query contains keys that are not used by the service (eg. limit=20&foo=bar). It is recommended that the service simply ignores the key and excludes it from the normalized query (eg. limit=20).

Support for linked resources

Issue

Models currently only supports primitive properties.
To fetch a complex resource structure, you have to do it with multiple requests:

Example:

Promise.all([
	api.getResource('userService.user.42'),
	api.getResource('userService.user.42.roles')
]).then(result => {
	api.getResource('orgService.org.'+result[0].orgId)
		.then(org => {
			console.log("Name : ", result[0].name);
			console.log("Roles: ", result[1]);
			console.log("Org. : ", org);
		});
});

Suggested solution

RES protocol could define a special object that is used for resource links, eg. {"rid":"orgService.org.3"}. If a property contains such a link, the gateway will fetch these resources, doing indirect subscriptions just like collections models, and return them together with the parent model in the response.

Example on get.userService.user.42 service result response:

{
	"id": 42,
	"name": "Jane Doe",
	"roles": {"rid":"userService.user.42.roles"},
	"org": {"rid":"orgService.org.3"}
}

In client:

api.getResource('userService.user.42').then(user => {
	console.log("Name : ", user.name);
	console.log("Roles: ", user.roles);
	console.log("Org. : ", user.org);
});

Integration tests

Story

Integration tests should be added to ensure that Resgate complies with the RES client-, and service protocol specification.

Notes

Mocking both the NATS client behaviour as well as the WebSocket client connection, tests can be made that sends client requests, validates and replies to requests posted on NATS, and validates the responses to the client.

Mocking NATS

By creating a testnats struct that implements the mq.Client interface, NATS client may be mocked.

Mocking WebSocket client

By using github.com/posener/wstest, the clients may be mocked as well.

Allow query on non-query get request

Issue

The RES protocol currently doesn't allow responding with a query resource on a non-query get request. In the RES Service Protocol - Get request, it states in the Result section:

query
MUST be omitted if the request had no query parameter.

This restriction should be removed as it may cause unnecessary complexity for service development of query resource that uses default values for omitted query parameters.

Use case

Assume we have a collection of a million items, but the service wish to only expose the collection as a query resource where a from and limit parameter may be provided to specify the span.

To simplify the request, omitting any of the two parameters will have the service fall back to the default values (eg. from=0&limit=50).

If both parameters were omitted, the query would be empty, and due to the restriction it would no longer accept a normalized query in the response. This would currently force the developers to either:

1. Add specific non-query handling on the service for the default params usecase.
2. Add a dummy property to the query (eg. ignore=this) on the request

This scenario would be solved if the restriction was lifted.

Example request

Request body for get.example.list:

{ }

Response:

{
    "result": {
        "collection": [{ "rid": "example.item.1" }, ... , { "rid": "example.item.50" }],
        "query": "from=0&limit=50"
    }
}

Compatibility

Removing the restriction would:

• be backwards compatible (running older services on a newer versions of Resgate)
• have no impact on client

different project but the same vision

the same protocol maybe interested for you
https://github.com/dunglas/mercure

document read and write path

Would be great to have doc folder so that as a developer i can see how te system works architecturally and its features architecturally.

Tasks:
In a doc folder, explain the Read and Write flow and interception points such as Security, Tracing and Logging).
It should use a common naming convention. A glossary might be needed.
It should delineate if else logic in how each part of the flow works that is pertinent to the outcome.
It can have links to code if its pertinent later.
It should be written for architectural analysis and not as a how to use perspective.

DOD ( Definition of done):

• Ability to understand the conceptual architecture
• Be enough to conceptualise a refactoring of the architecture in whatever way seems fit going forward.

No access call when indirect subscription becomes direct

Issue

Resgate does not make an access call on an indirectly subscribed resource after it turns to a solely directly subscribed resource.

Example

Let assume we have a resource test.parent with a resource reference to test.child:

• Client subscribes to test.parent and gets test.child as an indirectly subscribed resource.
• Client subscribes to test.child, which triggers no access request as it is also indirectly subscribed.
• Client unsubscribes to test.parent, which turns test.child to a solely directly subscribed resource.
• Resgate erroneously does not confirm client's access to test.child
• Service sends a reaccess on test.parent, and denies access.
• Client will continue to receive events for test.child even if access should have been revoked.

Impact

May have access control impact for solutions where reaccess is used on nested resources.

Solution

On unsubscribe, Resgate should check if any referenced resource is directly subscribed without any indirect reference. In such a case, Resgate should trigger a reaccess on the referenced resource.

Allow service to extend request timeout

Issue

When a request to a service takes a long time, there is currently no way for the service to signal back to the resgate that the request has been received and is being worked on. This may cause the request to timeout, giving the client a timeout error, even if the request has been processed.

Suggested solution

A meta response may be made by the service to any type of request. This response would contain the new desired timeout duration in milliseconds. To be easy to distinguish from the actual json response, it should not be json encoded.

It should be utf-8/latin-1 encoded and have the format timeout:"<number>". These sort of meta responses may later be extended with more key/values, using comma separation. resgate can quickly detect a meta response by analyzing the first byte - if it is a latin-1 letter, it is a meta response.

Example meta response, extending the timeout to 10 seconds:

timeout:"10000"

collection format

Hi,
Nice work with Resgate.
I am having fun with it since a few days.
Not I want to complicate a little bit my model.
But, on a GET request, complicated JSON struct don't seems to pass.

For instance I return this Json structure on a Get request:
{
"result": {
"collection": [
{
"description": "Room1",
"device.description": "Room1",
"device.icon_name": "audio-card",
"device.intended_roles": "music",
"device.model": "WX-010",
"device.string": "[10.0.140.181]:5000",
"index": 0,
"name": "raop_output.Room1.local",
"path": "system.audio.raop_output.Room1.local",
"subscribe": [
{
"format": "pa_sample_spec",
"methode": "createStream"
},
{
"format": "sample",
"methode": "writeStream"
}
],
"uuid": "raop_output.Room1.local"
}
]
}
}

But, the Get request return this:
{
code: "system.internalError",
message: "Internal error: Internal error: Invalid value"
}

Could you take a look at it please?
Thanks,

Michael

Pre-defined call methods

Issue

Some call methods needs to be pre-defined to allow specific behaviour by resgate and resclient.

Methods

new

Used for creating new resources.
Needed to be defined with the return value containing the newly created rid. This way a web resource request can return with a 201 Created and a Location. For the RES client protocol, the response may contain the newly created rid and all its linked resources.

set

Used for setting model properties.
Has already been implemented, but needs to be more clearly defined in resprotocol documentation.

Props field on change event

Issue

Model change events, as defined in RES-service specification V1.0, gives no opportunity to include meta data in the event. This is a design flaw which prevents the specification to adapt to requests such as version numbering of resources.

Since the number of Resgate deployments are still limited, this issue should be addressed, and a new specification version (v1.1) should be created.

Solution

Update specification to have the changed properties in a props field:

v1.0 model change event payload:

{
   "foo": "bar",
   "faz": 42
}

v1.1 model change event payload:

{
   "props": {
      "foo": "bar",
      "faz": 42
   }
}

Implementation

Resgate should detect legacy behaviour and handle it, while logging a Deprecated warning.
A change event payload is considered legacy behaviour unless all statements below are true:

• payload contains a property called props
• payload contains only a single property
• value for props is an object

The only time a v1.0 service might be mistakenly taken as non-legacy (v.1.1), is for the following two payloads:

{
    "props": { "rid": "example.model" }
}

or

{
   "props": { "action": "delete" }
}

The chances of anyone being affected by this is minimal.

Go 1.12

Issue

Go 1.12 is released.
Update Travis job to do the release build using latest stable Go 1.12 version.