dadi/api

A high-performance RESTful API layer designed in support of API-first development and COPE. Connects your content to the world

DADI API

• Overview
• Requirements
• Your First API Project
• Links

Overview

DADI API is built on Node.JS. It is a high performance RESTful API layer designed in support of API-first development and the principle of COPE. It can use virtually any database engine, such as MongoDB, CouchDB, RethinkDB or simply a JSON filestore.

You can consider it as the data layer within a platform (including the data model). It is designed to be plugged into a templating layer (such as DADI Web), a mobile application or to be used with any other data consumer.

Calls to a DADI API can contain your business/domain logic (the part of a platform that encodes the real-world business rules that determine how data is created, displayed, stored and changed). It has full support for searching, filtering, limiting, sorting, offsetting, input validation and data aggregation (through support for MongoDB's aggregation pipeline).

It has built-in support for oAuth2, includes full collection-level ACL, can connect to multiple databases out of the box, provides native document versioning at collection level, supports static endpoints, includes automatic indexing, has a caching layer and can be run in a clustered configuration.

DADI API provides a starting point that's further advanced than a framework. It allows you to get a complete data layer up and running in minutes.

It is part of DADI, a suite of components covering the full development stack, built for performance and scale.

Requirements

• Node.js (supported versions: 6.11.x, 8.9.x)
• A data connector module

Your first API project

Install API

The quickest way to get started with API is to use DADI CLI. See Creating an API for full installation details.

Configuration

API starts with some sensible defaults, so it's not necessary to understand all the configuration options available when first running the application.

Configuration is handled using JSON files specific to the application environment. For example in the production environment a file named config.production.json will be used. Configuration files must be placed in a config folder in your application root, for example config/config.production.json. The default start up environment is development, using the configuration file at config/config.development.json.

The bare minimum required for running the API is a server block. With only a server block, default values are used for all other properties.

Sample configuration

{
  "server": {
    "host": "127.0.0.1",
    "port": 3000
  }
}

Start the server

API can be started from the command line simply by issuing the following command:

$ npm start

Test the connection

With the default configuration, our API server is available at http://localhost:3000. If you've modified the configuration file's server block, your API will be available at the address and port you've chosen. Use cURL to check the server is running, if the connection can be made you will receive the following "Unauthorised" message.

$ curl http://localhost:3000

{"statusCode": 401}

Check the response headers

$ curl -I http://localhost:3000

HTTP/1.1 401 Unauthorized
content-type: application/json
content-length: 18
Date: Thu, 20 Apr 2017 23:42:25 GMT
Connection: keep-alive

Authentication

The HTTP 401 response received in the previous step shows that the server is running. To start using the REST endpoints you'll need a user account so you can obtain access tokens for interacting with the API.

User accounts provide an authentication layer for API. Each user account has a clientId and a secret. These are used to obtain access tokens for interacting with the API. See the Authentication section of the API documentation for full details.

Creating the first user

CLI contains an interactive "Client Record Generator" to help you create user accounts. Run the following command in the directory where you installed API:

cd my-new-api
dadi api clients:add

If you need to create user accounts in other environments (for example following a deployment to a live server), add the environment to the following command:

$ NODE_ENV=production npm explore @dadi/api -- npm run create-client

Run API as a service

To run your API application in the background as a service, install Forever and Forever Service:

$ npm install forever forever-service -g

$ sudo forever-service install -s index.js -e "NODE_ENV=production" api --start

You can now interact with the api service using the following commands:

$ [sudo] service api start
$ [sudo] service api stop
$ [sudo] service api status
$ [sudo] service api restart

Note: the environment variable NODE_ENV=production must be set to the required configuration version matching the configuration files available in the config directory.

Tests

To run the tests after cloning the repository, run the following command:

$ npm test

NOTE: API installs version 4.0.1 of Mocha and uses this when calling npm test. If you have Mocha installed globally and want to simply run mocha, if using version 4 or above, add --exit to the command so it becomes mocha --exit

Links

• API Documentation

Contributors

DADI API is based on an original idea by Joseph Denne. It is developed and maintained by the engineering team at DADI (https://dadi.cloud)

• Adam K Dean [email protected]
• Arthur Mingard [email protected]
• David Longworth [email protected]
• Eduardo Bouças [email protected]
• Francesco Iannuzzelli [email protected]
• James Lambie [email protected]
• Joe Wagner
• Joseph Denne [email protected]
• Kevin Sowers [email protected]
• Robert Stanford [email protected]
• Viktor Fero [email protected]

Licence

DADI is a data centric development and delivery stack, built specifically in support of the principles of API first and COPE.

Copyright notice
(C) 2018 DADI+ Limited [email protected]
All rights reserved

This product is part of DADI.
DADI is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version ("the GPL").

If you wish to use DADI outside the scope of the GPL, please contact us at [email protected] for details of alternative licence arrangements.

This product may be distributed alongside other components available under different licences (which may not be GPL). See those components themselves, or the documentation accompanying them, to determine what licences are applicable.

DADI is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

The GNU General Public License (GPL) is available at http://www.gnu.org/licenses/gpl-3.0.en.html.
A copy can be found in the file GPL.md distributed with these files.

This copyright notice MUST APPEAR in all copies of the product!

api's Issues

Replace status endpoint with repo version

Expected behavior

API has a status endpoint which can be replaced by the dadi/status repo version, rather than have it built into the codebase.

Export normal app exports in single threaded mode

Expected behavior

Prior to 1.8.x, API exported to it's parents several pieces including it's loaded components. This was used to generate menus in pyncheon, as well as potentially other endpoints.

Actual behavior

Latest update no longer exports that, and changes the form of the exports

Steps to reproduce the problem

Update to latest API, install as module, request v1/endpoints/collections

Export Controller object

Please can we export the Controller object in the same we do for Model, Log, etc.?

This will be useful for custom endpoints that need to replicate the default collection endpoint as part of their behaviour.

Cheers!

[Minor] Negative values in count and page provoke a 500 error

If count is set to a negative number, it is passed down to MongoDB provoking a "bad skip value in query" exception and a 500 error is returned by API. A similar situation happens with page <= 0.
The error is recorded in API log.

Queries with invalid count and page values should either be corrected with default values, or captured and logged as invalid, but not executed in MongoDB.

Convert string IDs to ObjectID when passing query to model

Currently only $in queries will have the IDs converted to ObjectIDs, this should be extended to strings also.

Every request to an API endpoint is logged twice, the second one as 404

Expected behavior

Requests made to an endpoint are logged

Actual behavior

When a request is made to an endpoint, it is logged twice, the first with a 200 status code, the second one as 404

Example of request sent from Web
[2016-03-14T13:07:15.781Z] INFO: dadi-web/8465 on ip-172-31-7-184: GET datasource 'article': http://api.test.lifestyle.one:80/1.0/article?count=1&page=1&filter={"published":"published","channel":"heat","category":"celebrity","subcategory":"news","title":"celebrity-dogs-famous"}&fields={}&sort={} (module=helper)

Requests logged in API

[2016-03-14T13:01:29.603Z]  INFO: dadi-api/17422 on ip-172-31-6-45: GET http://api.test.lifestyle.one:80/1.0/article?count=1&page=1&filter={"published":"published","channel":"heat","category":"celebrity","subcategory":"news","title":"celebrity-dogs-famous"}&fields={}&sort={} 404 23ms (module=router)```

Note that the second one is a 404 of a similar time length
### Steps to reproduce the problem

Request any article in the Lifestyle test environment and check API logs.
It can be reproduced in a local development environment running the same dataset and endpoints.

Only one query is executed on the underlying MongoDB, so it seems to be an issue with the API logger and not with its execution.
### Package details

dadi/web 1.1.1
dadi/api 1.3.0

Tokens expiring prematurely

Expected behavior

Bearer tokens should be invalidated according to the auth.tokenTtl value set in config.

Actual behavior

Tokens are being invalidated before the supposed expiry date.

Steps to reproduce the problem

Set auth.tokenTtl to a large value (e.g. 86400), request a bearer token and try to use it after a few hours.

Package details

API v1.6.4

Hooks: endpoint

There is a requirement for 3 endpoints to aid in the delivery of hook config and availability.

• /hooks to deliver a list of all available hooks, including a message/label, required and optional parameters, and available methods e.g. beforeCreate or beforeDelete
• /hooks/{hookSlug} used to deliver field values to aid with real-time output preview
• /hooks/{hookSlug}/config returns the full JS file used in the hook - TBD in Publish

WWW-Authenticate header

When returning a 401 due to either a missing or invalid token, the server must send back a WWW-Authenticate header indicating the error.

Too many open connections

Version 1.3.0 appears to assign a new MongoClient for every loaded collection. In practice this means it opens 5 connections per collection, as each MongoClient instantiates a connection pool (default poolsize 5)

A more ideal solution would be to have one MongoClient created at startup and allow that to handle the connection pool for the entire API instance.

Are we providing a guarantee against collisions in authentication tokens between clientId's?

And if not, would it even matter?

Allow `skip` parameter to be defined via the querystring

skip is currently determined by count * page but there is a case for allowing a configurable skip/offset

Article History

• Introduce /{version}/{db}/{collection}?history=true
• Return all historic versions including current
• Allow ?fields param to continue to define which fields to return
• Date range filtering

Further concepts

• Filter where fields have changed e.g. ?whereChange=['title', 'heroImage', 'content']

Run queries after compose

Currently queries are run against a document pre-compose.

There is a requirement for a boolean param to dictate whether they are run before or after aggregate resolve.

Request: Notify once done

Compose option for hooks

Currently hooks are only able to utilise the un-resolved values of a new or existing document.

A typical example of a new document as it would appear to a hook

{
  "foo": "bar",
  "referencedDocuments": [
    "576ba3602f5a57d1456c351f",
    "576ba3642f5a57d1456c365d"
  ]
}

Whilst it would not be possible to resolve the entire document because it doesn't exist, it would be possible to resolve the input Object by using the collection schema to resolve Reference fields, as compose does when retrieving a document.

{
  "foo": "bar",
  "referencedDocuments": [
     {
      "_id": "576ba3602f5a57d1456c351f",
      "name": "yours"
    },
    {
      "_id": "576ba3642f5a57d1456c365d",
      "name": "westfm"
    }
  ]
}

Use case: A URL hook that requires values from Reference documents to build

Bulk updating documents

It is currently possible to send an array of documents to the API for creation, but not to send an array of documents to be updated.

Currently to update a document the ID must be passed in the request URL and the update query as the body.

POST /1.0/library/notices/1234 HTTP/1.1
Host: api.example.com
content-type: application/json
Authorization: Bearer 171c8c12-6e9b-47a8-be29-0524070b0c65

{ "status": "published"  }

It would be useful to allow sending an array of IDs to be updated:

POST /1.0/library/notices/1234,5678,9012,3456 HTTP/1.1
Host: api.example.com
content-type: application/json
Authorization: Bearer 171c8c12-6e9b-47a8-be29-0524070b0c65

{ "status": "published"  }

Add cluster module for performance improvement

Addition of a status endpoint

A new endpoint at /status would enable a finer-grained approach to platform monitoring; would enable us make decisions relating to load (e.g. at load balancer level) on the basis of much more than just whether or not an API instance is responding to ping on port 80; and would allow us to more easily visualise stack performance.

/status would ideally return:

• An overall health indicator (green/amber/red)
• Messaging relating to the health indicator
• Report on endpoint status by self-testing to schedule (defaults to be provided, but ideally configurable on a per-deployment basis)
• Provide CPU load and load averages
• Provide memory utilisation as well as free and freeable memory
• Provide uptime
• Provide the version of DADI API being used
• Alert where new version(s) are available

/status should be authenticated, and also optional (enabled by default). The port for /status should also be configurable.

[minor] API system log is flooded with "number"

it's probably a debug command outputting some variable's type

Steps to reproduce the problem

run API as a service and check its log

Package details

dadi/api 1.3.1

Redis support

The addition of Redis support as a cache option. Redis support will allow a shared cache between multiple instances, required to ensure consistency in delivery in some product setups.

This will also bring API inline with Web and CDN in terms of cache support.

NPM registry check: uncaught 500

Expected behavior

NPM registry check failure to be caught

Actual behavior

App crash and error:

{
  "message": "Response code 500 (Internal Server Error)",
  "host": "registry.npmjs.org",
  "hostname": "registry.npmjs.org",
  "method": "GET",
  "path": "/@dadi%2Fapi",
  "statusCode": 500,
  "statusMessage": "Internal Server Error"
}

Steps to reproduce the problem

Repeat calls to the status api endpoint

Incorrect test for replicaSet

I believe

replicaSet: { doc: "", format: String, default: false },

should be

replicaSet: { doc: "", format: Boolean, default: false },

in config.js

npm test throws errors and app won't start

DateTime fieldtype

SODate and make use of existing MongoDB functionalities

Queries using dot notation, and targeting a field of type Object, should not have strings representing Mongo IDs converted into Mongo ID type

Expected behavior

Queries in "dotted notation", i.e. targeting a Mongo ID stored in sub-element of a field of type Object, match the way that Mongo ID is stored (i.e. as a string).

Actual behavior

Currently strings that resemble a Mongo ID are converted to a Mongo ID when stored in a field of type Object ID. In all other cases, they are stored as strings, for example also when they are part of an Object.

However, all queries including a string that resembles a Mongo ID are converted to a Mongo ID, regardless of the type of field they are targeting. So in the case of an Object, a query on object._id would be converted to Mongo ID, while object._id would have been stored as a string, and a match cannot happen.

Package details

API 1.4.1

Add file rotation option for standard log files

The access log can be rotated on a daily/weekly/etc basis, ensuring log files don't get too large. The same functionality would be useful for the standard log files.

Any filter condition including a dot (.) in the property name is stripped

Expected behavior

Any query targeting a Mixed field to be passed to the database

Actual behavior

Queries including a valid path are stripped

For example

localhost:3000/1.0/articles/celebrity?page=1&count=1&filter={"subtitle":"test","editor.editor_name":"test"}

is passed to Mongo as

        "query" : {
                "subtitle" : /^test$/i,
                "apiVersion" : "1.0"
        },

Note that editor.editor_name is not present

Steps to reproduce the problem

It seems that API queries are validated against the schema, and if they include an invalid reference they are stripped. However editor in this case is Mixed, and contains other subdocuments, which are individually validated in the schema
https://github.com/dadiplus/bauer-lifestyle-serama/blob/dev/workspace/collections/1.0/articles/collection.celebrity.json

It didn't seem possible to enforce validation on a subelement in the previous API version, but it had to be written at root level. The document in the sample request above includes this field

        "editor" : {
                "editor_id" : "566f8a14975babee0989528c",
                "editor_name" : "Francesco Iannuzzelli"
        },

In any case, validation shouldn't be enforced on queries targeting a Mixed field, as that could contain anything. This is particularly applicable to dates that are stored as Mongo dates, for example

        "publicationDate" : {
                "sec" : 1462119300,
                "usec" : 0
        },

(this is actually the origin of this issue, as queries targeting the publicationDate.sec timestamp fail in Bauer Lifestyle)

Package details

1.3.0

Add ENV variable names for all sensitive properties

Config settings can be loaded from the config files, from environment variables or from the command line when launching the app.

All sensitive settings need to have an "env": "xxx" property added to the config schema

Delete old records from history collections

History collections are growing indefinitely and creating un-necessary load, both in terms of write operations and storage.
As they are not accessible out of the API layer, we need a command-line tool shipped with API to perform regular clean up, such as extract a copy of all historical records older than a certain amount of time and delete them from the MongoDB collections.

Configurable parameters:

• timestamp to identify records to be deleted
• wether to extract a copy of the old records or simply delete them

Make the apiVersion filter configurable

Every API query passes an
"apiVersion" : "1.0"
down to Mongo

This adds unnecessary load (most applications in production to not distinguish by apiVersion).

Although it is important to store this value in each document, its filtering should be made configurable.

Temporary workaround is to add an index to the apiVersion field.

Package details

1.3.0

Cache pre warming

In large scale product implementations where a lot of collections and endpoints are in play, unpredictable traffic patterns can - and frequently do - lead to significant database load. This neccessatates high capacity infrastructure above and beyond normal operating thresholds.

The ability to pre warm API's cache could mitigate against this and reduce infrastructure costs.

The concept is that a API understands the breadth of collections and endpoints that it contains, and then iteratively caches the common requests responses, preventing load spikes as a result of unpredictable user activity. In the event that a cached object expires, API would repopulate the cache automatically.

Pre warming would need to be sequential in order to prevent it from triggering load issues itself. It would also need to be configurable.

Thoughts on this request appreciated.

Extend the concept of environment to allow config switching by domain

We need to be able to test the integration between DADI applications within a staging environment which is as-live; i.e. the instance set that is then imaged and pushed into production.

Rather than having to play to play with config files or environment variables manually, the ability to automatically switch based on domain is desired. This would enable the use of different config files based on the domain used, allowing us to test the integration between applications without modification immediately prior to deployment.

Example domain pattern:

client.product.env.app.dadi.technology

Example domains:

• Test: bauer.empire.test.api.dadi.technology
• QA: bauer.empire.qa.api.dadi.technology
• Live: empireonline.com

Add lastUpdated timestamp to collections

Expanding on the existing block passed back by api/collections endpoint, introduction of a lastUpdated value to prompt a refetch from publish.

Current

 {
      "version": "1.0",
      "database": "oolipo",
      "name": "Image",
      "slug": "images",
      "path": "/1.0/oolipo/images"
    }

Proposed

 {
      "version": "1.0",
      "database": "oolipo",
      "name": "Image",
      "slug": "images",
      "path": "/1.0/oolipo/images",
      "lastUpdated" : 1462362369
    }

Create client script throws error

Expected behavior

The script creates client and exits

Actual behavior

The script creates client and throws an error

Steps to reproduce the problem

node ./utils/create-client.js

Package details

Api revision e26a97f

/dadi/api/workspace/docs being created without api-docs being enabled

On start up I'm seeing Created directory /dadi/api/workspace/docs, indicating that the presence or otherwise of api-docs is not being factored.

Cache invalidation API

The addition of an cache invalidation API along the same lines as the invalidation API in DADI CDN to enable the easy flushing of either individual cached elements or the entire cache.

This will need to support local and Redis based cache setups.

Write operation queue

To reduce the load at times of peak read/write

• Monitor system hardware performance (average and 98th percentile)
• When latency detected, instantly capture API write calls to a queue in memory / localdb
• Resolve queue writes at a pace that suits the load
• Possible ability to return a reference to the memory resident entry for CMS to reference
• Possibly return lock state to temporary block CMS saves
• Possibly create empty entries to create ObjectID's and write field values from memory queue

Publish fields as standard

Expected behavior

Bake in publish state, date, expires date, publisher as standard.

Add logging level to config

Currently the logging level cannot be controlled, resulting in all log statements being sent to the log file.

Modify as per Web, which has the following config option:

logging: {
    level: {
      doc: "Sets the logging level.",
      format: ['debug','info','warn','error','trace'],
      default: 'info'
    }
}

Asynchronous hooks

Making hooks asynchronous would support more complex operations, e.g. looking up data from another collection, querying a 3rd party API, chaining further model creation.

The use case for this is avoiding using custom endpoints for model-specific pre/post-processing.

At present, if we want to lookup data before inserting a document we must add that logic in a custom endpoint. But doing so hides that logic from other endpoints and leaves open other routes (the default collection endpoint and model.create) that don't share the same logic.

Asynchronous hooks would help us to keep model-specific logic where it belongs.

Cache documentation needs updating

https://github.com/dadi/api/blob/docs/docs/cachePerformance.md

Conditional batch delete

Expected behavior

Requirement for a batch delete method, with suggested format of:

http://domain.com/1.0/databse/collection?where={"title": "Duplicate article title"}
or regex
http://domain.com/1.0/databse/collection?where={"title": {"$regex": "/pattern/"}}
or multiple match
http://domain.com/1.0/databse/collection?where={"title": {"$regex": "/pattern/"}, "pages": {"$lte": 5}}

Configuration docs

This file needs completing:
https://github.com/dadi/api/blob/docs/docs/configuration.md

Controller instance `get` method should allow querying for "null"

Controller instance get method should allow querying for "null":

     TypeError: Cannot convert undefined or null to object
      at Function.keys (native)
      at Object.module.exports.transformQuery (dadi/lib/help.js:119:10)

Authentication is bypassed in endpoints

Expected behavior

All endpoints to return 401 if no authentication bearer is passed in the HTTP headers

Actual behavior

Endpoints are returning data with a simple GET request

Steps to reproduce the problem

Just browse to these endpoints (GET, without authentication headers)

http://api.test.lifestyle.one/1.0/hubArticles?count=2&page=1
http://api.whatcar.tech/1.0/car-rivals

Package details

dadi/api 1.3.0

API crashes if the caching Redis server connection times out

Expected behavior

If the Redis server is unreachable, API should fall back to directory caching.

Actual behavior

Server crashes and will not respond.

Steps to reproduce the problem

Turn off the Redis server before starting the app, or while it's running.

Package details

@dadi/api 1.4.0

Add version number sequence to revision history

Capture and log database errors

Some database errors in production provoke a service restart, but no useful information is logged about the offending query (example below).

Error: Can't set headers after they are sent. at ServerResponse.OutgoingMessage.setHeader (_http_outgoing.js:335:11) at /dadi/api/bantam/lib/help.js:18:13 at /dadi/api/bantam/lib/model/index.js:304:25 at /dadi/api/node_modules/mongodb/lib/mongodb/cursor.js:180:16 at commandHandler (/dadi/api/node_modules/mongodb/lib/mongodb/cursor.js:734:16) at /dadi/api/node_modules/mongodb/lib/mongodb/db.js:1905:9 at Server.Base._callHandler (/dadi/api/node_modules/mongodb/lib/mongodb/connection/base.js:453:41) at /dadi/api/node_modules/mongodb/lib/mongodb/connection/server.js:488:18 at MongoReply.parseBody (/dadi/api/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5) at null.<anonymous> (/dadi/api/node_modules/mongodb/lib/mongodb/connection/server.js:446:20) error: Forever detected script exited with code: 1

Schema validation is enforced on all sub-elements of a Mixed fields

Expected behaviour

Anything can be stored in a Mixed field (similarly to an Object field)

(I may be missing something here, is there any specific difference between a Mixed and an Object field?)

Actual behaviour

All sub-elements of a Mixed field are checked against the schema, like they were at root level

Steps to reproduce the problem

Store a structured element in a Mixed field; operation fails

[Error: Model Validation Failed]\n statusCode: 400,\n json: { success: false, errors: [ [Object] ] } }

Package details

API 1.4.0

Ability to filter stats endpoint

Expected behavior

Either by using the existing /stats/ endpoint or an additional /meta endpoint, give the ability to get meta-like detail with filters e.g.
http://host.com:80/1.0/db/collection/metadata?filter={"answer": "yes"}

Expected response

{ "metadata": {
    "limit": 40,
    "totalCount": 434,
    "totalPages": 11
  }
}

Add Sentry logging

API needs the same Sentry log options as Web:

"logging": {
    "sentry": {
      "enabled": true,
      "dsn": "a dsn key generated from the Sentry server"
    }
}