Support nested values in config overrides

[strongloop]

Support nested values in config overrides

[raymondfeng]

In a plain json configuration file, it's impossible to tell if the developers' intention is 1, 2, 3, or 4 unless we introduce extra meta, in which case, the config file will become a diff/patch. IMO, overriding the whole value is a reasonable solution as long as we document it.

[bajtos]

IMO, overriding the whole value is a reasonable solution as long as we document it.

In my opinion, the most frequent use-case for complex values is customising datasource configuration. In which case it makes more sense to add/override key-values instead of replacing the whole object.

Example: to specify different MySQL settings for the production environment, one wants to add or change host, port, username and password. Other options (e.g. connnector: mysql) remain without change.

If we are going to support only one of 1-4, then I would definitely like to get more feedback from the community before making the decision.

[raymondfeng]

I'm fine with merging too.

See https://github.com/strongloop/loopback-datasource-juggler/blob/master/lib/utils.js#L138. We use it to merge model settings.

[britztopher]

Can we reuse this merge function from datasource-juggler to be used in loopback-boot for nested objects? If so, I can try to make this work, just need to know where to start looking for where in the loopback code this should go. Im guessing its somewhere in the executor or config-loader.js files.

[matejc]

I am using this one-level merging trick, it's not ideal but works for my usecase
matejc@4c88a5d

[raymondfeng]

@bajtos What's your take? We need a solution.

[bajtos]

@bajtos What's your take? We need a solution.

How about supporting two flavours of datasources.*.js files:

• a file exporting a config object (i.e. a dynamic version of *.json), only value-type values are supported (no change from the current state)

• a file exporting a function that will be called to modify the config object (a new feature)

  module.exports = function(config) {
    config.mail.transports = [{
      // configure SMTP
    }];
  
    switch (config.db.connector) {
      case 'mysql':
        config.db.host = process.env.MYSQL_HOST;
        config.db.port = process.env.MYSQL_PORT;
        break;
    }
    // etc.
  });

That way we don't have to worry whether the JSON format is expressing the intention (merge/replace) clearly, and the users can apply whatever changes they need.

[glesage]

So far when I attempt to change some variables (for ex, the password) like this I am having to do:

config.datasources.db.connector.settings.password = process.env.MY_PASS;
config.datasources.db.settings.password = process.env.MY_PASS;

Is this correct? It does not seem to work for me /: It is the correct spot where the password is stored but when using the DB, loopback fails with:

{
  "error": {
    "name": "Error",
    "status": 500,
    "message": "ER_ACCESS_DENIED_ERROR: Access denied for user 'starchup'@'localhost' (using password: YES)",
    "stack": "Error: ER_ACCESS_DENIED_ERROR: Access denied for user 'starchup'@'localhost' (using password: YES)\n    at Handshake.Sequence._packetToError (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/sequences/Sequence.js:48:14)\n    at Handshake.ErrorPacket (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/sequences/Handshake.js:101:18)\n    at Protocol._parsePacket (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:271:23)\n    at Parser.write (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Parser.js:77:12)\n    at Protocol.write (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:39:16)\n    at Socket.<anonymous> (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Connection.js:82:28)\n    at Socket.EventEmitter.emit (events.js:107:17)\n    at readableAddChunk (_stream_readable.js:159:16)\n    at Socket.Readable.push (_stream_readable.js:126:10)\n    at TCP.onread (net.js:514:20)\n    --------------------\n    at Protocol._enqueue (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:135:48)\n    at Protocol.handshake (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:52:41)\n    at PoolConnection.connect (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Connection.js:109:18)\n    at Pool.getConnection (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Pool.js:42:23)\n    at MySQL.query (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/lib/mysql.js:167:10)\n    at MySQL.all (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/lib/mysql.js:580:8)\n    at Function.find (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-datasource-juggler/lib/dao.js:735:34)\n    at Function.ACL.checkAccessForContext (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback/lib/models/acl.js:397:8)\n    at Function.Model.checkAccess (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback/lib/models/model.js:281:12)\n    at Function.<anonymous> (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback/lib/application.js:319:13)",
    "code": "ER_ACCESS_DENIED_ERROR",
    "errno": 1045,
    "sqlState": "28000",
    "fatal": true,
    "INVALID_EXPRESSION_ERR": 51,
    "TYPE_ERR": 52
  }
}

Please advise,
THanks!

[bajtos]

@glesage connector gets the settings object from datasource settings.

// server/datasources.local.js
module.exports = {
  "db" : {
    "password": process.env.MY_PASS
  }
}

[glesage]

oh alright thks, works perfect indeed (:

[frankcarey]

Have you guys considered using https://github.com/lorenwest/node-config instead of rolling your own, or at least as a base? It supports hierarchical configurations (something I needed to have loopback-passport working properly in multiple environments.)

[bajtos]

Have you guys considered using https://github.com/lorenwest/node-config instead of rolling your own, or at least as a base?

We did not investigate this option thoroughly. At the first glance, node-config does provide a fine-grained control over how the configs from different sources are merged.

It supports hierarchical configurations (something I needed to have loopback-passport working properly in multiple environments.)

loopback-boot supports hierarchical configurations too. Can you provide an example of what you are not able to achieve with our current solution?

[bajtos]

#52 implements a solution where both Object and Array values from .json files are recursively merged together.

[bajtos]

The feature is available as of v2.1.0.