Configure datasources via env variables

[strongloop]

Configure datasources via env variables

[davidcheung]

Results that came out of this issue:

• datasources.json now handles dynamic variables ${SOME_STRING}
• middleware.json and component-config now will resolve dynamic variables from env-vars too
• upon unresolved dynamic variables, undefined is returned ( this is so it can fallback to other settings )

Improvements to have on current implementation

• prefixable/suffix

Things still need to be addressed:

• Supporting cloud platforms out of the box with no need to go into settings and configure abunch of settings, for the use case of where DB services comes in a nested object rather than strings ( something that is extensible that brings more flexibility to dynamic configurations )
• Goal of having out-of-the-box working with the cloud solution.

[ashwinik001]

Hi,

To use the values from the environment variables, we do have support for configuration inside JS files.
But, still it made sense to me to have the same support through the json files as well for easy maintanence even by a JS-agnostic guy, probably a devop.

I have come up with first draft of my changes, which can be reviewed here.

Working:

• It looks for the pattern ${SOME_ENV_VAR} on the strigified configuration object and does the replacement with the process.env[SOME_ENV_VAR] if possible, else left intact. Please let me know if we should follow a different pattern like ${LB_SOME_ENV_VAR} etc.
• The variable types are intact during strigifying and parsing it again to the JSON. 👍
• The ENV_VARS are alwasy string. So, post replacement the data-type is always going to be string. Probably a limitation for the first draft. (We can think if the code should be intelligent enough to parse it to a number/boolean if needed/possible. e.g. "3306", "5432" or "true")

Please let me know if the approach/direction is good to proceed with. I shall go ahead incorporate the feedback, if any And write the test cases before I generate a PR.

Cheers :-)

[raymondfeng]

It might be better to use the dot notion (the json way), such as:

DATASOURCE.email.type=SMTP

[bajtos]

It might be better to use the dot notion (the json way), such as:

DATASOURCE.email.type=SMTP

That's an option too.

I prefer to stay consistent with existing conventions, which use underscores (see index.js):

appConfig.host =
    process.env.npm_config_host ||
    process.env.OPENSHIFT_SLS_IP ||
    process.env.OPENSHIFT_NODEJS_IP ||
    process.env.HOST ||
    appConfig.host ||
    process.env.npm_package_config_host ||
    app.get('host');

[rmg]

Underscores are definitely the convention on the environment variable side. You could take a cue from nconf and use the _'s to generate a hierarchical config tree like that of a JSON file.

[rmg]

Beyond the 12-factor app is a preference for the configuration to be stored in a separate service as a way of mitigating the risks of local-file-injection vulnerabilities (if someone tricked loopback into serving up /proc/self/environment and exposes all of the app's secrets, for example). In that scenario, the preference would probably be module.exports = remoteConfig(process.env.CONFIG_SERVER || 'localhost').

This is something I would like to use for our own servers, putting development defaults in a config.js of some kind, and having the production runtime configuration stored in Redis. To simplify even further, each VM could have a slave instance of Redis. Every app on every server queries localhost, removing the need for any production configuration being packaged for deployment.

[bajtos]

For mongodb data-sources, it would be nice to support Heroku env variables MONGOLAB_URI and MONGOHQ_URL too.

[sam-github]

Consider using URLs where possible to specify config, it reduces the number of variables, and also emphasizes that apps should be self-contained in terms of behaviour , but will need to know externalities: mostly where dependent services are, and where to offer it's own services (for example, PORT).

Opening example:

DATASOURCE_db_host=staging.mysql.com
DATASOURCE_db_user=webapp

DATASOURCE_email_type=SMTP
DATASOURCE_email_host=smtp.gmail.com

could be:

LOOPBACK_CONNECTOR_DB=mysql://[email protected]
LOOPBACK_CONNECTOR_EMAIL=smtp://smtp.gmail.com

I added namespacing under LOOPBACK_.

Also, @raymondfeng brought up more things like connection pool size, and more complex config than just endpoints, this is also URL representable:

DATASOURCE_DB=mysql://user:[email protected]:9886?poolSize=20&retryDelay=120

@raymondfeng also brought up the pain of managing configs, and modifying when you move your app around to different environments. Obivously, having values in your app.js that change depending on the deploy/run enviornment is awful.

Moving it out into files that contain nothing bug config (either json or .env environment specifications) is better, but managing them can be painful, this is true of env vars as well as .yml/json files. Some things we found help in StrongOps"

• all config should have a default, and the default should be development, so that for developers, things run as easily as possible from git clone
• the idea of having a grunt task that writes a .env (or other config) based on interactive questions is interesting, it could help devs or even devops during initial setup, a README that docs the env keys and their meaning isn't terrible, either!
• slc run now loads a .env if it finds one: strongloop/strong-supervisor#20 - I considered adding a --env=CFG and a --no-env argument options, but punted for now. Might do on request.
• since slc run is more of a deploy tool, than a local dev-time tool (when some kind of restart on file-change runner is probably being used), its loading of .env favours the "defaults are for developers, so they can run without env vars" approach.

[ritch]

DATASOURCE_DB=mysql://user:[email protected]:9886?poolSize=20&retryDelay=120

This makes a lot of sense, is easy to implement and we are already using for connector config. +1

[bajtos]

DATASOURCE_DB=mysql://user:[email protected]:9886?poolSize=20&retryDelay=120

This makes a lot of sense, is easy to implement and we are already using for connector config. +1

I agree it makes sense to pass the full config as a connection string.

However, not all connectors support connections strings - e.g. e-mail transport.

Therefore I'd prefer to implement a generic mechanism that would work for any datasource config option, not only the connection string.

Assuming the mysq connector expects the connection string in url property, the ENV variable would be

DATASOURCE_db_url=mysql://user:[email protected]:9886?poolSize=20&retryDelay=120

[gregpardo]

I'm not active on the project but is it a bad idea to convert all json configs to javascript files? That way we can inject our own environment variables where we need.

[bajtos]

@gregpardo We need to keep the main config file in JSON to allow tools like yo loopback and StrongLoop Studio edit them programatically.

You can provide another JS file to customise these initial settings - see Environment-specific configuration.

The goal of this enhancement is to make the environment-specific files redundant for people using ENV variables.

[doublemarked]

After discussion with @bajtos in #84 I took some time to look into the implementation of this feature. A few questions have emerged which I'd like to discuss. First though, a summary of my understanding of the objectives and approach to implementation. As I understand it from this thread,

1. The objective is to merge environment variables into the config produced by ConfigLoader in order to allow the environment to override directives. Ideally, it should be possible to override any config value using the environment.
2. Loopback environment variables should be marked with a magic prefix, likely corresponding to the config file it targets (main config, datasource, middleware, or component). For example, LB_DATASOURCE_db_host.
3. Environment directives targeting nested values would indicate this using an underscore delimiter. For example, the env-var LB_DATASOURCE_db_host would override the values for the datasource config's db.host.
4. Per @bajtos in #84, environment would be merged after local, dynamic, and master config.

After thinking about implementation from these conditions I have the following questions and comments,

1. Underscore as a delimiter will prevent the override of config directives that use underscores in the key name. While camelcase is typical by convention, using . as a delimiter (as @raymondfeng suggested) may avoid future conflicts.
2. Configuration may contain nested arrays. A common example is the email transports config. I propose that array indexes be supported in the following manner: LB_DATASOURCE_mail.transports.0.auth.pass.
3. Env-vars are always strings. I propose that they be coerced into the types of the values they are overriding.
4. @bajtos, you suggested that env vars be merged last, meaning they are processed after the dynamic config. This makes it possible for dynamic config to inject a non-primitive type which we cannot coerce the env-var value in to. I propose hard failure under any case where the target value is non-primitive. This will also prevents env-var config from clobbering non-leaf config nodes.
5. I propose that env-vars targeting non-existent configuration keys be dropped, not inserted.

Please help me refine this informal spec to meet your standards and expectations. Afterwards, I will put together a pull request including tests that enforce these conditions.

Thanks! :)

[bajtos]

@doublemarked thank you for thinking this through. The objectives 1-4 are correct.

Underscore as a delimiter will prevent the override of config directives that use underscores in the key name. While camelcase is typical by convention, using . as a delimiter (as @raymondfeng suggested) may avoid future conflicts.

Underscores are definitely the convention on the environment variable side. For example nconf uses a double-underscore and allows the user to provide a custom delimiter. Can we do the same here? Default to __ and allow the user to provide a different value like . if he prefers so?

Env-vars are always strings. I propose that they be coerced into the types of the values they are overriding.

That may be difficult to implement, as the target type is now always know. Can we leave this out of the first iteration and implement it as a second step?

I propose hard failure under any case where the target value is non-primitive. This will also prevents env-var config from clobbering non-leaf config nodes.

+1

I propose that env-vars targeting non-existent configuration keys be dropped, not inserted.

I suppose that's an acceptable limitation for the first iteration. In the long term, we should IMO allow users to add extra configuration via env. E.g. production settings of a db-backed datasource may add extra options to tweak pooling size or reconnect behaviour, while the development configuration does not include any of that.

Most importantly, while implementing the type coercion proposed above, we must not assume that the target type can be inferred from an existing value.

[doublemarked]

Hello! Thank you for your feedback. A few responses below.

Underscores are definitely the convention on the environment variable side. For example nconf uses a double-underscore and allows the user to provide a custom delimiter. Can we do the same here? Default to __ and allow the user to provide a different value like . if he prefers so?

Great, yes, agreed - __ to escape underscores, and potentially custom delimiters.

That may be difficult to implement, as the target type is now always know.
[...]
Most importantly, while implementing the type coercion proposed above, we must not assume that the target type can be inferred from an existing value.

Can you please elaborate on why this is the case? This would only apply to JSON primitive types (string, number, bool, null).

[bajtos]

Can you please elaborate on why this is the case? This would only apply to JSON primitive types (string, number, bool, null).

Here is a sample datasources.json file used in development:

{
  "db": {
    "database": "products"
  }
}

And the production ENV config:

DATASOURCE__db__url=mongodb://products:[email protected],backup.example.com/products?replicaSet=test&connectTimeoutMS=300000

As you can see, the configuration does not have any url property.

Does my previous comment make sense now?

[doublemarked]

Ah yes, I understand. Actually, this is specifically why I suggested that unknown keys be dropped instead of inserted. But this does not apply to existing values, only to keys that do not already exist in the configuration.

Would always inserting new values as strings be reasonable?

[bajtos]

Would always inserting new values as strings be reasonable?

Definitely! Let's start with that, we can always fine tune this initial implementation later.

[rmg]

It would be handy if a connector configuration could give the name of an environment variable to use if present, including a convention for specifying that value via an environment variable:

datasource.json:

{
  "db": {
    "env": "POSTGRES_URL"
  }
}

environment:

DATASOURCE_db_env=POSTGRES_URL

This would be useful for environments like Heroku where your environment will have injected into it:

HEROKU_POSTGRESQL_BROWN_URL=postgres://xxxx:yyyyy@some-host:5432/zzzz

Which could then be used without changing your config file:

$ heroku config:set DATASOURCE_db_env=HEROKU_POSTGRESQL_BROWN_URL

Also, just so it doesn't go unsaid: for maximum portability environment variable names should probably contain only letters, digits, and underscores http://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap08.html

[bajtos]

Hi @dopeddude, your approach looks reasonable. I have two concerns though:

1. From my experience, there are usually two configs needed: a) configuration for development environment (e.g. running everything on a laptop) b) configuration for staging/production. The dev config is usually static and same for all users (i.e. everybody is running MySQL on localhost), only the staging/production config needs to pick values from ENV. With your proposal, one will either have to maintain two config files or else force all developers to set env variables, which may not work well when developing multiple apps on the same machine.

I suppose a possible solution is to replace values not found in ENV with empty value:

// server/datasources.json
{
  "db": {
    "connector": "postgresql",

    // staging/production
    "url": "${HEROKU_POSTGRESQL_BROWN_URL}",

    // local development
    "host": "localhost",
    "user": "admin"
   }
}

Then we need to check connector implementation and verify that that url always overrides individual config options.

2. I am wondering what is the likelihood that somebody would actually want to use a string like ${FOO} in the configuration? If we implement string replacement based on environment variables, there won't be any reliable way for specifying such string in the config.

Thoughts?

[ashwinik001]

Hi @bajtos ,

Thanks for the response.

1. May be I could not understand your comment well, But I can not imagine why

one will either have to maintain two config files or else force all developers to set env variables, which may not work well when developing multiple apps on the same machine.

Let me try with a sample configuration.

//datasources.json
{
    "db": {
        "name": "db",
        "connector": "memory"
    },
    "myDb": {
        "host": "localhost",
        "database": "some-db-name",
        "password": "some-secret-key",
        "name": "myDb",
        "connector": "postgresql",
        "user": "some-user"
    }
}

//datasources.staging.json
{
    "myDb": {
        "host": "pg.pool.com",
        "password": "some-staging-secret-key",
        "user": "some-staging-user"
    }
}

//datasources.production.json
{
    "myDb": {
        "host": "${ENV_PROD_PG_URL}",
        "password": "some-super-secret-key-for-prod",
        "user": "some-prod-user"
    }
}   

At runtime

Development Machine:
    host === 'localhost'
Staging:
    host === 'pg.pool.com'
Production:
    host === 'prod.pg.pool.com'     //If the env variable `ENV_PROD_PG_URL` is set and is equal to 'prod.pg.pool.com'.
    host === '${ENV_PROD_PG_URL}'   //If the env varaible `ENV_PROD_PG_URL` is not set.

---

Then we need to check connector implementation and verify that that url always overrides individual config options.

This should be an existing problem, if it is not implemented so.
Reason: I assume at present anyways we are merging the configurations from datasources.json, datasources.local.json, datasources.staging.js || datasources.staging.json. Now, if after merging if the final configuration object has url and individual configs like host, port etc. The connector has to be intelligent to pick the url and not the individual configs.

2. I also doubt the likelihood

that somebody would actually want to use a string like ${FOO} in the configuration

But let's say if some use case requires this exact string, then I assume the user should not be setting this FOO variable on the environment. And in that case the string will be left intact and no substitution. Good for him and his use case.

To add another concern,
We should also be aware of the fact that there is another dynamic-configuration-params substituion in case of (Code Ref). In that case the placeholders patterns being similar can cause confusion. Like, if the user specifies a pattern like ${FOO}, We do not know if he wants it replace it with the ENV-VARS or he want to replace it with the APP-Config defined as part of the config.json to say.

[bajtos]

Hi @dopeddude, sorry for taking long to respond, we have an important deadline ahead us. I will be slow to respond for another two weeks or so.

1. May be I could not understand your comment well, But I can not imagine why

one will either have to maintain two config files or else force all developers to set env variables, which may not work well when developing multiple apps on the same machine.

Let me try with a sample configuration.

So I was thinking about the following configuration instead, which keeps everything in a single file:

// server/datasources.json
{
    "db": {
        "name": "db",
        "connector": "memory"
    },
    "myDb": {
        // used in production only
        "url": "${POSTGRESQL_URL}",
        // when the url is not set, fall back to development config
        "host": "localhost",
        "database": "some-db-name",
        "password": "some-secret-key",
        "name": "myDb",
        "connector": "postgresql",
        "user": "some-user"
    }
}

Of course, that's only one of possible approaches, and it may make sense to support different conventions too.

Then we need to check connector implementation and verify that that url always overrides individual config options.

This should be an existing problem, if it is not implemented so.

Yes, this is not strictly necessary for the changes proposed in your patch here. It's just that my example described above would not work without it, therefore the changes will have limited use in my opinion.

2. I also doubt the likelihood
   that somebody would actually want to use a string like ${FOO} in the configuration
   But let's say if some use case requires this exact string, then I assume the user should not be setting this FOO variable on the environment. And in that case the string will be left intact and no substitution. Good for him and his use case.

Relying on the environment to not contain FOO variable is very fragile. How is the developer supposed to know what are the all env variables defined on a production box, to ensure they don't accidentally use the same name?

I still think we should substitute unknown variables with empty strings.

Regardless of that, I think we will eventually need to come up with a way how to encode the value so that loopback-boot knows it should not substitute the value. For example $={$FOO}. However, let's defer that for later, when there is actually somebody asking for that feature.

To add another concern,
We should also be aware of the fact that there is another dynamic-configuration-params substituion in case of (Code Ref). In that case the placeholders patterns being similar can cause confusion. Like, if the user specifies a pattern like ${FOO}, We do not know if he wants it replace it with the ENV-VARS or he want to replace it with the APP-Config defined as part of the config.json to say.

I was thinking about this. In near future, I'd like to add support for VCAP_APPLICATION that is used by CloudFoundry to provide arbitrary configuration. That will add yet another source of values available for substitution.

I am proposing the following solution: When resolving a variable reference, let's say ${ENV_PROD_PG_URL}, we will check environment variables first, and if there is no variable with such name defined, the we will check app.get() variables from server/config.json. If there the variable is not found in app config either, then we will use undefined for the substituted value.

Thoughts?
/cc @rmg @sam-github

[ashwinik001]

@bajtos Thanks for the response.

As per you:

When resolving a variable reference, let's say ${ENV_PROD_PG_URL}, we will check environment variables first, and if there is no variable with such name defined, the we will check app.get() variables from server/config.json. If there the variable is not found in app config either, then we will use undefined for the substituted value.

This approach looks good to me. But, let's just keep our list of minor limitations ready with this approach just for our future revisit purpose:

• Currently user does not have support for using a value pattern like ${FOO}. And I agree as you said

"However, let's defer that for later, when there is actually somebody asking for that feature."

• As the ENV-VARS are essentially just strings, so the substitution for port like variables will also result in string. Which is also a potential candidate for getting deferred.

Let's keep adding more limitations to this list.

And others can also please express their thoughts on this approach, whenever possible.

Regards.

[davidcheung]

Information placeholder for implementation:

Current implementation of middleware dynamic variable

loopback-boot/lib/executor.js

Lines 338 to 388 in 90b9211

	function getUpdatedConfigObject(app, config) {
	var DYNAMIC_CONFIG_PARAM = /\$\{(\w+)\}$/;
	function getConfigVariable(param) {
	var configVariable = param;
	var match = configVariable.match(DYNAMIC_CONFIG_PARAM);
	if (match) {
	var appValue = app.get(match[1]);
	if (appValue !== undefined) {
	configVariable = appValue;
	} else {
	console.warn('%s does not resolve to a valid value. ' +
	'"%s" must be resolvable by app.get().', param, match[1]);
	}
	}
	return configVariable;
	}
	function interpolateVariables(config) {
	// config is a string and contains a config variable ('${var}')
	if (typeof config === 'string')
	return getConfigVariable(config);
	// anything but an array or object
	if (typeof config !== 'object' || config == null)
	return config;
	// recurse into array elements
	if (Array.isArray(config))
	return config.map(interpolateVariables);
	// Not a plain object. Examples: RegExp, Date,
	if (!config.constructor || config.constructor !== Object)
	return config;
	// recurse into object props
	var interpolated = {};
	Object.keys(config).forEach(function(configKey) {
	var value = config[configKey];
	if (Array.isArray(value)) {
	interpolated[configKey] = value.map(interpolateVariables);
	} else if (typeof value === 'string') {
	interpolated[configKey] = getConfigVariable(value);
	} else if (typeof value === 'object' && Object.keys(value).length) {
	interpolated[configKey] = interpolateVariables(value);
	} else {
	interpolated[configKey] = value;
	}
	});
	return interpolated;
	}

server/middleware.json

  "routes": {
    "loopback#rest": {
      "paths": ["${restApiRoot}"]
    }

server/config.json

{
  "restApiRoot": "/api"
}

Current implementation of host and port reading from dynamic sources

loopback-boot/lib/executor.js

Lines 102 to 142 in 90b9211

	function setHost(app, instructions) {
	// jscs:disable requireCamelCaseOrUpperCaseIdentifiers
	var host =
	process.env.npm_config_host ||
	process.env.OPENSHIFT_SLS_IP ||
	process.env.OPENSHIFT_NODEJS_IP ||
	process.env.HOST ||
	process.env.VCAP_APP_HOST ||
	instructions.config.host ||
	process.env.npm_package_config_host ||
	app.get('host');
	if (host !== undefined) {
	assert(typeof host === 'string', 'app.host must be a string');
	app.set('host', host);
	}
	}
	function setPort(app, instructions) {
	// jscs:disable requireCamelCaseOrUpperCaseIdentifiers
	var port = find([
	process.env.npm_config_port,
	process.env.OPENSHIFT_SLS_PORT,
	process.env.OPENSHIFT_NODEJS_PORT,
	process.env.PORT,
	process.env.VCAP_APP_PORT,
	instructions.config.port,
	process.env.npm_package_config_port,
	app.get('port'),
	3000
	], function(p) {
	return p != null;
	});
	if (port !== undefined) {
	var portType = typeof port;
	assert(portType === 'string' || portType === 'number',
	'app.port must be a string or number');
	app.set('port', port);
	}
	}

• should keep the precedence same as these

Unknown things that could be concerns (unknown by me only)

• will ARC overwrite these variables?
• does anywhere bypass boot and read these config files? loopback:refresh?

[sam-github]

I'm having trouble following the current state of this. Also, and perhaps this is NOT something to address with this issue, various service providers specify information on the configured DBs with provider-specific env vars. BlueMix instant runtimes use VCAP_... env vars, for example.

This should be supported either out of the box (possibly via some config mechanism), or via require of a supported module.

By some config mechanism, I mean like:

{
   name: "mysql",
   password: "$(VCAP_PROVIDER["mysql"].password)
  ...

^--- above is totally fake, I don't recall of the top of my head what the VCAP syntax is, but its possible that a survey of bluemix/heroku/openshift/etc. will find that there is some kindof core loopback json config file syntax we could implement that would allow env vars to be referenced and partially unparsed in order to have a generic way to configure an app to run in any (reasonable) cloud environment.

If not, then there will need to be an alternate way to get the ability for LB apps to be runnable easily in BlueMix/Heroku/etc., environments where the configuration is available in the environment, but the LB app developer does not control the env var naming or syntax.

@bajtos should I open another issue for this, or will this issue a good place to discuss that feature?

I also would like to echo @bajtos point about defaults, I would very much want this PR to ensure that the config can have defaults, and env overrides the defaults, so the defaults can be setup for dev envs. And using "named envs" like NODE_ENV=staging is an anti-pattern, IMO.

[bajtos]

@sam-github let's keep this issue focused on environment variables (the 12factor way) and open a new issue for VCAP and other mechanisms.

This is the current status: @jannyHou is improving the connectors to support url or dsn setting that allows users to pass all configuration in a single string, see strongloop/loopback#2089. @davidcheung is looking into loopback-boot and how to implement env-variable injection as described above (see #177).

[bajtos]

@davidcheung please consolidate the information from the comments above and the demo meeting, so that it's clear what's done and what still needs to be done.

[drewwiens]

I wanted to use this nice feature, but unfortunately there is no way I can see to interpret an environment variable as another type, eg boolean. For example, the Mongodb connector seems to interpret the SSL param as false if you pass it a string.

Ended up using a datasources.local.js file as below. If we used a URL connection string this feature would have worked, but then I'd have to change and re-deploy all our other (non-Loopback) microservices.

module.exports = { 
  mongo: { 
    name: 'mongo', 
    connector: 'mongodb', 
    host: process.env.MONGO_HOST || 'localhost', 
    port: process.env.MONGO_PORT || 27017, 
    user: process.env.MONGO_USER || '', 
    password: process.env.MONGO_PASSWORD || '', 
    database: process.env.MONGO_NAME || 'mydb', 
    ssl: (String(process.env.MONGO_SSL).toLowerCase() === 'true'), // default to false
    allowExtendedOperators: true
  }
}