Comments (28)
Results that came out of this issue:
datasources.json
now handles dynamic variables${SOME_STRING}
middleware.json
andcomponent-config
now will resolve dynamic variables from env-vars too- upon unresolved dynamic variables, undefined is returned ( this is so it can fallback to other settings )
Improvements to have on current implementation
- prefixable/suffix
Things still need to be addressed:
- Supporting cloud platforms out of the box with no need to go into settings and configure abunch of settings, for the use case of where DB services comes in a nested object rather than strings ( something that is extensible that brings more flexibility to dynamic configurations )
- Goal of having out-of-the-box working with the cloud solution.
from loopback-boot.
Hi,
To use the values from the environment variables, we do have support for configuration inside JS files.
But, still it made sense to me to have the same support through the json files as well for easy maintanence even by a JS-agnostic guy, probably a devop.
I have come up with first draft of my changes, which can be reviewed here.
Working:
- It looks for the pattern
${SOME_ENV_VAR}
on the strigified configuration object and does the replacement with theprocess.env[SOME_ENV_VAR]
if possible, else left intact. Please let me know if we should follow a different pattern like${LB_SOME_ENV_VAR}
etc. - The variable types are intact during strigifying and parsing it again to the JSON. 👍
- The ENV_VARS are alwasy string. So, post replacement the data-type is always going to be string. Probably a limitation for the first draft. (We can think if the code should be intelligent enough to parse it to a number/boolean if needed/possible. e.g. "3306", "5432" or "true")
Please let me know if the approach/direction is good to proceed with. I shall go ahead incorporate the feedback, if any And write the test cases before I generate a PR.
Cheers :-)
from loopback-boot.
It might be better to use the dot notion (the json way), such as:
DATASOURCE.email.type=SMTP
from loopback-boot.
It might be better to use the dot notion (the json way), such as:
DATASOURCE.email.type=SMTP
That's an option too.
I prefer to stay consistent with existing conventions, which use underscores (see index.js):
appConfig.host =
process.env.npm_config_host ||
process.env.OPENSHIFT_SLS_IP ||
process.env.OPENSHIFT_NODEJS_IP ||
process.env.HOST ||
appConfig.host ||
process.env.npm_package_config_host ||
app.get('host');
from loopback-boot.
Underscores are definitely the convention on the environment variable side. You could take a cue from nconf and use the _'s to generate a hierarchical config tree like that of a JSON file.
from loopback-boot.
Beyond the 12-factor app is a preference for the configuration to be stored in a separate service as a way of mitigating the risks of local-file-injection vulnerabilities (if someone tricked loopback into serving up /proc/self/environment
and exposes all of the app's secrets, for example). In that scenario, the preference would probably be module.exports = remoteConfig(process.env.CONFIG_SERVER || 'localhost')
.
This is something I would like to use for our own servers, putting development defaults in a config.js
of some kind, and having the production runtime configuration stored in Redis. To simplify even further, each VM could have a slave instance of Redis. Every app on every server queries localhost, removing the need for any production configuration being packaged for deployment.
from loopback-boot.
For mongodb data-sources, it would be nice to support Heroku env variables MONGOLAB_URI
and MONGOHQ_URL
too.
from loopback-boot.
Consider using URLs where possible to specify config, it reduces the number of variables, and also emphasizes that apps should be self-contained in terms of behaviour , but will need to know externalities: mostly where dependent services are, and where to offer it's own services (for example, PORT
).
Opening example:
DATASOURCE_db_host=staging.mysql.com
DATASOURCE_db_user=webapp
DATASOURCE_email_type=SMTP
DATASOURCE_email_host=smtp.gmail.com
could be:
LOOPBACK_CONNECTOR_DB=mysql://[email protected]
LOOPBACK_CONNECTOR_EMAIL=smtp://smtp.gmail.com
I added namespacing under LOOPBACK_
.
Also, @raymondfeng brought up more things like connection pool size, and more complex config than just endpoints, this is also URL representable:
DATASOURCE_DB=mysql://user:[email protected]:9886?poolSize=20&retryDelay=120
@raymondfeng also brought up the pain of managing configs, and modifying when you move your app around to different environments. Obivously, having values in your app.js that change depending on the deploy/run enviornment is awful.
Moving it out into files that contain nothing bug config (either json or .env environment specifications) is better, but managing them can be painful, this is true of env vars as well as .yml/json files. Some things we found help in StrongOps"
- all config should have a default, and the default should be development, so that for developers, things run as easily as possible from
git clone
- the idea of having a grunt task that writes a .env (or other config) based on interactive questions is interesting, it could help devs or even devops during initial setup, a README that docs the env keys and their meaning isn't terrible, either!
slc run
now loads a .env if it finds one: strongloop/strong-supervisor#20 - I considered adding a--env=CFG
and a--no-env
argument options, but punted for now. Might do on request.- since slc run is more of a deploy tool, than a local dev-time tool (when some kind of restart on file-change runner is probably being used), its loading of .env favours the "defaults are for developers, so they can run without env vars" approach.
from loopback-boot.
DATASOURCE_DB=mysql://user:[email protected]:9886?poolSize=20&retryDelay=120
This makes a lot of sense, is easy to implement and we are already using for connector config. +1
from loopback-boot.
DATASOURCE_DB=mysql://user:[email protected]:9886?poolSize=20&retryDelay=120
This makes a lot of sense, is easy to implement and we are already using for connector config. +1
I agree it makes sense to pass the full config as a connection string.
However, not all connectors support connections strings - e.g. e-mail transport.
Therefore I'd prefer to implement a generic mechanism that would work for any datasource config option, not only the connection string.
Assuming the mysq connector expects the connection string in url
property, the ENV variable would be
DATASOURCE_db_url=mysql://user:[email protected]:9886?poolSize=20&retryDelay=120
from loopback-boot.
I'm not active on the project but is it a bad idea to convert all json configs to javascript files? That way we can inject our own environment variables where we need.
from loopback-boot.
@gregpardo We need to keep the main config file in JSON to allow tools like yo loopback
and StrongLoop Studio edit them programatically.
You can provide another JS file to customise these initial settings - see Environment-specific configuration.
The goal of this enhancement is to make the environment-specific files redundant for people using ENV variables.
from loopback-boot.
After discussion with @bajtos in #84 I took some time to look into the implementation of this feature. A few questions have emerged which I'd like to discuss. First though, a summary of my understanding of the objectives and approach to implementation. As I understand it from this thread,
- The objective is to merge environment variables into the config produced by
ConfigLoader
in order to allow the environment to override directives. Ideally, it should be possible to override any config value using the environment. - Loopback environment variables should be marked with a magic prefix, likely corresponding to the config file it targets (main config, datasource, middleware, or component). For example,
LB_DATASOURCE_db_host
. - Environment directives targeting nested values would indicate this using an underscore delimiter. For example, the env-var
LB_DATASOURCE_db_host
would override the values for the datasource config'sdb.host
. - Per @bajtos in #84, environment would be merged after local, dynamic, and master config.
After thinking about implementation from these conditions I have the following questions and comments,
- Underscore as a delimiter will prevent the override of config directives that use underscores in the key name. While camelcase is typical by convention, using
.
as a delimiter (as @raymondfeng suggested) may avoid future conflicts. - Configuration may contain nested arrays. A common example is the email transports config. I propose that array indexes be supported in the following manner:
LB_DATASOURCE_mail.transports.0.auth.pass
. - Env-vars are always strings. I propose that they be coerced into the types of the values they are overriding.
- @bajtos, you suggested that env vars be merged last, meaning they are processed after the dynamic config. This makes it possible for dynamic config to inject a non-primitive type which we cannot coerce the env-var value in to. I propose hard failure under any case where the target value is non-primitive. This will also prevents env-var config from clobbering non-leaf config nodes.
- I propose that env-vars targeting non-existent configuration keys be dropped, not inserted.
Please help me refine this informal spec to meet your standards and expectations. Afterwards, I will put together a pull request including tests that enforce these conditions.
Thanks! :)
from loopback-boot.
@doublemarked thank you for thinking this through. The objectives 1-4 are correct.
Underscore as a delimiter will prevent the override of config directives that use underscores in the key name. While camelcase is typical by convention, using
.
as a delimiter (as @raymondfeng suggested) may avoid future conflicts.
Underscores are definitely the convention on the environment variable side. For example nconf uses a double-underscore and allows the user to provide a custom delimiter. Can we do the same here? Default to __
and allow the user to provide a different value like .
if he prefers so?
Env-vars are always strings. I propose that they be coerced into the types of the values they are overriding.
That may be difficult to implement, as the target type is now always know. Can we leave this out of the first iteration and implement it as a second step?
I propose hard failure under any case where the target value is non-primitive. This will also prevents env-var config from clobbering non-leaf config nodes.
+1
I propose that env-vars targeting non-existent configuration keys be dropped, not inserted.
I suppose that's an acceptable limitation for the first iteration. In the long term, we should IMO allow users to add extra configuration via env. E.g. production settings of a db-backed datasource may add extra options to tweak pooling size or reconnect behaviour, while the development configuration does not include any of that.
Most importantly, while implementing the type coercion proposed above, we must not assume that the target type can be inferred from an existing value.
from loopback-boot.
Hello! Thank you for your feedback. A few responses below.
Underscores are definitely the convention on the environment variable side. For example nconf uses a double-underscore and allows the user to provide a custom delimiter. Can we do the same here? Default to __ and allow the user to provide a different value like . if he prefers so?
Great, yes, agreed - __
to escape underscores, and potentially custom delimiters.
That may be difficult to implement, as the target type is now always know.
[...]
Most importantly, while implementing the type coercion proposed above, we must not assume that the target type can be inferred from an existing value.
Can you please elaborate on why this is the case? This would only apply to JSON primitive types (string, number, bool, null).
from loopback-boot.
Can you please elaborate on why this is the case? This would only apply to JSON primitive types (string, number, bool, null).
Here is a sample datasources.json
file used in development:
{
"db": {
"database": "products"
}
}
And the production ENV config:
DATASOURCE__db__url=mongodb://products:[email protected],backup.example.com/products?replicaSet=test&connectTimeoutMS=300000
As you can see, the configuration does not have any url
property.
Does my previous comment make sense now?
from loopback-boot.
Ah yes, I understand. Actually, this is specifically why I suggested that unknown keys be dropped instead of inserted. But this does not apply to existing values, only to keys that do not already exist in the configuration.
Would always inserting new values as strings be reasonable?
from loopback-boot.
Would always inserting new values as strings be reasonable?
Definitely! Let's start with that, we can always fine tune this initial implementation later.
from loopback-boot.
It would be handy if a connector configuration could give the name of an environment variable to use if present, including a convention for specifying that value via an environment variable:
datasource.json:
{
"db": {
"env": "POSTGRES_URL"
}
}
environment:
DATASOURCE_db_env=POSTGRES_URL
This would be useful for environments like Heroku where your environment will have injected into it:
HEROKU_POSTGRESQL_BROWN_URL=postgres://xxxx:yyyyy@some-host:5432/zzzz
Which could then be used without changing your config file:
$ heroku config:set DATASOURCE_db_env=HEROKU_POSTGRESQL_BROWN_URL
Also, just so it doesn't go unsaid: for maximum portability environment variable names should probably contain only letters, digits, and underscores http://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap08.html
from loopback-boot.
Hi @dopeddude, your approach looks reasonable. I have two concerns though:
- From my experience, there are usually two configs needed: a) configuration for development environment (e.g. running everything on a laptop) b) configuration for staging/production. The dev config is usually static and same for all users (i.e. everybody is running MySQL on localhost), only the staging/production config needs to pick values from ENV. With your proposal, one will either have to maintain two config files or else force all developers to set env variables, which may not work well when developing multiple apps on the same machine.
I suppose a possible solution is to replace values not found in ENV with empty value:
// server/datasources.json
{
"db": {
"connector": "postgresql",
// staging/production
"url": "${HEROKU_POSTGRESQL_BROWN_URL}",
// local development
"host": "localhost",
"user": "admin"
}
}
Then we need to check connector implementation and verify that that url
always overrides individual config options.
- I am wondering what is the likelihood that somebody would actually want to use a string like
${FOO}
in the configuration? If we implement string replacement based on environment variables, there won't be any reliable way for specifying such string in the config.
Thoughts?
from loopback-boot.
Hi @bajtos ,
Thanks for the response.
- May be I could not understand your comment well, But I can not imagine why
one will either have to maintain two config files or else force all developers to set env variables, which may not work well when developing multiple apps on the same machine.
Let me try with a sample configuration.
//datasources.json
{
"db": {
"name": "db",
"connector": "memory"
},
"myDb": {
"host": "localhost",
"database": "some-db-name",
"password": "some-secret-key",
"name": "myDb",
"connector": "postgresql",
"user": "some-user"
}
}
//datasources.staging.json
{
"myDb": {
"host": "pg.pool.com",
"password": "some-staging-secret-key",
"user": "some-staging-user"
}
}
//datasources.production.json
{
"myDb": {
"host": "${ENV_PROD_PG_URL}",
"password": "some-super-secret-key-for-prod",
"user": "some-prod-user"
}
}
At runtime
Development Machine:
host === 'localhost'
Staging:
host === 'pg.pool.com'
Production:
host === 'prod.pg.pool.com' //If the env variable `ENV_PROD_PG_URL` is set and is equal to 'prod.pg.pool.com'.
host === '${ENV_PROD_PG_URL}' //If the env varaible `ENV_PROD_PG_URL` is not set.
Then we need to check connector implementation and verify that that url always overrides individual config options.
This should be an existing problem, if it is not implemented so.
Reason: I assume at present anyways we are merging the configurations from datasources.json, datasources.local.json, datasources.staging.js || datasources.staging.json
. Now, if after merging if the final configuration object has url
and individual configs like host, port etc
. The connector has to be intelligent to pick the url
and not the individual configs
.
- I also doubt the likelihood
that somebody would actually want to use a string like
${FOO}
in the configuration
But let's say if some use case requires this exact string, then I assume the user should not be setting this FOO
variable on the environment. And in that case the string will be left intact and no substitution. Good for him and his use case.
To add another concern,
We should also be aware of the fact that there is another dynamic-configuration-params substituion in case of (Code Ref). In that case the placeholders patterns being similar can cause confusion. Like, if the user specifies a pattern like ${FOO}
, We do not know if he wants it replace it with the ENV-VARS or he want to replace it with the APP-Config defined as part of the config.json to say.
from loopback-boot.
Hi @dopeddude, sorry for taking long to respond, we have an important deadline ahead us. I will be slow to respond for another two weeks or so.
- May be I could not understand your comment well, But I can not imagine why
one will either have to maintain two config files or else force all developers to set env variables, which may not work well when developing multiple apps on the same machine.
Let me try with a sample configuration.
So I was thinking about the following configuration instead, which keeps everything in a single file:
// server/datasources.json
{
"db": {
"name": "db",
"connector": "memory"
},
"myDb": {
// used in production only
"url": "${POSTGRESQL_URL}",
// when the url is not set, fall back to development config
"host": "localhost",
"database": "some-db-name",
"password": "some-secret-key",
"name": "myDb",
"connector": "postgresql",
"user": "some-user"
}
}
Of course, that's only one of possible approaches, and it may make sense to support different conventions too.
Then we need to check connector implementation and verify that that url always overrides individual config options.
This should be an existing problem, if it is not implemented so.
Yes, this is not strictly necessary for the changes proposed in your patch here. It's just that my example described above would not work without it, therefore the changes will have limited use in my opinion.
- I also doubt the likelihood
that somebody would actually want to use a string like ${FOO} in the configuration
But let's say if some use case requires this exact string, then I assume the user should not be setting this FOO variable on the environment. And in that case the string will be left intact and no substitution. Good for him and his use case.
Relying on the environment to not contain FOO
variable is very fragile. How is the developer supposed to know what are the all env variables defined on a production box, to ensure they don't accidentally use the same name?
I still think we should substitute unknown variables with empty strings.
Regardless of that, I think we will eventually need to come up with a way how to encode the value so that loopback-boot knows it should not substitute the value. For example $={$FOO}
. However, let's defer that for later, when there is actually somebody asking for that feature.
To add another concern,
We should also be aware of the fact that there is another dynamic-configuration-params substituion in case of (Code Ref). In that case the placeholders patterns being similar can cause confusion. Like, if the user specifies a pattern like ${FOO}, We do not know if he wants it replace it with the ENV-VARS or he want to replace it with the APP-Config defined as part of the config.json to say.
I was thinking about this. In near future, I'd like to add support for VCAP_APPLICATION
that is used by CloudFoundry to provide arbitrary configuration. That will add yet another source of values available for substitution.
I am proposing the following solution: When resolving a variable reference, let's say ${ENV_PROD_PG_URL}
, we will check environment variables first, and if there is no variable with such name defined, the we will check app.get()
variables from server/config.json
. If there the variable is not found in app config either, then we will use undefined
for the substituted value.
Thoughts?
/cc @rmg @sam-github
from loopback-boot.
@bajtos Thanks for the response.
As per you:
When resolving a variable reference, let's say ${ENV_PROD_PG_URL}, we will check environment variables first, and if there is no variable with such name defined, the we will check app.get() variables from server/config.json. If there the variable is not found in app config either, then we will use undefined for the substituted value.
This approach looks good to me. But, let's just keep our list of minor limitations ready with this approach just for our future revisit purpose:
- Currently user does not have support for using a value pattern like
${FOO}
. And I agree as you said
"However, let's defer that for later, when there is actually somebody asking for that feature."
- As the ENV-VARS are essentially just strings, so the substitution for
port
like variables will also result in string. Which is also a potential candidate for getting deferred.
Let's keep adding more limitations to this list.
And others can also please express their thoughts on this approach, whenever possible.
Regards.
from loopback-boot.
Information placeholder for implementation:
Current implementation of middleware dynamic variable
Lines 338 to 388 in 90b9211
server/middleware.json
"routes": {
"loopback#rest": {
"paths": ["${restApiRoot}"]
}
server/config.json
{
"restApiRoot": "/api"
}
Current implementation of host and port reading from dynamic sources
Lines 102 to 142 in 90b9211
- should keep the precedence same as these
Unknown things that could be concerns (unknown by me only)
- will ARC overwrite these variables?
- does anywhere bypass boot and read these config files?
loopback:refresh
?
from loopback-boot.
I'm having trouble following the current state of this. Also, and perhaps this is NOT something to address with this issue, various service providers specify information on the configured DBs with provider-specific env vars. BlueMix instant runtimes use VCAP_...
env vars, for example.
This should be supported either out of the box (possibly via some config mechanism), or via require of a supported module.
By some config mechanism, I mean like:
{
name: "mysql",
password: "$(VCAP_PROVIDER["mysql"].password)
...
^--- above is totally fake, I don't recall of the top of my head what the VCAP syntax is, but its possible that a survey of bluemix/heroku/openshift/etc. will find that there is some kindof core loopback json config file syntax we could implement that would allow env vars to be referenced and partially unparsed in order to have a generic way to configure an app to run in any (reasonable) cloud environment.
If not, then there will need to be an alternate way to get the ability for LB apps to be runnable easily in BlueMix/Heroku/etc., environments where the configuration is available in the environment, but the LB app developer does not control the env var naming or syntax.
@bajtos should I open another issue for this, or will this issue a good place to discuss that feature?
I also would like to echo @bajtos point about defaults, I would very much want this PR to ensure that the config can have defaults, and env overrides the defaults, so the defaults can be setup for dev envs. And using "named envs" like NODE_ENV=staging
is an anti-pattern, IMO.
from loopback-boot.
@sam-github let's keep this issue focused on environment variables (the 12factor way) and open a new issue for VCAP and other mechanisms.
This is the current status: @jannyHou is improving the connectors to support url
or dsn
setting that allows users to pass all configuration in a single string, see strongloop/loopback#2089. @davidcheung is looking into loopback-boot and how to implement env-variable injection as described above (see #177).
from loopback-boot.
@davidcheung please consolidate the information from the comments above and the demo meeting, so that it's clear what's done and what still needs to be done.
from loopback-boot.
I wanted to use this nice feature, but unfortunately there is no way I can see to interpret an environment variable as another type, eg boolean. For example, the Mongodb connector seems to interpret the SSL param as false if you pass it a string.
Ended up using a datasources.local.js file as below. If we used a URL connection string this feature would have worked, but then I'd have to change and re-deploy all our other (non-Loopback) microservices.
module.exports = {
mongo: {
name: 'mongo',
connector: 'mongodb',
host: process.env.MONGO_HOST || 'localhost',
port: process.env.MONGO_PORT || 27017,
user: process.env.MONGO_USER || '',
password: process.env.MONGO_PASSWORD || '',
database: process.env.MONGO_NAME || 'mydb',
ssl: (String(process.env.MONGO_SSL).toLowerCase() === 'true'), // default to false
allowExtendedOperators: true
}
}
from loopback-boot.
Related Issues (20)
- Subfolders in boot folder HOT 7
- [forward-port] Fix resolution of middleware modules
- Boot scripts are loaded, but not actually run HOT 9
- organizing configuration files leads to problems HOT 3
- test failing on browserify due to form-data HOT 1
- Document refactored design of loopback-boot HOT 1
- datasources duplicated when booting using a config HOT 1
- models not loaded when using config on boot HOT 1
- 3.0.0
- Async boot scripts - silent failure with security risk HOT 6
- resolveAppScriptPath() is not pass scriptExtensions to fixFileExtensions() HOT 2
- [BUG] Middleware is loading .js.map of .js HOT 1
- The middleware "loopback#static" in phase "files"is not defined in the main config
- Async operations in component initialization HOT 3
- isomorphic and uglify causing "AssertionError: must provide a valid {{SharedClass}}" HOT 1
- custom middleware not found when boot options are just __dirname HOT 2
- Why next() requires to trigger process.nextTick in bootscript. HOT 2
- export-api-def doesn't work with loopback-boot@3 HOT 1
- Address usage of config/application on boot LBOptions HOT 2
- Loopback-boot updating to 3.3.1 producting "LoopBack server listening @ https://undefined:undefined/" HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from loopback-boot.