feathersjs-ecosystem/configuration

[MOVED] A plugin for configuring a Feathers application

@feathersjs/configuration

Important: The code for this module has been moved into the main Feathers repository at feathersjs/feathers (package direct link). Please open issues and pull requests there. No changes in your existing Feathers applications are necessary.

A small configuration module for your Feathers application.

About

@feathersjs/configuration is a module that wraps node-config to configure your Feathers application.

npm install @feathersjs/configuration

const feathers = require('@feathersjs/feathers');
const configuration = require('@feathersjs/configuration');

// Use the application root and `config/` as the configuration folder
let app = feathers().configure(configuration())

See the Feathers configuration docs for the full API usage.

License

Copyright (c) 2018

Licensed under the MIT license.

configuration's Issues

Configuration Management

overview

as per issue #24 we want to make this repo more focused on configuration management as a whole. When it comes to reading configs, we'll look to node-config for this rather then re-invent that functionality. We should also build out the repo to have support for more complex production configuration management storage such as vault, consul and etc so that we can not only protect secrets but also eventually support service discovery.

goals

• create base Configuration class
• create base Adapterclass
• Adapters should be able to read and write
• a configuration will have default Adapters
• a configuration can be extended with one or more storage Adapters
• support multiple Adapters
  • process.env (default)
  • json files (default)
  • vault
  • consul
  • etcd
• Adapters will be implemented with loading precedence
• proposed loading precedence: process.env.feathers, json files, storage (vault, consul, etcd)
• a value that is already set will not be overridden by subsequent Adapters
• conflicts will be logged for user transparency
• processes will manage config via app.get and app.set
• operators will manage storage configs via feathers config get and feathers config set
• operators will need to pass explicitly the --storage when managing a configuration
• configuration will default to always reading from process.env and json files
• if storage options are passed, then additional configurations will be merged
• an applications should be able to simply listen for new configurations added at any name space

examples

implement

const feathers  = require('feathers');
const configuration  = require('feathers-configuration');
const etcd =  require('feathers-etcd')(['http://127.0.0.1:2379']);
const vault = require('feathers-vault')(['http://127.0.0.1:8200', process.env.VAULT_TOKEN]);

feathers()
  .configure(configuration()) // process.env.feathers + json files by default
  .configure(etcd())
  .configure(vault())

service discovery

// listen to a specific storage adapter directly
vault.on('create', function(key, value) {
  console.log(key, '=>', value); // /databases/mongo/production => mongodb://mongo.stack.local:27017
});

//listen across all storage adapters that support pub sub
configuration.on('create /databases/mongo/*', function(key, value) {
  console.log(key, '=>', value); // /databases/mongo/production => mongodb://mongo.stack.local:27017
});

vault.create({ 
  key:'/databases/mongo/production', 
  value: 'mongodb://mongo.stack.local.27107'
});

workflow

// 1. loads from process.env
// 2. loads from json files and merges
// 3. loads from etcd and merges
// 4. loads from vault and merges
// 5. listens to each storage
// 6. notifies of changes in storage

<NODE_ENV>.json config need deep merge options

Issue related to: feathersjs-ecosystem/authentication#191.

the problem is that if you have a default.json with

{ "facebook": { "secret": "TESTSECRET", "scope": ["email"] } }

and then another environment configuration (production.json) with

{ "facebook": { "secret": "SECRET_FROM_ENV" } }

It blows away the scope instead of merging it

Needs an escape character

There needs to be an escape character (most likely \) if you do not want to convert a configuration value. E.g.

{
  "env": "\\NODE_ENV",
  "path": "\\./home"
}

Should set env to the actual string NODE_ENV and path to ./home

Investigate node-config

https://github.com/lorenwest/node-config seems to be doing almost the same thing so it might make sense to use it here.

Why use "NODE_ENV=development" with default.json?

My intuition is to have default.json for running locally w/o setting any NODE_ENV.
And to run on DEV environment, we need to do NODE_ENV=development which would use the config/development.json instead.

Stuck in configuration loophole...

Steps to reproduce

I'm trying to setup my global app logger (winston), independant from the logging middleware.
Something like that:

logger.ts

import winston = require('winston');
import { app } from '../app';

const config = app.get('logger');
const transports = [];

if (config.console) {
  transports.push(new winston.transports.Console({
    colorize: true,
    level: config.console.level
  }));
}
export const logger = new winston.Logger({ transports });

Obviously I need the app config for this.

But on the other hand, to setup my logging middleware express-winston, I need the logger:

app.ts

import path = require('path');
import favicon = require('serve-favicon');
import compress = require('compression');
import helmet = require('helmet');
import cors = require('cors');
import bodyParser = require('body-parser');
import feathers = require('feathers');
import configuration = require('feathers-configuration');
import hooks = require('feathers-hooks');
import rest = require('feathers-rest');
import expressWinston = require('express-winston');
import errorHandler = require('feathers-errors/handler');

import { logger } from './core/logger';
import { notFoundHandler } from './middleware/not-found-handler';
import { services } from './services';

export const app = feathers();

app.configure(configuration(path.join(__dirname, '..')));

app.use(compress())
  .options('*', cors())
  .use(cors())
  .use(helmet())
  .use(favicon( path.join(app.get('public'), 'favicon.ico') ))
  .use(bodyParser.urlencoded({ extended: true }))
  .use(bodyParser.json())
  .use(expressWinston.logger({
    winstonInstance: logger,
    level: 'debug'
  }))
  .use('/', feathers.static(app.get('public')))
  .configure(hooks())
  .configure(rest())
  .configure(services)
  .use(expressWinston.errorLogger({
    winstonInstance: logger
  }))
  .use(notFoundHandler())
  .use(errorHandler() as any);

And boom, app.ts needs logger and logger.ts needs app for config. How to get out of this?

I never had similar issues with other frameworks, since usually the config is decoupled from everything else. Isn't this a major issue with your current approach?
Having the config exposed in the app is nice, but I think also having access to the config independently is also needed, for case like this, or maybe I'm missing something?

Getting Env name

I think it is usefull to know which enviroment is loaded.
Example for doing something in default or development mode. (not using directly env vars in node).
@daffl what do you think about?

Configuration should recursively go through the values

Something like this doesn't work:

{...
  "auth" : {
      github: {
        "clientID": "ENV_VARIABLE_NAME"
        ...
      }
      ...
    }
    ...
  }
  ...
}

Missing TypeScript declaration file

Would you accept a PR or be willing to add a index.d.ts TypeScript declaration file for this lib? Even if the definitions are not complete yet, adding the following line to a declaration file would prevent a TS error "File '/path-to/feathers-configuration/index.d.ts' is not a module.".

declare module 'feathers-configuration';

A way to have local override

I see that NODE_ENV.json override is disabled only for development mode. Is there specific reason for that?

It's already checking for file existence, so it will be good if it just override configs from development.json if the file exists.

That will be helpful to have some configs local only by having default.json in version control and development.json ignored.

Make this repo more about managing configuration

As others have mentioned in #8 we should look at using something like Vault or making a configuration service. This might be a way to re-purpose feathers-configuration. It should be more for managing config keys and secrets.

feathers-configuration could be a configuration management service that can store data wherever and just has hooks to encrypt/decrypt values. I think we have a couple options here:

1. Create a feathers-vault adapter. Vault can be used to read/write to many different backends like AWS IAM users, SQL DBs, Files System, etc. I would need to investigate further but @slajax might know more. Could be use like this:

   const vault = require('feathers-vault');
   app.use('config', vault(options));

2. Create our own hooks that encrypt and decrypt keys and allow you to CRUD them as well. It's possibly a little more feathersy. Could look something like this:

   const service = require('feathers-mongodb');
   const options = {
      Model: db.collection('config')
   };
   
   app.use('/config', service(options));
   
   app.service('config').hooks({
     before: {
       create: [encrypt()]
     },
     before: {
        get: [decrypt()]
     }
   });

Thoughts @feathersjs/core-team?

Feathers writing in typescript fails to boot on configuration

Where it fails

I'm writing feathers in typescript but it fails on configuration so it won't boot.
Also, I can't find any information about writing feathers in typescript or es6 so I think I'm on my own here. I just gonna follow the steps that I have for Express.

Expected behavior

That the app boots

Actual behavior

It fails to boot

System configuration

just a normal laptop

Module versions (especially the part that's not working):

    "debug": "^2.6.8",
    "express": "^4.15.3",
    "feathers": "^2.1.7",
    **"feathers-configuration": "^0.4.1",**
    "feathers-errors": "^2.9.1",
    "feathers-hooks": "^2.0.2",
    "feathers-hooks-common": "^3.7.2",
    "feathers-knex": "^2.7.2",
    "feathers-objection": "^0.6.0",
    "feathers-rest": "^1.8.0",
    "feathers-sequelize": "^2.3.0",
    "feathers-socketio": "^2.0.0",

NodeJS version:
node 8.4.0
npm 5.3.0
yarn 0.27.5

Operating System:
windows 10

Module Loader:

import fs from 'fs';
import path from 'path';
import webpack from 'webpack';
import NodemonPlugin from 'nodemon-webpack-plugin';
import WebpackNotifierPlugin from 'webpack-notifier';

let nodeModules = {};
fs.readdirSync('node_modules')
  .filter(function(x) {
    return ['.bin'].indexOf(x) === -1;
  })
  .forEach(function(mod) {
    nodeModules[mod] = 'commonjs ' + mod;
  });


let webpackConfig = {
	target: 'node',
  entry: `./server/index.ts`,
  output: {
		path: (process.env.NODE_ENV === 'production') ? path.join(__dirname, 'dist') : path.join(__dirname, 'bin'),
		libraryTarget: 'commonjs',
    filename: 'server.js'
	},
	externals: nodeModules,
  resolve: {
		extensions: ['.ts']
	},
  module: {
    rules: [
      { test: /\.json$/, use: 'json-loader' },
      { test: /\.tsx?$/, use: 'awesome-typescript-loader?configFileName=./tsconfig.server.json', exclude: /(node_modules|bower_components)/ }
    ]
	},
  plugins: [
    new webpack.SourceMapDevToolPlugin(),
		new NodemonPlugin()
  ]
};

export default webpackConfig;

Prevent automatic expansion of environment variables

This is usually fine to be able to use environment variables in config files without the need of creating an additional file as with node-config : https://github.com/lorenwest/node-config/wiki/Environment-Variables#custom-environment-variables.

However it has a side effect that if you want to have a string value corresponding to the name of an environment variable you can't because you have no mean to prevent expansion. I found the specific case of a string value required to be TMP but this corresponds to the TMP environment variable, so expanded to my temporary directory path !

I would recommend to prefix environment variables with eg $ so that we have control over the expansion.

PR: Support modules in config

We've been throwing a ton of stuff into our FeathersJS install and have outgrown purely static config files. Needed some flexibility and options for computed values so that derived config wasn't pushed up to the app initialization or higher layers.

I took a look at feathers-configuration and since it is using require() to pull in the .json files, there's no reason I can think of not to support both json and node module exports.

I'm going to send over the PR. I don't see anything in these minor mods which breaks existing code; only expands the options. But let me know if I missed some detail.

Short summary of what I remember changing:

• Remove explicit .json requirement
• changed the <env> test file to be testing.js instead of testing.json (as shortest path to test both old functionality and new w/out creating more test data files).
• modification to package.json test script because jshint was failing w/out telling me why it failed. Only saw node errors and not what jshint reported (deja vu?: feathersjs-ecosystem/generator-feathers@f006e4d)
• Readme updates for new functionality

Deprecate v1 in favour of node-config

Related to #8. We've decided that for v2 of the generator and CLI we're going to use node-config and phase this out in it's current form.

Docs are wrong

Docs say:

require('feathers-configuration')()

I need to do

require('feathers-configuration')()()

I'd like to do

require('feathers-configuration')

just like in node-config.

Move config options to app.config instead of the Express app object.

In order to decouple us from Express we might want to namespace config options to app.config instead of using app.set() and app.get(). The rationale:

1. node-config already exposes the .get and .set functionality
2. with Express it’s kind of a weird legacy API inconsistency because you can do app.get for a route as well which is confusing
3. It might help us decouple from Express a bit more instead of relying on the .get and .set functionality.
4. It gives a central place to access config values so you can dynamically set and get them. Basically a basic synchronous in memory service.
5. app.config is a bit more explicit, especially for people that are new to Feathers/Express

To get this done

• Assign the node-config instance to app.config
• Alias Express' app.get and app.set to app.config.get and app.config.set except for when registering an Express get route. This might mean implementing with the Buzzard release when the engine adapters are more established.

Related

feathersjs/feathers#258

Config adding a value of userName in runtime its overwritten to the OS name

I added the auth config values to the config files, and for the local when I set the usernameField to use userName instead of email in run-time it replaces the usernameField value to my PC name. If I add anything else it works but not username.

 local: {
      entity: 'user',
      usernameField: 'userName',
      passwordField: 'password'
  }

// becomes 

local: {
      entity: 'user',
      usernameField: 'MyPcName',
      passwordField: 'password'
  }

Nested configuration

Hi,

It is said that feathers-configuration is a simple wrapper on node-config. However, with node-config, if you have nested configuration, like that:

{
  "Customer": {
    "dbConfig": {
      "host": "prod-db-server"
    },
    "credit": {
      "initialDays": 30
    }
  }
}

you can access the host attribute by using config.get('Customer.dbConfig.host'). However, with feathers-configuration, this dot notation do not work.