goliatone/application-core

Application core

application-core's Issues

Enable installation

Install core.io cli tool belt:
$ npm i -g core.io-cli
not working.

Add makeId function to context

In order to have a consistent way of making ids across modules, provide a makeId helper function.

logger: we should be able to format output by column width

If we have a multiline message, we need to format spacing, i.e. column width:

ERROR [16:16:12] ├── datamanager  :  _importModel.iterate error
preference.updateOrCreate failed: [Error (E_UNKNOWN) Encountered an unexpected error] Details:  Error: .findOne() requires a criteria. If you want the first record try .find().limit(1)

DataManager does not know what to do with this error.
They will be bubbled up, you should handle them. 

ERROR [16:16:12] ├── datamanager  :  _importModel.iterate error
                                     preference.updateOrCreate failed: [Error (E_UNKNOWN) 
                                     Encountered an unexpected error] Details:  Error: .findOne() requires a 
                                     criteria. If you want the first record try .find().limit(1)

                                     DataManager does not know what to do with this error.
                                     They will be bubbled up, you should handle them. 

context.resolve should timeout and throw error

If we are trying to resolve a missing dependency we need to throw an error or at the very least notify the user, our code might be dependent on the dependency.

this.context.resolve('datamanager').then(()=> {
});

error: catch unhandled promises

Add:

• "unhandledRejection" (corresponds to the local Promise.onPossiblyUnhandledRejection)
• "rejectionHandled" (corresponds to the local Promise.onUnhandledRejectionHandled)

process.on('rejectionHandled', handleError);
process.on('unhandledRejection', handleError);

logger: add getChildLogger

Extended loggers should have a method that returns a child logger. The only thing that distinguish a child logger is the name: parent:child.

ie:

INFO  [15:12:19] ├── admin      : Loading strategy bean "bearer".
INFO  [15:12:19] ├── admin:auth : Bearer strategy loaded

Either modify getLogger to work this way, or add a new method.

Feature: I want to be able to search each module for a file type, ie commands.

I want to be able to structure projects around modules, rather than roles:

.
|-- commands
|-- models
| -- modules
|     |-- user
|          |-- models
|          |-- commands

The idea is that when looking for commands, we create a list of possible directories to look for:

let cmdPaths = ['./commands'];
let mpath;
this.modulePaths.map((p)=>{
    mpath = join(p, 'commands');
    if(exists(mpath)) cmdPaths.push(mpath);
});

Then we will scan all of those dirs for our files.

Document module configuration object

Document the attributes that coreio will add to the configuration object, like moduleid.
Also, if muteLogger is present, the logger for that module will be muted.

resolve: Add timeout to ensure we are not stuck on a promise that does not resolve

Related to #23, we should be able to timeout a call to resolve after a given period of time. As it is right now, if a call to resolve never executes the error would not be noticed.

We could use either Bluebird's timeout feature or build our own implementation.

function myPromise(ms, callback) {
    return new Promise(function(resolve, reject) {
        // Set up the timeout
        let timer = setTimeout(function() {
            reject('Promise timed out after ' + ms + ' ms');
        }, ms);
        let cancelTimer = _ => {
            if (timer) {
                clearTimeout(timer);
                timer = 0;
            }
        };

        // Set up the real work
        callback(
            value => {
                cancelTimer();
                resolve(value);
            },
            error => {
                cancelTimer();
                reject(error);
            }
        );
    });
}

from here

Command

We should aim to normalize/formalize the return format for commands:

KO:

{success: false, mesasge: '', id: event.id}

OK:

{success: true, ..., id: event.id}

Some of this we could augment in the handler in the same way we normalize events before being delivered.

Ensure that commands are optional

If we don't have a commands folder the application fails, ensure that commands are optional.

Send error during deregisterApplication

If our application is crashing and we are closeing, we should send the current error along to the registry service.

moduleid or moduleId: pick a convention and stick to it

Should properties be camel cased or all lower caps? We should pick up one and stick to it through the code base.

Post process each config namespace before init module

Use case:

• Inject a logger instance on each configuration object so that it trickles down.

Accept functions, iterate over them before registering:

function(app, id, instance, config){
    config.logger = app.getLogger(id);
}

Take option during creation, or in app.js:

let core = new Application({
    configurers: [
         function(app, id, instance, config){
             config.logger = app.getLogger(id);
         }
   ]
});

This probably should be handled by in module...

remove debug calls in dispatcher

Remove calls to debug in dispatcher module:

app.getLogger('dispatcher').debug

Use info instead or something that is available in console.

Enable getLogger for child loggers

Enable child loggers to create new loggers.

Event/Command IDs and session ID matching strategy

We should come up with a workflow or set of best practices to enable consistent session ids across different actions so that they can be grouped together logically.

If clients start the interaction, we could reuse the ID of the initiator (HTTP request, WS request, etc) and it should propagate across sections.
What are the boundaries? Should we create a new ID per request ? etc

Add support for hot-reloading

It would be ideal to have support for hot-reloading, specifically for the persistence module. When the persistence module changes, it will update the context. If we have clients connected during that period they will have a stale context in their REPL.

We should add a way to invalidate the context. That is, enable partial update of already loaded modules, or at the very minimum add a context.invalidate function, that would trigger an event.

Track features in core.io-persistence

Normalize events

Normalize event payloads:

context.emit(CustomEvent.MY_TYPE, {
    type: CustomEvent.MY_TYPE,
    source: context, 
});

context.emit(CustomEvent.MY_TYPE, new CustomEvent(CustomEvent.MY_TYPE, payload));

Fix handleError

errors.js handleError, fix #L-16:

function handleError(err){
    app.logger.error(err);
    cleanup('ERROR', 1);
}

Relative path resolution when application started from different directory

If the process.cwd does not match the directory where the application's package resides, things break.

1. We want to catch this during config load and give some useful information at static loadConfig
2. We could force to change to the result of getPathToMain

For point 2 we could use this:

process.chdir( getPathToMain() );

Add version attribute to application instance

Enable some sort of introspection at the instance level for applications so that .e.g. modules can check for feature support, etc.

Add paths manager

Add a module to manage paths. We can add/get paths and build new paths from them.
This way we can make global changes in paths without having to chase individual instances.

Similar to how we manage paths in BaseCommand

Normalize Errors

Create custom errors, use those.

Error handling in mountDirectory

Given the following scenario:
We changed to use ascoltatori-lowfat from previously using ascoltatori but we did not update the require in the module. require won't be able to find ascoltatori since is not installed anymore...

If we have an error in one of our modules, the loading cycle breaks:

ERROR [13:20:06] ├── core         :  true 
ERROR [13:20:06] ├── core         :  Cannot read property 'dependencies' of undefined 
ERROR [13:20:06] ├── core         :  TypeError: Cannot read property 'dependencies' of undefined
  at beans.map (/Users/guest/my-dashboard/src/node_modules/in/lib/sortByDependencies.js:146:35)
  at Array.map (native)
  at createGraph (/Users/guest/my-dashboard/src/node_modules/in/lib/sortByDependencies.js:145:11)
  at PluginLoader.sort [as sortFilter] (/Users/guest/my-dashboard/src/node_modules/in/lib/sortByDependencies.js:8:17)
  at PluginLoader.sort (/Users/guest/my-dashboard/src/node_modules/in/lib/in.js:179:21)
  at PluginLoader.load (/Users/guest/my-dashboard/src/node_modules/in/lib/in.js:150:14)
  at find.then.then (/Users/guest/my-dashboard/src/node_modules/in/lib/in.js:228:37) 

Add core-command

Add a core-command that extends cli-command with the added functionality of loading dependencies of command. We might need this functionality to ensure the command is not executed before dependencies are loaded.

Also, note that you have to return a promise from the execute method or the cli will exit before finishing loading things...

'use strict';
const CoreCommand = require('base-cli-commands').CliCommand;

class FindUserByIdCommand extends CoreCommand {


    execute(event) {
        const context = event.context;

        return context.resolve(this.constructor.dependencies).then(_ => {
            console.log('resolved....')
            return this.doExecute(event);
        });
    }

    doExecute(event) {
        const context = event.context;
        const logger = context.getLogger('service id insights');

        const id = event.id;

        logger.info('Get latency for service %s at interval %s', id);

        const User = context.models.User;

        return User.findOne(id).then(user => {
            console.log(user);
        });
    }

    static describe(prog, cmd) {
        cmd.argument(
            '<id>',
            'User id'
        );

        cmd.option(
            '--interval <slice>',
            'Aggregate interval <slice> that will partition our aggregates.'
        );
    }
}

FindUserByIdCommand.DEFAULTS = {};

FindUserByIdCommand.COMMAND_NAME = 'user.find';
FindUserByIdCommand.DESCRIPTION = 'Find user by id';

FindUserByIdCommand.dependencies = ['persistence'];

We are now able to call the command from the terminal loading the application as context:

./bin/cli user.find 23

Logger: Make error output more prominent

Currently we are not giving error output enough visibility. Output text should be red, or something...

config.basepath should have a sensible default

Currently, if you config object does not specify a basepath attribute, we can not resolve it in config resolver:

app.js:

module.exports = {
    banner,
//    basepath: __dirname,
    name: 'auth-server',
    environment: process.env.NODE_ENV || 'development',
};

config/filesync.js:

module.exports = {
seed: {
        path: '${app.basepath}/data/seed',
        options:{
            depth: 0,
            //This will add a delay of 2s and 100ms polling
            awaitWriteFinish: true,
            //Ignore all dot files.
            ignored: /(^|[\/\\])\../
        },
        moveAfterDone: false,
        historyPath: '${app.basepath}/data/history'
    }
};

Output:

$ npm start

> @0.0.0 start /Users/eburgos/Development/WW/authentication-server/src
> node index.js

GSolver max recursion for: ${app.basepath}/data/seed
GSolver max recursion for: ${app.basepath}/data/history
...

Add function to extend context

Formalize a way to extend context with new functionality. Instead of directly attaching properties to the app context, provide a function to do so.

app.use = function(name, fn) {
   Object.defineProperty(app, name, {
      value: fn,
      writeble: false,
      configurable: true
   });
};

This way we can prevent core extensions to be overwritten unnoticed but still be able to do so if we need to- configurable.

Empty modules break import

If we have an empty folder in the modules dir- it breaks loading. Related to in module.

Make `repl.context.app` configurable

Make app configurable, want to make this depends on feature #28 :

//TODO: Make name of exposed context configurable.
repl.context.app = context;

repl.expose(config.contextName, context);

Fix default logger message make easy to understand

Currently if we log an empty message we have a default text which is not very helpful.
Either change behavior or text.

Change:

this._message = 'No message';

To:

this._message = '<Empty Logger Message>';

Use `context.dispatch` to dispatch command events

Provide a normalized interface in context to dispatch commands.

command function: response to wrap out in Promise.resolve

Right now we assume that what we return from our command handler is a promise, we should ensure that by wrapping using promise resolve, from:

if (_isFunction(e.respondTo)) {
    out.then(response => {

To:

Promise.resolve(out).then(response => {

Logger: get tty width and use to calculate max characters

We have a two column layout:

+----------+ +----------------------------+
---  INFO ---  ---------- CONTENT --------

Make content be sized based on tty size

Document last bootloader event

After application core goes over all the tasks, the last event it fires is modules.ready. Add to documentation:

1. coremodules.ready
2. modules.ready
3. modules.resolved
4. hook run:
   4.1. run.pre
   4.2. run.post
   4.3. run.complete
5. context.ready

Update examples.

Enable module models

We should be able to bundle modes within modules and have them work. Currently we have to manually move models to the models directory.

We could have a registration process, we call it from the module init method. It would either link or copy the models file to the models directory.

Register command, handler support for function or Class

We should be able to register command functions or classes. The criteria would be to check if the handler's prototype has an execute function. If so, then we treat it as a command

if(typeof handler === 'function'){
  if(typeof handler.prototype.execute === 'function'){
    let command = new handler();
    handler = command.execute;
  }
}

Fix error monitoring

The error monitoring module was initially intended to handle errors in a data pipeline, save whatever we had in memory at that point, and exit.

Basically, providing a small window to clean up responsibly, however, if not done properly this will mess everything up.

Does it make sense to have monitoring as a core module?

register: Add timeout to ensure we are not stuck on a promise that does not resolve

We should have a mechanism to check that a module is not stuck for ever.

We could use something like p-timeout

Promise.resolve(out).then((module)=> {
            _register(this, name, module);
        }).catch(this.onErrorHandler.bind(this, true, 'register: Error registering module ' + name));

Fix links in guide.md and GETTING_STARTED_GUIDE.md

LINKS

GETTING_STARTED_GUIDE.md

re[core-persistence]:https://github.com/goliatone/core.io-persistence
re[core-server]:https://github.com/goliatone/core.io-express-server
re[core-data]:https://github.com/goliatone/core.io-data-manager
re[core-sync]:https://github.com/goliatone/core.io-filesync
re[core-auth]:https://github.com/goliatone/core.io-express-auth
re[core-crud]:https://github.com/goliatone/core.io-crud

ok[envset]:https://github.com/goliatone/envset
re[taskfile]:https://github.com/adriancooney/Taskfile
re[module-instantiation]:modules.md#module-instantiation
re[config-docs]:guide.md

ok[node]:http://nodejs.org/
ok[npm]: http://npmjs.com
404[core.io-cli]:https://www.npmjs.com/package/core.io-cli
404[core.io-cli-docs]:https://github.com/goliatone/core.io-cli/tree/master/docs
ok[core.io-starter-template]:https://github.com/goliatone/core.io-starter-template
re[scl]:https://github.com/goliatone/simple-config-loader
re[poke]:https://github.com/goliatone/poke-repl
re[noop-console]:https://github.com/goliatone/noop-console

guide.md

ok[core-persistence]:https://github.com/goliatone/core.io-persistence
ok[core-server]:https://github.com/goliatone/core.io-express-server
ok[core-data]:https://github.com/goliatone/core.io-data-manager
ok[core-sync]:https://github.com/goliatone/core.io-filesync
ok[core-auth]:https://github.com/goliatone/core.io-express-auth
404[core-crud]:https://github.com/goliatone/core.io-crud

ok[poke]:https://github.com/goliatone/poke-repl
ok[noop-console]:https://github.com/goliatone/noop-console
ok[scl]:https://github.com/goliatone/simple-config-loader
ok[ioc]:https://en.wikipedia.org/wiki/Inversion_of_control
ok[envset]:https://github.com/goliatone/envset
ok[mixin]:https://www.joezimjs.com/javascript/javascript-mixins-functional-inheritance/
ok[winston]:https://github.com/winstonjs/winston
ok[npm-packages]:https://docs.npmjs.com/how-npm-works/packages
ok[poke-repl-banner]:https://github.com/goliatone/poke-repl/tree/master/examples
ok[ascii-art]:http://www.network-science.de/ascii/

should be (#application-context-1)application context
okGetting Started
okReference
okConcepts
okProject Structure
okmoduleId
okREPL
should be (#connect)?[connect][connect]
okconfiguration object
okextending context
ok? config/
okconfiguration interpolated values
ok loaded
okconfiguration dependencies have been solved
ok core modules

Add support for command hot-reloading

Related to #58 but scoped to command registration.
Watch commands directory, if a file is added/updated then:

• clear cache for that command module
• unregister previos event listener
• register module command again

NOTE: As it stands right now it might not be possible to unregister a command since we don't have it's listener. We might have to store the handler and eventtype together.

There's one edge case. If we enable module sub-commands, and one of those sub-commands has the same name as a command.

Look at clear-module, or:

function requireUncached(module){
    delete require.cache[require.resolve(module)]
    return require(module)
}

Or this so answer.

errors.js exit on error

Add to #L-40:

process.exit(0);

Fix run: this.once(`coreplugins.${this.registerReadyEvent}`, this.run);

This never get's fired:

this.once(`coreplugins.${this.registerReadyEvent}`, this.run);

It should be:

this.once('coreplugins.ready', this.run);

Handle close on application.js

What are we going to do with close #L-354?

close(code, label){
    //here we could run a close routing. Whatever it is, it should not
    //take more than 10 seconds- Serene's default
    return Promise.resolve();
}

As it is right now, is a hook so we can override with config so that we implement any cleaning strategy we need. i.e maybe a server needs to handle pending requests, etc. We should, at least, document it.

Extend REPL with function to extend context

Currently to add anything to the REPL context we need to access the context directly:

context.resolve('repl').then((repl) => {
    repl.context.myCustomCommand = function() {
        console.log('TODO: We should really thing of a better example!');
    };
});

Make it so that we use a method exposes in the repl itself:

context.resolve('repl').then((repl) => {
    repl.export('myCustomCommand', function myCustomCommand() {
        console.log('TODO: We should really thing of a better example!');
    };
});

Increate resolveTimeout

Currently resolveTimeout is set to 10s which is to low. Some modules, like the persistence/ORM module, can take longer than 10 seconds to get started or they might have a higher timeout before they throw. We can end up aborting the app before the problematic module has had a chance to throw en Error thus not really having any insights on what's going on.

We could have a lower value for prod, higher value for dev, and switch between to figure out issues.

(Add to docs)

monitoring module

Move signals.js to its own package- Serene.
Move signal related functionality away from monitoring into errors module. Import serene package there and handle errors.

Review when/how we create logger instance for modules

We should make it available in the config object for all modules.
We should rewrite modules so that they do not use context.getLogger and use config instead. Best practice?

There are two modules that are requried

It looks like there are two modules which are required right now even tho we mark them as optionals:

• logger
• dispatcher

We can not load logger if we do some modifications:

• remove calls to logger.debug which are not part of the console.
  • patch console: console.debug = console.log
• add a default getLogger implementation.

getLogger: function() {
    return console;
}

dispatcher is needed to run the hook mechanism... figure out how to run an app without hook.