idexlabs/idex.nucleus.core

IDEX Nucleus Core is the framework to build flexible, scalable and structured back-end architecture

idex.nucleus.core's Issues

Allow file streaming

The retrieve all by type feature is too slow.

The current benchmark is ~200 resource retrieved in 3 seconds.

Implement action's event propagation

Log action execution elapsed time

The engine doesn't throw an error if it has no name

Trying to instantiate an engine without a name doesn't throw an error, but part of the initialization code expects the value to be set.

const $engine = new NucleusEngine();

This throws a Nucleus Datastore error NucleusError: The item must be a string.

The name should be mandatory and throw a clear error when omitted.

Implement resource versioning as part of the Resource API

When updating a resource using the Resource API, the changes made to the object and persist it along with resource. As part of this feature, an action to rewind the resource to a specific version should also be implemented.

Implement action's event propagation for distributed system

Part of the mechanism is in place, but not leveraged.
When an event is published, its key is pushed to a sorted list that should be used to make sure only one engine instance handles a given event.

The engine datastore and the resource datastore can't be separated

An engine requires a datastore connection which it uses to store engine specific data and passes to the Resource API; if a user chooses to pass a datastore adapter (not Redis), the engine won't be able to perform some low level tasks. The engine should accept, optionally a $resourceDatastore and a $cacheDatastore... Also, the engine should verify that the engine datastore is a Redis adapter.

Allow an engine to send warning events to a specific engine.

Engine should accept a resource datastore and a cache datastore

Refactor direct usage of the datastore's direct connection to Redis in the Resource API

Refactor direct usage of the datastore's direct connection to Redis in the Resource API;

Input validation adapter

Like the datastore, an input validator should be interchangeable; right now, the default is the Nucleus validator.

An action moved to another engine is not removed from the original engine

When using autodiscovery, an engine's action's configuration are stored locally.
If an action was to be moved to another engine, the action's configuration would be stored under the second engine and the first.
The first engine should be able to clean after itself.

Share Redis connection for non-hanging Nucleus Datastore

It would be interesting to attempt to reduce the number of Redis connection in a given controlled ecosystem. If one or many engines are instantiated on the same node process, there is no point of having multiple Redis connection since they are both single threaded.
Hanging connection like those sued to subscribe to Redis commands or events should not share that connection for obvious reason.

class NucleusDatastore {
  constructor (datastoreName = 'Untitled', options = {}) {
    const {
      $logger = console,
      $$server,
      index: datastoreIndex = 0,
      port: datastorePort = 6379,
      URL: datastoreURL = 'localhost'
    } = options;

    this.name = datastoreName;
    this.index = datastoreIndex;

    this.$$handlerCallbackListByChannelName = {};
    this.scriptSHAbyScriptName = {};

    this.$$server = $$server || redis.createClient({
      db: datastoreIndex,
      host: datastoreURL,
      port: datastorePort
    });
  // ...
  }
}

Implement method hooks

The feature would allow a user to specify code to be executed before and after a specific method.

Failed action publication error doesn’t log the action ID

The Resource API methods doesnt accept the extended signatures.

Implement a resource archive API

Errors rejected from a publish action request does not contain the meta

When using the #publishActionByNameAndHandleResponse method to publish an action and the action rejects with a Nucleus error containing some meta data, the error that is caught does not contain the meta data.

/**
 * @Nucleus ActionName ThrowErrorWithMetaData
 */
function () {

  throw new NucleusError(`This is a Nucleus error with meta data`, { testID: uuid.v4() }); 
}

...

$engine. publishActionByNameAndHandleResponse('ThrowErrorWithMetaData', {}, uuid.v4())
  .catch((error) => {
    error.meta.testID === undefined
  });

The Nucleus error is correctly store in the action, so the issue probably comes from the NucleusError constructor. library/Error.nucleus.js

The Nucleus error is rebuilt around library/Engine.nucleus.js:740.

Might be related to: #14

Implement an action middleware

There are action events stored in the event datastore without a TTL.

Nucleus Datastore used as queue handler (blocking connections) are not destroyed properly

When a Nucleus Datastore uses the blocking command BRPOP (BLPOP), the client cannot be destroyed properly.

Registered anchor node (indexed node) can't be unregistered.

When creating a relationship with a resource using the is-member-of predicate, the system registers the object as anchor node to speed up queries; this relationship is never unregistered, ie: the anchor node's resource is removed.

Can’t collapse to dot notation an empty array

The Nucleus Datastore API has a method to collapse an object to dot notation, currently it returns undefined when passed an object with an empty array.

Deep hash structure are not expanded properly

When retrieving a deep hash structure, the datastore tries to spread the data using dot notation to allow for the data to be searchable. A structure containing an array should use "bracket notation" and be expanded to an array as expected.

const object = {
  key: [
    'value1',
    'value2'
  ];

const expandedObject = NucleusDatastore.expandDotNotationObject(object);

chai.expect(expandedObject.key[0]).to.equal('value1');

Publishing an action should accept a correlation ID.

https://blog.rapid7.com/2016/12/23/the-value-of-correlation-ids/

Abstract the Datastore API for common persistence task

The Datastore API has to be more generic so it can be used interchangeably with the Resource API.

A user that is a member of the SYSTEM node can't retrieved all resources by type as expected.