hipages/inceptum

NodeJS Enterprise Applications Framework

Inceptum – Enterprise Node.js Framework

What is it?

Inceptum is a framework built at hipages to support our move towards microservices.

Why?

The move to microservices may or may not make sense for your company. In our case as our Engineering team keeps on growing we feel that having clear ownership boundaries of different parts of the system will be beneficial.

But the microservices journey is filled with a lot of operational complexities:

• Multiple moving pieces that make debugging more difficult,
• Cascading failure across dependencies,
• Monitoring of multiple applications,
• and many, many more.

It is for this reason that we wanted to embed as many operational concerns as possible into a common framework that would power all of our microservices. This is where Inceptum comes in.

What do you get?

These are some of the benefits of using Inceptum to build your apps:

• Typescript ready: Javascript is a great language, and V8 (which powers Node.js) is a great engine with outstanding performance. But Typescript is better! Having a strongly typed language can prove very beneficial in preventing difficult to find errors.
• Inversion of Control (IoC): Inspired by the Spring Framework and the support for Decorators, you can use annotations like @Autowire or @AutowireConfig to inject the necessary dependencies in your classes. This provides much better testability and isolation of concerns.
• Use of express.js: Built from the ground up to support Rest/HTTP microservices and APIs, it has a sensible setup for express.
• DB enabled: With support for MySQL and Postgress built in, you can easily start integrating DBs in your apps in a matter of seconds.
• Solid logging framework: Based on Bunyan, Inceptum makes logging management easy, it allows for easy configuration of log levels per logger and stream.
• Swagger/OpenAPI enabled: Because API support is critical in microservices and documentation is hard. Swagger/OpenAPI solves for those two problems and Inceptum makes it easy by allowing you to easily route endpoints specified in the swagger file to controllers defined in the context.
• Prometheus enabled: Understanding how your app is doing is critical for successful operations of your microservices. Inceptum is heavily instrumented using Prometheus and will, out of the box gather and publish: all the basic node metrics, http requests metrics, db connection pools metrics, and many more.
• New Relic enabled: You can very easily use New Relic as your APM to gain insights of your app.
• Health Checks Enabled: A typical miss on many frameworks, your app will automatically expose a health check endpoint that will show whether your app is running properly. If you use a DB, RabbitMQ, or other backend provided by Inceptum, each of them will register a health check automatically.

And many more

So, how mature is Inceptum

We're currently using Inceptum in production supporting more than 5 microservices and the number is growing by the day. We're very close to reaching a 1.0.0 release and we have put quite a bit of effort into test coverage (it could be better, and you can help us ;).

Most importantly, though, there's a lot of commitment from the hipages Engineering team to keep supporting and building the framework as we continue our move to microservices.

How do I get started?

You can check the online documentation available in https://inceptum.io.

Questions, issues, etc?

Please open your tickets in Github.

inceptum's Issues

Create metrics instrumentation

Create utility classes we can use to push metrics (counters, gauges, histograms, etc) to a system that collects the stats.
We want to go with Promoetheus, se we should use the datadog protocol (extension of statsd protocol)

Add an ignore option to inceptumApp.addDirectory method

Calling addDirectory on a new InceptumApp will add all files in the directory in a recursive manner. However, there can be instances where a user wants to ignore some files or subdirectories in a given path.

For instance, if we write tests alongside the source files, with a .test.js/.spec.js prefix OR if we use a test directory as __test__ /test, inceptum will try to add these files to the app.

On the other hand, inceptum app will crash, if test specific configurations are there in the test files. (configs or globals which will be loaded by a test runner)

Thus there should be an options to Ignore sub-directories/files from the addDirectory method.

Define the API standard to use

Should we use HAL? or is Swagger enough?
What autodiscovery capabilities do we want

Update `config` dependency

The config dependency can be updated to v1.28 or later to make the configuration management more flexible.

Check best redis comms method

The logging framework uses Redis. It currently uses a "list", but also has the option of using a redis channel instead.
Figure out which one is better and if it's channel do the change and test.

Define how to handle rate limits

Design the Front-end architecture

• How to do routing
• using Redux on both sides
• CSRF
• Authentication in general
• Caching

Instrument mysqlclient with logging and metrics

Depends on #6
Instrument the mysql client so that we get per pool:

• Connection pool stats: established, active, idle, max size
• Connection borrowing stats: histogram of for how long we borrow connections from the pool
• Query execution time: histogram of execution time
• Slow queries: log any query that takes longer than a configurable number of millis (allow for Env override)

Create circuit breaker capable http client

We want to talk to the outside world without dying if they do

Move Inceptum to circleCI

Now that we are building on circleci, I think we should be moving this to circleci

Load balancing of services

Registration of services in Consul

Merge transaction management into the MysqlClient

So that the MySQL client is "free standing" and can be used in or out of the IoC and without external dependencies

Test bunyan redis interaction

I've settled on using bunyan as the logging framework as it allows for easier publishing of structure data (in principle).
One thing we need to test is whether the module that publishes logs to redis that is included with bunyan is compatible with the redis plugin of logstash to extract logs and push to ELK.

If this works, the next step will be to create a wrapper around the bunyan config to make it easier to manage.

Health checks

Create swagger middleware

We want swagger to own the routing, but we don't want it to take over the platform.

Build a "simple" middleware for express that uses the Swagger file to manage:

• Request routing
• Validation of the request against the schema
• Validation of the response against the schema

Instrument with metrics:

• General count of response codes
• Counts of response codes per path
• Response time per path
• Slow requests log: Log any request that takes longer than a specified millis to fulfil (this should be overridable by path)
• Counter of errors by error class

Make @types/mocha an peer dependency

I tried to use jest for testing and it makes the build fail because of conflicting global types.

node_modules/@types/mocha/index.d.ts:66:18 - error TS2300: Duplicate identifier 'afterEach'.

66 declare function afterEach(description: string, callback: (this: Mocha.IBeforeAndAfterContext, done: MochaDone) => any): void;
                    ~~~~~~~~~

  node_modules/@types/jest/index.d.ts:31:13
    31 declare var afterEach: jest.Lifecycle;
                   ~~~~~~~~~
    'afterEach' was also declared here.

The solution would be to make @types/mocha a peer dependency, rather than adding it directly.

Correlation Ids on logs

Create HTTP client library

We want our microservices to be able to talk with other microservices that follow our standards in an easy way.
It should use whatever auto-discovery capabilities we decide upon (reading the swagger of the service, or using HAL, etc)
The client should have a high-level API that resembles a RPC and not a low-level Rest-like API

It should be built on top of a circuit-breaker capable http client

Support for factories on the context

Currently the IoC framework supports singletons (which is what you normally need). But being able to register factories would be good.
A factory is an object definition that returns a new instance every time its getInstance method is called.

Symfony: http://symfony.com/doc/current/service_container/factories.html