trajano/app-ms

Microservices based application stack using Vert.X, Resteasy and Spring-boot

Trajano

Trajano is an IT Consulting/Contracting delivery organization specialising in end-to-end enterprise architecture, systems design and development. It is founded by Archimedes Trajano [LinkedIn] [StackOverflow].

About this project

This project provides the Trajano organizational POM. In this project, the services used, overall reports, core plugins and managed plugin versions are specified along with branding details. In a way this codifies the Standards Information Base for the organization.

Coding standards

Coding standards are referenced on this plugin to define the rulesets that are used by Checkstyle, PMD, FindBugs and m2e codestyle Maven plugins. The activation of these plugins are done on individual projects and not on this level.

Versioning

The organizational POM version numbering scheme is a single numeric number. This is similar to the way Apache does their POM versioning. The only exception is when there is a build issue, but the core plugin versions and configurations have not changed and it had already been released to central.

app-ms's Issues

2.0.0 High level ideas

The concept of 2.0.0 is to architect things so that Vert.X event bus and Hazelcast becomes a core part of the architecture.

In addition, we'd be simplifying the security since we'd be relying on the secure VLAN capabilities of a Dockerized environment.

In addition gateway will become a support library rather than an implementation.

Fundamentally the application side API should be retained as much as possible.

In addition, Spring will be systematically removed from the code base. It will still be used for the actual JAX-RS code but will be removed from the non-JAX-RS infrastructure code as much as possible.

Most likely the JWKS code would be removed for efficiency, but we'll see. It could just have a permanently stored AES key that is generated once in startup rather than rotating over and over again.

Add X-Request-ID from the gateway

Generate a random token and pass it to every request down as part of the header.

In ms common a new prefilter is needed to call slf4j MDC put to store the request ID and the default logback message should add it in as well.

Data Resource API

Something that provides CRUDL and Search

Create resources-ms

Ms-Resource is a configurable microservice that will provide i18n type of data

It uses Excel as the source data and will do the conversion.

The request looks like

/{context}/{locale}

Context represents a set of excel sheets.

Configuration looks like

Default locale : en
Contexts:

• name: context name
  . Sheets:
  • file: excel file
    . Sheet: sheet name
• name: another context
  . File: excel file
  . Sheet sheet name

The second is a short cut for more common usages where there is only one file and sheet

The data looks like

Key locale locale locale

The key is dot notation and each dot will be represented as a json path

The default locale will be used as a fall back when the resource was not found for a given locale.

If a locale does not have a value then the output of the value is (locale requested) + default locale value

If the default locale value is also missing then the output is

(locale requested) key name

Add caching support to Gateway

If an ETag and Expires was provided by the backend service, store it in the gateway's local cache, the cache does not need to be in a Hazelcast as an in-memory cache will suffice.

Refactor to service provider pattern

Rather than have ms-common include ms-engine and a whole slew of dependencies. Separate it out to two different projects. One for just API ang the other for the actual service implementation.

By doing so the application name space does not get too polluted with implementation classes.

The API should contain

• spring-boot-autoconfiguration for the spring and spring-boot annotations.
• Jackson annotations
• Microservice class to bootstrap the whole process
• JAX-RS API
• swagger-annotations
• annotations provided by the framework if any

Some parts that may be needed but I am debating on

• resteasy multipart?

Gateway should only accept requests with a Bearer token or a client ID/Secret and use that for CORS

The gateway needs to be able to use the client ID+secret or bearer token to build the CORS data. For example quasarAppId may have a CORS specific for ms.trajano.net.

Don't use the Origin header or Referrer header matches the expected value since those will not be set correctly when using OpenID. Instead use a HttpOnly cookie with a verifiable token to represent the fact that we are coming from the same source.

Remove `file://` workaround

Use the proxy to do it, it seems to be an issue with some mobile frameworks and not normal browser behaviour.

Cookies should not be passed down by the gateway

Cookie and Set-Cookie headers should be filtered out by the gateway. https://github.com/spring-cloud/spring-cloud-netflix/blob/master/docs/src/main/asciidoc/spring-cloud-netflix.adoc#cookies-and-sensitive-headers but only to the final handlers (not the authorization)

Convert authorization microservice to use Hazelcast messaging

Rather than HTTP calls everywhere just use Hazelcast or some internal messaging system.

Duplicate initialization of application context

Create Sample-OIDC

Basically entails making the core part of OIDC be kept in ms-oidc

The sample will utilize JSON configuration files to provide the issuer configuration. The usage of JSON would be part of the Sample.

Another part of the sample would be the ClientValidator implementation The validator will be relocate to core since it is now going to be used by both authn and authz

the ClientValidator will check for the grant type: openid

TBD: ClientValidator will remove the `getJwksUri to a separate interface since it is not part of authentication.

It's main class would have

@SpringBootApplication(
    scanBasePackageClasses = {
        SampleOidcMS.class,
        OpenIdConnect.class
    })
public class SampleOidcMS extends Application {

    public static void main(final String[] args) {

        Microservice.run(SampleOidcMS.class, args);
    }

}

mvn install failing

mvn clean install is failing. Although npm install runs successfully in folder

"/c/git projects/app-ms/app-ms/quasar-ms"

.

All other modules are compiling successfully.

[DEBUG] -- end configuration --
[INFO] Running 'npm install' in C:\git projects\app-ms\app-ms\quasar-ms
[DEBUG] Executing command line [C:\git projects\app-ms\app-ms\quasar-ms\node\node.exe, C:\git projects\app-ms\app-ms\quasar-ms\node\node_modules\npm\bin\npm-cli.js, install]
[ERROR] module.js:538
[ERROR]     throw err;
[ERROR]     ^
[ERROR]
[ERROR] Error: Cannot find module 'C:\git projects\app-ms\app-ms\quasar-ms\node\node_modules\npm\bin\npm-cli.js'
[ERROR]     at Function.Module._resolveFilename (module.js:536:15)
[ERROR]     at Function.Module._load (module.js:466:25)
[ERROR]     at Function.Module.runMain (module.js:676:10)
[ERROR]     at startup (bootstrap_node.js:187:16)
[ERROR]     at bootstrap_node.js:608:3
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Microservice Application ........................... SUCCESS [  5.650 s]
[INFO] Microservice Common ................................ SUCCESS [  9.739 s]
[INFO] ms-engine-spring-jaxrs ............................. SUCCESS [  2.796 s]
[INFO] Microservice Engine ................................ SUCCESS [  7.808 s]
[INFO] Common Microservice Implementation ................. SUCCESS [  5.325 s]
[INFO] Microservices Base ................................. SUCCESS [  2.074 s]
[INFO] ms-common-auth ..................................... SUCCESS [  3.195 s]
[INFO] ms-engine-manifest ................................. SUCCESS [  0.421 s]
[INFO] ms-engine-swagger .................................. SUCCESS [  0.350 s]
[INFO] API Gateway ........................................ SUCCESS [  3.677 s]
[INFO] OpenID Connect ..................................... SUCCESS [  2.357 s]
[INFO] ms-resource ........................................ SUCCESS [  0.350 s]
[INFO] Swagger Microservice ............................... SUCCESS [  1.421 s]
[INFO] Microservice Web UI ................................ FAILURE [  1.048 s]
[INFO] Sample Authentication Microservice ................. SKIPPED
[INFO] Sample Authorization Microservice .................. SKIPPED
[INFO] Sample Microservice ................................ SKIPPED
[INFO] Sample OIDC Microservice ........................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 48.422 s
[INFO] Finished at: 2018-02-12T23:51:29+05:30
[INFO] Final Memory: 72M/569M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.6:npm (npm install) on project quasar-ms: Failed to run task: 'npm install' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.6:npm (npm install) on project quasar-ms: Failed to run task
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:497)

Add a listener for Hazelcast events when logging is debug

See http://docs.hazelcast.org/docs/latest-development/manual/html/Distributed_Events/Event_Listener_for_Members/Listening_for_Map_Events.html

the concept is to detect how the token are progressing in the hazelcast cache.

Handlers.java is getting too large

The file should be broken down into smaller files.

Open API 3.0

Open API 3.0 adds a few things like Bearer token. However, there is no annotation support for it as of yet so this will be postponed until the supporting technologies come up.

Separate the authentication filter logic from ms-gateway

This will make the gateway a bit more general. However, it still needs to perform just as well.

URL not reachable

URL is not reachable. Moreover the site https://site.trajano.net/ is not available as well.

Add health check to docker

Add a new well known router end point. ping that will be called by curl periodically to check if the server is up using health check from docker.

clientinfo-ms

This microservice will manage the client information rather than having it replicated a few times

https://github.com/trajano/app-ms/blob/master/sample-oidc/clients.json
https://github.com/trajano/app-ms/blob/master/sample-authz/clients.json

This will also provide the CORS implementation on a per client ID basis.

JWT Only Gateway

Instead of going through the access token approach to get the JWT. Pass the entirety of the JWT to the client and let it send it back, the gateway can perform the initial JWT validation.

The JWT can be encrypted as well.

This goes on the theory that the lookup for the JWT token associated with the access token inside a cache is slower than passing the token and checking the validity of the token. The token itself can become quite large and will have to get sent on every request.

Add metrics

Deprecate JwtNotRequired

Instead of having a custom annotation for specifying jwt is not required.

Check for the presence of a Swagger annotation that would represent the security constraint.

Or if there is a @context SecurityContext

The value specified by the swagger annotation will take priority

Create a separate set of docker-compose files for "production"

Right now the docker-compose files are written in such a way that it is close to production. However, the production one should be the only ones with gw.trajano.net and ms.trajano.net

see

• https://docs.docker.com/compose/production/#modify-your-compose-file-for-production
• https://docs.docker.com/compose/extends/#multiple-compose-files

File Download Endpoint

This is for non-public downloads. Do it similar to how Amazon and Google does it. An endpoint resource will be used to request the creation of the download file the URI will have a "key" associated with it.

A external endpoint would be /download/{accesstoken}/filename.ext

The data will be retrieved from a Cache which will expire after a given time

There are two internal endpoints

/store which will store the data into the cache and provide an access token. It passes the following meta data in along with formdata which contains the file contenxt

{ expiresIn. meta }

There can be multiple files stored for a given request. Invoking with the same meta will update the data

The content disposition portion will say the file name and content type.

The meta is used to provide information on how the record was built

/check-meta which will check if the meta is present the store. This is used to prevent double processing. It returns the following:

{ accessToken, expires, files [ { filename, contentType } ] }