ibmstreams/endpoint-monitor

Nginx reverse proxy sample application to Streams REST operators.

endpoint-monitor's Issues

change name of the default branch to develop

as per guidelines the "main" branch shall no longer have the name "master"

Images.

Support signature verification at the nginx entry

Style of signing: https://forum.level1techs.com/t/signing-http-requests-to-make-sure-that-the-client-is-not-an-attacker-or-bot/111821

Sample use for Segment. https://segment.com/docs/destinations/webhooks/#authentication

Signature check has side effect to performance

An active signature check in actual implementation has side effect of request performance.

js_set $signatureSecret readSecret;

This set's the readSecret function as handler being called for EACH!!! request accessing the nginx variable $signatureSecret.

It doesn't set a value already in this line!

The handler is called when the nginx variable is used in an nginx 'location' AND!!! also in the JS code via r.variables.signatureSecret. This is by design of this nginx extension, accessing a JS_SET nginx variable during processing of request will call the handler with the request as argument and returns e.g. a string which is used in 'location' config or JS code. It's a somehow strange implementation as nothing points to this fact as var secret_key = r.variables.signatureSecret; looks as normal JS assignment, but it isn't.

I suppose that the intention was a one-time-only call to the readSecret function to read the secret from only once, isn't it? So another way of reading the signature secret only once should be used (may be using SET and assigning the secret value as string).

Application configuration not accessible by non-admin user

On the current CP4D Blitz test machine the application configuration for endpoint-monitor is not accessible by a non-admin user, so a call to endpoint.inject(...) fails during initialization.

Upgrade to 1.14 nginx

https://github.com/sclorg/nginx-container/tree/master/1.14

Support UDP for injection operators.

For injection operators UDP can be supported by having nginx route the requests across the multiple injection points.

Support SSL with client certs between nginx and Jetty operartors.

From https://blog.knoldus.com/easiest-way-to-setup-ssl-on-nginx-using-pfx-files/ (which is server side):

Need cert files from pfx file:

openssl pkcs12 -in ./YOUR-PFX-FILE.pfx -clcerts -nokeys -out client.crt
openssl pkcs12 -in ./YOUR-PFX-FILE.pfx -nocerts -nodes -out client.rsa

Then the nginx conf looks like (last two lines are the key):

location /streams/job/0/ {
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-Proto https ;
  proxy_set_header  X-Forwarded-For $remote_addr;
  proxy_set_header  X-Forwarded-Host $remote_addr;
  proxy_pass https://172.17.0.12:37336/;

  proxy_ssl_certificate /opt/streams_job_configs/certs/client.crt ;
  proxy_ssl_certificate_key /opt/streams_job_configs/certs/client.rsa ;
}

Support multiple endpoints within a single Streams application.

While a common case is for all rest operators to be fused together, this should not be a requirement, so we could have:

• injection with context path sensors on ip 172.0.0.2 and port 32255
• tuple view with context path state on ip 172.0.0.4 and port 38321

Since the lead in path is the same, we need entries that are logically

streams/job/123/sensors --> 172.0.0.2:32255/sensors
streams/job/123/state --> 172.0.0.4: 38321/state

May need changes so that the context path is visible through metrics, though maybe the existing REST api to get the info is enough?

Note this is not the case where a single logical rest operator is replicated using UDP, e.g. multiple injection operators for performance that feed into the same logical stream.

Support fixed path based upon job name.

Define pattern for a path based upon job name.

Signature check should be disabled if signature secret is not available

We should implement a means to disable the signature check. Even in Segment Webhook, the signature is optional.

The signature check must be clearely documented

Maintain information about PEs in a job.

In endpoint_monitor.py for each operator that is a rest operator we get its IP address through its PE and Resource.

https://github.com/IBMStreams/endpoint-monitor/blob/master/endpoint_monitor.py#L13

We need to maintain this information to support more efficient update job checking.

Ie. we can see if an existing job's ip addresses/ports (may) have changed by just checking PE launch counts, if the same as the last time then no change occurred.

We need to ensure that the startup period where the operators have not yet set a port in their metrics (port=0) and the port becoming visible (port !=0) are caught since in this case the PE's launch count will be the same.

Support basic authentication at the Nginx server.

Allow endpoint-monitor to use a .htpasswd file, most likely set through a secret.

Verified manually that it works

Invalid location modifier "~^" in nginx job configuration

We were playing around on the Blitz machine (ngb8) and wanted to try if with the merged signature patch we get to exchanging messages yet. So we triggered a new build for "streams-endpoint-monitor".

Unfortunately, we didn't get far, as we ran into a new error:
Invalid location modifier "~^" in nginx job configuration

Support update of operator configuration due to PE restart.

Job submission and cancelation is supported at the moment.

Need to support update handling the case when PE restarts and the operator has a new ip and/or port.

Pass Streams user secret name to template.

Have the template create the linkage between the secret and the monitor container within the pod.

setupProxy.sh creates an streams-endpoint-monitor POD being not working

I used the setupProxy.sh script to install the streams enpoint-monitor on CPD 3.5.
I only provided the mandatory parameters.
The script run was successful. When checking the POD logs in the OC console I saw authentication errors on endpoint-monitor POD.

I checked this and saw that the DeploymentConfig for the endpoint-monitor is handling the streams-user secret differently than other secrets.

envFrom:
- secretRef:
name: "${STREAMS_USER_SECRET}"

I don't know the reason doing so. But Adding the user secret in the volumes section and the streams-endpoint-monitor containers volumeMounts section solved the problem

Support SSL client certificate pass through.

From client through nginx to Jetty operator.

Signature check is incorrect

The signature check should compute the signature over request body as is.
The current implementation computes the the signature over a stringified JSON object, which is not identically to the request body.
E.g. the request body is:
{"item1":"xxxx","item2":"yyyy"}

the stringified request body is:
"{"item1":"xxxx","item2":"yyyy"}"

But the sender computes the signature over the plain body.

Make nginx server use SSL always.

In any production environment HTTPS/SSL will be required and since it's easy to do (automated) makes sense for demo environments as well.

SSL certificate and key provisioning

I'm still not sure if I understood the certificate provisioning for the communication between proxy and streams.inetserver correct.

In /var/run/secrets/streams-endpoint-monitor/streams-certs/streams-certs/client.pfx
Is the client key + certificate.

But where is the truststore for the server certitficate? Or is the client configured to trust all certificates?