ibmstreams / endpoint-monitor Goto Github PK
View Code? Open in Web Editor NEWNginx reverse proxy sample application to Streams REST operators.
License: Apache License 2.0
Nginx reverse proxy sample application to Streams REST operators.
License: Apache License 2.0
as per guidelines the "main" branch shall no longer have the name "master"
An active signature check in actual implementation has side effect of request performance.
js_set $signatureSecret readSecret;
This set's the readSecret function as handler being called for EACH!!! request accessing the nginx variable $signatureSecret
.
It doesn't set a value already in this line!
The handler is called when the nginx variable is used in an nginx 'location' AND!!! also in the JS code via r.variables.signatureSecret. This is by design of this nginx extension, accessing a JS_SET nginx variable during processing of request will call the handler with the request as argument and returns e.g. a string which is used in 'location' config or JS code. It's a somehow strange implementation as nothing points to this fact as var secret_key = r.variables.signatureSecret;
looks as normal JS assignment, but it isn't.
I suppose that the intention was a one-time-only call to the readSecret function to read the secret from only once, isn't it? So another way of reading the signature secret only once should be used (may be using SET and assigning the secret value as string).
On the current CP4D Blitz test machine the application configuration for endpoint-monitor is not accessible by a non-admin user, so a call to endpoint.inject(...) fails during initialization.
For injection operators UDP can be supported by having nginx route the requests across the multiple injection points.
From https://blog.knoldus.com/easiest-way-to-setup-ssl-on-nginx-using-pfx-files/ (which is server side):
Need cert files from pfx file:
openssl pkcs12 -in ./YOUR-PFX-FILE.pfx -clcerts -nokeys -out client.crt
openssl pkcs12 -in ./YOUR-PFX-FILE.pfx -nocerts -nodes -out client.rsa
Then the nginx conf looks like (last two lines are the key):
location /streams/job/0/ {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto https ;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $remote_addr;
proxy_pass https://172.17.0.12:37336/;
proxy_ssl_certificate /opt/streams_job_configs/certs/client.crt ;
proxy_ssl_certificate_key /opt/streams_job_configs/certs/client.rsa ;
}
While a common case is for all rest operators to be fused together, this should not be a requirement, so we could have:
sensors
on ip 172.0.0.2 and port 32255state
on ip 172.0.0.4 and port 38321Since the lead in path is the same, we need entries that are logically
streams/job/123/sensors --> 172.0.0.2:32255/sensors
streams/job/123/state --> 172.0.0.4: 38321/state
May need changes so that the context path is visible through metrics, though maybe the existing REST api to get the info is enough?
Note this is not the case where a single logical rest operator is replicated using UDP, e.g. multiple injection operators for performance that feed into the same logical stream.
Define pattern for a path based upon job name.
We should implement a means to disable the signature check. Even in Segment Webhook, the signature is optional.
In endpoint_monitor.py
for each operator that is a rest operator we get its IP address through its PE and Resource.
https://github.com/IBMStreams/endpoint-monitor/blob/master/endpoint_monitor.py#L13
We need to maintain this information to support more efficient update job checking.
Ie. we can see if an existing job's ip addresses/ports (may) have changed by just checking PE launch counts, if the same as the last time then no change occurred.
We need to ensure that the startup period where the operators have not yet set a port in their metrics (port=0) and the port becoming visible (port !=0) are caught since in this case the PE's launch count will be the same.
Allow endpoint-monitor to use a .htpasswd
file, most likely set through a secret.
Verified manually that it works
We were playing around on the Blitz machine (ngb8) and wanted to try if with the merged signature patch we get to exchanging messages yet. So we triggered a new build for "streams-endpoint-monitor".
Unfortunately, we didn't get far, as we ran into a new error:
Invalid location modifier "~^" in nginx job configuration
Job submission and cancelation is supported at the moment.
Need to support update handling the case when PE restarts and the operator has a new ip and/or port.
Have the template create the linkage between the secret and the monitor container within the pod.
I used the setupProxy.sh script to install the streams enpoint-monitor on CPD 3.5.
I only provided the mandatory parameters.
The script run was successful. When checking the POD logs in the OC console I saw authentication errors on endpoint-monitor POD.
I checked this and saw that the DeploymentConfig for the endpoint-monitor is handling the streams-user secret differently than other secrets.
envFrom:
- secretRef:
name: "${STREAMS_USER_SECRET}"
I don't know the reason doing so. But Adding the user secret in the volumes
section and the streams-endpoint-monitor
containers volumeMounts
section solved the problem
From client through nginx to Jetty operator.
The signature check should compute the signature over request body as is.
The current implementation computes the the signature over a stringified JSON object, which is not identically to the request body.
E.g. the request body is:
{"item1":"xxxx","item2":"yyyy"}
the stringified request body is:
"{"item1":"xxxx","item2":"yyyy"}"
But the sender computes the signature over the plain body.
In any production environment HTTPS/SSL will be required and since it's easy to do (automated) makes sense for demo environments as well.
I'm still not sure if I understood the certificate provisioning for the communication between proxy and streams.inetserver correct.
In /var/run/secrets/streams-endpoint-monitor/streams-certs/streams-certs/client.pfx
Is the client key + certificate.
But where is the truststore for the server certitficate? Or is the client configured to trust all certificates?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.