instana/otel-shop

OpenTelemetry Demo App

Instana OpenTelemetry Demo

This OpenTelemetry demo consists of the following services:

Application:

• cart: (Auto-Instrumented) NodeJS application talking to MongoDB and Redis
• catalogue: (Auto-Instrumented) NodeJS application talking to MongoDB
• dispatch: A golang application consuming from RabbitMQ
• front: An nginx frontend proxy server
• payment: (Auto-Instrumented) Python application talking to RabbitMQ, receiving and calling via HTTP
• ratings: PHP/Symfony Application reading and writing to MySQL
• shipping: (Auto-Instrumented) Java Application
• user: (Auto-Instrumented) NodeJS application reading and writing to MongoDB
• web: Apache web server for static assets

Infrastructure:

• mongodb
• redis
• rabbitmq
• mysql

(Kinda) Optional:

• agent: Instana Agent configured to receive OpenTelemetry via OTLP
• load-gen: Load Generator for the whole thing

Usage

The Otel-Shop can be deployed to Kubernetes or plain docker-compose.

docker-compose

1. Copy .env.template to .env and set the values straight in there.
2. docker-compose -f docker-compose.yaml -f docker-compose-agent.yaml -f docker-compose-load.yaml pull
3. docker-compose -f docker-compose.yaml -f docker-compose-agent.yaml -f docker-compose-load.yaml up

Kubernetes

Deploy the Instana Agent via our helm chart or operator and enable the OpenTelemetry collector endpoint:

Helm Chart:

helm install --create-namespace instana-agent --namespace instana-agent \
    --repo https://agents.instana.io/helm \
    --set agent.key='<your agent key>' \
    --set agent.endpointHost='<your host agent endpoint>' \
    --set cluster.name='<your-cluster-name>' \
    --set zone.name='<your-zone-name>' \
    --set opentelemetry.enabled=true \ # this is the setting
    instana-agent

Operator:

Note: You need cert-manager in your cluster. You can install it as follows, if you dont have it:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml

Install the Instana Agent operator

# Install the operator
kubectl apply -f https://github.com/instana/instana-agent-operator/releases/latest/download/instana-agent-operator.yaml

Create a agent.yaml file with the following content and apply it: kubectl apply -f agent.yaml

# Deploy Instana Agents with OTel
apiVersion: instana.io/v1
kind: InstanaAgent
metadata:
  name: instana-agent
  namespace: instana-agent
spec:
  zone:
    name: '<your-zone-name>'
  cluster:
    name: '<your-cluster-name>'
  # this setting enables the OpenTelemetry collector endpoint
  opentelemetry:
    enabled: true
  agent:
    key: <your agent key>
    endpointHost: <your host agent endpoint>
    endpointPort: "443"
    env: {}
    configuration_yaml: |
      # Disable Python AutoInstrumentation through Instana
      com.instana.plugin.python:
        autotrace:
          enabled: false

Install OTel-Shop in your cluster

helm install otel-shop otel-shop \
  --create-namespace \
  --namespace otel-shop \
  --repo https://instana.github.io/otel-shop/

Upgrading OTel-Shop to the latest version

helm upgrade otel-shop otel-shop \
  --namespace otel-shop \
  --repo https://instana.github.io/otel-shop/

Sending OTLP somewhere else

This chart can as well as adressing Instana, send the data to some other destination, via OTLP.

The following settings can be used:

opentelemetry.exporter.otlp.endpoint - Set the OTLP endpoint to send the data to (defaulting to Instana Agent) opentelemetry.exporter.otlp.headers - Set a list of comma-separated headers for the OTLP exporter (defaulting to blanks)

License

Apache 2.0

otel-shop's Issues

TypeError: requestCount.bind is not a function

running on my openshift cluter and having some trouble with a few of the robot-shop microservices with errors like

$ oc logs -f -n otel-shop otel-shop-catalogue-59cd5f9f7f-lwncq 
{"level":30,"time":1653170515740,"pid":1,"hostname":"otel-shop-catalogue-59cd5f9f7f-lwncq","msg":"Started on port"}
{"level":30,"time":1653170515947,"pid":1,"hostname":"otel-shop-catalogue-59cd5f9f7f-lwncq","msg":"MongoDB connected"}
{"level":30,"time":1653170518238,"pid":1,"hostname":"otel-shop-catalogue-59cd5f9f7f-lwncq","req":{"id":1,"method":"GET","url":"/health-check","query":{},"params":{},"headers":{"host":"10.129.6.151:8080","user-agent":"kube-probe/1.21","accept":"*/*","connection":"close"},"remoteAddress":"::ffff:10.129.6.1","remotePort":47904},"trace_id":"a2d29bb106b9e5ab6dd69b938f7a23c8","span_id":"0bacb1ac719cfe17","trace_flags":"01","res":{"statusCode":500,"headers":{"x-powered-by":"Express","content-security-policy":"default-src 'none'","x-content-type-options":"nosniff","content-type":"text/html; charset=utf-8","content-length":1194}},"err":{"type":"Error","message":"failed with status code 500","stack":"Error: failed with status code 500\n    at ServerResponse.onResFinished (/opt/server/node_modules/pino-http/logger.js:107:40)\n    at /opt/server/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:49:55\n    at AsyncLocalStorage.run (node:async_hooks:327:14)\n    at AsyncLocalStorageContextManager.with (/opt/server/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)\n    at ServerResponse.contextWrapper (/opt/server/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:49:32)\n    at ServerResponse.emit (node:events:539:35)\n    at onFinish (node:_http_outgoing:830:10)\n    at callback (node:internal/streams/writable:553:21)\n    at afterWrite (node:internal/streams/writable:498:5)\n    at afterWriteTick (node:internal/streams/writable:485:10)"},"responseTime":25,"msg":"request errored"}
TypeError: requestCount.bind is not a function
    at /opt/server/monitoring.js:25:41
    at /opt/server/node_modules/@opentelemetry/instrumentation-express/build/src/instrumentation.js:207:41
    at Layer.handle [as handle_request] (/opt/server/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/opt/server/node_modules/express/lib/router/index.js:328:13)
    at /opt/server/node_modules/express/lib/router/index.js:286:9
    at Function.process_params (/opt/server/node_modules/express/lib/router/index.js:346:12)
    at next (/opt/server/node_modules/express/lib/router/index.js:280:10)
    at /opt/server/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:49:55
    at AsyncLocalStorage.run (node:async_hooks:327:14)
    at AsyncLocalStorageContextManager.with (/opt/server/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
...

and of course the pods are not becoming ready

$ oc get pods -n otel-shop
NAME                                   READY   STATUS    RESTARTS   AGE
otel-shop-cart-657cc7d647-qwqbr        0/1     Running   0          61m
otel-shop-catalogue-59cd5f9f7f-lwncq   0/1     Running   0          61m
otel-shop-dispatch-6f789d4f75-dpfxw    1/1     Running   0          34h
otel-shop-front-5fcd4d7ddd-8bwnw       1/1     Running   0          81m
otel-shop-load-5d6c6cf968-pmzfh        1/1     Running   0          34h
otel-shop-mongodb-5d7b969b4c-7j9l6     1/1     Running   0          82m
otel-shop-payment-84787cb46f-qnskd     1/1     Running   0          31h
otel-shop-rabbitmq-5c79c47764-q4cgn    1/1     Running   0          29h
otel-shop-ratings-6686bfbd79-4pcz6     0/1     Running   0          61m
otel-shop-redis-0                      1/1     Running   0          29h
otel-shop-shipping-67b45478f8-89jjg    0/1     Running   3          61m
otel-shop-user-6df576467c-jmldb        0/1     Running   1          61m
otel-shop-web-85c784cfc7-j5k72         1/1     Running   0          81m

is there anything that I should configured/deployed differently?

WARN io.opentelemetry.sdk.metrics.internal.state.MetricStorageRegistry - Found duplicate metric definition: jvm.threads.states

due to a known issue in opentelemetry 1.14.0 with springboot java applications, we are hitting it in the shipping service

$ oc logs -f -n otel-shop otel-shop-shipping-5b99f8459c-xf44t 
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.14.0

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.7.0)

2022-05-24 11:47:42.475  INFO 1 --- [           main] c.i.r.s.ShippingServiceApplication       : Starting ShippingServiceApplication v0.1.149 using Java 18.0.1.1 on otel-shop-shipping-5b99f8459c-xf44t with PID 1 (/opt/shipping/shipping.jar started by root in /opt/shipping)
2022-05-24 11:47:42.493  INFO 1 --- [           main] c.i.r.s.ShippingServiceApplication       : No active profile set, falling back to 1 default profile: "default"
2022-05-24 11:47:50.680  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2022-05-24 11:47:50.904  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 209 ms. Found 2 JPA repository interfaces.
2022-05-24 11:47:53.076  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'shippingServiceApplication' of type [com.instana.robotshop.shipping.ShippingServiceApplication$$EnhancerBySpringCGLIB$$41b10b5a] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-05-24 11:47:55.596  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2022-05-24 11:47:55.886  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2022-05-24 11:47:55.886  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.63]
2022-05-24 11:47:56.389  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2022-05-24 11:47:56.403  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 13313 ms
2022-05-24 11:47:57.482  INFO 1 --- [           main] c.instana.robotshop.shipping.JpaConfig   : jdbc url jdbc:mysql://otel-shop-mysql/cities?useSSL=false&autoReconnect=true&allowPublicKeyRetrieval=True
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.

[main] WARN io.opentelemetry.sdk.metrics.internal.state.MetricStorageRegistry - Found duplicate metric definition: jvm.threads.states
	at unknown source
		To enable better debugging, run your JVM with -Dotel.experimental.sdk.metrics.debug=true
Causes
- Description [The current number of threads having RUNNABLE state] does not match [The current number of threads having NEW state]
- InstrumentDescription [The current number of threads having RUNNABLE state] does not match [The current number of threads having NEW state]
Original instrument registered with same name but is incompatible.
	at unknown source
		To enable better debugging, run your JVM with -Dotel.experimental.sdk.metrics.debug=true

this is caused by open-telemetry/opentelemetry-java#4381
which is fixed in 1.14.1 (not yet released) by open-telemetry/opentelemetry-java#4457

Issues with frontend

I use helm to spin up the version otel-shop-0.1.174 and otel-shop-0.1.175, but I'm getting same error.

 [helio@fedora ~]$ kubectl logs otel-shop-front-85758df788-s6tmb -n otel-shop
nginx: [emerg] module "/usr/share/nginx/modules/otel_ngx_module.so" version 1021006 instead of 1023001 in /etc/nginx/nginx.conf:1

php.WARNING: Uncaught User Notice: Class "OpenTelemetry\SDK\SystemClock " is deprecated and will be removed in a future release

This is just a function of the alpha version of the opentelemetry php library, but unfortunately it throws exceptions for depracation warnings, which seems to break the ratings service

$ oc logs -f -n otel-shop otel-shop-ratings-566cb8d56b-d2sfg
[2022-05-24T11:26:56.923220+00:00] php.WARNING: Uncaught User Notice: Class "OpenTelemetry\SDK\SystemClock " is deprecated and will be removed in a future release. Please, use "OpenTelemetry\SDK\Common\Time\SystemClock" instead. {"exception":"[object] (ErrorException(code: 0): User Notice: Class \"OpenTelemetry\\SDK\\SystemClock \" is deprecated and will be removed in a future release. Please, use \"OpenTelemetry\\SDK\\Common\\Time\\SystemClock\" instead. at /var/www/html/vendor/open-telemetry/opentelemetry/src/SDK/Common/Dev/Compatibility/Util.php:51)\n[previous exception] [object] (ErrorException(code: 0): User Notice: Class \"OpenTelemetry\\SDK\\SystemClock \" is deprecated and will be removed in a future release. Please, use \"OpenTelemetry\\SDK\\Common\\Time\\SystemClock\" instead. at /var/www/html/vendor/open-telemetry/opentelemetry/src/SDK/Common/Dev/Compatibility/Util.php:51)"} []
$ oc get pods -n otel-shop otel-shop-ratings-566cb8d56b-d2sfg
NAME                                 READY   STATUS    RESTARTS   AGE
otel-shop-ratings-566cb8d56b-d2sfg   0/1     Running   0          46m

open-telemetry/opentelemetry-php@99fba0d