instana/robot-shop

Sample microservices application for playing with

Sample Microservice Application

Stan's Robot Shop is a sample microservice application you can use as a sandbox to test and learn containerised application orchestration and monitoring techniques. It is not intended to be a comprehensive reference example of how to write a microservices application, although you will better understand some of those concepts by playing with Stan's Robot Shop. To be clear, the error handling is patchy and there is not any security built into the application.

You can get more detailed information from my blog post about this sample microservice application.

This sample microservice application has been built using these technologies:

• NodeJS (Express)
• Java (Spring Boot)
• Python (Flask)
• Golang
• PHP (Apache)
• MongoDB
• Redis
• MySQL (Maxmind data)
• RabbitMQ
• Nginx
• AngularJS (1.x)

The various services in the sample application already include all required Instana components installed and configured. The Instana components provide automatic instrumentation for complete end to end tracing, as well as complete visibility into time series metrics for all the technologies.

To see the application performance results in the Instana dashboard, you will first need an Instana account. Don't worry a trial account is free.

Build from Source

To optionally build from source (you will need a newish version of Docker to do this) use Docker Compose. Optionally edit the .env file to specify an alternative image registry and version tag; see the official documentation for more information.

To download the tracing module for Nginx, it needs a valid Instana agent key. Set this in the environment before starting the build.

$ export INSTANA_AGENT_KEY="<your agent key>"

Now build all the images.

$ docker-compose build

If you modified the .env file and changed the image registry, you need to push the images to that registry

$ docker-compose push

Run Locally

You can run it locally for testing.

If you did not build from source, don't worry all the images are on Docker Hub. Just pull down those images first using:

$ docker-compose pull

Fire up Stan's Robot Shop with:

$ docker-compose up

If you want to fire up some load as well:

$ docker-compose -f docker-compose.yaml -f docker-compose-load.yaml up

If you are running it locally on a Linux host you can also run the Instana agent locally, unfortunately the agent is currently not supported on Mac.

There is also only limited support on ARM architectures at the moment.

Marathon / DCOS

The manifests for robotshop are in the DCOS/ directory. These manifests were built using a fresh install of DCOS 1.11.0. They should work on a vanilla HA or single instance install.

You may install Instana via the DCOS package manager, instructions are here: https://github.com/dcos/examples/tree/master/instana-agent/1.9

Kubernetes

You can run Kubernetes locally using minikube or on one of the many cloud providers.

The Docker container images are all available on Docker Hub.

Install Stan's Robot Shop to your Kubernetes cluster using the Helm chart.

To deploy the Instana agent to Kubernetes, just use the helm chart.

Accessing the Store

If you are running the store locally via docker-compose up then, the store front is available on localhost port 8080 http://localhost:8080

If you are running the store on Kubernetes via minikube then, find the IP address of Minikube and the Node Port of the web service.

$ minikube ip
$ kubectl get svc web

If you are using a cloud Kubernetes / Openshift / Mesosphere then it will be available on the load balancer of that system.

Load Generation

A separate load generation utility is provided in the load-gen directory. This is not automatically run when the application is started. The load generator is built with Python and Locust. The build.sh script builds the Docker image, optionally taking push as the first argument to also push the image to the registry. The registry and tag settings are loaded from the .env file in the parent directory. The script load-gen.sh runs the image, it takes a number of command line arguments. You could run the container inside an orchestration system (K8s) as well if you want to, an example descriptor is provided in K8s directory. For End-user Monitoring ,load is not automatically generated but by navigating through the Robotshop from the browser .For more details see the README in the load-gen directory.

Website Monitoring / End-User Monitoring

Docker Compose

To enable Website Monioring / End-User Monitoring (EUM) see the official documentation for how to create a configuration. There is no need to inject the JavaScript fragment into the page, this will be handled automatically. Just make a note of the unique key and set the environment variable INSTANA_EUM_KEY and INSTANA_EUM_REPORTING_URL for the web image within docker-compose.yaml.

Kubernetes

The Helm chart for installing Stan's Robot Shop supports setting the key and endpoint url required for website monitoring, see the README.

Prometheus

The cart and payment services both have Prometheus metric endpoints. These are accessible on /metrics. The cart service provides:

• Counter of the number of items added to the cart

The payment services provides:

• Counter of the number of items perchased
• Histogram of the total number of items in each cart
• Histogram of the total value of each cart

To test the metrics use:

$ curl http://<host>:8080/api/cart/metrics
$ curl http://<host>:8080/api/payment/metrics

robot-shop's Issues

Shipping Deployment - Probe Unhealthy

Warning Unhealthy 3s (x22 over 108s) kubelet Readiness probe failed: Get "http://10.42.136.89:8080/health": dial tcp 10.42.136.89:8080: connect: connection refused

Failing to start shipping pods due to health probe failing

Redis pod keeps status "pending"

Hello,
I want to use the robot-shop as an autoscaling testing application for my thesis and have some trouble deploying it in Kubernetes.
I use Kubernetes from Docker Desktop and have linkerd (via script) and Prometheus (via helm) installed.
I deployed the robot-shop as described via terminal with the command: helm install robot-shop --namespace robot-shop .

But when I want to check the pods status via kubectl get pods -n robot-shop i get the following response:

NAME                         READY   STATUS    RESTARTS   AGE
cart-65bddd8879-rtpjg        2/2     Running   0          5m50s
catalogue-55f4558d8d-dmfw8   2/2     Running   0          5m50s
dispatch-7c97898876-jfd5r    2/2     Running   0          5m50s
mongodb-7fcc49887c-w549g     2/2     Running   0          5m49s
mysql-6fbdd859fd-j7w57       2/2     Running   0          5m50s
payment-6d69b97956-cjg7v     2/2     Running   0          5m49s
rabbitmq-86f8d8c475-hrq9q    2/2     Running   0          5m49s
ratings-57498966c6-vgqrv     2/2     Running   0          5m50s
redis-0                      0/2     Pending   0          5m50s
shipping-8454757cf6-gk5tv    2/2     Running   0          5m50s
user-7c4c4f96d4-fpbct        2/2     Running   0          5m49s
web-6589588994-ljxjk         2/2     Running   0          5m50s

Which tells me that the Redis container is not running.

When inspecting the pod via kubectl describe pods redis-0 -n robot-shop I get:
Warning FailedScheduling 46s (x9 over 8m41s) default-scheduler 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.

What does that mean and how can I change that?

Instana EUM javascript generated incorrectly in Open Shift helm deployment

Deploying Robot Shop to Open Shift using the helm chart generates incorrect EUM javascript, causing Instana EUM reporting to fail.

This is the debug from running the helm chart. Note that the eum.url value passed has the trailing slash.
Release "robot-shop" has been upgraded. Happy Helming!
NAME: robot-shop
LAST DEPLOYED: Thu Oct  7 09:27:40 2021
NAMESPACE: robot-shop
STATUS: deployed
REVISION: 2
TEST SUITE: None
USER-SUPPLIED VALUES:
eum:
  key: XXXXXXXXXXXXXXXXXXXXX
  url: https://instana-eum.XXXXXX.XXXXX.com/eum/
ocCreateRoute: true
openshift: true

The helm chart deployment results in this eum.html page being created in the container. Note that the trailing slash has been droped from reportingUrl.

<script>   (function(s,t,a,n){s[t]||(s[t]=a,n=s[a]=function(){n.q.push(arguments)},   n.q=[],n.v=2,n.l=1*new Date)})(window,"InstanaEumObject","ineum");   ineum('reportingUrl', 'https://instana-eum.XXXXXX.XXXXX.com/eum');   ineum('key', 'XXXXXXXXXXXXXXXXXXXXX');   ineum('trackSessions');   ineum('page', 'splash'); </script> <script defer crossorigin="anonymous" src="https://instana-eum.XXXXXX.XXXXX.com/eum/eum.min.js"></script>

As a result, Robot Shop does not transmit data to Instana EUM. After modifying eum.html in the running container to add the trailing slash to reportingUrl, EUM data begins appearing in Instana.

This is using the most recent commit in the master branch.

$ git log
commit b1adf4c (HEAD -> master, origin/master, origin/HEAD)
Author: SteveWW [email protected]
Date: Wed Sep 1 12:27:45 2021 +0100

EUM endpoint needs tailing / for backend to capture client IP address

In the robot-shop-web-1 container the /usr/share/nginx/html/eum.html reportingUrl requires a tailing / when sending EUM data to :86/eum/. Otherwise the GET request to :86/eum/ is responded with HTTP 301.

After adding a / in the line, it worked and IP addresses were captured and correctly geolocated.

Example of the corresponding line:
ineum('reportingUrl', 'http://:86/eum/');

Feel free to Slack me IBM internally for more information

How to install the agent to monitor?

Hi,

I have deployed the application by docker-compose up command, and I got a trial account at this website. But i am confusing the way to get the INSTANA_EUM_KEY and INSTANA_EUM_REPORTING_URL, which seems useful for another docker-compose up.

I have no idea the logic of the agent's installation, and what is the INSTANA_AGENT_KEY?

Is needed to install the agent first (in my application running host?), and then get the INSTANA_EUM_KEY and INSTANA_EUM_REPORTING_URL.

Thankyou

Can this be made to enable ssl/tls between the microservices?

helm deployment redis stays pending

I'm trying to deploy the app on a kubernetes cluster built using kubeadm and when helm install redis stays in a pending status

/root/config.sh in mysql Dockerfile

The permission to file config.sh aren't given in docker file
building the container gives error code 126

COPY config.sh /root/
RUN /root/config.sh

How to monitor the robotshop in instana

Hi,

I have registered an Instana Trail account and successfully add the robotshop website in it. However it doesn't show the infrastructure and metric. Anything i need to do in addition of runing the "docker-compose up" Thanks.

Regards,
Thomas

root/config.sh: not found

I am trying to docker build in Windows. I am getting below error. I tried chmod to the sh file, still the issue persists.

root/config.sh: not found

Create Template for Mesos Orchestrator

@steveww is this already a WIP? I wouldn't mind taking a 🔪 at it.

Unable to install Robo-shop

root@kmaster:# helm install robot-shop --namespace robot-shop .
walk.go:74: found symbolic link in path: /root/snap/kubectl/current resolves to /root/snap/kubectl/1931
walk.go:74: found symbolic link in path: /root/snap/snap-store/current resolves to /root/snap/snap-store/518
Error: Chart.yaml file is missing
root@kmaster:#

Couldn't build robot shop from source

Hi, guys

Please check attached output from building docker image. There is a problem with command docker-compose build and asking for files from link https://oss.risingstack.com/repo/v1/@risingstack/profiler/dist/v5.7.11

So we can't make build.

robot-shop-output-install.txt

Getting connection error when calling to payment api in Openshft

I've simply put all the services in openshift (with deploy.sh in Openshift dir), but when I click on pay now, it throws connection error -

Please investigate it as it is coming always, in any openshift version.

Log

[2020-01-27 11:25:48,138] ERROR in payment: HTTPSConnectionPool(host='paypal.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff9e85cc0b8>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))

[Payment service] Missing attaching Formatter and StreamHandler to application logger

Hi team, I believe I've found a small bug in the payment service logging. I've been using your demo app (thanks btw for creating it and sharing it) as a "guinea pig" of sorts and particularly focused on the logs. When using the load generator with forced errors (bad calls to /api/payment/pay, I looked into the payment service.

There seems to be missing the part where the Formatter is attached to the StreamHandler and the handler to the logger.

robot-shop/payment/payment.py

Lines 151 to 154 in e2cfded

	sh = logging.StreamHandler(sys.stdout)
	sh.setLevel(logging.INFO)
	fmt = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
	app.logger.info('Payment gateway {}'.format(PAYMENT_GATEWAY))

It seems to be missing something like this:

sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.INFO)
fmt = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
sh.setFormatter(fmt)
app.logger.addHandler(sh)

Also, this part should be done before instantiating the RabbitMQ publisher since it passes the app logger as an argument:

robot-shop/payment/payment.py

Lines 147 to 148 in e2cfded

	# RabbitMQ
	publisher = Publisher(app.logger)

Thanks again for the work to have this demo tool publicly open!

Cannot access the web UI when deploying in k8s

I am using helm to deploy robot-shop in k8s: https://github.com/instana/robot-shop/blob/master/K8s/helm/README.md#helm-v3x

However, I cannot access to the web UI via 192.168.49.2:31241

$ minikube ip
192.168.49.2
$ kubectl get svc web -n robot-shop
NAME   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
web    LoadBalancer   10.104.90.129   <pending>     8080:31241/TCP   3m44s

In the web service pod, it continuously generates logs like these:

│ 2023-10-20 22:47:19, 19 [lis] Error: lookup requests for all agent host candidates failed                                                                 │
│ 2023-10-20 22:47:19, 29 [lis] Error: lookup requests for all agent host candidates failed                                                                 │
│ 2023-10-20 22:47:20, 18 [lis] Error: lookup requests for all agent host candidates failed

For the dispatch pod, I got:

2023/10/20 23:41:21 instana: INFO: failed to send a request to http://192.168.49.2:42699/: Get "http://192.168.49.2:42699/": dial tcp 192.16 
│ 2023/10/20 23:41:21 instana: INFO: failed to send a request to http://10.244.0.1:42699/: Get "http://10.244.0.1:42699/": dial tcp 10.244.0.1 
│ 2023/10/20 23:41:21 instana: ERROR: cannot connect to the agent through localhost or default gateway, scheduling retry                      
│ 2023/10/20 23:41:51 instana: INFO: failed to send a request to http://192.168.49.2:42699/: Get "http://192.168.49.2:42699/": dial tcp 192.16
│ 2023/10/20 23:41:51 instana: INFO: failed to send a request to http://10.244.0.1:42699/: Get "http://10.244.0.1:42699/": dial tcp 10.244.0.1 
│ 2023/10/20 23:41:51 instana: ERROR: cannot connect to the agent through localhost or default gateway, scheduling retry                      
│ 2023/10/20 23:41:56 instana: ERROR: Failed sending profiles, backing off next sending                                                        
│ 2023/10/20 23:41:56 instana: ERROR: sender not ready 

I can start robot-shop when running it locally. https://github.com/instana/robot-shop#run-locally

JSONDecodeError while load gen with locust

Hello,

now that I have successfully deployed the robot-shop to my local Kubernetes cluster, I get some error messages during the load generation with locust.

Sometimes when it tries to get the area code with the line:
code = choice(self.client.get('/api/shipping/codes, headers={'x-forwarded-for': fake_ip}).json())
It raises an JSONDecode error, which tells me that it probably did not get the expected response from the api.
It also seems that this is unattached to what IP is used.

here is the full error message:

ERROR:locust.user.task:Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "C:\Users\y509019\OneDrive - Software AG\Documents\Projekte\Thesis\venv\lib\site-packages\locust\user\task.py", line 290, in run
    self.execute_next_task()
  File "C:\Users\y509019\OneDrive - Software AG\Documents\Projekte\Thesis\venv\lib\site-packages\locust\user\task.py", line 315, in execute_next_task
    self.execute_task(self._task_queue.pop(0))
  File "C:\Users\y509019\OneDrive - Software AG\Documents\Projekte\Thesis\venv\lib\site-packages\locust\user\task.py", line 428, in execute_task
    task(self.user)
  File "C:\Users\y509019\OneDrive - Software AG\Documents\Projekte\Thesis\data\loadtest\robotshop.py", line 78, in load
    code = choice(self.client.get('/api/shipping/codes', headers={'x-forwarded-for': fake_ip}).json())
  File "C:\Users\y509019\OneDrive - Software AG\Documents\Projekte\Thesis\venv\lib\site-packages\requests\models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2032.0_x64__qbz5n2kfra8p0\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2032.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2032.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Support for arm64 platform.

I have been working on deploying the instana/robot-shop on the ARM64 platform using the docker-compose method. But the project does not work on the arm64 platform. I have made some changes in some files to get this work on the arm64 platform. With these changes, it works fine on the x64 and Arm64 platforms. Please share your suggestions on the same.

Commit Link: - odidev@4776c88

Web Pod Keeps crashing.

1. Created an EKS cluster having 4 nodes.
2. Pulled the latest and Installed robot-shop to test my K8S.
3. Web-pod keeps crashing as below. Describing pod says just Error.

NAME                         READY   STATUS    RESTARTS   AGE
cart-7f78759dc-ltmbd         1/1     Running   0          5m21s
catalogue-67f57b7b56-4z2cp   1/1     Running   0          5m21s
dispatch-5c54f84c7d-87xdg    1/1     Running   0          5m21s
mongodb-79fd96896c-2mwpj     1/1     Running   0          5m20s
mysql-849b8c895b-l6lfz       1/1     Running   0          5m20s
payment-fcfc6477b-h9xxz      1/1     Running   0          5m20s
rabbitmq-57f9cd66d9-p6zn6    1/1     Running   0          5m20s
ratings-76979dc658-jxvkd     1/1     Running   0          5m19s
redis-5dccc57554-q8jfg       1/1     Running   0          5m19s
shipping-7f8cf9f599-7cjp7    1/1     Running   0          5m19s
user-7588d6c948-8tvtc        1/1     Running   0          5m19s
web-57fbfdfcc4-g7qc2         0/1     Error     5          5m18s

Namespace:          robot-shop
Priority:           0
PriorityClassName:  <none>
Node:               ip-192-168-92-252.us-west-2.compute.internal/192.168.92.252
Start Time:         Tue, 03 Sep 2019 06:00:55 -0700
Labels:             pod-template-hash=57fbfdfcc4
                    service=web
Annotations:        kubernetes.io/psp: eks.privileged
Status:             Running
IP:                 192.168.103.131
Controlled By:      ReplicaSet/web-57fbfdfcc4
Containers:
  web:
    Container ID:   docker://c6890fe67301630a41f1cabdeb43234a81ee5db59cf281f726635cbe5cd91ff8
    Image:          robotshop/rs-web:latest
    Image ID:       docker-pullable://robotshop/rs-web@sha256:ec0cf5f4562668e260f2fd168b6f8fd8e8d0e5e6d85a77d57d95f1122308c4d7
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 03 Sep 2019 06:08:51 -0700
      Finished:     Tue, 03 Sep 2019 06:09:11 -0700
    Ready:          False
    Restart Count:  6
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:        100m
      memory:     50Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kmdjk (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True

Should the memory be increased for webpod ?

Running on aws ecs

What should i do to run robotshop application on to aws ecs ?

Robot-shop is not reporting anymore into Instana websiteMonitoring view

I remember some months ago I could fire up robot-shop and get it reporting to any Instana environment (release, qa, pink, it wouldn't matter...) just uncommenting these docker-compose.yaml lines:

# environment:
      # INSTANA_EUM_KEY: <your eum key>
      # INSTANA_EUM_REPORTING_URL: <your reporting url>

and filling them with the values reported in the configuration section of the websiteMonitoring view.
Firing up robot-shop with this docker-compose.yaml file, with the command docker-compose up would have been enough.
Now this procedure is not working.
Did something change?

Can you please help here?

DC/OS deployment files for new PHP service

New PHP ratings service needs matching DC/OS deployment files.