danwahlin/codewithdandockerservices

Demonstration of using Docker and/or Kubernetes to orchestrate multiple services

Docker Services Orchestration Demonstration (Kubernetes Orchestration demo also available)

This is a demo application from the Docker for Web Developers course on Pluralsight that demonstrates how multiple services can be integrated and orchestrated using Docker and Docker Compose.

Running Docker with WSL (Windows Subsystem for Linux) on Windows?

Open a WSL (Linux) command prompt and clone the repo from there into a folder of your choosing. For example you can cd into desktop and put it there if you'd like.

git clone https://github.com/danwahlin/codewithdandockerservices

Ensure that Node is installed into your Linux instance. Details here.

Note that if you're NOT using WSL at all with Docker on Windows then you can clone the repo from any Windows command prompt.

Instructions

1. Install Docker Desktop for Mac or Docker Desktop for Windows from https://docker.com and the lastest LTS version of Node.js from https://nodejs.org.

2. Set the environment variables in your command window.

   export APP_ENV=development

   export DOCKER_ACCT=codewithdan

   NOTE: For the Windows DOS command shell use set instead of export. For Windows Powershell use $env:KEY = "value":

   $env:APP_ENV="development"

   $env:DOCKER_ACCT="codewithdan"

3. Run npm install to install the Node.js dependencies for the project (when running containers in development mode since a volume is defined docker-compose.yml file)

4. Run docker-compose build

5. Run docker-compose up

6. Wait until everything has finished loading in the console (may take 60 seconds or so) and then visit http://localhost in a browser. You may need to refresh a few times to see the data displayed (typically a timing issue if you try it too soon).

7. Live long and prosper

To run in Kubernetes with Docker Desktop

1. Enable Kubernetes in Docker Desktop.

   Note: You MUST have Docker Desktop for this particular demo to work or another local Kubernetes option such as Minikube.

2. Do a production Docker Compose build (see docker-compose.yml for instructions on doing the build) to create the local images. Ensure that you set APP_ENV=production as mentioned in the compose file.

3. Open a command-prompt at the root of the project

4. Run the following to add the database passwords as secrets (yes - these are simple passwords for the demo :-)):

   kubectl create secret generic db-passwords --from-literal=db-password=password --from-literal=db-root-password=password

   Note: password is being used here purely to keep things very simple for the demo. Use strong passwords for a "real" setup!!!!

5. Run kubectl create -f .k8s to create the Kubernetes Services, Deployments, Pods, etc.

6. Once the deployments are applied several pods will be created.

7. Open the browser and go to http://localhost. Read note below.

NOTE: You'll need to wait since it'll take a little bit for the DB to start up. Once the Pods are ready you should see data in the app (hit refresh if needed).

1. When you're done run kubectl delete -f .k8s to delete the Kubernetes resources.

Local Storage Hostname Volume

NOTE: The local storage hostname volume is commented out in the .k8s/mongo.deployment.yml StatefulSet because MongoDB doesn't support that type of volume correctly on Docker for Windows. It does work on Mac/Linux. You'd need to create a /tmp/data/db directory and then uncomment the volumes and volumeMounts properties (and sub-properties) in the StatefulSet to actually use the volume on Mac/Linux.

Load Balancer versus Port Forwarding

This demo includes a LoadBalancer service for the nginx Pod which is why you can hit http://localhost. To expose a specific port for localhost for the nginx Pod, get the name of the nginx pod by running kubectl get pods and use the pod name in the following command:

sudo kubectl port-forward [name-of-nginx-pod] 8080:80

Note that sudo is needed to enable port 80 in this case on Mac. You can choose a different port as well such as 8081:80.

Running with Skaffold

1. Open a command-prompt at the root of the project

2. Run the following to add the database passwords as secrets (yes - these are simple passwords for the demo :-)):

   kubectl create secret generic db-passwords --from-literal=db-password='password' --from-literal=db-root-password='password'

   Note: password is being used here purely to keep things very simple for the demo. Use strong passwords for a "real" setup!!!!

3. Install Skaffold from https://skaffold.dev.

4. Run the following command at the root of the project:

   skaffold dev
   

codewithdandockerservices's Issues

More of some questions really...

• Currently there are multiple dockerfiles in the /.docker folder - however it might be an idea to build a multi stage build with named targets? you could pass the environment name through and get the appropriate image from the build.
• Installation notes say I have to install a node environment to build the code before packaging - but this could be solved using the multistage build process.
• I found the inclusion of pm2 confusing - I wasn't sure that it added anything that you don't already get from the docker container restart policies - We could remove this right and use the restart policies right? If we could remove the pm2 use, then this would simplify the build image (particularly for production).
• Found it strange that the secrets were only used to set the password for mongo - when they should also be used to configure the node containers (?)

Secrets mongo deployment does not create admin user

In the course: "Kubernetes for Developers: Core Concepts, 7 Creating ConfigMaps and Secrets, Secrets in Action", the mongo.deployment.yml does not create an admin user or allow authentication at the shell level or otherwise.

I followed the steps create the secrets with kubectl and using the current version of mongo.deployment.yml: https://github.com/DanWahlin/DockerAndKubernetesCourseCode/blob/main/samples/secrets/mongo.deployment.yml

Output after pod creation:

➜  secrets k apply -f mongo.deployment.yml
configmap/mongo-secrets-env created
storageclass.storage.k8s.io/local-storage created
persistentvolume/mongo-pv created
persistentvolumeclaim/mongo-pvc created
statefulset.apps/mongo created
➜  secrets k get pods
NAME                              READY   STATUS    RESTARTS   AGE
mongo-0                           1/1     Running   0          3s
node-configmap-577f5d6b98-qlwwb   1/1     Running   0          140m

Output attempting to auth inside mongo shell in the pod:

➜  ckad_training k exec mongo-0 -it sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
# mongo 
MongoDB shell version v5.0.6
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("a66eee69-b066-4639-b06f-422745204d16") }
MongoDB server version: 5.0.6
================
Warning: the "mongo" shell has been superseded by "mongosh",
which delivers improved usability and compatibility.The "mongo" shell has been deprecated and will be removed in
an upcoming release.
For installation instructions, see
https://docs.mongodb.com/mongodb-shell/install/
================
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        https://docs.mongodb.com/
Questions? Try the MongoDB Developer Community Forums
        https://community.mongodb.com
> db.auth("admin", "password")
Error: Authentication failed.
0
> use codeWithDan
switched to db codeWithDan
> db.auth("admin", "password")
Error: Authentication failed.
0

Output from Studio T3 (using legacy SHA-1, but basic SHA-256 has same result):

Would love to know what the solution is here as I was not able to find anything. Thanks!

Mongo Exit Code 1

Starting mongo ...
Starting mongo ... done
Recreating node-codewithdan ... done
nginx is up-to-date
Attaching to redis, mongo, node-codewithdan, nginx
mongo | standard_init_linux.go:211: exec user process caused "no such file or directory"
mongo exited with code 1

Connection error: MongoNetworkError: failed to connect to server [mongo:27017] on first connect [Error: getaddrinfo ENOTFOUND mongo
node-codewithdan | 22:26:39 0|codewithdan | at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26) {
node-codewithdan | 22:26:39 0|codewithdan | name: 'MongoNetworkError',
node-codewithdan | 22:26:39 0|codewithdan | [Symbol(mongoErrorContextSymbol)]: {}
node-codewithdan | 22:26:39 0|codewithdan | }]

Question: Which Commit

@DanWahlin - Can you tell me which commit would be best to checkout for following along with the Pluralsight course? I'm on Module 8 (Managing Containers with Docker Compose) and would like to take a look at the code prior to the Kubernetes updates and other things not yet covered in the lessons. Thanks - enjoying the course.

host not found in upstream "node1:8080" in /etc/nginx/nginx.conf:9

Hello Sir,
Thanks for plurasight tutorial. It help me a lot.

I am getting above exception on docker-compose up. I did share you a git hub link to access my code.

It's very simple code. I would really appreciate your help.

https://github.com/tusharbudhe0302/dockernode

Error: Cannot find module 'express'

HI Dan,
thank you for the excellent tutorial on Docker.

I am getting the error 'Error: Cannot find module 'express'' when I try to run 'docker-compose up'. I added 'npm install -g express' in node-codeqwithdan.$app-env.dockerfile, it didn't help.

Please help.

Rafeeq

Unable to run apt-get for mongo image from corporate firewall

Hi Dan,

Thanks for good code. I am able to build other images properly but build for mongo image is not working from corporate firewall proxy. Please find below the errors.

Errorrs:
_---> Running in 24eb6fa4206e
Err http://repo.mongodb.org jessie/mongodb-org/3.4 InRelease

Err http://repo.mongodb.org jessie/mongodb-org/3.4 Release.gpg
Unable to connect to repo.mongodb.org:http:
---> 25ea78fb5b0b
Err http://deb.debian.org jessie InRelease

Err http://deb.debian.org jessie-updates InRelease

Err http://deb.debian.org jessie Release.gpg
Unable to connect to deb.debian.org:http: [IP: 149.20.4.15 80]
Err http://deb.debian.org jessie-updates Release.gpg
Unable to connect to deb.debian.org:http: [IP: 149.20.4.15 80]
Err http://security.debian.org jessie/updates InRelease

Err http://security.debian.org jessie/updates Release.gpg
Unable to connect to security.debian.org:http: [IP: 195.20.242.89 80]
Removing intermediate container 24eb6fa4206e
Step 7/8 : EXPOSE 27017
Reading package lists... ---> Running in 47c172f04e22

---> e36b88f91680
Removing intermediate container 47c172f04e22
Step 8/8 : ENTRYPOINT /mongo_scripts/run.sh
W: Failed to fetch http://deb.debian.org/debian/dists/jessie/InRelease

W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/InRelease

W: Failed to fetch http://security.debian.org/dists/jessie/updates/InRelease

W: Failed to fetch http://repo.mongodb.org/apt/debian/dists/jessie/mongodb-org/3.4/InRelease

W: Failed to fetch http://repo.mongodb.org/apt/debian/dists/jessie/mongodb-org/3.4/Release.gpg Unable to connect to repo.mongodb.org:http:

W: Failed to fetch http://deb.debian.org/debian/dists/jessie/Release.gpg Unable to connect to deb.debian.org:http: [IP: 149.20.4.15 80]

W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/Release.gpg Unable to connect to deb.debian.org:http: [IP: 149.20.4.15 80]

W: Failed to fetch http://security.debian.org/dists/jessie/updates/Release.gpg Unable to connect to security.debian.org:http: [IP: 195.20.242.89 80]

W: Some index files failed to download. They have been ignored, or old ones used instead.
---> Running in 9abd7a5c214b
Reading package lists... ---> 914a5eed5c5e

Removing intermediate container 9abd7a5c214b
Successfully built 914a5eed5c5e
Successfully tagged maheshk172/mongo:latest
Building redis
Building dependency tree...
Reading state information...
E: Unable to locate package cron
E: Unable to locate package netcat-traditional
E: Unable to locate package netcat-openbsd
Service 'mongo' failed to build: The command '/bin/sh -c apt-get update && apt-get install -y cron netcat-traditional netcat-openbsd' returned a non-zero code: 100_

I tried adding the http_proxy, https_proxy under the mongo.development.env but it did not helped.
I also tried adding a /etc/apt/apt.conf file with a proxy configuration and copy them before running apt-get update commands, as suggested on the UBUNTU forums, but it did not helped as well

Could you please help with this issue?

Development: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28

Software versions:

• Docker Compose version v2.5.1
• Docker version 20.10.14, build a224086
• M1 Pro on macOS Monterey version 12.4

Environment variables

• APP_ENV=development
• DOCKER_ACCT=codewithdan

Steps Taken from comments on docker-compose.yml

1. Update config values (localhost --> mongo and localhost --> redis) in config/config.development.json if necessary.

-> Did not edit

2. Set APP_ENV environment variable by running the following command in your commnand window
   export APP_ENV=development or export APP_ENV=production
   export DOCKER_ACCT=codewithdan

-> Done

3. Remove "node" service volume (below) if doing a production build

-> Did not remove, using development mode

4. Run docker-compose build

-> Done

5. Run docker-compose up

-> Done

Screenshots:

nginx:

Logs

Running docker logs nginx:

2022/05/31 01:25:15 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:25:15 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:25:16 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:25:16 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:25:17 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:25:19 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:25:23 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:25:29 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:25:43 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:26:08 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:27:00 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:28:00 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:29:01 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:30:01 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:31:01 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:32:02 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:33:02 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:34:02 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:35:02 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:36:03 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:37:03 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:38:03 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:39:04 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:40:04 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:41:04 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
2022/05/31 01:42:05 [emerg] 1#1: host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28
nginx: [emerg] host not found in upstream "node:8080" in /etc/nginx/nginx.conf:28

Please help, I'm doing my best to learn docker,

Thanks in advance @DanWahlin