dwyl/learn-devops

🚧 Learn the craft of "DevOps" (Developer Operations) to Deploy your App and Monitor it so it stays "Up"!

Learn DevOps

Learn the craft of "DevOps" (Developer Operations) to reliably deploy your App and keep it Up!

Why?

You should learn more "advanced" DevOps if:

• You / your team have "out-grown" Heroku (e.g: your Heroku bill is more than $100/month)
• You want "more control" over your infrastructure e.g: to run a specific version of software or database.
• Your client/boss has instructed you to use a specific "cloud" provider.
• Curiosity to extend your "back end infrastructure" knowledge to be a more "well-rounded" developer.

What?

"DevOps integrates developers and operations teams in order to improve collaboration and productivity by automating infrastructure, automating workflows and continuously measuring application performance." from: "What is DevOps?" by RackSpace

Who?

Everyone that wants to seriously consider/call themself a "Full Stack" Developer must know how to deploy, secure and monitor their app on their chosen infrastructure.

How?

Over the years we @dwyl have deployed/managed Apps, both our own and those of our clients, on a wide variety of infrastructure and platform providers.

Most of our Apps have been deployed to Amazon Web Services ("AWS") e.g: https://www.sciencemuseum.org.uk
We have several clients who use (and love) Heroku e.g: https://www.ellenmacarthurfoundation.org
For the National Health Service (NHS) in England, (who have a major contract with Microsoft) we deployed to Microsoft Azure.
We have clients who still own their own "Bare Metal" Servers.

Provider-Specific Guides

We have produced a guide for each of our most-used infrastructure/platform providers:

• AWS: https://github.com/dwyl/learn-amazon-web-services
• AWS Lambda: https://github.com/dwyl/learn-aws-lambda
• Azure: https://github.com/dwyl/learn-microsoft-azure
• Heroku: https://github.com/dwyl/learn-heroku
• Fly.io: fly-phoenix-setup.md
• Linode: linode-setup.md
• DigitalOcean: nodejs-digital-ocean-centos-dokku.md

Gogs Git Server

We needed to run a our own Git server for one of our projects. So we deployed Gogs both locally and to Fly.io and documented it: /gogs

Node.js

• Node.js with Dokku: nodejs-digital-ocean-centos-dokku.md
• Node.js "Zero Downtime" Production Deployment with PM2: nodejs-pm2-zero-downtime.md

If you would like to see a guide for a different service provider, please open an issue with your suggestion(s): https://github.com/dwyl/learn-devops/issues

Testing, Continuous Integration & Delivery

Deployment is what you do with your app once you have built, tested and documented it.

If you are unfamiliar with Test-Driven Development (TDD), please see: https://github.com/dwyl/learn-tdd

Next you should be proficient with Continuous Integration. For that we recommend using Travis-CI, see: https://github.com/dwyl/learn-travis

https://github.com/dwyl/learn-travis/blob/master/encrypted-ssh-keys-deployment.md

Resources

Videos

• What is DevOps: https://youtu.be/I7vHqXY22gg
• What is DevOps? - In Simple English: https://youtu.be/_I94-tJlovg (good info but last minute is a RackSpace pitch)
• DevOps for Beginners Course Introduction: https://youtu.be/v7ZcZfGBFcU

Background Reading

• https://en.wikipedia.org/wiki/DevOps
• https://theagileadmin.com/what-is-devops
• https://newrelic.com/devops/what-is-devops
• https://logit.io/blog/post/what-is-devops

Linode-specific How-tos

• SSH with Public Keys: https://www.linode.com/docs/security/use-public-key-authentication-with-ssh
• Using Vagrant for Environment Management: https://www.linode.com/docs/applications/configuration-management/vagrant-linode-environments

Using a Different Cloud Infrastructure Provider?

While this tutorial has focussed on using Linode, we @dwyl have experience of using several infrastructure providers:

• Amazon Web Services: https://github.com/dwyl/learn-amazon-web-services
• Azure: https://github.com/dwyl/learn-microsoft-azure
• Digital Ocean: https://github.com/dwyl/DigitalOcean-Setup
• Heroku: https://github.com/dwyl/learn-heroku

if you have a question specific to using Linode or one of the other "cloud" providers, please open an issue and we will attempt to help!

Thanks for learning with us! Your feedback/questions is always welcome.

learn-devops's Issues

Can we use Amazon Aurora PostgreSQL with Phoenix?

https://aws.amazon.com/rds/aurora/

Mastering Elixir Releases with Distillery — A (Pretty) Complete Guide

This looks like a good one:
https://hackernoon.com/mastering-elixir-releases-with-distillery-a-pretty-complete-guide-497546f298bc
shared by @Shouston3 in a Private project. (thanks!)
will read laters and update our tutorials accordingly 📚

How to setup a Subdomain and Serve a Web App via NGINX?

Todo

• Create Subdomain on Registrar (iWantMyName)
• Forward Subdomain to VM IP Address
• Create NGINX config file for subdomain
• Test serving basic html page via subdomain
• Serve a basic Node.js app on the subdomain
• Serve a Phoenix App on Subdomain
• Setup Let's Encrypt (SSL) for the Subdomain

Create Linode User for Rob

Create user on Linode so that Rob can follow along with the migration from Node.js to Phoenix:
https://manager.linode.com/user/edit

At what point should you *automate* infrastructure creation?

If you only have 2 servers and a load balancer, is it worth it to "automate" anything?

What are the circumstances that make it worth automating?

Can (should) we run multiple versions of the same app on the same server/instance?

The only issue i have with running both staging and prod on the same VM is the naming of environment variables ...

Cloud Computing without Containers

https://blog.cloudflare.com/cloud-computing-without-containers/
Via: https://news.ycombinator.com/item?id=18415708

How to migrate a PostgreSQL Database from Heroku to another Server/Service?

Three reasons:

• We want to have control over where our data is stored for our Product Projects.
• Heroku PostgreSQL Databases are Silly Expensive:
  https://devcenter.heroku.com/articles/heroku-postgres-plans
  
• Clients want to be on a specific hosting/service provider (NHS wants Azure!!)

Three dots (...) as host in phoenix deloyment tutorial

Hey,

I have been following the Phoenix production deployment guide, and I managed to get it running, so kudos for that!

However, I have a question about the endpoint config that the guide gives. This is the excerpt:

config :hello_world_edeliver, HelloWorldEdeliver.Endpoint,
  http: [port: {:system, "PORT"}],
  url: [host: "...", port: {:system, "PORT"}],

I'm confused because of the three dots as the host. As far as I know, this has no special meaning as IP address. Does it do anything in particular or is it just ignored?

Thanks in advance!

“There’s a Heroku buildpack for that…”

Optimize Dokku Deployment Speed for Ruby on Rails with Dockerfile
https://pawelurbanek.com/optimize-dokku-deployment-speed

Some of this will be applicable to both Node and Phoenix apps so worth synthesising.

Why Linode?

I've been keeping my eye on Linode for a while as an alternative to AWS/DigitalOcean.
Specifically their pricing, features and independence (no VCs to please!)
Given that Linode offers much better value-for-money and performance I think it's worth considering!

at the time of writing the Linode Pricing is: https://www.linode.com/pricing#all

Which is exactly double the RAM compared to Digital Ocean (all other things equal) https://www.digitalocean.com/pricing/

@joshtronic wrote a superb comparison between Linode, DigitalOcean, Amazon Lightsail & Vultr: https://joshtronic.com/2017/02/14/five-dollar-showdown-linode-vs-digitalocean-vs-lightsaild-vs-vultr/ PDF Snapshot: $5-Showdown-Linode-vs.-DigitalOcean-vs.-Amazon-Lightsail-vs-Vultr.pdf
On virtually every dimension Linode comes out top when compared to alternatives!

So, today I'm trying Linode! (with Josh's Referral Code!)
https://www.linode.com/?r=5f682793582e82ce686747c851b998dc1f86a55b

mesg: ttyname failed: Inappropriate ioctl for device

keep getting this error message when running ssh commands:

mesg: ttyname failed: Inappropriate ioctl for device

Time to fix it!!

How to Setup SSL for a Phoenix Web Application hosted on a VM or "Bare Metal"?

http://www.phoenixframework.org/docs/configuration-for-ssl
relates to: dwyl/learn-microsoft-azure#8

Can we use Vagrant for Production Deployments?

https://www.linode.com/docs/applications/configuration-management/vagrant-linode-environments

How to setup Logging for Phoenix Web Applications

Which open source logging system should we use?
Leaning towards ELK ... but if anyone knows a better one I'm all ears!

Let's Encrypt certificate expiration notice for domain "*.ademo.app"

How to check Disk Usage on Linux ("no space left on device") ?

reading: https://unix.stackexchange.com/questions/125429/tracking-down-where-disk-space-has-gone-on-linux
most popular answer recommends ncdu ("NCurses Disk Usage"): https://dev.yorhel.nl/ncdu
which uses: https://en.wikipedia.org/wiki/Ncurses

So I need to install both ncurses and ncdu for Centos ...

Install EPEL (Extra Packages for Enterprise Linux)
https://fedoraproject.org/wiki/EPEL

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Unfortunately, I get the No space left on device error when attempting to install (duh!)

So I need to use (good old) du first!

du -Pshx /* 2>/dev/null

In the /home directory, run: du -sm * | sort -n

You have new mail in /var/spool/mail/root

to read the "mail" run:

cat /var/spool/mail/root

via: https://superuser.com/questions/306163/what-is-the-you-have-new-mail-message-in-linux-unix

After deleting a few of the "stale" apps:

I can re-attempt to install ncdu:

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

followed by:

yum install ncdu -y

Run ncdu and then navigate!

Turns out the disk hog is the node_modules cache ... 🙄

Where to save SSH keys for Dokku PaaS

In the pre-requisites of the Dokku Paas tutorial, we need to add an SSH key to our droplet" https://github.com/dwyl/learn-devops/blob/bd4489096cf6ed31eb2fb6a8886fb1eae4bfc41f/nodejs-digital-ocean-centos-dokku.md#0-pre-requisites

Part of the process is to create one.

As someone who is totally new to devops, are there any conventions I should be aware of around where I should save this to? Does scope matter? Can I just make up any name for a file within the project folder where I'm working (without committing it, obviously)?

How to Add Environment Variables to a Dokku App

• investigate the "right" way to add environment variables to an app deployed with Dokku.
  relates to #24

LetsEncrypt Wildcard SSL Certificate

LetsEncrypt announced Wildcard SSL Certificates on March 13, 2018 🎉

• Investigate how to configure a Wildcard Certificate for use with multiple apps proxied by nginx

Note: at the time of opening this issue, there is still no end-to-end "tutorial" for doing this ...
So I'm investigating it as a "SPIKE".

*single* edeliver command to build and "hot upgrade" a Elixir/Phoenix App?

picks up from: dwyl/learn-elixir#54
moved here to reduce noise in learn-elixir repo which has more watchers...

after updating distillery to latest version (1.4.1): nelsonic/hello_world_edeliver@3c8f493
we are still getting an error:

** (EXIT from #PID<0.73.0>) an exception was raised:
    ** (File.Error) could not write to file "/home/ubuntu/hello_world_edeliver/builds/_build/prod/lib/edeliver/ebin/Elixir.Releases.Plugin.LinkConfig.beam": no such file or directory
        (elixir) lib/file.ex:831: File.write!/3
        (mix) lib/mix/compilers/elixir.ex:431: anonymous fn/4 in Mix.Compilers.Elixir.write_manifest/5
        (elixir) lib/enum.ex:1811: Enum."-reduce/3-lists^foldl/2-0-"/3
        (mix) lib/mix/compilers/elixir.ex:427: Mix.Compilers.Elixir.write_manifest/5
        (mix) lib/mix/compilers/elixir.ex:166: anonymous fn/4 in Mix.Compilers.Elixir.compile_manifest/8
        (elixir) lib/agent/server.ex:31: Agent.Server.handle_cast/2
        (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:686: :gen_server.handle_msg/6


15:59:25.377 [error] GenServer #PID<0.269.0> terminating
** (File.Error) could not write to file "/home/ubuntu/hello_world_edeliver/builds/_build/prod/lib/edeliver/ebin/Elixir.Releases.Plugin.LinkConfig.beam": no such file or directory
    (elixir) lib/file.ex:831: File.write!/3
    (mix) lib/mix/compilers/elixir.ex:431: anonymous fn/4 in Mix.Compilers.Elixir.write_manifest/5
    (elixir) lib/enum.ex:1811: Enum."-reduce/3-lists^foldl/2-0-"/3
    (mix) lib/mix/compilers/elixir.ex:427: Mix.Compilers.Elixir.write_manifest/5
    (mix) lib/mix/compilers/elixir.ex:166: anonymous fn/4 in Mix.Compilers.Elixir.compile_manifest/8
    (elixir) lib/agent/server.ex:31: Agent.Server.handle_cast/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
Last message: {:"$gen_cast", {:cast, #Function<8.64051268/1 in Mix.Compilers.Elixir.compile_manifest/8>}}

A remote command failed on:

  [email protected]

Output of the command is shown above and the command executed
on that host is printed below for debugging purposes:

FAILED with exit status 1:

    [ -f ~/.profile ] && source ~/.profile
    set -e

    cd '/home/ubuntu/hello_world_edeliver/builds'
    APP='hello_world_edeliver' MIX_ENV='prod' mix phoenix.digest

nginx: [warn] conflicting server name "app" on [::]:80, ignored

-----> Creating https nginx.conf
-----> Running nginx-pre-reload
       Reloading nginx
nginx: [warn] conflicting server name "ci." on [::]:80, ignored
nginx: [warn] conflicting server name "hello-world-node." on [::]:80, ignored
nginx: [warn] conflicting server name "hello." on [::]:80, ignored
nginx: [warn] conflicting server name "ci." on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "hello-world-node." on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "hello." on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "ci." on [::]:443, ignored
nginx: [warn] conflicting server name "hello-world-node." on [::]:443, ignored
nginx: [warn] conflicting server name "hello." on [::]:443, ignored
nginx: [warn] conflicting server name "ci." on 0.0.0.0:443, ignored
nginx: [warn] conflicting server name "hello-world-node." on 0.0.0.0:443, ignored
nginx: [warn] conflicting server name "hello." on 0.0.0.0:443, ignored
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
nginx: [warn] conflicting server name "ci." on [::]:80, ignored
nginx: [warn] conflicting server name "hello-world-node." on [::]:80, ignored
nginx: [warn] conflicting server name "hello." on [::]:80, ignored
nginx: [warn] conflicting server name "ci." on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "hello-world-node." on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "hello." on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "ci." on [::]:443, ignored
nginx: [warn] conflicting server name "hello-world-node." on [::]:443, ignored
nginx: [warn] conflicting server name "hello." on [::]:443, ignored
nginx: [warn] conflicting server name "ci." on 0.0.0.0:443, ignored
nginx: [warn] conflicting server name "hello-world-node." on 0.0.0.0:443, ignored
nginx: [warn] conflicting server name "hello." on 0.0.0.0:443, ignored

remote: Error response from daemon: Error processing tar file(exit status 1): write /app/.heroku/node/bin/node: no space left on device

When attempting to auto-deploy from CircleCI we are seeing the following error:

remote: Error response from daemon: Error processing tar file(exit status 1): write /app/.heroku/node/bin/node: no space left on device

Full Stack Trace:

#!/bin/bash -eo pipefail
sh bin/deploy.sh
CWD /root/repo/bin
BRANCH => close-modal
DOKKU_APP => ci
DOKKU_APP => ci-close-modal
URL => 138.68.163.126:ci-close-modal
CREATE => dokku apps:create ci-close-modal
Warning: Permanently added '138.68.163.126' (ECDSA) to the list of known hosts.

Name is already taken
 WARNING:  env2 was required to load an .env file:  /root/repo/.env  NOT FOUND!  Please see: http://git.io/vG3UZ
ENV => NODE_VERSION=8.11.3 HOSTNAME=a406bd28a9d6 CI_PULL_REQUESTS=https://github.com/charterindex/application/pull/320 SERVER_IP_ADDRESS=138.68.163.126 ENVIRONMENT-VARIABLES-REDACTED!
-----> Setting config vars
       YARN_VERSION:          1.6.0
       PORT:                  5000
       JWT_SECRET:            thisissecret
       HOSTNAME:              a406bd28a9d6
       SERVER_IP_ADDRESS:     138.68.163.126
       DOKKU_APP:             ci-close-modal
       CI:                    true
       NODE_VERSION:          8.11.3
       CI_PULL_REQUESTS:      https://github.com/charterindex/application/pull/320
       BASH_ENV:              /tmp/.bash_env-5b3bfdabb05ef50013c42a5b-0-build
       TRAVIS:                false
       CIRCLECI:              true
-----> Restarting app ci-close-modal
 !     App ci-close-modal has not been deployed
REMOTE => dokku [email protected]:ci-close-modal
fatal: No such remote 'dokku'
dokku	[email protected]:ci-close-modal (fetch)
dokku	[email protected]:ci-close-modal (push)
origin	[email protected]:charterindex/application.git (fetch)
origin	[email protected]:charterindex/application.git (push)
KILL_NGINX => pkill nginx
SYSTEMCTL_START_NGINX => systemctl start nginx
COMMIT_HASH => bedba5b4f404b4f94b7c69eb211eaf9844ba7f28
PUSH => git push dokku bedba5b4f404b4f94b7c69eb211eaf9844ba7f28:refs/heads/master
Counting objects: 3711, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (1213/1213), done.
Writing objects:  26% (965/3711)   
Writing objects: 100% (3711/3711), 6.78 MiB | 5.13 MiB/s, done.
Total 3711 (delta 2415), reused 3698 (delta 2408)
remote: -----> Cleaning up...
remote: -----> Building ci-close-modal from herokuish...
remote: -----> Adding BUILD_ENV to build environment...
remote:        -----> Node.js app detected
remote:        
remote: -----> Creating runtime environment
remote:        
remote:        NPM_CONFIG_LOGLEVEL=error
remote:        YARN_VERSION=1.6.0
remote:        NODE_VERSION=8.11.3
remote:        NODE_VERBOSE=false
remote:        NODE_ENV=production
remote:        NODE_MODULES_CACHE=true
remote:        
remote: -----> Installing binaries
remote:        engines.node (package.json):  8.11.2
remote:        engines.npm (package.json):   5.6.0
remote:        
remote:        Resolving node version 8.11.2...
remote:        Downloading and installing node 8.11.2...
remote:        npm 5.6.0 already installed with node
remote:        
remote: -----> Restoring cache
remote:        Loading 2 from cacheDirectories (default):
remote:        - node_modules
remote:        - bower_components (not cached - skipping)
remote:        
remote: -----> Building dependencies
remote:        Installing node modules (package.json)
remote:        
remote:        > [email protected] postinstall /tmp/build
remote:        > npm run elm-install && npm run elm-compile && npm run css-compile
remote:        
remote:        
remote:        > [email protected] elm-install /tmp/build
remote:        > cd elm && ../node_modules/elm/binwrappers/elm-package install -y && cd ..
remote:        
remote:        Starting downloads...
remote:        
remote:        ● dwyl/elm-datepicker 2.2.0
remote:        ● NoRedInk/elm-decode-pipeline 3.0.0
remote:        ● elm-lang/core 5.1.1
remote:        ● elm-lang/dom 1.1.1
remote:        ● elm-lang/html 2.0.0
remote:        ● elm-lang/http 1.0.0
remote:        ● elm-lang/keyboard 1.0.1
remote:        ● elm-lang/navigation 2.1.0
remote:        ● elm-lang/virtual-dom 2.0.4
remote:        ● justinmimbs/elm-date-extra 3.0.0
remote:        
remote:        Packages configured successfully!
remote:        
remote:        > [email protected] elm-compile /tmp/build
remote:        > cd elm && ../node_modules/elm/binwrappers/elm-make *.elm --output ../public/js/elm.js --yes --warn && cd ..
remote:        
remote:        Success! Compiled 77 modules.
remote:        Successfully generated ../public/js/elm.js
remote:        
remote:        > [email protected] css-compile /tmp/build
remote:        > postcss routes/css/index.css --output public/stylesheets/style.min.css --config postcss.config.js --verbose
remote:        
remote:        Processing routes/css/index.css...
remote:        Finished routes/css/index.css in 3.41 s
remote:        up to date in 32.714s
remote:        
remote: -----> Caching build
remote:        Clearing previous node cache
remote:        Saving 2 cacheDirectories (default):
remote:        - node_modules
remote:        - bower_components (nothing to cache)
remote:        
remote: -----> Pruning devDependencies
remote:        Skipping because npm 5.6.0 sometimes fails when running 'npm prune' due to a known issue
remote:        https://github.com/npm/npm/issues/19356
remote:        
remote:        You can silence this warning by updating to at least npm 5.7.1 in your package.json
remote:        https://devcenter.heroku.com/articles/nodejs-support#specifying-an-npm-version
remote:        
remote: -----> Build succeeded!
remote:        -----> Discovering process types
remote:        Procfile declares types -> web
remote: Error response from daemon: Error processing tar file(exit status 1): write /app/.heroku/node/bin/node: no space left on device
To [email protected]:ci-close-modal
 ! [remote rejected] bedba5b4f404b4f94b7c69eb211eaf9844ba7f28 -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:ci-close-modal'
server.crt
server.key
-----> Unsetting DOKKU_PROXY_PORT
-----> Unsetting DOKKU_PROXY_SSL_PORT
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:5000
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:5000 https:443:5000
-----> Setting config vars
       DOKKU_PROXY_PORT:  80
-----> Setting config vars
       DOKKU_PROXY_SSL_PORT:  443
-----> No matching configured domains for ci-close-modal found in SSL certificate. Your app will show as insecure in a browser if accessed via SSL
-----> Please add appropriate domains via the dokku domains command
-----> Configured domains for app:
=====> A
=====> *.ademo.app
=====> 138.68.163.126
=====> ademo.app
-----> Domains found in SSL certificate:
=====> *.ademo.app
=====> ademo.app
grep: /home/dokku/ci-close-modal/VHOST: No such file or directory
-----> Configuring 138.68.163.126...(using built-in template)
-----> Configuring A...(using built-in template)
-----> Configuring *.ademo.app...(using built-in template)
-----> Configuring ademo.app...(using built-in template)
-----> App ci-close-modal has not been deployed. Skipping nginx config creation
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Source/Create Diagram to "Explain" DevOps to a Beginner (or NonTech)

Most diagrams are too simple and uninformative:
https://www.google.pt/search?q=devops&tbm=isch

Others are detailed diagram but will/would "confuse" most people e.g:

from: http://www.enhops.com/images/infographics/full/DevOps.png

This buzzword fest is an informative summary to people who already have DevOps experience:

But is overwhelming for everyone else!

I think we can do a much better job ...

How to Auto-Renew Let's Encrypt SSL Certificate?

We have a few projects which use Let's Encrypt SSL Certificates and I intend to continue using them.

Tasks

• Research, document & implement certificate renewal for:
  • hits.dwyl.io (Debian)
  • ademo.app #43 (Centos)

I feel like this should be a standalone .md file similar to:
https://github.com/dwyl/learn-devops/blob/master/letsencrypt-wildcard-certificate.md

Starting point: https://community.letsencrypt.org/search?q=auto

related to: #43
required by: dwyl/learn-travis#45

pm2: conflict with ecosystem.config.js

I had an issue while using pm2 for deployement:
The master branch already had a ecosystem.config.js but when I tried to create a new PR this issue appears:

I tried to rebase/merge my branch with master but it was already up to date
I figure out that the issue was due to the git pull command defined in predeploy:

module.exports = {
  apps: [{
    name: "appname",
    script: "./bin/www"
  }],
  deploy: {
    // "production" is the environment name
    production: {
      // SSH key path, default to $HOME/.ssh
      key: "./deploy_key",
      // SSH user
      user: "dev",
      // SSH host
      host: ["ip-address-host"],
      // SSH options with no command-line flag, see 'man ssh'
      // can be either a single string or an array of strings
      ssh_options: "StrictHostKeyChecking=no",
      // GIT remote/branch
      ref: "origin/master",
      // GIT remote
      repo: "[email protected]:repo_app",
      // path on the server
      path: "/home/dev",
      // Pre-setup command or path to a script on your local machine
      "pre-setup": "ls -la",
      // Post-setup commands or path to a script on the host machine
      // eg: placing configurations in the shared dir etc
      "post-setup": "ls -la",
      // pre-deploy action
      "pre-deploy":"git pull",
      // "pre-deploy-local": "echo 'This is a local executed command'",
      // post-deploy action
      "post-deploy": "npm install && pm2 startOrRestart ecosystem.config.js"
    },
  }
}

To resolve this issue I replace git pull by git fetch --all; git reset --hard origin/master. This will force the pre-deploy to get the master version and discard any conflict occuring with the branch.

This solution is working in my case because I only want to deploy the master branch, however I'm not 100% sure why the conflict occurs in the first place.

Dokku Cannot Deploy (Too Many Docker Containers Exhausting RAM/SSD!)

When we review the main graphs for the DigitalOcean Droplet:
https://cloud.digitalocean.com/droplets/94540890/graphs?i=dc34d1

we see that both Disk (SSD) and Memory (RAM) are near 100% utilisation!

This is super frustrating!

When we run docker ps
to list all the docker processes, we see that there are many:

the docker stats command https://docs.docker.com/engine/reference/commandline/stats/
shows the following for Memory Usage:

This does not appear to "add up" to the total allocated RAM of the VM instance ("droplet") ...
Further investigation required.

Relates to: https://github.com/charterindex/application/issues/79
and https://github.com/charterindex/application/issues/568 ([P1])

Deploying App(s) to Digital Ocean using Dokku

@iteles in case you're wondering why I'm "working late" today ...
I'm working on "Staging" Continuous (Automatic) Deployment for CI. 🚀

But I don't want to create "noise" on the "official" issue, instead I will just create a PR here
(so that other people can benefit from the knowledge)
and then when it's all working I will copy over the "how-to" as "documentation" for the project.

• Writeup how to deploy multiple apps to the same VM/instance
• Setup & Use a Let's Encrypt Wildcard SSL certificate #28
• Create a new App for the current Git branch #30

Diagnosing Failure to Deploy Dokku App to Digital Ocean Instance

The 2GB DigitalOcean VM ("droplet") we are using is running out of memory (RAM) ... 😞
When I run the free command on the instance we see:

Which is bonkers considering we are only running 3 "hello world" style apps on it! (or so I thought...!)

When I run docker ps I see that there are actually 40 Apps running!!

short-hand to count apps: docker ps -q $1 | wc -l via: https://gist.github.com/ggtools/af819efc6b8e3287616c

Most of these Docker containers are "test" apps I created using dokku apps:create <appname> while testing my deployment script(s) ... once I confirmed the app (deployment) was working I would "destroy" the app with the command dokku apps:destroy <appname>
thinking that the underlying Docker container would be destroyed also. But, as it turns out the container is still running!!

So, we need a way of clearing out all the "old" docker containers we no longer need in order to free up the memory on the instance for the Apps we do want to be running!

Insert SSH (RSA) Private Key during Vagrant Configuration

when config.ssh.insert_key = 'true' we see the following error:

so I propose that we do not insert the public key on the instance using Vagarant.
Relates to using Vagrant to deploy VM nodes: #12

Broken Links in readme

Record *detailed* debug log of Vagrant provisioning

If we want to debug the output of the vagrant up command use:

 vagrant up --debug &> vagrant.log

sample output:
vagrant.log.txt

Todo:

• add vagrant.log to .gitignore to avoid "noise".

How to use PM2 to Run Node.js Apps on a VPS

As a node.js developer wanting to deploy a web app to !Heroku
I need a way to deploy and run my App on a "commodity" VPS.

Todo

• Automatically deploy from CI to Digital Ocean VPS (we will demo other providers later)
• Use GitHub Deploy Keys on the Server to git pull from GH.
  see: https://developer.github.com/v3/guides/managing-deploy-keys
• NPM install the latest dependencies
• Use PM2 to restart the Server once latest dependencies installed.

Reading

• https://mrvautin.com/running-nodejs-applications-in-production-forever-vs-supervisord-vs-pm2/

Deploy master branch to Production and any other branch to Staging

It's occurred to me that edeliver is currently only configured to deploy the master branch and only deploys to a single environment.
Could we write a script that checks the branch and sets the target deployment environment to production if the branch is master and deploy to staging for every other branch?

Why? What? Who? How?

Why?

You have "out-grown" Heroku (or more specifically) you're reached the limit of Heroku's value-for-money and want to get way more power for the money you are spending on infrastructure.

Most companies/organisations will never reach this point. Most organisations only have a couple of thousand "users" and for that Heroku is great because the cost will be below $50/month, which means hiring a "DevOps" consultant to do an "advanced deploy" will take 1-2 year(s) to pay off...
The teams who already have a few thousand active users
or companies who are replacing/updating an existing system that has several thousand users,
will benefit from "advanced" deployment

What?

• Continuous Integration
• Deployment
• Security & Secrets
• Clustering, Load Balancing and "Failover"
• Monitoring
• Logging (ELK)?
• Alerts

Who?

Developers (or Technology-savvy Entrepreneurs) who realise that having an intimate understanding of the infrastructure their application is running on will give them a
real advantage.

How?

• Amazon Web Services: https://github.com/dwyl/learn-amazon-web-services
• Azure: https://github.com/dwyl/learn-microsoft-azure
• Digital Ocean: https://github.com/dwyl/DigitalOcean-Setup

cURL for *just* the HTTP Status Code

as part of "Dynamic CI" we need to know if an app has already been deployed
to avoid re-creating it if it exists. the simplest way I can think of for doing this is to

curl https://branch-name.ademo.app

and, based on the HTTP Status Code, decide if we need to create a new app!

Upgrade Linode Instance to Avoid 502 on Hits

Todo

• Login to Linode
  • Increase instance from current $5/month to $10/month (double disk space)
• ssh to restart the instance/app
• test service has resumed

related to dwyl/hits#62

How to use Deploy Keys to *git pull* from GitHub

At present our deployment method requires us to have an (encrypted) RSA key to perform deployment from Travis-CI/CircleCI/Codeship etc.
This is undesirable because it means we have to "Trust" the CI-as-a-Service provider to not lose their private keys (which can decrypt our RSA key...)
Instead of git push from CI we could git pull from the deployment server using a Deploy Key:
https://developer.github.com/v3/guides/managing-deploy-keys/

! [remote rejected] master -> master (pre-receive hook declined)

While attempting to deploy from Travis-CI using SSH to a Digital Ocean Server with Dokku,
We are seeing the following error:

 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:13'

https://travis-ci.org/nelsonic/hello-world-node-http-server/builds/384311558#L549-L550

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

when attempting to delete a Dokku app ("hello") with the command:

 sudo dokku apps:destroy hello

I get the following error:

 !     WARNING: Potentially Destructive Action
 !     This command will destroy hello (including all add-ons).
 !     To proceed, type "hello"

> hello
Destroying hello (including all add-ons)
App image (dokku/hello:latest) not found
App hello has not been deployed
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Add (encrypted) SSH Key to Travis-CI

in order to deploy "review apps" from Travis-CI we need to:

• add an encrypted SSH key to the .travis.yml file
  thankfully, I've done this before (and docummented it): https://github.com/healthlocker/healthlocker/blob/a57d2741ae703b45ac8eb8465305a6e327baa152/continuous-deployment.md#encrypted-ssh-key-for-deployment
  so it should be reasonably "easy" (there are a couple more steps this time...)
• ensure that the travis user has the necessary permissions to perform the deploy.
• do a "dry run" using: https://github.com/nelsonic/hello-world-node-http-server/blob/b4b49a11cb7839af5b7449cdb425553e1b61e9f2/.bin/deploy.sh#L1-L26
• Update: https://stackoverflow.com/questions/27444891/how-to-add-ssh-key-in-travis-ci

Read Git Branch on PR

As part of building "Dynamic CI" #24 to allow us to deploy apps with a single command we need to be able to read the Git Branch.

• read git branch on Travis-CI

Similar to: dwyl/learn-elixir#53

error: failed to push some refs to 'dokku@...'

-----> Build succeeded!
-----> Discovering process types
       Default types for  -> web
-----> Releasing hello (dokku/hello:latest)...
-----> Deploying hello (dokku/hello:latest)...
-----> Attempting to run scripts.dokku.predeploy from app.json (if defined)
-----> App Procfile file found (/home/dokku/hello/DOKKU_PROCFILE)
-----> DOKKU_SCALE file not found in app image. Generating one based on Procfile...
-----> New DOKKU_SCALE file generated
=====> web=1
-----> Attempting pre-flight checks
       For more efficient zero downtime deployments, create a file CHECKS.
       See http://dokku.viewdocs.io/dokku/deployment/zero-downtime-deploys/ for examples
       CHECKS file not found in container: Running simple container check...
-----> Waiting for 10 seconds ...
-----> Default container check successful!
-----> Running post-deploy
-----> Creating new /home/dokku/hello/VHOST...
-----> Setting config vars
       DOKKU_PROXY_PORT:  80
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:5000
-----> Configuring hello....(using built-in template)
-----> Configuring hello.ademo.app...(using built-in template)
-----> Creating http nginx.conf
-----> Running nginx-pre-reload
       Reloading nginx
remote: Job for nginx.service invalid.
To 138.68.163.126:hello
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:hello'

** (FunctionClauseError) no function clause matching in Postgrex.Messages.decode_fields/1

11:20:21.091 [error] GenServer #PID<0.438.0> terminating
** (FunctionClauseError) no function clause matching in Postgrex.Messages.decode_fields/1
    (postgrex) lib/postgrex/messages.ex:339: Postgrex.Messages.decode_fields("")
    (postgrex) lib/postgrex/messages.ex:344: Postgrex.Messages.decode_fields/1
    (postgrex) lib/postgrex/messages.ex:344: Postgrex.Messages.decode_fields/1
    (postgrex) lib/postgrex/messages.ex:131: Postgrex.Messages.parse/3
    (postgrex) lib/postgrex/protocol.ex:1842: Postgrex.Protocol.msg_decode/1
    (postgrex) lib/postgrex/protocol.ex:1816: Postgrex.Protocol.msg_recv/3
    (postgrex) lib/postgrex/protocol.ex:560: Postgrex.Protocol.auth_recv/3
    (postgrex) lib/postgrex/protocol.ex:475: Postgrex.Protocol.handshake/2
Last message: nil

fatal: protocol error: expected old/new/ref, got 'shallow

while attempting to git push from Travis to Dokku PaaS for #24
I'm seeing the following error:

fatal: protocol error: expected old/new/ref, got 'shallow 5d269f1b1dabb757134e0efea85055f755e2c482'
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly

See: https://travis-ci.org/nelsonic/hello-world-node-http-server/builds/384299693#L537

SSH

as a person learning how to do "DevOps"
I need to know how to use SSH to login to a remote machine
and execute commands e.g. installing dependencies

Todo

• How to setup SSH on your computer (configure RSA PSK)
• breakdown the parts of an ssh command.
• illustrate with examples
• when to use sudo ?
• how not to use ssh
• good habits when using ssh (record all your commands!!)

Dokku: "We're sorry, but something went wrong"

At present when the App is being updated, we see:

The template is this:
https://github.com/dokku/dokku/blob/master/plugins/nginx-vhosts/templates/500-error.html

ideally this should be a more friendly message such as:
"A New Version of the App is currently being deployed, please check back in a few minutes. Thanks!"

build tools issue?

I am trying to deploy to amazon ec2 when i run the mix edeliver build release --verbose, i am getting error like follow

-----> Authorizing hosts
-----> Ensuring hosts are ready to accept git pushes
A remote command failed on:
[email protected]
Output of the command is shown above and the command executed
on that host is printed below for debugging purposes:
FAILED with exit status 255:
set -e
if [ ! -d /home/ec2-user/xxxx/builds ]
then
mkdir -p /home/ec2-user/xxxx/builds
cd /home/ec2-user/xxxx/builds
git init
git config receive.denyCurrentBranch ignore
else
cd /home/ec2-user/xxxx/builds
git config receive.denyCurrentBranch ignore
fi

Investigate options for Private Key / Certificate / Environment Variable Management

At present, provisioning a VM instance requires several manual steps. Specifically:

• SSL/TLS Certificate Creation
• SSL/TLS Certificate location (on the file system)
• Updating the NGiNX config file to reflect the location of the Certificate see: nginx.config#L37

These manual steps ok initially because "automation" is _expensive when uncertainty is high (e.g. in a "startup") read: http://paulgraham.com/ds.html

However I counter that automating the steps (using scripts) is not that much more work than documenting them (in words for humans to read).

Possibly invest time in https://github.com/dwyl/akey ? ⏳ / 💰
Which would be a general purpose solution to the problem ... 🤔

Get List of users on Linux/Unix VPS/VM Instance

What is the command to get the the list of users on a Linux Virtual Machine (VM) instance?

aparently:

getent passwd

Not a very descriptive command ...
basically getent is "get entries" https://en.wikipedia.org/wiki/Getent
in the passwd file which contains a list of all the users.

via: https://askubuntu.com/questions/410244/a-command-to-list-all-users-and-how-to-add-delete-modify-users