devopsloft/dpline

Out of the box Delivery Pipeline

DevOps Loft

Detailed prerequisites and instructions for spinning DEV/STAGE/PROD environments

DEV environment

Prerequisites

• Install Docker (version 19.03.xx or higher)
• Install Docker Compose (version 1.25.5 or higher)
• AWS account
• AWS Profile
• Create an envfile '.env.dev' from the example '.env.dev.example'
• Chrome - Allows requests to localhost over HTTPS even when an invalid certificate is presented. `chrome://flags/#allow-insecure-localhost`

Spin DEV environment

Execute the following commands:

1. make up
2. Browse: https://localhost:8443

Teardown DEV environment

Execute the following commands:

1. ./build/build.sh dev
2. source .env.dev
3. docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ${NAMESPACE}/spinner:latest ./spin-docker.py --action destroy
4. docker image prune -af
5. docker volume prune -f

---

STAGE environment

Prerequisites

• Dockerhub account
• AWS account
• AWS Profile
• Terraform
• Docker
• Docker Compose
• AWS ~/.aws or %UserProfile%\.aws folder
• subnet ID
• Security Group with inbound ports for SSH (22), HTTP (80), HTTPS (443), and 8200
• AWS S3 Bucket
• Create an envfile '.env.stage' from the example '.env.stage.example'

Spin STAGE environment

Execute the following:

1. ./build/build.sh stage
2. source .env.stage
3. terraform -chdir=deploy init
4. terraform -chdir=deploy apply --var-file=aws-stage.tfvars
5. docker run --rm -v $HOME/.aws:/root/.aws -v /var/run/docker.sock:/var/run/docker.sock ${NAMESPACE}/spinner:latest ./spin-docker.py --environment $ENVIRONMENT
6. Locate the EC2 instance Public DNS: AWS Consule->EC2->Insance->Public DNS (IPv4)
7. Browse

Teardown STAGE environment

Execute the following:

1. ./build/build.sh stage
2. docker run --rm -v ~/.aws:/root/.aws -v /var/run/docker.sock:/var/run/docker.sock ${NAMESPACE}/spinner:latest ./spin-docker.py --environment $ENVIRONMENT --action destroy
3. docker image prune -af

---

PROD environment

Prerequisites

• Dockerhub account
• AWS account
• AWS Profile
• AWS ~/.aws or %UserProfile%\.aws folder
• keypair
• subnet ID
• Security Group with inbound ports for SSH (22), HTTP (80), HTTPS (443), and 8200
• AWS S3 Bucket
• Elastic IP Address (EIP)
• Create an envfile '.env.prod' from the example '.env.prod.example'

Teardown PROD environment

Execute the following:

1. ./build/build.sh prod
2. source .env.prod
3. docker run --rm -v ~/.aws:/root/.aws -v /var/run/docker.sock:/var/run/docker.sock ${NAMESPACE}/spinner:latest ./spin-docker.py --environment $ENVIRONMENT --action destroy
4. docker image prune -af

Spin PROD environment

Execute the following:

1. ./build/build.sh prod
2. terraform -chdir=deploy init
3. terraform apply -chdir=deploy --var-file=aws-prod.tfvars
4. docker run --rm -v $HOME/.aws:/root/.aws -v /var/run/docker.sock:/var/run/docker.sock ${NAMESPACE}/spinner:latest ./spin-docker.py --environment $ENVIRONMENT
5. Browse www.devopsloft.org

dpline's Issues

[epic-Continuous Integration]

jenkins is the tool

Fix the CI build label in the README

deploy_dpline.sh breaks on OSX

host OS - OSX

./deploy_dpline.sh

• virtualenv --system-site-packages /tmp/dpline_venv
  New python executable in /tmp/dpline_venv/bin/python
  Installing setuptools, pip, wheel...done.
• source /tmp/dpline_venv/bin/activate
  ++ deactivate nondestructive
  ++ unset -f pydoc
  ++ '[' -z '' ']'
  ++ '[' -z '' ']'
  ++ '[' -n /bin/bash ']'
  ++ hash -r
  ++ '[' -z '' ']'
  ++ unset VIRTUAL_ENV
  ++ '[' '!' nondestructive = nondestructive ']'
  ++ VIRTUAL_ENV=/tmp/dpline_venv
  ++ export VIRTUAL_ENV
  ++ _OLD_VIRTUAL_PATH=/Users/liora/git/lioramilbaum/devopsloft/venv/bin:/Users/liora/.minishift/cache/oc/v3.11.0/darwin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Wireshark.app/Contents/MacOS
  ++ PATH=/tmp/dpline_venv/bin:/Users/liora/git/lioramilbaum/devopsloft/venv/bin:/Users/liora/.minishift/cache/oc/v3.11.0/darwin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Wireshark.app/Contents/MacOS
  ++ export PATH
  ++ '[' -z '' ']'
  ++ '[' -z '' ']'
  ++ _OLD_VIRTUAL_PS1=
  ++ '[' x '!=' x ']'
  +++ basename /tmp/dpline_venv
  ++ PS1='(dpline_venv) '
  ++ export PS1
  ++ alias pydoc
  ++ '[' -n /bin/bash ']'
  ++ hash -r
• pip install ansible
  Collecting ansible
  Using cached https://files.pythonhosted.org/packages/db/3e/04d67f7798432542dcf0258b59634f4f60f1f9ba9505cc71d69af88673d6/ansible-2.7.6.tar.gz
  Collecting jinja2 (from ansible)
  Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
  Collecting PyYAML (from ansible)
  Using cached https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz
  Collecting paramiko (from ansible)
  Using cached https://files.pythonhosted.org/packages/cf/ae/94e70d49044ccc234bfdba20114fa947d7ba6eb68a2e452d89b920e62227/paramiko-2.4.2-py2.py3-none-any.whl
  Collecting cryptography (from ansible)
  Downloading https://files.pythonhosted.org/packages/76/69/3464878ad1211f1f842e89217d907645849c5bfe20e20c695a1d7e4a7c88/cryptography-2.4.2-cp27-cp27m-macosx_10_6_intel.whl (1.5MB)
  100% |████████████████████████████████| 1.5MB 1.5MB/s
  Requirement already satisfied: setuptools in /private/tmp/dpline_venv/lib/python2.7/site-packages (from ansible) (40.6.3)
  Collecting MarkupSafe>=0.23 (from jinja2->ansible)
  Downloading https://files.pythonhosted.org/packages/cd/52/927263d9cf66a12e05c5caef43ee203bd92355e9a321552d2b8c4aee5f1e/MarkupSafe-1.1.0-cp27-cp27m-macosx_10_6_intel.whl
  Collecting pynacl>=1.0.1 (from paramiko->ansible)
  Downloading https://files.pythonhosted.org/packages/51/83/2db5b919bf9848fe25d301225a16faabc378419e7eaf00da0b7d200fe801/PyNaCl-1.3.0-cp27-cp27m-macosx_10_6_intel.whl (283kB)
  100% |████████████████████████████████| 286kB 1.1MB/s
  Requirement already satisfied: pyasn1>=0.1.7 in /Library/Python/2.7/site-packages (from paramiko->ansible) (0.1.9)
  Collecting bcrypt>=3.1.3 (from paramiko->ansible)
  Downloading https://files.pythonhosted.org/packages/14/1b/f6f05b80783ff84c58bca5f2b7a4cef7c02570e4bb2e40d326abb2ba41d3/bcrypt-3.1.6-cp27-cp27m-macosx_10_6_intel.whl (53kB)
  100% |████████████████████████████████| 61kB 2.7MB/s
  Collecting idna>=2.1 (from cryptography->ansible)
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
  100% |████████████████████████████████| 61kB 512kB/s
  Collecting enum34; python_version < "3" (from cryptography->ansible)
  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
  Collecting cffi!=1.11.3,>=1.7 (from cryptography->ansible)
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)",)': /simple/cffi/
  Downloading https://files.pythonhosted.org/packages/7e/4a/b647e46faaa2dcfb16069b6aad2d8509982fd63710a325b8ad7db80f18be/cffi-1.11.5-cp27-cp27m-macosx_10_6_intel.whl (238kB)
  100% |████████████████████████████████| 245kB 666kB/s
  Collecting asn1crypto>=0.21.0 (from cryptography->ansible)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
  Collecting ipaddress; python_version < "3" (from cryptography->ansible)
  Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
  Requirement already satisfied: six>=1.4.1 in /Library/Python/2.7/site-packages (from cryptography->ansible) (1.10.0)
  Collecting pycparser (from cffi!=1.11.3,>=1.7->cryptography->ansible)
  Building wheels for collected packages: ansible, PyYAML
  Running setup.py bdist_wheel for ansible ... done
  Stored in directory: /Users/liora/Library/Caches/pip/wheels/db/aa/61/9ec7362320b5db8d37593b35848c7a726574360f264bfeb4ea
  Running setup.py bdist_wheel for PyYAML ... done
  Stored in directory: /Users/liora/Library/Caches/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f
  Successfully built ansible PyYAML
  Installing collected packages: MarkupSafe, jinja2, PyYAML, idna, enum34, pycparser, cffi, asn1crypto, ipaddress, cryptography, pynacl, bcrypt, paramiko, ansible
  Successfully installed MarkupSafe-1.1.0 PyYAML-3.13 ansible-2.7.6 asn1crypto-0.24.0 bcrypt-3.1.6 cffi-1.11.5 cryptography-2.4.2 enum34-1.1.6 idna-2.8 ipaddress-1.0.22 jinja2-2.10 paramiko-2.4.2 pycparser-2.19 pynacl-1.3.0
• ansible-playbook setup/prepare_host.yml
  [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Setup Host] **************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************
ok: [localhost]

TASK [install_docker : Install Docker community version] ***********************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "sudo: a password is required\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
to retry, use: --limit @/Users/liora/git/dpline/dpline/setup/prepare_host.retry

PLAY RECAP *********************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=1

[Task][Dpline] Add CLI

Add proper CLI like:

dpline --deploy

or

dpline --setup

Probably using Python

vagrant - switch to fedora or centos box

Use packer vagrant builder - template

hashicorp/packer#8239

docker network is required for sub-services integration

[Task][CI] Add basic CI coverage for the project

Dpline is going to be mostly Python and Ansible. We should start adding CI tests based on these technologies.

Use sample dataset for the kibana dashboard in dev environment

When I spin a dev environment, I want to use a test dataset when viewing the dashboard in kibana

[epic-deployment]

[epic-Continuous Integration][story-Deploy Jenkins][task-Jenkins deployment script]

Use one command to deploy Jenkins based on Jenkins docker image

[epic-deployment][story-deploy Jenkins,RabbitMQ,Prometheus][task-deploy script]

adding all sub-services to the deploy script

[epic-Continuous Integration][story-Deploy Jenkins][task-Jenkins avoid running setup wizard]

[epic-Incident Management] [story-Choose a tool]task-Consider Jira]

#1
#2

[epic-Incident Management][story-Choose a tool]

Go over the existing tools in the market with the following requirements:

• open source

• already used in Red Hat

• large echo system

I understand that Jira/Bugzilla/Trello were not meant to be used as incident management services. We can consider using one of them for MVP as a starter to on-board the first customer as soon as possible.

[Task][Dpline] Allow to customize delivery pipeline

Let the user pick system and don't choose for him.
User will be able to define the following yaml file:

---

ci: jenkins
messaging: rabbitmq
monitoring: promethus

and based on his choice, we will deploy a delivery pipeline

[epic-deployment][story-deploy Jenkins,RabbitMQ,Prometheus]

[Task][CLI] Allow to remove Dpline services

Right now you can use the CLI only to deploy, not to remove.

Ability to deploy selective part of micro-services

[Task][Jenkins] Setup instance with sample jobs

Deploy instance with some sample jobs in order to to be able to test the delivery pipeline

Note to future me (or future contributors): add --demo flag which will control it.

[Task] Allow to deploy directly on the environment

VM deployment is SLOW.
If user wants to deploy directly on the host, allow him to do it.

Jenkins - Install Rabbit-MQ Publisher and JDK Tool plugins

[epic-Continuous Integration][story-Deploy Jenkins]

Automate Jenkins Deployment

[task-Jenkins/RabbitMQ integration]

install Jenkins plugin
Configure jenkins to know rabbitmq
Deploy a job which send notifications to rabbitmq
Run the job
Verify rabbitmq received the messages

everything automated

[epic-Incident Management] [story- Prometheus][task-deploy]

[epic-deployment][story-deploy Jenkins,RabbitMQ,Prometheus][task-Deploy RabbitMQ]

[epic-Incident Management]

Delivery Pipeline is a multi sub-services service. Each sub-service may through alerts when hitting a failure or close to hitting one.

The alerts should be pushed to a one and only one queue. This is the first sub-service.

The queue should be able to:

• Consume the alerts and transform them to incidents

• Analyze the alerts - removing false/positives and aggregation of similar alerts

• Tracking incidents are resolved according to SLA

• Remediation by automation

fix docker install playbook

environment spawning - ./dpline-env.sh

docker install playbook is broken

[Task][Dpline] Print access summary

Once Dpline deployment is done, print out a summary to the user of the systems deployed and how to access them.

For example:

Jenkins 1.2.3, Address: http://localhost:8080
Prometheus 2.3.1, Address: http:/localhost:5050