devopsloft/dpline

Out of the box Delivery Pipeline

dpline's Issues

[epic-Incident Management][story-Choose a tool]

Go over the existing tools in the market with the following requirements:

• open source

• already used in Red Hat

• large echo system

I understand that Jira/Bugzilla/Trello were not meant to be used as incident management services. We can consider using one of them for MVP as a starter to on-board the first customer as soon as possible.

[Task][Dpline] Add CLI

Add proper CLI like:

dpline --deploy

or

dpline --setup

Probably using Python

[Task][CI] Add basic CI coverage for the project

Dpline is going to be mostly Python and Ansible. We should start adding CI tests based on these technologies.

Fix the CI build label in the README

[Task][CLI] Allow to remove Dpline services

Right now you can use the CLI only to deploy, not to remove.

Use packer vagrant builder - template

hashicorp/packer#8239

Jenkins - Install Rabbit-MQ Publisher and JDK Tool plugins

[task-Jenkins/RabbitMQ integration]

install Jenkins plugin
Configure jenkins to know rabbitmq
Deploy a job which send notifications to rabbitmq
Run the job
Verify rabbitmq received the messages

everything automated

[epic-Incident Management]

Delivery Pipeline is a multi sub-services service. Each sub-service may through alerts when hitting a failure or close to hitting one.

The alerts should be pushed to a one and only one queue. This is the first sub-service.

The queue should be able to:

• Consume the alerts and transform them to incidents

• Analyze the alerts - removing false/positives and aggregation of similar alerts

• Tracking incidents are resolved according to SLA

• Remediation by automation

vagrant - switch to fedora or centos box

docker network is required for sub-services integration

[epic-deployment]

[Task][Jenkins] Setup instance with sample jobs

Deploy instance with some sample jobs in order to to be able to test the delivery pipeline

Note to future me (or future contributors): add --demo flag which will control it.

Use sample dataset for the kibana dashboard in dev environment

When I spin a dev environment, I want to use a test dataset when viewing the dashboard in kibana

[epic-Continuous Integration]

jenkins is the tool

[epic-Continuous Integration][story-Deploy Jenkins][task-Jenkins deployment script]

Use one command to deploy Jenkins based on Jenkins docker image

[epic-Incident Management] [story- Prometheus][task-deploy]

fix docker install playbook

environment spawning - ./dpline-env.sh

docker install playbook is broken

[Task][Dpline] Allow to customize delivery pipeline

Let the user pick system and don't choose for him.
User will be able to define the following yaml file:

---

ci: jenkins
messaging: rabbitmq
monitoring: promethus

and based on his choice, we will deploy a delivery pipeline

[epic-Continuous Integration][story-Deploy Jenkins]

Automate Jenkins Deployment

[epic-deployment][story-deploy Jenkins,RabbitMQ,Prometheus]

[epic-Incident Management] [story-Choose a tool]task-Consider Jira]

#1
#2

deploy_dpline.sh breaks on OSX

host OS - OSX

./deploy_dpline.sh

• virtualenv --system-site-packages /tmp/dpline_venv
  New python executable in /tmp/dpline_venv/bin/python
  Installing setuptools, pip, wheel...done.
• source /tmp/dpline_venv/bin/activate
  ++ deactivate nondestructive
  ++ unset -f pydoc
  ++ '[' -z '' ']'
  ++ '[' -z '' ']'
  ++ '[' -n /bin/bash ']'
  ++ hash -r
  ++ '[' -z '' ']'
  ++ unset VIRTUAL_ENV
  ++ '[' '!' nondestructive = nondestructive ']'
  ++ VIRTUAL_ENV=/tmp/dpline_venv
  ++ export VIRTUAL_ENV
  ++ _OLD_VIRTUAL_PATH=/Users/liora/git/lioramilbaum/devopsloft/venv/bin:/Users/liora/.minishift/cache/oc/v3.11.0/darwin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Wireshark.app/Contents/MacOS
  ++ PATH=/tmp/dpline_venv/bin:/Users/liora/git/lioramilbaum/devopsloft/venv/bin:/Users/liora/.minishift/cache/oc/v3.11.0/darwin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Wireshark.app/Contents/MacOS
  ++ export PATH
  ++ '[' -z '' ']'
  ++ '[' -z '' ']'
  ++ _OLD_VIRTUAL_PS1=
  ++ '[' x '!=' x ']'
  +++ basename /tmp/dpline_venv
  ++ PS1='(dpline_venv) '
  ++ export PS1
  ++ alias pydoc
  ++ '[' -n /bin/bash ']'
  ++ hash -r
• pip install ansible
  Collecting ansible
  Using cached https://files.pythonhosted.org/packages/db/3e/04d67f7798432542dcf0258b59634f4f60f1f9ba9505cc71d69af88673d6/ansible-2.7.6.tar.gz
  Collecting jinja2 (from ansible)
  Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
  Collecting PyYAML (from ansible)
  Using cached https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz
  Collecting paramiko (from ansible)
  Using cached https://files.pythonhosted.org/packages/cf/ae/94e70d49044ccc234bfdba20114fa947d7ba6eb68a2e452d89b920e62227/paramiko-2.4.2-py2.py3-none-any.whl
  Collecting cryptography (from ansible)
  Downloading https://files.pythonhosted.org/packages/76/69/3464878ad1211f1f842e89217d907645849c5bfe20e20c695a1d7e4a7c88/cryptography-2.4.2-cp27-cp27m-macosx_10_6_intel.whl (1.5MB)
  100% |████████████████████████████████| 1.5MB 1.5MB/s
  Requirement already satisfied: setuptools in /private/tmp/dpline_venv/lib/python2.7/site-packages (from ansible) (40.6.3)
  Collecting MarkupSafe>=0.23 (from jinja2->ansible)
  Downloading https://files.pythonhosted.org/packages/cd/52/927263d9cf66a12e05c5caef43ee203bd92355e9a321552d2b8c4aee5f1e/MarkupSafe-1.1.0-cp27-cp27m-macosx_10_6_intel.whl
  Collecting pynacl>=1.0.1 (from paramiko->ansible)
  Downloading https://files.pythonhosted.org/packages/51/83/2db5b919bf9848fe25d301225a16faabc378419e7eaf00da0b7d200fe801/PyNaCl-1.3.0-cp27-cp27m-macosx_10_6_intel.whl (283kB)
  100% |████████████████████████████████| 286kB 1.1MB/s
  Requirement already satisfied: pyasn1>=0.1.7 in /Library/Python/2.7/site-packages (from paramiko->ansible) (0.1.9)
  Collecting bcrypt>=3.1.3 (from paramiko->ansible)
  Downloading https://files.pythonhosted.org/packages/14/1b/f6f05b80783ff84c58bca5f2b7a4cef7c02570e4bb2e40d326abb2ba41d3/bcrypt-3.1.6-cp27-cp27m-macosx_10_6_intel.whl (53kB)
  100% |████████████████████████████████| 61kB 2.7MB/s
  Collecting idna>=2.1 (from cryptography->ansible)
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
  100% |████████████████████████████████| 61kB 512kB/s
  Collecting enum34; python_version < "3" (from cryptography->ansible)
  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
  Collecting cffi!=1.11.3,>=1.7 (from cryptography->ansible)
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)",)': /simple/cffi/
  Downloading https://files.pythonhosted.org/packages/7e/4a/b647e46faaa2dcfb16069b6aad2d8509982fd63710a325b8ad7db80f18be/cffi-1.11.5-cp27-cp27m-macosx_10_6_intel.whl (238kB)
  100% |████████████████████████████████| 245kB 666kB/s
  Collecting asn1crypto>=0.21.0 (from cryptography->ansible)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
  Collecting ipaddress; python_version < "3" (from cryptography->ansible)
  Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
  Requirement already satisfied: six>=1.4.1 in /Library/Python/2.7/site-packages (from cryptography->ansible) (1.10.0)
  Collecting pycparser (from cffi!=1.11.3,>=1.7->cryptography->ansible)
  Building wheels for collected packages: ansible, PyYAML
  Running setup.py bdist_wheel for ansible ... done
  Stored in directory: /Users/liora/Library/Caches/pip/wheels/db/aa/61/9ec7362320b5db8d37593b35848c7a726574360f264bfeb4ea
  Running setup.py bdist_wheel for PyYAML ... done
  Stored in directory: /Users/liora/Library/Caches/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f
  Successfully built ansible PyYAML
  Installing collected packages: MarkupSafe, jinja2, PyYAML, idna, enum34, pycparser, cffi, asn1crypto, ipaddress, cryptography, pynacl, bcrypt, paramiko, ansible
  Successfully installed MarkupSafe-1.1.0 PyYAML-3.13 ansible-2.7.6 asn1crypto-0.24.0 bcrypt-3.1.6 cffi-1.11.5 cryptography-2.4.2 enum34-1.1.6 idna-2.8 ipaddress-1.0.22 jinja2-2.10 paramiko-2.4.2 pycparser-2.19 pynacl-1.3.0
• ansible-playbook setup/prepare_host.yml
  [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Setup Host] **************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************
ok: [localhost]

TASK [install_docker : Install Docker community version] ***********************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "sudo: a password is required\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
to retry, use: --limit @/Users/liora/git/dpline/dpline/setup/prepare_host.retry

PLAY RECAP *********************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=1

[epic-deployment][story-deploy Jenkins,RabbitMQ,Prometheus][task-deploy script]

adding all sub-services to the deploy script

[Task] Allow to deploy directly on the environment

VM deployment is SLOW.
If user wants to deploy directly on the host, allow him to do it.

[epic-deployment][story-deploy Jenkins,RabbitMQ,Prometheus][task-Deploy RabbitMQ]

Ability to deploy selective part of micro-services

[Task][Dpline] Print access summary

Once Dpline deployment is done, print out a summary to the user of the systems deployed and how to access them.

For example:

Jenkins 1.2.3, Address: http://localhost:8080
Prometheus 2.3.1, Address: http:/localhost:5050

[epic-Continuous Integration][story-Deploy Jenkins][task-Jenkins avoid running setup wizard]