osthanes/activedeploy_common

Active Deploy common initialization for activedeploy_start and activedeploy_finish projects *** Please note: *** This service is being deprecated. For more information about the deprecation of this service, see https://www.ibm.com/blogs/bluemix/2017/05/retirement-ibm-active-deploy-beta-service/

activedeploy_common's Issues

Active Deploy unmaps route of wrong app

@ptaube
The Active Deploy - Begin deployer type scripts in for the delivery pipelines for containers unmaps the route of a wrong app. I have two apps: appetizers and desserts and if I map the route used for the dessert app to the appetizer app, the AD - Begin script will consider the appetizer app as an older version of the dessert app when I kick off the dessert delivery pipeline. It does so because the script looks at apps mapped with dessert's route without considering the PATTERN (as it appropriately does in the rampdown for deleting apps). As a result, the appetizer app will get its route unmapped. Also, the appetizer app is used for naming the AD instance as well as the original_gr_id.

I believe that a user should be able to use a common route for different apps and yet AD should upgrade apps appropriately without unmapping the route of any of them when a delivery pipeline for one of the apps is kicked off.

Please have a look at the log and specially to the bold text:

++ logDebug 'wfd-dessert-ic-ad-1479989940988.mybluemix.net is routed to wfd-dessert-ic-ad_4' wfd-appetizer-ic-ad_2
++ [[ -n 3 ]]
++ [[ 3 -ge 4 ]]
++ echo wfd-dessert-ic-ad_4 wfd-appetizer-ic-ad_2

• logDebug 2 of original groups routed to wfd-dessert-ic-ad-1479989940988.mybluemix.net: wfd-dessert-ic-ad_4 wfd-appetizer-ic-ad_2
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• (( 1 < 2 ))
• logWarning 'More than one app routed to wfd-dessert-ic-ad-1479989940988.mybluemix.net; updating the oldest'
• [[ -n 3 ]]
• [[ 3 -ge 2 ]]
• echo -e '\e[0;33m--- WARNING: More than one app routed to wfd-dessert-ic-ad-1479989940988.mybluemix.net; updating the oldest\e[0m'
  --- WARNING: More than one app routed to wfd-dessert-ic-ad-1479989940988.mybluemix.net; updating the oldest
• (( 0 < 2 ))
• readarray -t srtd
  ++ for e in '"${ROUTED[@]}"'
  ++ echo wfd-dessert-ic-ad_4
  ++ for e in '"${ROUTED[@]}"'
  ++ echo wfd-appetizer-ic-ad_2
  ++ sort
• **original_grp=wfd-appetizer-ic-ad_2
• original_grp_id=2
• logDebug 'Original_grp: wfd-appetizer-ic-ad_2 - Original_grp_id: 2'**
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• [[ 1 = 7 ]]
• [[ -z wfd-appetizer-ic-ad_2 ]]
• logInfo 'Not initial version'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: Not initial version\e[0m'
  --- INFO: Not initial version
• [[ -n '' ]]
• **successor_grp=wfd-dessert-ic-ad_5
• logDebug 'Original group is wfd-appetizer-ic-ad_2 (2)'**
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• logDebug 'Successor group is wfd-dessert-ic-ad_5 (5)'
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• [[ -n wfd-appetizer-ic-ad_2 ]]
• [[ mybluemix.net == mybluemix.net ]]
• logInfo 'ROUTE_DOMAIN is: mybluemix.net'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: ROUTE_DOMAIN is: mybluemix.net\e[0m'
  --- INFO: ROUTE_DOMAIN is: mybluemix.net
• logInfo 'check if AD instance exists with cf services, if not create it.'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: check if AD instance exists with cf services, if not create it.\e[0m'
  --- INFO: check if AD instance exists with cf services, if not create it.
• cf services
• grep activedeploy
  ++ cat mp.output
• foundservice='wfd_Active_Deploy activedeploy free update succeeded'
• [[ -z wfd_Active_Deploy activedeploy free update succeeded ]]
• logInfo 'Found Active Deploy Instance: activedeploy-for-pipeline'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: Found Active Deploy Instance: activedeploy-for-pipeline\e[0m'
  --- INFO: Found Active Deploy Instance: activedeploy-for-pipeline
• logInfo 'Beginning update with cf active-deploy-create ...'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: Beginning update with cf active-deploy-create ...\e[0m'
  --- INFO: Beginning update with cf active-deploy-create ...
• create_args='wfd-appetizer-ic-ad_2 wfd-dessert-ic-ad_5 --manual --quiet --timeout 60s'
• [[ -n 1s ]]
• create_args='wfd-appetizer-ic-ad_2 wfd-dessert-ic-ad_5 --manual --quiet --timeout 60s --rampup 1s'

Thanks

Bug in Active Deploy Clean function

@BirgitHeubach @ptaube Working on setting up a delivery pipeline with the Active Deploy scripts that come out of the box to create a red/black deployment for a container based application, I have found that the clean function do not filter container group names appropriately based on the PATTERN.
That is, I have a container group called wfd-appetizer-ac_2 and the active deploy scripts were deploying a newer version wfd-appetizer-ac_3. During the Active Deploy - Complete stage (after the test phase), the active deploy scripts try to figure out what is the most recent version of the application and then delete older versions of it based on the PATTERN variable and the CONCURRENT_VERSIONS.
The clean function calls the groupList function in order to get a list of all application versions. Then, it will delete the oldest versions based on CONCURRENT_VERSIONS value.

However, the groupList returns all container groups. As a result, incorrect version number is used for determining what versions to delete.

Please see log below: (check bold text)

• clean
  ++ echo wfd-appetizer-ac_3
  ++ rev
  ++ cut -d_ -f2-
  ++ rev
• PATTERN=wfd-appetizer-ac
  ++ echo wfd-appetizer-ac_3
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• VERSION=3
• unset IFS
• candidates=($(groupList))
  ++ groupList
  ++ __baseUrl=https://containers-api.ng.bluemix.net/v3/containers
  ++ python -
  2016-11-14 15:20:13,543 - ccs - DEBUG - [30] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'Accept: application/json;charset=utf-8' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X GET 'https://containers-api.ng.bluemix.net/v3/containers/groups'
  2016-11-14 15:20:13,559 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:13,581 - urllib3.connectionpool - DEBUG - Setting read timeout to 30
  2016-11-14 15:20:20,241 - urllib3.connectionpool - DEBUG - "GET /v3/containers/groups HTTP/1.1" 200 4526
  groupList: [{u'Status': u'CREATE_COMPLETE', u'Name': u'wfd-appetizer-ac_3', u'Creation_time': u'2016-11-14T15:06:54', u'Id': u'5a2f161f-4863-4458-84f6-892fb5448136', u'Env': None, u'Routes': [u'wfd-appetizer-ac.mybluemix.net'], u'Updated_time': None, u'Port': 8082}, {u'Status': u'UPDATE_COMPLETE', u'Name': u'wfd-appetizer-ac_2', u'Creation_time': u'2016-11-14T14:48:40', u'Id': u'aaa90396-b4b2-4bfd-86c8-b76ec2a5d18c', u'Env': None, u'Routes': None, u'Updated_time': u'2016-11-14T14:51:07', u'Port': 8082}, {u'Status': u'CREATE_COMPLETE', u'Name': u'wfd-menu-ui_2', u'Creation_time': u'2016-11-14T14:13:51', u'Id': u'47b7b01c-304d-48db-b152-191350f34838', u'Env': None, u'Routes': [u'wfd-toolchain-automation.mybluemix.net'], u'Updated_time': None, u'Port': 8181}, {u'Status': u'CREATE_COMPLETE', u'Name': u'wfd-zuul_3', u'Creation_time': u'2016-11-14T14:13:47', u'Id': u'cec9305f-6dc0-4a15-bd1b-3bc4c05e7780', u'Env': None, u'Routes': None, u'Updated_time': None, u'Port': 8080}, {u'Status': u'CREATE_COMPLETE', u'Name': u'wfd-entree_3', u'Creation_time': u'2016-11-14T14:13:17', u'Id': u'0b35e333-1cac-4068-bfe9-1c1154776ca2', u'Env': None, u'Routes': None, u'Updated_time': None, u'Port': 8081}, {u'Status': u'CREATE_COMPLETE', u'Name': u'wfd-menu_3', u'Creation_time': u'2016-11-14T14:13:11', u'Id': u'25479b13-bf98-4524-9a7c-aa5c369f45ca', u'Env': None, u'Routes': None, u'Updated_time': None, u'Port': 8180}, {u'Status': u'CREATE_COMPLETE', u'Name': u'wfd-dessert_3', u'Creation_time': u'2016-11-14T14:12:48', u'Id': u'25555d3f-9c6f-4815-b8a6-600b45913c1c', u'Env': None, u'Routes': None, u'Updated_time': None, u'Port': 8083}, {u'Status': u'CREATE_COMPLETE', u'Name': u'wfd-appetizer_2', u'Creation_time': u'2016-11-14T14:12:34', u'Id': u'4be59031-8210-418a-ab39-750d622f17fd', u'Env': None, u'Routes': None, u'Updated_time': None, u'Port': 8082}, {u'Status': u'CREATE_COMPLETE', u'Name': u'wfd-config_7', u'Creation_time': u'2016-11-14T13:59:53', u'Id': u'a71c2a14-d97a-4558-a994-9543434e4101', u'Env': None, u'Routes': None, u'Updated_time': None, u'Port': 8888}, {u'Status': u'CREATE_COMPLETE', u'Name': u'wfd-eureka_2', u'Creation_time': u'2016-11-14T12:24:01', u'Id': u'7b43203b-1c36-488f-8429-5238f79cb818', u'Env': None, u'Routes': [u'eureka-jesus.mybluemix.net'], u'Updated_time': None, u'Port': 8761}, {u'Status': u'CREATE_COMPLETE', u'Name': u'micro-socialreview-microservices-whats-for-dinner_1', u'Creation_time': u'2016-10-14T15:37:01', u'Id': u'7d97bacb-017a-4d7d-bbb4-4f1e1fd616cc', u'Env': None, u'Routes': None, u'Updated_time': None, u'Port': 8080}, {u'Status': u'CREATE_COMPLETE', u'Name': u'micro-inventory-microservices-whats-for-dinner_1', u'Creation_time': u'2016-10-14T15:26:11', u'Id': u'114d803d-3d65-4eb0-b732-91dc14e83e72', u'Env': None, u'Routes': None, u'Updated_time': None, u'Port': 8080}, {u'Status': u'CREATE_COMPLETE', u'Name': u'netflix-zuul-microservices-whats-for-dinner_1', u'Creation_time': u'2016-10-14T15:15:56', u'Id': u'b92dd907-c00a-4e18-9b9e-9a9e2020e799', u'Env': None, u'Routes': [u'netflix-zuul-microservices-whats-for-dinner.mybluemix.net'], u'Updated_time': None, u'Port': 8080}, {u'Status': u'CREATE_COMPLETE', u'Name': u'netflix-eureka-microservices-whats-for-dinner_1', u'Creation_time': u'2016-10-14T14:53:39', u'Id': u'9e9af417-29f5-4397-a51d-d890f2f749e4', u'Env': None, u'Routes': [u'netflix-eureka-microservices-whats-for-dinner.mybluemix.net'], u'Updated_time': None, u'Port': 8761}]
• logDebug 'clean(): Found 14 versions: wfd-appetizer-ac_3' wfd-appetizer-ac_2 wfd-menu-ui_2 wfd-zuul_3 wfd-entree_3 wfd-menu_3 wfd-dessert_3 wfd-appetizer_2 wfd-config_7 wfd-eureka_2 micro-socialreview-microservices-whats-for-dinner_1 micro-inventory-microservices-whats-for-dinner_1 netflix-zuul-microservices-whats-for-dinner_1 netflix-eureka-microservices-whats-for-dinner_1
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• VERSIONS=()
• for c in '"${candidates[@]}"'
  ++ echo wfd-appetizer-ac_3
  ++ rev
  ++ rev
  ++ cut -d_ -f1
• v=3
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo wfd-appetizer-ac_2
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=2
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo wfd-menu-ui_2
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=2
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo wfd-zuul_3
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=3
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo wfd-entree_3
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=3
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo wfd-menu_3
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=3
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo wfd-dessert_3
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=3
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo wfd-appetizer_2
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=2
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo wfd-config_7
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=7
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo wfd-eureka_2
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=2
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo micro-socialreview-microservices-whats-for-dinner_1
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=1
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo micro-inventory-microservices-whats-for-dinner_1
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=1
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo netflix-zuul-microservices-whats-for-dinner_1
  ++ rev
  ++ rev
  ++ cut -d_ -f1
• v=1
• VERSIONS+=(${v})
• for c in '"${candidates[@]}"'
  ++ echo netflix-eureka-microservices-whats-for-dinner_1
  ++ rev
  ++ cut -d_ -f1
  ++ rev
• v=1
• VERSIONS+=(${v})
• SORTED_VERSIONS=($(for i in ${VERSIONS[@]}; do echo $i; done | sort -n))
  ++ for i in '${VERSIONS[@]}'
  ++ echo 3
  ++ for i in '${VERSIONS[@]}'
  ++ echo 2
  ++ for i in '${VERSIONS[@]}'
  ++ echo 2
  ++ for i in '${VERSIONS[@]}'
  ++ echo 3
  ++ for i in '${VERSIONS[@]}'
  ++ echo 3
  ++ for i in '${VERSIONS[@]}'
  ++ echo 3
  ++ for i in '${VERSIONS[@]}'
  ++ echo 3
  ++ for i in '${VERSIONS[@]}'
  ++ echo 2
  ++ for i in '${VERSIONS[@]}'
  ++ echo 7
  ++ for i in '${VERSIONS[@]}'
  ++ echo 2
  ++ for i in '${VERSIONS[@]}'
  ++ echo 1
  ++ for i in '${VERSIONS[@]}'
  ++ sort -n
  ++ echo 1
  ++ for i in '${VERSIONS[@]}'
  ++ echo 1
  ++ for i in '${VERSIONS[@]}'
  ++ echo 1
• logDebug 'clean(): Found sorted 14 versions: 1' 1 1 1 2 2 2 2 3 3 3 3 3 7
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• CURRENT_VERSION=
• MOST_RECENT=
• KEPT=()
• (( idx=14-1 ))
• (( idx>=0 ))
• candidate=wfd-appetizer-ac_7
• logDebug 'clean(): Considering candidate wfd-appetizer-ac_7'
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• [[ -z '' ]]
  ++ getRoutes wfd-appetizer-ac_7
  ++ __name=wfd-appetizer-ac_7
  ++ __baseUrl=https://containers-api.ng.bluemix.net/v3/containers
  ++ python -
  2016-11-14 15:20:20,434 - ccs - DEBUG - inspect_group called
  2016-11-14 15:20:20,434 - ccs - DEBUG - [30] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'Accept: application/json;charset=utf-8' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X GET 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_7'
  2016-11-14 15:20:20,450 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:20,471 - urllib3.connectionpool - DEBUG - Setting read timeout to 30
  2016-11-14 15:20:21,049 - urllib3.connectionpool - DEBUG - "GET /v3/containers/groups/wfd-appetizer-ac_7 HTTP/1.1" 404 35
  Can't read group: No such group as 'wfd-appetizer-ac_7'
• [[ -n '' ]]
• (( 7 > 3 ))
• logInfo 'clean(): Deleting group wfd-appetizer-ac_7 from previous pipeline'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: clean(): Deleting group wfd-appetizer-ac_7 from previous pipeline\e[0m'
  --- INFO: clean(): Deleting group wfd-appetizer-ac_7 from previous pipeline
• groupDelete wfd-appetizer-ac_7
• __name=wfd-appetizer-ac_7
• __baseUrl=https://containers-api.ng.bluemix.net/v3/containers
• python -
  2016-11-14 15:20:21,138 - ccs - DEBUG - delete_group called
  2016-11-14 15:20:21,138 - ccs - DEBUG - [90] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X DELETE 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_7?force=true'
  2016-11-14 15:20:21,154 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:21,174 - urllib3.connectionpool - DEBUG - Setting read timeout to 90
  2016-11-14 15:20:22,517 - urllib3.connectionpool - DEBUG - "DELETE /v3/containers/groups/wfd-appetizer-ac_7?force=true HTTP/1.1" 404 322
  2016-11-14 15:20:22,518 - ccs - DEBUG - Group 'wfd-appetizer-ac_7' does not exist; exiting
• (( idx-- ))
• (( idx>=0 ))
• candidate=wfd-appetizer-ac_3
• logDebug 'clean(): Considering candidate wfd-appetizer-ac_3'
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• [[ -z '' ]]
  ++ getRoutes wfd-appetizer-ac_3
  ++ __name=wfd-appetizer-ac_3
  ++ __baseUrl=https://containers-api.ng.bluemix.net/v3/containers
  ++ python -
  2016-11-14 15:20:22,606 - ccs - DEBUG - inspect_group called
  2016-11-14 15:20:22,607 - ccs - DEBUG - [30] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'Accept: application/json;charset=utf-8' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X GET 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_3'
  2016-11-14 15:20:22,622 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:22,642 - urllib3.connectionpool - DEBUG - Setting read timeout to 30
  2016-11-14 15:20:24,302 - urllib3.connectionpool - DEBUG - "GET /v3/containers/groups/wfd-appetizer-ac_3 HTTP/1.1" 200 3226
• [[ -n wfd-appetizer-ac.mybluemix.net ]]
• CURRENT_VERSION=wfd-appetizer-ac_3
• KEPT+=(${candidate})
• logInfo 'clean(): Identified current version: wfd-appetizer-ac_3; keeping'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: clean(): Identified current version: wfd-appetizer-ac_3; keeping\e[0m'
  --- INFO: clean(): Identified current version: wfd-appetizer-ac_3; keeping
• (( idx-- ))
• (( idx>=0 ))
• candidate=wfd-appetizer-ac_3
• logDebug 'clean(): Considering candidate wfd-appetizer-ac_3'
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• [[ -z wfd-appetizer-ac_3 ]]
• (( 3 > 3 ))
• [[ -z wfd-appetizer-ac_3 ]]
  ++ isStopped wfd-appetizer-ac_3
  ++ local __name=wfd-appetizer-ac_3
  ++ echo false
• [[ true == \f\a\l\s\e ]]
• (( 1 >= 1 ))
• logInfo 'clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)\e[0m'
  --- INFO: clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)
• groupDelete wfd-appetizer-ac_3
• __name=wfd-appetizer-ac_3
• __baseUrl=https://containers-api.ng.bluemix.net/v3/containers
• python -
  2016-11-14 15:20:24,405 - ccs - DEBUG - delete_group called
  2016-11-14 15:20:24,405 - ccs - DEBUG - [90] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X DELETE 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_3?force=true'
  2016-11-14 15:20:24,421 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:24,440 - urllib3.connectionpool - DEBUG - Setting read timeout to 90
  2016-11-14 15:20:26,361 - urllib3.connectionpool - DEBUG - "DELETE /v3/containers/groups/wfd-appetizer-ac_3?force=true HTTP/1.1" 204 0
  2016-11-14 15:20:26,362 - ccs - DEBUG - Waiting for group 'wfd-appetizer-ac_3' deletion
  2016-11-14 15:20:26,362 - ccs - DEBUG - inspect_group called
  2016-11-14 15:20:26,362 - ccs - DEBUG - [30] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'Accept: application/json;charset=utf-8' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X GET 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_3'
  2016-11-14 15:20:26,363 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:26,382 - urllib3.connectionpool - DEBUG - Setting read timeout to 30
  2016-11-14 15:20:27,336 - urllib3.connectionpool - DEBUG - "GET /v3/containers/groups/wfd-appetizer-ac_3 HTTP/1.1" 200 3058
  2016-11-14 15:20:27,347 - ccs - DEBUG - _wait_for discovered group {u'AntiAffinity': False, u'BluemixApp': u'container_bridge_app', u'Status': u'DELETE_IN_PROGRESS', u'AvailabilityZone': u'docker', u'Name': u'wfd-appetizer-ac_3', u'Autorecovery': False, u'Image': u'sha256:9179c2b2ba65f4a9a2b31d505f2f523bc3949f1194d021090c3ed5f8427397de', u'Cmd': [], u'Creation_time': u'2016-11-14T15:06:54', u'Port': None, u'NumberInstances': {u'Desired': 1, u'Max': 2, u'Min': 1, u'CurrentSize': 1}, u'ImageName': u'registry.ng.bluemix.net/chrisking/wfd-appetizer:activeDeploy2', u'Route_Status': {u'successful': True, u'in_progress': False, u'message': u'registered route successfully'}, u'Env': [u'CCS_BIND_APP=container_bridge_app', u'HTTP_MONITOR=True', u'HTTP_MONITOR_PATH=/', u'HTTP_MONITOR_RC_LIST=200,201,202,204,300,301,302,401,403,404', u'VCAP_SERVICES={"user-provided": [{"credentials": {"uri": "http://172.32.0.185:8761/eureka/"}, "tags": [], "name": "eureka-service-discovery", "syslog_drain_url": "", "label": "user-provided"}, {"credentials": {"uri": "http://172.32.0.191:8888/"}, "tags": [], "name": "config-server-ic", "syslog_drain_url": "", "label": "user-provided"}]}', u'VCAP_SERVICES_USER_PROVIDED_0_CREDENTIALS_URI=http://172.32.0.185:8761/eureka/', u'VCAP_SERVICES_USER_PROVIDED_0_LABEL=user-provided', u'VCAP_SERVICES_USER_PROVIDED_0_NAME=eureka-service-discovery', u'VCAP_SERVICES_USER_PROVIDED_0_SYSLOG_DRAIN_URL=', u'VCAP_SERVICES_USER_PROVIDED_1_CREDENTIALS_URI=http://172.32.0.191:8888/', u'VCAP_SERVICES_USER_PROVIDED_1_LABEL=user-provided', u'VCAP_SERVICES_USER_PROVIDED_1_NAME=config-server-ic', u'VCAP_SERVICES_USER_PROVIDED_1_SYSLOG_DRAIN_URL=', u'group_id=5a2f161f-4863-4458-84f6-892fb5448136', u'ibmcon_UL_nofile=', u'loadbalancer_fip=100.100.70.225', u'loadbalancer_vip=172.32.0.207', u'logging_password=', u'logstash_target=logmet.opvis.bluemix.net:9091', u'metrics_target=logmet.opvis.bluemix.net:9095', u'routing_mode=map', u'sgroup_id=5a2f161f-4863-4458-84f6-892fb5448136', u'sgroup_name=wfd-appetizer-ac_3', u'space_id=8b0398b1-1b77-4a3d-9e3a-45819622ed6c', u'tenant_id=c2c66e7a29d74c59a03c02f2cd67f45c'], u'Memory': 256, u'Routes': [u'wfd-appetizer-ac.mybluemix.net'], u'Updated_time': None, u'Id': u'5a2f161f-4863-4458-84f6-892fb5448136', u'Loadbalancer': {u'private_ip_address': u'172.32.0.207', u'intermediate_ip_address': u'100.100.70.225', u'health_monitors': [{u'type': u'tcp', u'enabled': True}, {u'response_codes': u'200,201,202,204,300,301,302,401,403,404', u'path': u'/', u'enabled': True, u'type': u'http'}]}}
  2016-11-14 15:20:27,348 - ccs - DEBUG - Waiting for group 'wfd-appetizer-ac_3' deletion: sleeping 5s
  2016-11-14 15:20:32,353 - ccs - DEBUG - inspect_group called
  2016-11-14 15:20:32,353 - ccs - DEBUG - [30] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'Accept: application/json;charset=utf-8' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X GET 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_3'
  2016-11-14 15:20:32,354 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:32,401 - urllib3.connectionpool - DEBUG - Setting read timeout to 30
  2016-11-14 15:20:32,912 - urllib3.connectionpool - DEBUG - "GET /v3/containers/groups/wfd-appetizer-ac_3 HTTP/1.1" 200 2045
  2016-11-14 15:20:32,913 - ccs - DEBUG - _wait_for discovered group {u'AntiAffinity': False, u'BluemixApp': u'container_bridge_app', u'Status': u'DELETE_IN_PROGRESS', u'AvailabilityZone': u'docker', u'Name': u'wfd-appetizer-ac_3', u'Autorecovery': False, u'Image': u'sha256:9179c2b2ba65f4a9a2b31d505f2f523bc3949f1194d021090c3ed5f8427397de', u'Cmd': [], u'Creation_time': u'2016-11-14T15:06:54', u'Port': None, u'NumberInstances': {u'Desired': 1, u'Max': 2, u'Min': 1, u'CurrentSize': 0}, u'ImageName': u'registry.ng.bluemix.net/chrisking/wfd-appetizer:activeDeploy2', u'Env': [u'CCS_BIND_APP=container_bridge_app', u'HTTP_MONITOR=True', u'HTTP_MONITOR_PATH=/', u'HTTP_MONITOR_RC_LIST=200,201,202,204,300,301,302,401,403,404', u'VCAP_SERVICES={"user-provided": [{"credentials": {"uri": "http://172.32.0.185:8761/eureka/"}, "tags": [], "name": "eureka-service-discovery", "syslog_drain_url": "", "label": "user-provided"}, {"credentials": {"uri": "http://172.32.0.191:8888/"}, "tags": [], "name": "config-server-ic", "syslog_drain_url": "", "label": "user-provided"}]}', u'VCAP_SERVICES_USER_PROVIDED_0_CREDENTIALS_URI=http://172.32.0.185:8761/eureka/', u'VCAP_SERVICES_USER_PROVIDED_0_LABEL=user-provided', u'VCAP_SERVICES_USER_PROVIDED_0_NAME=eureka-service-discovery', u'VCAP_SERVICES_USER_PROVIDED_0_SYSLOG_DRAIN_URL=', u'VCAP_SERVICES_USER_PROVIDED_1_CREDENTIALS_URI=http://172.32.0.191:8888/', u'VCAP_SERVICES_USER_PROVIDED_1_LABEL=user-provided', u'VCAP_SERVICES_USER_PROVIDED_1_NAME=config-server-ic', u'VCAP_SERVICES_USER_PROVIDED_1_SYSLOG_DRAIN_URL=', u'routing_mode=map', u'logstash_target=logmet.opvis.bluemix.net:9091', u'metrics_target=logmet.opvis.bluemix.net:9095', u'space_id=8b0398b1-1b77-4a3d-9e3a-45819622ed6c'], u'Memory': 256, u'Routes': [], u'Updated_time': None, u'Id': u'5a2f161f-4863-4458-84f6-892fb5448136'}
  2016-11-14 15:20:32,913 - ccs - DEBUG - Waiting for group 'wfd-appetizer-ac_3' deletion: sleeping 5s
  2016-11-14 15:20:37,918 - ccs - DEBUG - inspect_group called
  2016-11-14 15:20:37,918 - ccs - DEBUG - [30] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'Accept: application/json;charset=utf-8' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X GET 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_3'
  2016-11-14 15:20:37,919 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:37,937 - urllib3.connectionpool - DEBUG - Setting read timeout to 30
  2016-11-14 15:20:38,279 - urllib3.connectionpool - DEBUG - "GET /v3/containers/groups/wfd-appetizer-ac_3 HTTP/1.1" 200 2045
  2016-11-14 15:20:38,280 - ccs - DEBUG - _wait_for discovered group {u'AntiAffinity': False, u'BluemixApp': u'container_bridge_app', u'Status': u'DELETE_IN_PROGRESS', u'AvailabilityZone': u'docker', u'Name': u'wfd-appetizer-ac_3', u'Autorecovery': False, u'Image': u'sha256:9179c2b2ba65f4a9a2b31d505f2f523bc3949f1194d021090c3ed5f8427397de', u'Cmd': [], u'Creation_time': u'2016-11-14T15:06:54', u'Port': None, u'NumberInstances': {u'Desired': 1, u'Max': 2, u'Min': 1, u'CurrentSize': 0}, u'ImageName': u'registry.ng.bluemix.net/chrisking/wfd-appetizer:activeDeploy2', u'Env': [u'CCS_BIND_APP=container_bridge_app', u'HTTP_MONITOR=True', u'HTTP_MONITOR_PATH=/', u'HTTP_MONITOR_RC_LIST=200,201,202,204,300,301,302,401,403,404', u'VCAP_SERVICES={"user-provided": [{"credentials": {"uri": "http://172.32.0.185:8761/eureka/"}, "tags": [], "name": "eureka-service-discovery", "syslog_drain_url": "", "label": "user-provided"}, {"credentials": {"uri": "http://172.32.0.191:8888/"}, "tags": [], "name": "config-server-ic", "syslog_drain_url": "", "label": "user-provided"}]}', u'VCAP_SERVICES_USER_PROVIDED_0_CREDENTIALS_URI=http://172.32.0.185:8761/eureka/', u'VCAP_SERVICES_USER_PROVIDED_0_LABEL=user-provided', u'VCAP_SERVICES_USER_PROVIDED_0_NAME=eureka-service-discovery', u'VCAP_SERVICES_USER_PROVIDED_0_SYSLOG_DRAIN_URL=', u'VCAP_SERVICES_USER_PROVIDED_1_CREDENTIALS_URI=http://172.32.0.191:8888/', u'VCAP_SERVICES_USER_PROVIDED_1_LABEL=user-provided', u'VCAP_SERVICES_USER_PROVIDED_1_NAME=config-server-ic', u'VCAP_SERVICES_USER_PROVIDED_1_SYSLOG_DRAIN_URL=', u'routing_mode=map', u'logstash_target=logmet.opvis.bluemix.net:9091', u'metrics_target=logmet.opvis.bluemix.net:9095', u'space_id=8b0398b1-1b77-4a3d-9e3a-45819622ed6c'], u'Memory': 256, u'Routes': [], u'Updated_time': None, u'Id': u'5a2f161f-4863-4458-84f6-892fb5448136'}
  2016-11-14 15:20:38,281 - ccs - DEBUG - Waiting for group 'wfd-appetizer-ac_3' deletion: sleeping 5s
  2016-11-14 15:20:43,286 - ccs - DEBUG - inspect_group called
  2016-11-14 15:20:43,286 - ccs - DEBUG - [30] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'Accept: application/json;charset=utf-8' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X GET 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_3'
  2016-11-14 15:20:43,287 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:43,304 - urllib3.connectionpool - DEBUG - Setting read timeout to 30
  2016-11-14 15:20:43,617 - urllib3.connectionpool - DEBUG - "GET /v3/containers/groups/wfd-appetizer-ac_3 HTTP/1.1" 404 35
  2016-11-14 15:20:43,619 - ccs - DEBUG - _wait_for discovered group None
  2016-11-14 15:20:43,619 - ccs - INFO - Group 'wfd-appetizer-ac_3' deletion completed successfully in 16.9238450527
• (( idx-- ))
• (( idx>=0 ))
• candidate=wfd-appetizer-ac_3
• logDebug 'clean(): Considering candidate wfd-appetizer-ac_3'
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• [[ -z wfd-appetizer-ac_3 ]]
• (( 3 > 3 ))
• [[ -z wfd-appetizer-ac_3 ]]
  ++ isStopped wfd-appetizer-ac_3
  ++ local __name=wfd-appetizer-ac_3
  ++ echo false
• [[ true == \f\a\l\s\e ]]
• (( 1 >= 1 ))
• logInfo 'clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)\e[0m'
  --- INFO: clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)
• groupDelete wfd-appetizer-ac_3
• __name=wfd-appetizer-ac_3
• __baseUrl=https://containers-api.ng.bluemix.net/v3/containers
• python -
  2016-11-14 15:20:43,713 - ccs - DEBUG - delete_group called
  2016-11-14 15:20:43,713 - ccs - DEBUG - [90] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X DELETE 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_3?force=true'
  2016-11-14 15:20:43,729 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:43,749 - urllib3.connectionpool - DEBUG - Setting read timeout to 90
  2016-11-14 15:20:44,426 - urllib3.connectionpool - DEBUG - "DELETE /v3/containers/groups/wfd-appetizer-ac_3?force=true HTTP/1.1" 404 321
  2016-11-14 15:20:44,426 - ccs - DEBUG - Group 'wfd-appetizer-ac_3' does not exist; exiting
• (( idx-- ))
• (( idx>=0 ))
• candidate=wfd-appetizer-ac_3
• logDebug 'clean(): Considering candidate wfd-appetizer-ac_3'
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• [[ -z wfd-appetizer-ac_3 ]]
• (( 3 > 3 ))
• [[ -z wfd-appetizer-ac_3 ]]
  ++ isStopped wfd-appetizer-ac_3
  ++ local __name=wfd-appetizer-ac_3
  ++ echo false
• [[ true == \f\a\l\s\e ]]
• (( 1 >= 1 ))
• logInfo 'clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)\e[0m'
  --- INFO: clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)
• groupDelete wfd-appetizer-ac_3
• __name=wfd-appetizer-ac_3
• __baseUrl=https://containers-api.ng.bluemix.net/v3/containers
• python -
  2016-11-14 15:20:44,517 - ccs - DEBUG - delete_group called
  2016-11-14 15:20:44,517 - ccs - DEBUG - [90] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X DELETE 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_3?force=true'
  2016-11-14 15:20:44,532 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:44,553 - urllib3.connectionpool - DEBUG - Setting read timeout to 90
  2016-11-14 15:20:44,852 - urllib3.connectionpool - DEBUG - "DELETE /v3/containers/groups/wfd-appetizer-ac_3?force=true HTTP/1.1" 404 323
  2016-11-14 15:20:44,852 - ccs - DEBUG - Group 'wfd-appetizer-ac_3' does not exist; exiting
• (( idx-- ))
• (( idx>=0 ))
• candidate=wfd-appetizer-ac_3
• logDebug 'clean(): Considering candidate wfd-appetizer-ac_3'
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• [[ -z wfd-appetizer-ac_3 ]]
• (( 3 > 3 ))
• [[ -z wfd-appetizer-ac_3 ]]
  ++ isStopped wfd-appetizer-ac_3
  ++ local __name=wfd-appetizer-ac_3
  ++ echo false
• [[ true == \f\a\l\s\e ]]
• (( 1 >= 1 ))
• logInfo 'clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)'
• [[ -n 3 ]]
• [[ 3 -ge 3 ]]
• echo -e '\e[0;37m--- INFO: clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)\e[0m'
  --- INFO: clean(): Deleting group wfd-appetizer-ac_3 (already identified sufficient versions to keep)
• groupDelete wfd-appetizer-ac_3
• __name=wfd-appetizer-ac_3
• __baseUrl=https://containers-api.ng.bluemix.net/v3/containers
• python -
  2016-11-14 15:20:44,943 - ccs - DEBUG - delete_group called
  2016-11-14 15:20:44,943 - ccs - DEBUG - [90] curl -H 'X-Auth-Project-Id: 8b0398b1-1b77-4a3d-9e3a-45819622ed6c' -H 'X-Auth-Token: [PRIVATE DATA HIDDEN]' -X DELETE 'https://containers-api.ng.bluemix.net/v3/containers/groups/wfd-appetizer-ac_3?force=true'
  2016-11-14 15:20:44,959 - urllib3.connectionpool - INFO - Starting new HTTPS connection (1): containers-api.ng.bluemix.net
  2016-11-14 15:20:45,084 - urllib3.connectionpool - DEBUG - Setting read timeout to 90
  2016-11-14 15:20:45,390 - urllib3.connectionpool - DEBUG - "DELETE /v3/containers/groups/wfd-appetizer-ac_3?force=true HTTP/1.1" 404 323
  2016-11-14 15:20:45,390 - ccs - DEBUG - Group 'wfd-appetizer-ac_3' does not exist; exiting
• (( idx-- ))
• (( idx>=0 ))
• candidate=wfd-appetizer-ac_2
• logDebug 'clean(): Considering candidate wfd-appetizer-ac_2'
• [[ -n 3 ]]
• [[ 3 -ge 4 ]]
• [[ -z wfd-appetizer-ac_3 ]]
• (( 2 > 3 ))
• [[ -z wfd-appetizer-ac_3 ]]
  ++ isStopped wfd-appetizer-ac_2
  ++ local __name=wfd-appetizer-ac_2