[DO_NOT_CLOSE] Automate Satellite upgrade process

[satelliteqe]

[DO_NOT_CLOSE] Automate Satellite upgrade process

[jyejare]

Referring PR #196 which covers the basic infrastructure for Product Upgrade. Currently Satellite upgrade process has been implemented using Openstack images.

[jyejare]

Things to be taken care while creating images of satellite and Capsule:

1. Create CV as 'rhel<major_version>_cv' for capsule.
2. Create AK as 'rhel<msjor_version>_ak' only for capsule.
3. Create 'DEV' environment and promote CV to 'DEV'(only capitals).
4. Remove entries from /etc/hosts while saving the images for satellite and capsule.

Automated Satellite Upgrade Process:

- Pre-requisite:
1. Get ready an image in openstack with old satellite version and capsule version, e.g 6.0.8.
2. Add public ssh key of execution machine to openstack manually well before running automation.
- Steps Automated:
3. Create an instance from already created openstack image of satellite and capsule.
4. Get an hostname from satellite/capsule instance IP to login into instance for upgrade process.
5. Run upgrade commands on the step(3) instance.
- Optional:
6. Delete the satellite and capsule instance before upgrade starts. (Automated)

For playing with openstack, I have used novaclient python API. So for running the whole upgrade automation, it is very much required to install novaclient python package. So the same has been added into requirements.txt.
For more info on novaclient API, visit http://docs.openstack.org/developer/python-novaclient/api.html.

Also, please note that before installing novaclient python package, it is required to have python-devel package installed in execution system. Otherwise novaclient package installation fails.
You can install python-devel package by simply,

yum -y install python-devel

Q&A:
How to run upgrade automation ?
Answer : You can run upgrade automation using fab framework, as following command,

#FROM_VERSION=<'6.1'/'6.0'> RHN_USERNAME=<cdn_username> RHN_PASSWORD=<cdn_password> RHN_POOLID=<cdn_pool_id> USERNAME=<openstack_username> PASSWORD=<password> AUTH_URL=http://<openstack_server>:<port>/<version> PROJECT_ID=<project_id> BASE_URL=http://example.com/Satellite/x86_64/os/  CAPSULE_REPO=http://example.com/Capsule/x86_64_os/ TOOLS_REPO=http://example.com/Sat_Tools/x86_64_os/
fab -i <local private_ssh_key> product_upgrade:<product_name e.g 'capsule'>, <ssh_key>,<sat_instance_name>,<sat_image_name>,<sat_instance_flavor>,<cap_instance_name>,<cap_image_name>,<cap_instance_flavor>

Note: If you provide <product_name> as satellite then only satellite will be upgraded else if you provide <product_name> as capsule then both satellite as well as capsule will be upgraded.
All the parameters and environment variable to fab command are mandatory, else the upgrade will fail.

Required Parameters: product, ssh_key, satellite_instance_name, satellite_image_name, satellite_flavor_name, (In case of capsule upgrade ->)capsule_instance_name, capsule_image_name, capsule_flavor_name
Required Environment Variables: RHN_USERNAME, RHN_PASSWORD, RHN_POOLID, USERNAME, PASSWORD, AUTH_URL, PROJECT_ID, BASE_URL
In
Optional Environment Variable: ADMIN_PASSWORD, BASE_URL, CAPSULE_REPO, TOOLS_REPO

[jyejare]

Referring PR #214 , for updates in upgrade process and few updates related to running automation Openstack instance.

[jyejare]

Refering PR #225, For removing/replacing the static delay for resolving the DNS issue in Openstack instance.

[jyejare]

Also the BIG THING is from now onwards, We can run satellite upgrade automation on RHEL 6 instances. By simply adding following script in /etc/rc.local file in RHEL 6 image.

#Setting Network Services
if [ -f /etc/sysconfig/network-scripts/ifcfg-eth0 ]; then
        for (( netini=0; netini<=3; netini++ ))
        do
                cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth$((netini+1))
                sed -i "s/DEVICE=\"eth0\"/DEVICE=\"eth$((netini+1))\"/" /etc/sysconfig/network-scripts/ifcfg-eth$((netini+1))
                sleep 5
                service network restart
                sleep 5
        done
else
        echo "The network interface eth0 is not present!" > ~/networkServiceIssue.txt
fi

[jyejare]

Referring #228, Generates Foreman Debug at the end of upgrade. So that it can be download as build artefact in Jenkins.

[jyejare]

A new great addition to the Upgrade feature is: Capsule Upgrade
Refering PR: #236
Updating the Product Upgrade process.

[jyejare]

Only Client Upgrade is pending here.

[jyejare]

The Client Upgrade is now automated and so closing this issue. Very Happy and Glad to have all these three automated.