sdake/heat-jeos

The Heat JEOS tools

HEAT-JEOS

IMPORTANT NOTE: This project has been superceeded by further community efforts to use diskimage-builder created images for heat. This work is underway. In the meantime, please use the heat-templates repository for template examples and jeos creation tools:

https://github.com/openstack/heat-templates https://github.com/openstack/heat-templates/blob/master/tools/heat-jeos.sh

Future development will occur here: https://github.com/stackforge/diskimage-builder

Only security updates will be maintained until Fedora 20 when this software will be deprecated. See the schedule here: https://fedoraproject.org/wiki/Schedule

Getting Started

If you'd like to run from the master branch, you can clone the git repo:

git://github.com/sdake/heat-jeos.git

Related projects

• http://www.heat-api.org
• http://wiki.openstack.org/Heat

heat-jeos's Issues

port cfn-hup to use heat Cloudformation API

As part of the ongoing security rework, we need to move the data source for the cfn-hup utility to be the heat CloudFormation API, ref https://github.com/heat-api/heat/wiki/Cloudwatch-Architecture-rework

This is similar to the rework carried out for cfn-push-stats, and will allow us to remove the metadata-read functionality from the (unauthenticated) heat-metadata service

heat-jeos will not run without glance installed

heat-jeos should only import glance if register-with-glance is used.

Openshift tdl broken

The current openshift tdl is broken, because the openshift-origin (formerly crankcase) repo has moved on github.

Probably need to remove the F16 build-from-source based TDL, and replace it with a install-from-rpm TDL (and/or template) based on RHEL6.3 (which seems to be the most well tested/documented platform for openshift deployment AFAICT):

Refs:
https://openshift.redhat.com/community/wiki/build-your-own
https://gist.github.com/3901379
https://gist.github.com/3933438
https://github.com/openshift/crankcase

The old build-from-source link is still in the wiki, but it points the the old github location, so I assume it's outdated, the build-from-rpms approach seems like it will be better anyway and the instructions for this method did not exist when I first looked at this:

https://openshift.redhat.com/community/wiki/build-openshift-origin-from-source

files section creates an error on the vm (No such file or directory: ''")

It's trying to create a dir with name ""

Jul 2 22:20:12 localhost cloud-init-cfg[909]: ERROR [2012-07-02 22:20:12,827] [Errno 2] No such file or directory: ''"
Jul 2 22:20:12 localhost cloud-init-cfg[909]: Traceback (most recent call last):
Jul 2 22:20:12 localhost cloud-init-cfg[909]: File "/opt/aws/bin/cfn_helper.py", line 521, in apply_files
Jul 2 22:20:12 localhost cloud-init-cfg[909]: os.makedirs(os.path.dirname(dest))
Jul 2 22:20:12 localhost cloud-init-cfg[909]: File "/usr/lib64/python2.7/os.py", line 157, in makedirs
Jul 2 22:20:12 localhost cloud-init-cfg[909]: mkdir(name, mode)

Port cfn-push-stats to CloudWatch API

As part of the CloudWatch security rework, cfn-push-stats needs to be ported to use the (authenticated) cloudwatch API, not the heat-metadata server (which has no auth and is planned to be removed)

cfntools need to be able to use secure transport

This was posted to LP as well, as it needs changes both in heat and heat-jeos:

https://bugs.launchpad.net/heat/+bug/1117594

Currently the cfntools in heat-jeos hard code 'is_secure=False' to boto.cloudformation.CloudFormationConnection()

This should be configurable instead by the heat engine using the boto config that is dropped on disk.

heat-jeos image should print out the location of the generated image

unless you know the location of the images you don't know the location of it.

Also what is the point of register? Isn't using glance just as easy?

How about:

sudo -E heat-jeos image F16 x86_64 cfntools ./heat_jeos/jeos/F16-x86_64-cfntools-jeos.tdl
...
Generated image: /var/lib/libvirt/images/F16-x86_64-cfntools-jeos.qcow2

Now register with glance using:
glance add name=F16-x86_64-cfntools is_public=true disk_format=qcow2 container_format=ovf < /var/lib/libvirt/images/F16-x86_64-cfntools-jeos.qcow2

heat-jeos fails silently when passed an invalid name

If you pass an invalid template name to heat-jeos create, it silently fails - the INFO should probably be an ERROR log so the user sees something went wrong

[root@heatlt heat-jeos]# heat-jeos create F16-x86_64-cfntools-openshift-jeos --register-with-glance[root@heatlt heat-jeos]# echo $?
1
[root@heatlt heat-jeos]# heat-jeos --debug create F16-x86_64-cfntools-openshift-jeos --register-with-glance
DEBUG:Debug level logging enabled
INFO:You must specify a correct template name or path.

cfn-push-stats "Units" should be "Unit"

If it's easy and not too risky to change this, it would be helpful (for cloudwatch) if we could change the "Units" key in cfn-push stats to "Unit"

This will allow us to more easily align with the AWS MetricDatum specification:

http://docs.amazonwebservices.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html

Also "Counter" should be "Count"

I can mangle this in the CW API, but it would make things easier if these changes can easily be made (looks like a quick sed job but would appreciate input from asalkeld before I change anything)

cfn-signal reports success even when curl request fails

cfn-signal (and probably also cfn-push-stats since it uses the same CommandRunner class) fails to check return code for the curl request (which it runs in a subshell via CommandRunner).

This means that the cfn-signal return code (and log output in /var/log/cfn-signal.log) indicates success, when really we could not connect to the metadata server.

(Stopped heat-metadata service on host)
[root@openshiftbrokerserver ~]# /opt/aws/bin/cfn-signal -s SUCCESS -r "foobar" -i 0000 http://10.0.0.1:8002/stacks/4/resources/WaitHandle
DEBUG [2012-08-01 04:35:09,845] cfn-signal called Namespace(data='Application has completed configuration.', exit_code=None, reason='foobar', success='SUCCESS', unique_id='0000', url='http://10.0.0.1:8002/stacks/4/resources/WaitHandle')
SHDEBUG cmd_str = curl -X PUT -H 'Content-Type:' --data-binary '{"Status": "FAILURE", "Reason": "foobar", "Data": "Application has completed configuration.", "UniqueId": "0000"}' http://10.0.0.1:8002/stacks/4/resources/WaitHandle
DEBUG [2012-08-01 04:35:09,847] Running command: curl -X PUT -H 'Content-Type:' --data-binary '{"Status": "FAILURE", "Reason": "foobar", "Data": "Application has completed configuration.", "UniqueId": "0000"}' http://10.0.0.1:8002/stacks/4/resources/WaitHandle

[root@openshiftbrokerserver ~]# echo $?
0

[root@openshiftbrokerserver ~]# curl -X PUT -H 'Content-Type:' --data-binary '{"Status": "FAILURE", "Reason": "foobar", "Data": "Application has completed configuration.", "UniqueId": "0000"}' http://10.0.0.1:8002/stacks/4/resources/WaitHandle
curl: (7) couldn't connect to host

[root@openshiftbrokerserver ~]# echo $?
7

cfn-hup doesn't log to it's logfile

cfn-hup doesn't log to it's logfile under /var/log, /var/log/cfn-hup.log is always zero bytes even after cfn-hup has been run.

heat-jeos always tries to delete an existing image

On Folsom, heat-jeos will always try to delete the first available image before adding a new one. It should only delete the one with the same name, if it exists.

Empty cfntools files in instance generated with usability branch

Steps:

Checkout the usability branch.

Build the image:

sudo -E ./bin/heat-jeos -d -y create F16-x86_64-cfntools-jeos --register-with-glance

Launch a stack:

./bin/heat -d create wordpress --template-file=templates/WordPress_Single_Instance.template --parameters="InstanceType=m1.xlarge;DBUsername=${USER};DBPassword=verybadpass;KeyName=${USER}_key"

SSH into the stack.

Look at the /opt/aws/bin/* files. They are all empty.

Authentication credentials in database should be encrypted.

It's not a good scenario for the authentication information to be held in the database in plain text. I did some reading on the topic and it sounds like the best solution for us would be to do symmetrical encryption with a key held in a readable-only-by-root file.

no man page

Detect instance type (gold/cfntools) from the TDL

Since we already ship different templates based on the gold/cfntools type, heat-jeos should detect the type from the template and not require that we pass it as a commandline argument.

What would be the best way to detect this?

Deciding by the template name seems inflexible.

We could look for the appropriate <file> tags (e.g. <file name='/opt/aws/bin/cfn-init' type='base64'></file>).

Make the commands in the tdl more readable

yum -y update --skip-broken;yum -y install yum-plugin-fastestmirror;yum -y update;/usr/sbin/useradd ec2-user;echo -e 'ec2-user\tALL=(ALL)\tNOPASSWD: ALL' >> /etc/sudoers;yum -y install cloud-init;cat >> /etc/rc.d/rc.local << EOF;chmod +x /etc/rc.d/rc.local;chmod +x /opt/aws/bin/cfn-*
#!/bin/bash
setenforce 0
EOF

Assuming that still does as advertised

yum -y update --skip-broken
yum -y install yum-plugin-fastestmirror
yum -y update
/usr/sbin/useradd ec2-user
echo -e 'ec2-user\tALL=(ALL)\tNOPASSWD: ALL' >> /etc/sudoers
yum -y install cloud-init
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=permissive/" /etc/sysconfig/selinux
chmod +x /opt/aws/bin/cfn-*

python-glanceclient code does not actually work

The commit 3940132 introduces support for the python-glanceclient package that shipped in the Folsom timeframe, if it is present. However, when python-glanceclient is installed, I cannot actually use glance:

ERROR:Request returned failure status.
An existing JEOS was found on disk. Do you want to build a fresh JEOS? (y/n) n
ERROR:Request returned failure status.
ERROR: Failed to add image. Got error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/heat_jeos-7-py2.7.egg/EGG-INFO/scripts/heat-jeos", line 157, in command_create
    image = glance.find_image_by_name(client, image_name)
  File "/usr/lib/python2.7/site-packages/heat_jeos-7-py2.7.egg/heat_jeos/glance_clients.py", line 100, in find_image_folsom
    return images.next()
  File "/usr/lib/python2.7/site-packages/python_glanceclient-0.6.0-py2.7.egg/glanceclient/v1/images.py", line 130, in paginate
    images = self._list(url, "images")
  File "/usr/lib/python2.7/site-packages/python_glanceclient-0.6.0-py2.7.egg/glanceclient/common/base.py", line 53, in _list
    resp, body = self.api.json_request('GET', url)
  File "/usr/lib/python2.7/site-packages/python_glanceclient-0.6.0-py2.7.egg/glanceclient/common/http.py", line 196, in json_request
    resp, body_iter = self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/python_glanceclient-0.6.0-py2.7.egg/glanceclient/common/http.py", line 180, in _http_request
    raise exc.from_response(resp)
HTTPNotFound: HTTPNotFound (HTTP 404)
WARNING:Note: Your image metadata may still be in the registry, but the image's status will likely be 'killed'.

Port cfn-signal to CloudWatch API

As part of the CloudWatch security rework, we need to port cfn-signal to use the CloudWatch API (along with some corresponding rework to the wait-condition resource implementation in the heat engine). This will allow us to remove the insecure writeable heat-metadata service.

cfn_helper.py has boto debugging on

I left boto debug on in cfn_helper.py which makes the logging noisy:

        self.client = CloudFormationConnection(
                         aws_access_key_id=self.access_key,
                         aws_secret_access_key=self.secret_key,
                         is_secure=False, port=self.port, path="/v1", debug=1)

When the problem with cfn-init is resolved this should be set to debug=0, leaving until then as the output is useful while debugging

heat-jeos should check credentials earlier

Just finished building an image using heat-jeos, after running a long while it tried to register the image with glance and got permission denied. We should check permissions sooner in the cycle to give users an early warning.

systemd timeout on cloud-init-final job

Depending on network speed and the complexity of the userdata script the cloud-init service is timing out.
Causing the job to be terminated and the instance to be in an undefined state.
Short term: set
TimeoutSec=
using sed?

Make a fedora bug so it can be fixed there (or upstream).

build images from Heat templates

see: https://github.com/heat-api/heat/wiki/Roadmap-Feature:-Prebuilding-Images-From-Templates

Port cfn-hup to use nova metadata

Need to port cfn-hup to pull latest nova instance metadata (rather than reading from our own heat-metadata service, which is planned to be removed). Investigation required as I'm not sure how instance metadata update works in nova

Authenticated metadata breaks cfn-init

The new auth scheme breaks the initial run of cfn-init because the sedding of the /etc/boto.cfg doesn't happen until after it first tries to run.

cloud-init is giving us an initial copy of the instance metadata, so we either need to modify cfn-init to populate /etc/boto.cfg with the value from /var/lib/cloud/data/cfn-metadata-server if it finds the IP in boto.cfg invalid, or make it just use the local metadata on the first run:

#!/bin/bash -v
# Helper function
function error_exit
{
  /opt/aws/bin/cfn-signal -e 1 -r "$1" 'http://192.168.250.1:8000/stacks/5/resources/WaitHandle'
  exit 1
}
/opt/aws/bin/cfn-init -s wordpress_ha2 -r WikiDatabase --access-key 58554c0766e940b7a892c4479c89feed --secret-key 2a9d5112e0cd4354b616e3596795e2f6 --region ap-southeast-1 || error_exit 'Failed to run cfn-init'
DEBUG [2012-10-31 14:23:10,817] Method: GET
DEBUG [2012-10-31 14:23:10,821] Path: /v1/
DEBUG [2012-10-31 14:23:10,822] Data:
DEBUG [2012-10-31 14:23:10,822] Headers: {}
DEBUG [2012-10-31 14:23:10,823] Host: __GATEWAY_IP__:8000
DEBUG [2012-10-31 14:23:10,823] establishing HTTP connection: kwargs={}
DEBUG [2012-10-31 14:23:10,823] Token: None
DEBUG [2012-10-31 14:23:10,823] using _calc_signature_2
DEBUG [2012-10-31 14:23:10,823] query string: AWSAccessKeyId=58554c0766e940b7a892c4479c89feed&Action=DescribeStackResource&ContentType=JSON&LogicalResourceId=WikiDatabase&SignatureMethod=HmacSHA256&SignatureVersion=2&StackName=wordpress_ha2&Timestamp=2012-10-31T18%3A23%3A10Z&Version=2010-05-15
DEBUG [2012-10-31 14:23:10,823] string_to_sign: GET
__gateway_ip__:8000
/v1/
AWSAccessKeyId=58554c0766e940b7a892c4479c89feed&Action=DescribeStackResource&ContentType=JSON&LogicalResourceId=WikiDatabase&SignatureMethod=HmacSHA256&SignatureVersion=2&StackName=wordpress_ha2&Timestamp=2012-10-31T18%3A23%3A10Z&Version=2010-05-15
DEBUG [2012-10-31 14:23:10,827] len(b64)=44
DEBUG [2012-10-31 14:23:10,827] base64 encoded digest: UVVPWtW1hvHKYsTI5OsYNtfKn1de/JTXhxualmDgOLg=
DEBUG [2012-10-31 14:23:10,827] query_string: AWSAccessKeyId=58554c0766e940b7a892c4479c89feed&Action=DescribeStackResource&ContentType=JSON&LogicalResourceId=WikiDatabase&SignatureMethod=HmacSHA256&SignatureVersion=2&StackName=wordpress_ha2&Timestamp=2012-10-31T18%3A23%3A10Z&Version=2010-05-15 Signature: UVVPWtW1hvHKYsTI5OsYNtfKn1de/JTXhxualmDgOLg=
DEBUG [2012-10-31 14:23:11,060] encountered gaierror exception, reconnecting
DEBUG [2012-10-31 14:23:11,060] establishing HTTP connection: kwargs={}
DEBUG [2012-10-31 14:23:12,029] Token: None
DEBUG [2012-10-31 14:23:12,030] using _calc_signature_2
DEBUG [2012-10-31 14:23:12,030] query string: AWSAccessKeyId=58554c0766e940b7a892c4479c89feed&Action=DescribeStackResource&ContentType=JSON&LogicalResourceId=WikiDatabase&SignatureMethod=HmacSHA256&SignatureVersion=2&StackName=wordpress_ha2&Timestamp=2012-10-31T18%3A23%3A12Z&Version=2010-05-15
DEBUG [2012-10-31 14:23:12,030] string_to_sign: GET
__gateway_ip__:8000
/v1/

cfn_helper.py exposes credentials through world-readable tmpfile

cfn_helper.py creates /tmp/last_metadata which is world-readable and can contain the AWS credentials (as stored in /etc/cfn/cfn-credentials), we need to fix the permissions and/or stop storing raw metadata in temporary locations.

heat-jeos can NOT run against latest glance

Since the latest glance has "Drop Glance Client" (openstack/glance@0cd4a4e), the heat-jeos can NOT run against that, due to this line in bin/heat-jeos:

from glance import client as glance_client

create rpm

This will be just for heat-jeos.

cfntools imports

All the executable scripts in cfntools import from heat. This does not seem correct since heat-jeos is not supposed to depend on heat.

need better help with options

At the moment we have:

sudo -E heat-jeos help image 

    Create a new JEOS image.

    Usage:
    heat-jeos image <distribution> <architecture> <image type> <tdl>

It is not obvious what distro's are supported and what "image type" is.
Also where are the tdl's?

Maybe 2 or 3 examples would be good?

cfn-get-metadata does not make use of --credential-file argument

The argument is totally ignored.

cfn-hup does not work

I tried AutoScalingMultiAZSample.template. Then first, F17 LB_instance is started, but cfn-hup does not work. So HAProxy is not set.
I confirm LB_instance's credential file(/etc/cfn/cfn-credential) is blank(AWSAccessKeyId and AWSSecretKey are not set). Then boto client fails authorized.

My environment is baed on OpenStack Folsom(2012.2.3).
heat is commit:44ea0380041fd353a22f3d99a266d9cd3399ea5c.
heat-jeos is commit:88e0f4154d0539757085cb14de0e2f1bd864ba23

Resolve the '-jeos' suffix in image/template names

When the user interacts with heat-jeos, they see the template name as specified in the TDL. All our TDLs use the '-jeos' suffix: F16-x86_64-cfntools-jeos.

However, when we actually register the image with Glance and use it in Heat Templates, we drop the sufiix: F16-x86_64-cfntools.

I think we should get rid of the inconsistency. It's causing issues with the template translation feature1, it's confusing to the users and may cause trouble in the future.

I see two solutions:

1. remove the suffix from all the TDLs (in the template/name section)
2. keep the suffix, change heat-jeos to not remove the suffix during Glance registration and add the suffix to all the Heat templates we ship.

Unless we really want to keep using the '-jeos' suffix, I vote for number 1 since it's much less disruptive (all the already built & registered images and templates will continue to work).

--heartbeat flag not present in cfn-push-stats

Angus,

I think you missed a commit on cfn-push-stats

[root@wikidatabase bin]# ./cfn-push-stats --heartbeat --watch HeartbeatFailureAlarm
WARNING [2012-08-06 12:34:41,749] psutil not available. If you want process and memory statistics, you need to install it.
usage: cfn-push-stats [-h] [-v] [--service-failure] [--mem-util] [--mem-used]
[--mem-avail] [--swap-util] [--swap-used]
[--disk-space-util] [--disk-space-used]
[--disk-space-avail] [--memory-units MEMORY_UNITS]
[--disk-units DISK_UNITS] [--disk-path DISK_PATH]
[--haproxy] --watch WATCH
cfn-push-stats: error: unrecognized arguments: --heartbeat

IHA template doesn't work as a result

Add a Debian v6 JEOS tdl

Add a Debian v6 JEOS tdl that meets minimum requirements

Moved from https://github.com/heat-api/heat/issues/24

Update non-Fedora TDLs to install required version of boto

The cfntools depend on a recent (>=2.4.0) version of boto, so we get the required changes discussed under:
boto/boto#742
boto/boto#843

The correct version is included in F16/17/18 but not in recent versions of Ubuntu (not checked 12.10 yet). So we need to implement a workaround (install boto from source) in the tdls