jcvi-cloud/cloudman

clone of main hg cloudman repository https://bitbucket.org/galaxy/cloudman in order to add eucalyptus functionality

CloudMan

Easily create a compute cluster on top of a cloud computing infrastructure.

Overview

CloudMan is a cloud manager that orchestrates all of the steps required to provision a complete compute cluster environment on a cloud infrastructure; subsequently, it allows one to manage the cluster, all through a web browser. Although CloudMan can be used in any domain and for any purpose that calls for a compute cluster, it is primarily used in the context of Galaxy Cloud and CloudBioLinux and, along with the infrastructure, ensures a complete Next Generation Sequencing (NGS) analysis toolset is instantly available. CloudMan is currently available on the AWS EC2 cloud.

Use

To instantiate a CloudMan cluster, simply visit biocloudcentral.org.

Local deployment

For basic testing and some development, CloudMan can be run locally. Start by cloning CloudMan source, installing virtualenv, and adding Python libraries required by CloudMan. Then, run it:

$ cd <project root dir>
$ hg clone https://bitbucket.org/galaxy/cloudman
$ virtualenv --no-site-packages .
$ source bin/activate
$ pip install -r cloudman/requirements.txt
$ sh cloudman/run.sh [--reload]

Custom cloud deployment

If you would like to deploy CloudMan and all of its dependencies on a cloud infrastructure where a public image does not already exist, take a look at mi-deployment scripts (mi_fabfile.py in particular), which enable an easy way to do so.

LICENSE

The code is freely available under the MIT license.

cloudman's Issues

web interface does not permit nodes to be terminated

Adding a new worker node does not properly add it to the SGE grid

Essentially, it's the same problem as getting the hostname from the original. It probably also needs to be added to /etc/hosts

Log:

[DEBUG] sge:144 2012-05-03 22:39:45,751: Adding instance 'i-35A80758' w/ private IP '172.19.1.5' to SGE.
[INFO] sge:147 2012-05-03 22:39:45,752: Adding instance 'i-35A80758' as SGE administrative host.
[DEBUG] sge:150 2012-05-03 22:39:45,752: Add SGE admin host cmd: 'export SGE_ROOT=/opt/sge;. $SGE_ROOT/default/common/settings.sh; /
opt/sge/bin/lx24-amd64/qconf -ah 172.19.1.5'
[ERROR] sge:156 2012-05-03 22:39:45,998: Process encountered problems adding instance 'i-35A80758' as administrative host. Process r
eturned code 1
[DEBUG] sge:158 2012-05-03 22:39:45,999: Adding instance 'i-35A80758' SGE administrative host stdout (private IP: '172.19.1.5'): ''
[DEBUG] sge:159 2012-05-03 22:39:45,999: Adding instance 'i-35A80758' SGE administrative host stderr (private IP: '172.19.1.5'): 'ca
n't resolve hostname "172.19.1.5"
'
[INFO] sge:169 2012-05-03 22:39:46,167: Adding instance 'i-35A80758' to SGE execution host list.
[DEBUG] sge:179 2012-05-03 22:39:46,173: Created SGE host configuration template as file '/opt/sge/host_confs/i-35A80758'.
[DEBUG] sge:183 2012-05-03 22:39:46,174: Add SGE exec host cmd: 'export SGE_ROOT=/opt/sge; . $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -Ae /opt/sge/host_confs/i-35A80758'
[ERROR] sge:189 2012-05-03 22:39:46,362: Process encountered problems adding instance 'i-35A80758' as execution host. Process returned code 1
[DEBUG] sge:192 2012-05-03 22:39:46,362: Adding instance 'i-35A80758' SGE execution host stdout (private IP: '172.19.1.5'): ''
[DEBUG] sge:193 2012-05-03 22:39:46,363: Adding instance 'i-35A80758' SGE execution host stderr (private IP: '172.19.1.5'): 'can't resolve hostname "172.19.1.5"
'
[DEBUG] sge:237 2012-05-03 22:39:46,531: to_add: '172.19.1.5'
[DEBUG] sge:238 2012-05-03 22:39:46,532: to_remove: 'None'
[DEBUG] sge:240 2012-05-03 22:39:46,532: Composing SGE's @allhosts group config file '/tmp/ah_add_22_39_46':
[DEBUG] sge:241 2012-05-03 22:39:46,532: - adding master instance; IP '172.19.1.3'
[DEBUG] sge:246 2012-05-03 22:39:46,532: - adding instance with IP '172.19.1.5' (instance state: 'Startup')
[DEBUG] sge:263 2012-05-03 22:39:46,533: new_allhosts:
group_name @allhosts
hostlist 172.19.1.3
172.19.1.5 \

[DEBUG] sge:264 2012-05-03 22:39:46,533: New SGE @allhosts file written successfully to /tmp/ah_add_22_39_46.
[ERROR] misc:504 2012-05-03 22:39:46,705: Problems updating @allhosts aimed at adding 'i-35A80758', running command 'export SGE_ROOT=/opt/sge;. $SGE_ROOT/default/common/settings.sh; /opt/sge/bin/lx24-amd64/qconf -Mhgrp /tmp/ah_add_22_39_46' returned code '1' and following stderr: 'unable to resolve host "172.19.1.5"

New worker instances don't get eucalyptus connection info

In cm.util.master.ConsoleManager.add_instances(), it creates user-info that only includes

• access_key
• secret_key
• password
• cluster_name
• role : worker
• master_ip

but not

• s3_url
  or
• ec2_url
  So the workers download the wrong cm_boot and cm.tar.gz

Fix filtering of query blocking filesystem status

After mounting a drive, the status check is failing.
Log shows:
[ERROR] connection:894 2012-04-29 14:21:15,837: 400 Bad Request
[ERROR] connection:895 2012-04-29 14:21:15,837: Failure: 400 Bad Request
Failed to bind the following fields:
Filter.1.Name = attachment.instance-id
Filter.1.Value.1 = i-3CDF07E1
Filter.2.Name = attachment.device
Filter.2.Value.1 = None

[ERROR] filesystem:565 2012-04-29 14:21:15,838: STATUS CHECK: Exception checking status of FS 'galaxyData': EC2ResponseError: 400 Bad Request
None
[DEBUG] filesystem:567 2012-04-29 14:21:15,838: (0, '/dev/vda /mnt/galaxyData')
[DEBUG] master:1525 2012-04-29 14:21:15,839: S&S: SGE..OK; FS-galaxyData..Error;

web interface will not add more than one node at a time