galaxyproject/cloudlaunch

Easily launch cloud applications.

cloudlaunch's Issues

dev-mongo branch todo's

This is relation to #49 as one possible avenue for implementing the new launcher. This branch is based on the following stack: Django, Django Rest Framework, MongoDB, Django Rest Framework Mongoengine. The following is a fluid list of current tasks:

• Figure out how to POST a nested field (e.g., an Application with a Category)
• Define the models
• Integrate Django Social Auth
• Setup testing with Travis
• Write setup docs
• Decide how the Admin side of things will be handled given no default Django Admin site
• Add a Celery task queue

Specify VPC (or generic equivalent) when launching cluster [Enhancement]

Hello again,
I'm working under a corporate AWS account with an individual IAM user. I can't specify a default VPC and I have to run Galaxy in a particular VPC. Could VPC selection be implemented in Cloudman and/or Cloudlaunch? I see in the Cloudman Changelog that in August, support for VPC was added in some sense, but I'm not sure how to leverage this and/or specify it while using Cloudlaunch. Will this be part of the larger October Galaxy release?

Make backend plugins standalone

This documents a discussion on how we plan to handle backend plugins moving forward:
Currently, backend plugins access some cloudlaunch models, and therefore need a running cloudlaunch instance to be developed, tested or run against. However, there is no real reason to have this dependency, and we should be able to full decouple the two, allowing backend plugins to be executed in a standalone fashion, as well as from within cloudlaunch.

The planned strategy is to send in required objects as dicts. These objects can thus be provided through a commandline runner, say as json docs, as well as CloudLaunch itself. This should make plugin development much easier, and also allow for re-use outside of cloudlaunch.

Further along the line, we will also need to provide a way to bundle a backend plugin, along with its frontend component, and provide it together as a versioned, installable object within CloudLaunch.

Upgrade Django to 1.11

Django 1.11 is an LTS, which is what we want to stick with (and 1.9 is no longer supported: https://www.djangoproject.com/download/) so time to upgrade.

terminate instances from within CloudLaunch interface

README.md file unclear on using postgresql

While sample commands are provided, little explanation is provided of what they do, and they don't quite work on this instance of ubuntu 13.04 (I know, the docs say to specifically use 12.04, but, I figured I would give raring a try).

I got right up to the point where I need to create a database, and it's giving this error:

(bcc)janderson@ip-10-223-0-144:/gvl/bcc/biocloudcentral$ createdb --username bcc --port 5432 biocloudcentral
createdb: could not connect to database postgres: FATAL: Peer authentication failed for user "bcc"

In the meantime, I'll start a 12.04 LTS instance and try there.

dedicated server install prepends "None" to the page url

...which is a 404.

(e.g. instead of being redirected from http://my.ip.addr.here/ to http://my.ip.addr.here/launch, the user is redirected from http://my.ip.addr.here/ to http://my.ip.addr.here/None/launch )

I've tried setting "Redirect_Base" equal to "None", None (without quotes) and simply blank, but it doesn't seem to help. I've also restarted the service, and wiped/regenerated the static pages between each iteration, in case some cached value was tripping me up.

Should I instead set my redirect base to http://my.ip.addr.here/ ?

Add GCE cloud

Add the ability to interface with the Google Cloud. This will require storing the cloud connection properties in the model and creating appropriate serializers.

Note that support for the Google Cloud in CloudBridge has not been completed but it is sufficient to make progress here. See the gce branch: https://github.com/gvlproject/cloudbridge/tree/gce

add upload config file to recreate an existing cluster

It would be good to have a upload field to upload a ???-userdata.txt that you can download upon first creation of a cluster. The cloudlounch then should prefill all the values from the values in that file.

Allow users to add private clouds

It would be nice to allow for users to add their own private clouds, and optionally publish/share them, after being approved.

This is mostly useful for OpenStack clouds, but could also be useful for EC2 compatible clouds, and in future, Azure Stack based clouds.

Tasks

• Extend backend to store private cloud connection information.
• Add front-end component to define a cloud connection. Could be called My Connections, available through the My Profile page.

Add EBS optimized setting for master node

Enabling EBS optimization on the master node helps with NFS performance.

clear list of instances

It would be nice to remove entries from the displayed list, especially failed once or instances that are not running anymore.

Zone mismatches

Dear Brad Chapman,

Great tool you made! I've been playing with it a lot yesterday and today. I launches and relaunches a customized galaxy cloud instance in most cases.

There are however 2 issues:

1. Sometimes the amazon EC2 instance is launched in a different availability zone us-east1c instead of us-east-1b, therefore the stored snapshots of tools and data cannot be attached to the new instance. Here is the error:

13:16:56 - Retrieved file 'persistent_data.yaml' from bucket 'cm-e90ee86008a58c16f04d9ec29eebdb46' to 'pd.yaml'.
13:16:56 - Master starting
13:16:57 - Completed initial cluster configuration.
13:17:21 - PostgreSQL data directory '/mnt/galaxyData/pgsql/data' does not exist (yet?)
13:17:21 - Prerequisites OK; starting service 'SGE'
13:17:26 - Configuring SGE...
13:17:36 - Successfully setup SGE; configuring SGE
13:18:14 - Attaching volume 'vol-65c8be08' to instance 'i-7ea60e1c' as device '/dev/sdg3' failed. Exception: EC2ResponseError: 400 Bad Request InvalidVolume.ZoneMismatchThe volume 'vol-65c8be08' is not in the same availability zone as instance 'i-7ea60e1c'fd7371c2-acd6-43d1-9a16-8b5bd73f6df2

1. The size of the data partition is to small, it uses the full 2 GB. I therefore created a snapshot, created a volume of this snapshot of 6GB, created a snapshot of this again and managed to refer to this new snapshot in the yaml file. After this i could resize the new partition using xfs_resize -D 10000000 /dev/xdg2 (or whatever the volume of the galaxy-data partition). It would be usefull to have a bigger partition per default.

Keep up the great work, I hope I can contribute towards this.

Thomas

Unknown command celeryd for biocloudcentral/manage.py

Hello again,
I took advice from #42, setting up a fresh instance on a linux workstation, RHEL6 with pyenv-virtualenv. I took your suggestion @afgane and decided to install according to the repository's instructions. I manage fine until I receive the following error while starting supervisor. I notice that the celeryd command line tool is available(see below), although this subcommand is not available to manage.py

From the supervisor log (/srv/cloudlaunch/cloudlaunch/supervisor.log):

14:01:51 system   | web.1 started (pid=1543)
14:01:51 system   | celery.1 started (pid=1544)
14:01:52 celery.1 | Unknown command: 'celeryd'
14:01:52 celery.1 | Type 'manage.py help' for usage.
14:01:52 system   | celery.1 stopped (rc=1)
14:01:52 system   | sending SIGTERM to web.1 (pid 1543)
14:01:52 web.1    | Traceback (most recent call last):
14:01:52 web.1    |   File "../.cl/bin/gunicorn", line 11, in <module>
14:01:52 web.1    |     sys.exit(run())
14:01:52 web.1    |   File "/home/ralstonm/.pyenv/versions/cl/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
14:01:52 web.1    |     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
14:01:52 web.1    |   File "/home/ralstonm/.pyenv/versions/cl/lib/python2.7/site-packages/gunicorn/app/base.py", line 189, in run
14:01:52 web.1    |     super(Application, self).run()
14:01:52 web.1    |   File "/home/ralstonm/.pyenv/versions/cl/lib/python2.7/site-packages/gunicorn/app/base.py", line 72, in run
14:01:52 web.1    |     Arbiter(self).run()
14:01:52 web.1    |   File "/home/ralstonm/.pyenv/versions/cl/lib/python2.7/site-packages/gunicorn/arbiter.py", line 197, in run
14:01:52 web.1    |     self.halt()
14:01:52 web.1    |   File "/home/ralstonm/.pyenv/versions/cl/lib/python2.7/site-packages/gunicorn/arbiter.py", line 297, in halt
14:01:52 web.1    |     self.stop()
14:01:52 web.1    | 
14:01:52 system   | web.1 stopped (rc=1)

/srv/cloudlaunch >cloudlaunch/biocloudcentral/manage.py --help
Usage: manage.py subcommand [options] [args]

Options:
  -v VERBOSITY, --verbosity=VERBOSITY
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=all output
  --settings=SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
  --pythonpath=PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Print traceback on exception
  --version             show program's version number and exit
  -h, --help            show this help message and exit

Type 'manage.py help <subcommand>' for help on a specific subcommand.

Available subcommands:
  cleanup
  compilemessages
  createcachetable
  dbshell
  diffsettings
  dumpdata
  flush
  inspectdb
  loaddata
  makemessages
  reset
  runfcgi
  runserver
  shell
  sql
  sqlall
  sqlclear
  sqlcustom
  sqlflush
  sqlindexes
  sqlinitialdata
  sqlreset
  sqlsequencereset
  startapp
  startproject
  syncdb
  test
  testserver
  validate

/src/cloudlaunch >cd cloudlaunch

/srv/cloudlaunch/cloudlaunch >/bin/ls ../.cl/bin
activate      bundle_image  celeryd        cwutil        dynamodb_load      fetch_file   gunicorn_paster  list_instances  pidproxy    python     sdbadmin
activate.csh      camqadm   celeryd-multi  django-admin.py   easy_install       get-pip.py   honcho       lss3        pip         python2    supervisorctl
activate.fish     celery    celeryev       django-admin.pyc  easy_install-2.7   glacier      instance_events  mturk       pip2        python2.7  supervisord
activate_this.py  celerybeat    cfadmin        djcelerymon   echo_supervisord_conf  gunicorn     kill_instance    nosetests   pip2.7      route53    taskadmin
asadmin       celeryctl cq         dynamodb_dump     elbadmin       gunicorn_django  launch_instance  nosetests-2.7   pyami_sendmail  s3put      wheel

Login page does not exist (500 Internal Service Error)

Hello everyone. I have a fresh production install of cloudlaunch on RHEL6 configured with PostgreSQL and Nginx, following the README. When I access "my-server:portnum/admin" page, I receive a 500 error indicating that the login.html page could not be found.

Django error

Environment:


Request Method: GET
Request URL: http://localhost:8000/admin/

Django Version: 1.3
Python Version: 2.7.8
Installed Applications:
['django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.admin',
 'django.contrib.admindocs',
 'south',
 'bootstrap_toolkit',
 'biocloudcentral',
 'djcelery',
 'kombu.transport.django']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware')

Template Loader Error:
Django tried loading these templates, in this order:
Using loader django.template.loaders.filesystem.Loader:
/srv/cloudlaunch/cloudlaunch/templates/admin/login.html (File does not exist)
Using loader django.template.loaders.app_directories.Loader:
/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/bootstrap_toolkit/templates/admin/login.html (File does not exist)



Traceback:
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  111.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/contrib/admin/sites.py" in wrapper
  214.                 return self.admin_view(view, cacheable)(*args, **kwargs)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
  93.                     response = view_func(request, *args, **kwargs)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  79.         response = view_func(request, *args, **kwargs)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/contrib/admin/sites.py" in inner
  196.                 return self.login(request)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  79.         response = view_func(request, *args, **kwargs)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/contrib/admin/sites.py" in login
  331.         return login(request, **defaults)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
  93.                     response = view_func(request, *args, **kwargs)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  79.         response = view_func(request, *args, **kwargs)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/contrib/auth/views.py" in login
  69.                               context_instance=RequestContext(request, current_app=current_app))
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/shortcuts/__init__.py" in render_to_response
  20.     return HttpResponse(loader.render_to_string(*args, **kwargs), **httpresponse_kwargs)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/template/loader.py" in render_to_string
  181.         t = get_template(template_name)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/template/loader.py" in get_template
  157.     template, origin = find_template(template_name)
File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/django/template/loader.py" in find_template
  138.     raise TemplateDoesNotExist(name)

Exception Type: TemplateDoesNotExist at /admin/
Exception Value: admin/login.html

Contents of django bootstrap_toolkit (no admin directory):

>/bin/ls ~/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/bootstrap_toolkit/templates
bootstrap_toolkit
>/bin/ls ~/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/bootstrap_toolkit/templates/*
field_checkbox.html  field_errors.html      field.html             field_search.html    form.html      nav.html       pagination.html
field_choices.html   field_help.html        field_inline.html          field_vertical.html  formset.html   non_field_error.html   pills.html
field_default.html   field_horizontal.html  field_prepend_append.html  field_visible.html   messages.html  non_field_errors.html  tabs.html
>pip freeze
/srv/cloudlaunch/cloudlaunch >pip freeze
amqp==1.0.12
anyjson==0.3.3
billiard==2.7.3.31
bioblend==0.6.1
boto==2.38.0
celery==3.0.21
dj-database-url==0.2.1
Django==1.3
django-bootstrap-toolkit==2.12.0
django-celery==3.0.17
gunicorn==19.3.0
honcho==0.6.6
Jinja2==2.7
kombu==2.5.12
MarkupSafe==0.18
meld3==1.0.2
mock==1.0.1
nose==1.3.0
poster==0.8.1
psycopg2==2.4.2
python-dateutil==2.1
python-memcached==1.48
pytz==2013b0
PyYAML==3.10
requests==1.2.3
requests-toolbelt==0.4.0
simplejson==3.3.0
six==1.9.0
South==0.7.3
supervisor==3.1.3
wheel==0.24.0

how difficult would it be to have an "enter your own image id" on the drop-down menu ?

Create a CloudLaunch Docker image

We already have an Ansible playbook for deploying CloudLaunch: https://github.com/galaxyproject/ansible-cloudlaunch/
We should apply it towards a Docker container and see what needs to be done to make that happen.

As a first step, building a single image is fine but ideally we could build this as a Helm chart separating out the web server, the message queue, the application, and the database into dedicated services.

share an instance cluster field on biocloudcentral site

I am providing the share-an-instance cluster id on the corresponding filed on the biocloucentral page:

cm-04768f56d19d9ed91ac5e236b669f5ce/shared/2013-08-09--20-48
(this is a cluster I've shared previously through the cloudman interface)

However, when I click on the ip address of the started instance and get to the cloudman interface, it still asks me to choose option (Galaxy, share-an-instance, or data cluster).

Isn't it supposed to go ahead and start a cluster with volumes from my snapshots mounted based on the cm-04768f56d19d9ed91,,, id I've provided on the biocloudcentral site ?

Paging not implemented for cloud resources

Most viewsets in cloudlaunch simply call resource.list() to fetch objects.
e.g.

    def list_objects(self):
        provider = view_helpers.get_cloud_provider(self)
        return provider.compute.instances.list()

Since the limit and marker parameters are not being passed to cloudbridge, this will always only return the first 50 records. This needs to be integrated with DRF paging to pass in the limit and marker so that client side paging will work correctly.

cloudbridge no longer needs individual values for ec2 connection

Currently, cloudlaunch asks for ec2 connection details as a list of individual values, as follows:

                "ec2_region_name": "ap-southeast-2",
                "ec2_region_endpoint": "ec2.ap-southeast-2.amazonaws.com",
                "ec2_conn_path": "/",
                "ec2_port": null,
                "ec2_is_secure": true,
                "s3_host": "s3-ap-southeast-2.amazonaws.com",
                "s3_conn_path": "/",
                "s3_port": null,
                "s3_is_secure": true

However, after the boto3 upgrade, it's no longer required to provide these individually. Simply providing
ec2_endpoint_url
s3_endpoint_url
use_ssl
ssl_verify
and ec2_region_name is sufficient.

So we should be able to collapse ec2_region_endpoint, ec2_conn_path and ec2_is_secure to a single value. Dito for s3_host, s3_conn_path and s3_port.

dev branch todo's

This is relation to #49 for implementing the new launcher. This branch is based on the following stack: Django, Django Rest Framework, Django Social Auth, Celery. The app is intended to supply a REST API without an embedded UI (the UI will be a developed as a separate project to consume the API). The following is a fluid list of current tasks:

• Define the base app models
• Add a Celery task queue
• Decide if we want to use DRF-Nested-Routers for nicer URls
• Do we want to use slugs or IDs for resources?
• Define the following base API endpoints
  • /applications
  • /deployments
  • /infrastructure
• Stored credential encryption
• Provide ability to retrieve credentials from request (directly through headers or perhaps through middleware)
• Add OAuth support
• Tests and coverage
• Implement support for POST requests to be able to create resources
• Add support to track usage

Support for Neutron Networking?

Hello, just wondering if this has been tested with OpenStack's Neutron networking? Specifically on OpenStack Kilo.

Add the ability to update a deployment

'My Appliances' page is currently a log of launched instances but should be more of a dashboard allowing live view of the deployments and more control.

For starters, enable the status of an appliance to be updated according to the state of the cloud resource. This will require access to the user's credentials across all providers used for deployments. To limit the number of requests and credentials required, perhaps it would be good to just update the status of deployments launched in the past week, or month, or? and then allow each individual deployment to be explicitly updated.

Add support for OpenAPI and client bindings for other languages

With Django Rest Framework Swagger, we should be able to generate an OpenAPI spec for the current API. We can then use that spec to generate client bindings for multiple languages.

• Generate OpenAPI spec from Django Rest Framework, possibly using DRF Swagger.
  References:
  a. http://www.django-rest-framework.org/api-guide/schemas/#schemas
  b. https://github.com/marcgibbons/django-rest-swagger
• Generate a python client library using available code generators. Compare and choose a suitable generator:
  References:
  a. https://github.com/Azure/autorest
  b. https://github.com/swagger-api/swagger-codegen
• Make OpenAPI spec and client library available for download
• Stretch: Create a commandline client for cloudlaunch

Additional references:
https://apievangelist.com/2015/06/06/comparison-of-automatic-api-code-generation-tools-for-swagger/

Documentation on adding AWS cloud, instance types to fresh install?

Hi, I'm using the dockerized version from @afgane. I'd like to add the AWS cloud and some instance types, but I don't know where to find informtion about the "EC2 port", "S3" port, or others. Would someone let me know where I can find this information? Thanks for your time!

Security group create/update fails (thanks boto), no error provided to user in a GUI.

This error occurs after ICMP in the except block in the section concerning inter-instance communication. The POST request fails from the boto authorize statement stating that protocol is null, which is the first parameter is the classical method. However, my boto library seems to be up to date (2.38.0) I understand that this is mostly a boto failure, although this could provide the illusion to the user that a new security group was created or that an existing security group was updated when in reality, the inter-instance communication is not successfully authorized.

            # Add rule that allows communication between instances in the same SG
            g_rule_exists = False  # A flag to indicate if group rule already exists
            for rule in cmsg.rules:
                for grant in rule.grants:
                    if grant.name == cmsg.name:
                        g_rule_exists = True
                        bioblend.log.debug("Group rule already exists in the SG.")
                if g_rule_exists:
                    break
           if not g_rule_exists:
                try:
                    cmsg.authorize(src_group=cmsg)
                except EC2ResponseError as e:
                    err_msg = "A problem with security group authorization: {0} " \
                              "(code {1}; status {2})" \
                              .format(e.message, e.error_code, e.status)
                    bioblend.log.exception(err_msg)
                    progress['err_msg'] = err_msg
            bioblend.log.info("Done configuring '%s' security group" % cmsg.name)

[2015-10-23 15:08:19,451: DEBUG/MainProcess] Security group 'CloudMan' already exists; will add authorizations next.
[2015-10-23 15:08:19,451: DEBUG/MainProcess] Rule (20:21) already exists in the SG
[2015-10-23 15:08:19,451: DEBUG/MainProcess] Rule (22:22) already exists in the SG
[2015-10-23 15:08:19,452: DEBUG/MainProcess] Rule (80:80) already exists in the SG
[2015-10-23 15:08:19,452: DEBUG/MainProcess] Rule (443:443) already exists in the SG
[2015-10-23 15:08:19,452: DEBUG/MainProcess] Rule (8800:8800) already exists in the SG
[2015-10-23 15:08:19,452: DEBUG/MainProcess] Rule (9600:9700) already exists in the SG
[2015-10-23 15:08:19,452: DEBUG/MainProcess] Rule (30000:30100) already exists in the SG
[2015-10-23 15:08:19,452: DEBUG/MainProcess] ICMP rule already exists in CloudMan SG.
[2015-10-23 15:08:19,452: DEBUG/MainProcess] Method: POST
[2015-10-23 15:08:19,452: DEBUG/MainProcess] Path: https://ec2.amazonaws.com/
[2015-10-23 15:08:19,452: DEBUG/MainProcess] Data: 
[2015-10-23 15:08:19,453: DEBUG/MainProcess] Headers: {}
[2015-10-23 15:08:19,453: DEBUG/MainProcess] Host: ec2.amazonaws.com
[2015-10-23 15:08:19,453: DEBUG/MainProcess] Port: 443
...
[2015-10-23 15:08:19,572: DEBUG/MainProcess] <?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid value 'null' for protocol. VPC security group rules must specify protocols explicitly.</Message></Error></Errors><RequestID>f36e61be-64fe-40fc-bac5-09227f981253</RequestID></Response>
[2015-10-23 15:08:19,573: ERROR/MainProcess] 400 Bad Request
[2015-10-23 15:08:19,573: ERROR/MainProcess] <?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid value 'null' for protocol. VPC security group rules must specify protocols explicitly.</Message></Error></Errors><RequestID>f36e61be-64fe-40fc-bac5-09227f981253</RequestID></Response>
[2015-10-23 15:08:19,574: ERROR/MainProcess] A problem with security group authorization: Invalid value 'null' for protocol. VPC security group rules must specify protocols explicitly. (code InvalidParameterValue; status 400)
Traceback (most recent call last):
  File "/home/ralstonm/Projects/bioblend/bioblend/cloudman/launch.py", line 301, in create_cm_security_group
    cmsg.authorize(src_group=cmsg)
  File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/boto/ec2/securitygroup.py", line 203, in authorize
    dry_run=dry_run)
  File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/boto/ec2/connection.py", line 3191, in authorize_security_group
    params, verb='POST')
  File "/home/ralstonm/.pyenv/versions/cloudlaunch/lib/python2.7/site-packages/boto/connection.py", line 1227, in get_status
    raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid value 'null' for protocol. VPC security group rules must specify protocols explicitly.</Message></Error></Errors><RequestID>f36e61be-64fe-40fc-bac5-09227f981253</RequestID></Response>

Pip freeze

amqp==1.0.12
anyjson==0.3.3
billiard==2.7.3.31
-e git+https://github.com/MatthewRalston/bioblend.git@126ea70c3ad90ac02406b21d032d1b1d12d3858d#egg=bioblend-master
boto==2.38.0
celery==3.0.21
dj-database-url==0.2.1
Django==1.3
django-bootstrap-toolkit==2.12.0
django-celery==3.0.17
gunicorn==19.3.0
honcho==0.6.6
Jinja2==2.7
kombu==2.5.12
MarkupSafe==0.18
mock==1.0.1
nose==1.3.0
poster==0.8.1
psycopg2==2.4.2
python-dateutil==2.1
python-memcached==1.48
pytz==2013b0
PyYAML==3.10
requests==2.8.1
requests-toolbelt==0.4.0
simplejson==3.3.0
six==1.9.0
South==0.7.3
wheel==0.24.0

Make tasks an endpoint for deployments

At the moment, CloudLaunch assumes one Celery task per deployment, namely to launch the app. To allow other actions on for a deployment, let's allow multiple tasks to be associated with a deployment. This will allow other actions (e.g., update status, restart, shut down, etc.) to be performed more naturally. It will also provide provenance of actions.

Cloudlaunch monitor view hangs with no errors

Hi, I am launching an AMI I built on CentOS according to an adapted Galaxy-Cloudman playbook. I then built the Galaxy filesystem as well and saved the snapshot to an S3 bucket. I supplied this bucket (which contains galaxyFS-latest.tar.gz) and AMI to a Cloudlaunch instance setup using @afgane 's cloudlaunch-docker repo. When I submit this information, the "monitor" page hangs during the "Requesting" stage. Is there a log where I can monitor why this is hanging?

HTTP 400 Bad Request when configuring security group

Private Cloud Launch instance, commit 8d537a8. CloudMan security group (sg-8b3b99f3) exists (created earlier by https://launch.usegalaxy.org). Relevant entries from tasks.log:

[2016-04-11 16:46:01,747: DEBUG/MainProcess] Launching cluster 'cm-test' on Amazon - Virginia cloud from image ami-b45e59de on instance type c4.large.
[2016-04-11 16:46:02,018: ERROR/MainProcess] 400 Bad Request
[2016-04-11 16:46:02,018: ERROR/MainProcess] <?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidPermission.Duplicate</Code><Message>the specified rule "peer: sg-8b3b99f3, TCP, from port: 0, to port: 65535, ALLOW" already exists</Message></Error></Errors><RequestID>0b0a4d57-00f7-46d1-9be2-4da85221a3db</RequestID></Response>
[2016-04-11 16:46:02,019: ERROR/MainProcess] A problem with security group authorization: the specified rule "peer: sg-8b3b99f3, TCP, from port: 0, to port: 65535, ALLOW" already exists (code InvalidPermission.Duplicate; status 400)
Traceback (most recent call last):
  File "/srv/cloudlaunch/.cl/local/lib/python2.7/site-packages/bioblend/cloudman/launch.py", line 307, in create_cm_security_group
    cmsg.authorize(src_group=cmsg, ip_protocol='tcp', from_port=0, to_port=65535)
  File "/srv/cloudlaunch/.cl/local/lib/python2.7/site-packages/boto/ec2/securitygroup.py", line 203, in authorize
    dry_run=dry_run)
  File "/srv/cloudlaunch/.cl/local/lib/python2.7/site-packages/boto/ec2/connection.py", line 3191, in authorize_security_group
    params, verb='POST')
  File "/srv/cloudlaunch/.cl/local/lib/python2.7/site-packages/boto/connection.py", line 1227, in get_status
    raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidPermission.Duplicate</Code><Message>the specified rule "peer: sg-8b3b99f3, TCP, from port: 0, to port: 65535, ALLOW" already exists</Message></Error></Errors><RequestID>0b0a4d57-00f7-46d1-9be2-4da85221a3db</RequestID></Response>
[2016-04-11 16:46:02,019: INFO/MainProcess] Done configuring 'CloudMan' security group

List of available clouds

(this refers to, currently, the dev branch)

The following endpoint http://127.0.0.1:8000/api/v1/infrastructure/clouds/ will list all clouds registered on the instance, even if a user does not have credentials associated with all the available clouds. Should this be so or should the list be filtered based on the creds a user has registered?

docs need additional data, re custom AMIs and launch targets

I'm curious as to how I use biocloudcentral to launch my own custom AMIs, and also how I launch those AMIs into a VPC.

Thanks for any help you can lend me.

documentation unclear on accessing admin interface.

Does one use the "admin" user created as part of the django install?
I've been through it several times, but cannot seem to successfully login to the admin interface.

Securing Cloudman Galaxy master...

I'm tasked with ensuring that the cloud-based Galaxy instance launched from launch.usegalaxy.org passes the same security scans a permanent server would. I'm using the Cluster with Galaxy option. I've dug around quite a lot, but haven't quite managed to figure out the special magic that would resolve the following issues. I hit most of these issues with the 15.x AMI/Galaxy version I think, but details below were uncovered on the 16.x AMI/Galaxy version.

1. start up in HTTPS mode from the get go (rather than logging into the admin user-interface over HTTP and then toggling ssl mode).

2. update the Ubuntu instance with the latest security patches. I run "sudo apt-get updates; sudo unattended-upgrades", which seems to be the recommended recipe, and reboot the cluster (via the cloudman/admin link).

   1. it is unfortunate that there is no hook for a script to run earlier than any Galaxy services run which might save the reboot.

   2. Unfortunately, apache2 is upgraded and configured to start at reboot - its links must be removed from /etc/rc*.d to be consistent with the "before" state of the instance.

   3. Unfortunately, nginx is upgraded too, and as best I can make out, the upgraded version of ubuntu nginx does not provide the nginx upload module which is used by Galaxy and this makes it fail at startup. The error is "nginx: [emerg] unknown directive "upload_store" in /etc/nginx/..." in /var/log/cloudman/cm_boot.log.

3. If I stop ProFTPd, this change gets written to the s3 configuration bucket, no problem. However, after the reboot, the Galaxy service is never started by the supervisor, since it is waiting for ProFTPd to be started (and it never is, because it was stopped).

4. Where/how can I manipulate the ports opened by the security group? What is on the other end of each of these ports - there seems like a lot? I'd like to shut down everything other than HTTPS and SSH (at least to the outside world).

5. ssh access is permitted with either the password /or/ the key-pair - can I turn off password only access and require the key-pair?

6. (Not strictly a security issue, but observed when rebooting...) a new galaxyIndices volume gets created with each reboot.

Thanks,

-- n

Internal server error when fetching key pairs

Commit afbae38 running at http://cloudman.aws.stemcellcommons.org/launch.

Steps to reproduce

Enter access and secret keys, choose a saved cluster then fetch the key pairs.

Observed results

The key pair drop down is disabled and filled with some Javascript code.
Browser console:

jquery-2.0.2.min.js:6 POST http://cloudman.aws.stemcellcommons.org/get-key-pairs 500 (INTERNAL SERVER ERROR)

/srv/cloudlaunch/cloudlaunch/access.log:

127.0.0.1 - - [30/Aug/2016:18:24:45 -0400] "POST /get-key-pairs HTTP/1.0" 500 365 "http://cloudman.aws.stemcellcommons.org/launch" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"

However, there is nothing in /srv/cloudlaunch/cloudlaunch/error.log or in /var/log/nginx/error.log. Is there a way to see the actual server error (traceback)?

Cannot use Social Login

I have setup locally. When I try to use social login to sign up I get Internal server error.

Environment:


Request Method: GET
Request URL: http://localhost:4200/accounts/google/login/

Django Version: 1.11.5
Python Version: 3.5.1
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.sites',
 'nested_admin',
 'smart_selects',
 'corsheaders',
 'rest_framework.authtoken',
 'rest_auth',
 'allauth',
 'allauth.account',
 'rest_auth.registration',
 'allauth.socialaccount',
 'allauth.socialaccount.providers.facebook',
 'allauth.socialaccount.providers.github',
 'allauth.socialaccount.providers.google',
 'allauth.socialaccount.providers.twitter',
 'baselaunch',
 'rest_framework',
 'kombu.transport.django',
 'djcelery',
 'django_countries']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "/Users/shakeel/Bioinformatics/biocentral/venv/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/Users/shakeel/Bioinformatics/biocentral/venv/lib/python3.5/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/Users/shakeel/Bioinformatics/biocentral/venv/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/Users/shakeel/Bioinformatics/biocentral/venv/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/Users/shakeel/Bioinformatics/biocentral/venv/lib/python3.5/site-packages/allauth/socialaccount/providers/oauth2/views.py" in view
  73.                 return self.dispatch(request, *args, **kwargs)

File "/Users/shakeel/Bioinformatics/biocentral/venv/lib/python3.5/site-packages/allauth/socialaccount/providers/oauth2/views.py" in dispatch
  96.         app = provider.get_app(self.request)

File "/Users/shakeel/Bioinformatics/biocentral/venv/lib/python3.5/site-packages/allauth/socialaccount/providers/base.py" in get_app
  52.         return SocialApp.objects.get_current(self.id, request)

File "/Users/shakeel/Bioinformatics/biocentral/venv/lib/python3.5/site-packages/allauth/socialaccount/models.py" in get_current
  40.                 provider=provider)

File "/Users/shakeel/Bioinformatics/biocentral/venv/lib/python3.5/site-packages/django/db/models/manager.py" in manager_method
  85.                 return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/Users/shakeel/Bioinformatics/biocentral/venv/lib/python3.5/site-packages/django/db/models/query.py" in get
  380.                 self.model._meta.object_name

Exception Type: DoesNotExist at /accounts/google/login/
Exception Value: SocialApp matching query does not exist.

What's missing for me?

running 'sudo service bcc start' never returns to command line...

following the instructions for a dedicated server build, everything appears to have gone successfully, but I'm now at ten minutes of waiting for service bcc to start...

/var/log/bcc_server.log contains this:

2013-10-30 14:26:28 [23861] [INFO] Listening at: http://127.0.0.1:8000 (23861)
2013-10-30 14:26:28 [23861] [INFO] Using worker: sync
2013-10-30 14:26:28 [23871] [INFO] Booting worker with pid: 23871
2013-10-30 14:26:28 [23872] [INFO] Booting worker with pid: 23872
2013-10-30 14:26:28 [23873] [INFO] Booting worker with pid: 23873

choose custom instance type for master

would be great to have a text box for this.
With manipulating the HTML code and POST request, it works so far.

Add private cloud type

Hello, I've been working towards having this operate on my private Openstack cloud. I have Cloudlaunch running on my local machine, just wondering how I can configure it to spawn a Cloudman instance on my dev cluster instead of Amazon?

Make obscured passwords visible on demand

Follow up to #37

Cloud connection problem ?

I get

Cloud connection problem:
A problem starting your instance. Check the Amazon cloud's console: Problem launching an instance: EC2ResponseError: 400 Bad Request UnsupportedThe requested Availability Zone is currently constrained and we are no longer accepting new customer requests for t1/m1/c1/m2/m3 instance types. Please retry your request by not specifying an Availability Zone or choosing us-east-1c, us-east-1b, us-east-1a.899c81cf-cfdb-4b00-9cd8-a2e5bc06e545

Attached is a screenshot showing all the parameters.

Add multi-cloud browser frontend

Since the backend infrastructure is already available, it would be nice to see a frontend that can allow browsing through multiple cloud providers, and carrying out common tasks on them, such as launching instances, deleting them, rebooting etc.

The functionality would be similar to the AWS Console or OpenStack Dashboard, except that it would have multi-cloud support.

A sort of "Cloud Commander".

Split components into django modules

There are several parts to cloudlaunch that can be split into separate, installable Django modules. In addition, the current module organisation is not that great.

For example, the entire API for browsing and interacting with cloudbridge-based cloud resources, could possible be a separately installable module. That separate module alone would be useful to implement: #92

In addition, we could also have a separate module for the public appliances/map data component.

Finally, the rest of cloudlaunch could integrate these two modules and add the extra functionality for launching appliances.

Tasks

• Factor out "Cloud Commander"/resource browser component.
  Complications here include task management, and cloud credentials. These should probably
  be moved into this module. Use this to implement: #92
• Factor out public appliances module
• Compose cloudlaunch out of the other two modules, adding the extra functionality for launching appliances.

boto Dependency Problem

The following bioblend changeset by @nuwang added a validate_certs argument when connecting to EC2.

afgane/bioblend@b585245

A problem is this was introduced with boto somewhere between versions 2.5 and 2.6. Since biocloudcentral explicitly targets boto 2.3 (in requirements.txt), biocloudcentral seems broken out of the box.

I guess biocloudcentral should target a newer version of boto. I can try to update this, but I thought I would give someone else the opportunity if there was some specific tests that should be run.

The above changeset is problematic though also, right? Those checks should happen in most cases. I understand that for some private clouds this is going to be problematic, but the deactivation of this should be optional. Additionally along those lines, perhaps bioblend should set a minimum required version of boto with its own dependency so problems like this are less likely to occur in downstream projects (this is possible right?).

Thanks all.

Problem selecting a Cloud and entering an Access key

Clicking on the Cloud dropdown results in opening of https://github.com/galaxyproject/cloudman/blob/master/CHANGELOG.md in a new window and clicking on the Access key input box results in opening of https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key in a new window.

Need to enforce RFC1035 + spaces and underscores to match cloudbridge

All cloudbridge names need to be validated against the above, once the following issue is addressed:
CloudVE/cloudbridge#55

RFC: all new CloudLaunch

The process of launching Galaxy cloud instances has gone through several iterations over the past few years, starting with the manual process via the cloud’s web dashboard to the current CloudLaunch web app. This evolution was largely driven by the goal of making the launch process easier and quicker. Going forward, we have several improvements in mind that are summarized here in hope they get assessed and evaluated by a larger community with possibly new improvements suggested.

• Unify launch options: combine the Docker offering with the cloud offering and further enhance it with the Flavor Generator. We should allow users to launch Galaxy instances on a range of infrastructures, using different technologies from a single location/app. Namely, it should be possible to launch a cloud cluster on any one of a range of cloud providers as well as a Docker instance on a cloud container service or a cluster that supports container scheduling. Further, allow users to optionally build their own flavors using the app’s UI. Given the launcher app can already provision various resources, add support to automatically supply the infrastructure required to build a flavor. The user will then have an option to store, launch, and share their flavors.
• Implement a REST API: follow modern web app design and create a full-featured API that any UI can then leverage. Most likely, implement this using the Django REST framework.
• Allow user login: this will allow users to see all of their deployments regardless of when they were launched. For the cloud cluster case, it will also be possible to delete the clusters without needing to launch them first. Optionally, we can add the ability for users to store their cloud creds (at least the access key).
• Start with a fresh codebase: the current CloudLaunch app has been around for a while and has gone through a name change that has not been consistently updated in the codebase. Django v1.3 is still used (v1.9 is expected any day now). With the availability of Cloudbridge, much of the cloud-related code will need to be changed. It seems that starting with a blank slate and reusing code snippets (e.g., the model) will be easier, quicker, and lead to better code structure than going through the code migration process. Along these lines, develop this using Python 3.5 and drop support for Heroku.

Anyone, please feel free to comment on this topic but explicitly pinging @bgruening @nuwang @dannon @wookoouk.

Impossible to select a placement zone

Attempting to fetch the available zone list (after filling out the required fields) results in error message "tuple index out of range".

doesn't run... :-(

• followed all steps from :
  http://bcbio.wordpress.com/2011/11/29/making-next-generation-sequencing-analysis-pipelines-easier-with-biocloudcentral-and-galaxy-integration/

after hitting "Process" in the very last step from within Galaxy pipelines, it simply returns in the main Galaxy and doesn't show any running jobs there. Running qstat after having logged in does not show anything either.

• while logged in I also tried :
  ubuntu@ip-10-116-201-41:/export/data/work/2worker-run$ distributed_nextgen_pipeline.py /export/data/galaxy/post_process.yaml /export/data/exome_example/fastq /export/data/exome_example/config/run_info.yaml
  Starting manager
  .... but it just keeps hanging here forever.....

Where could I pull more data from to help you debug it ?

EBS optimization is not enabled when selected

Running commit afbae38 in a private Cloud Launch instance. Starting a cluster with a c4.large master and EBS-optimized setting enabled under advanced options. However, ec2 describe-instances reports "EbsOptimized": false for that instance. There are no errors in the tasks.log except for the one described in #56.

problem with how data are passed via biocloudcentral to the booted instance ?

This is what I see as user data to my instance, I am not sure if this is a problem or not. Is the "!!python.." in front of each supposed to be there ?

access_key: !!python/unicode 'AKIAJLBRCY******'
bucket_default: !!python/unicode 'cm-04768f56d19d9ed91ac5e236b669f5ce/shared/2013-08-09--20-48'
cloud_name: !!python/unicode 'Amazon'
cluster_name: !!python/unicode 'papercluster6'
freenxpass: !!python/unicode 'ubuntu'
password: !!python/unicode 'ubuntu'
secret_key: !!python/unicode 'i2By73KBXjot2eJP******'
share_string: !!python/unicode 'cm-04768f56d19d9ed91ac5e236b669f5ce/shared/2013-08-09--20-48'

cloud_type: !!python/unicode 'ec2'
ec2_conn_path: !!python/unicode '/'
ec2_port: null
is_secure: true
region_endpoint: !!python/unicode 'ec2.amazonaws.com'
region_name: !!python/unicode 'us-east-1'
s3_conn_path: !!python/unicode '/'
s3_host: !!python/unicode 's3.amazonaws.com'
s3_port: null