I'm seeing that the application launches pretty well except for the AMQP connection. Towards the bottom, a service failure is mentioned where the "hostname" service fails to restart (i've read this is different on RedHat machines). Is this causing the connection to fail? Alternatively, I am behind a proxy which could be disrupting the connection. Could someone suggest how to enable the AMQP connection behind my proxy? Is there a line in a profile shell script or in /etc/hosts/
that I can edit to enable this?
On a separate note, I'm running on CentOS 6. Why does "Running on Ubuntu: True" get set?
Python version: (2, 7)
2015-11-02 12:49:43,132 DEBUG app:80 Initializing app
2015-11-02 12:49:43,132 DEBUG ec2:115 Gathering instance zone, attempt 0
2015-11-02 12:49:43,136 DEBUG ec2:121 Instance zone is 'us-east-1a'
2015-11-02 12:49:43,137 DEBUG ec2:44 Gathering instance ami, attempt 0
2015-11-02 12:49:43,139 DEBUG app:83 Running on 'ec2' type of cloud in zone 'us-east-1a' using image 'ami-123456789'.
2015-11-02 12:49:43,139 DEBUG app:106 Looking for existing cluster persistent data (PD).
2015-11-02 12:49:43,139 DEBUG ec2:357 No S3 Connection, creating a new one.
2015-11-02 12:49:43,141 DEBUG ec2:361 Got boto S3 connection.
2015-11-02 12:49:43,857 DEBUG misc:597 Failed to get file 'persistent_data.yaml' from bucket 'my_company_bucket': S3ResponseError: 404 Not Found
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>persistent_data.yaml</Key><RequestId>bobloblaw</RequestId><HostId>123456789</HostId></Error>
2015-11-02 12:49:43,858 DEBUG app:125 No PD to go by. Setting deployment_version to 2.
2015-11-02 12:49:43,858 DEBUG app:132 Master process starting.
2015-11-02 12:49:43,938 DEBUG master:51 Initializing console manager - cluster start time: 2015-11-02 17:49:43.938619
2015-11-02 12:49:43,938 DEBUG comm:30 Setting up a new AMQP connection
2015-11-02 12:49:43,939 DEBUG comm:49 AMQP Connection Failure: [Errno 111] Connection refused
2015-11-02 12:49:43,940 DEBUG master:938 Trying to discover any worker instances associated with this cluster...
2015-11-02 12:49:43,940 DEBUG ec2:336 Establishing boto EC2 connection
2015-11-02 12:49:44,472 DEBUG ec2:328 Got region as 'RegionInfo:us-east-1'
2015-11-02 12:49:45,391 DEBUG ec2:345 Got boto EC2 connection for region 'us-east-1'
2015-11-02 12:49:45,507 DEBUG ec2:82 Gathering instance id, attempt 0
2015-11-02 12:49:45,510 DEBUG ec2:88 Instance ID is 'i-c1234567'
2015-11-02 12:49:45,837 DEBUG ec2:384 Adding tag 'clusterName:cloudlaunch-test-launch-v5' to resource 'i-c1234567'
2015-11-02 12:49:46,086 DEBUG ec2:384 Adding tag 'role:master' to resource 'i-c1234567'
2015-11-02 12:49:46,341 DEBUG ec2:384 Adding tag 'Name:master: cloudlaunch-test-launch-v5' to resource 'i-1234567'
2015-11-02 12:49:46,636 DEBUG registry:164 Initiating loading of services
2015-11-02 12:49:46,637 DEBUG registry:128 Loading service class in module 'cm/services/autoscale.py'
2015-11-02 12:49:46,890 DEBUG registry:145 Importing service name AutoscaleService as module cm.services.autoscale
2015-11-02 12:49:46,893 DEBUG registry:170 Loaded service Autoscale
2015-11-02 12:49:46,895 DEBUG registry:128 Loading service class in module 'cm/services/apps/clouderamanager.py'
2015-11-02 12:49:47,906 DEBUG registry:145 Importing service name ClouderaManagerService as module cm.services.apps.clouderamanager
2015-11-02 12:49:48,261 DEBUG registry:170 Loaded service ClouderaManager
2015-11-02 12:49:48,262 DEBUG registry:128 Loading service class in module 'cm/services/apps/pulsar.py'
2015-11-02 12:49:48,280 DEBUG registry:145 Importing service name PulsarService as module cm.services.apps.pulsar
2015-11-02 12:49:48,286 DEBUG registry:170 Loaded service Pulsar
2015-11-02 12:49:48,286 DEBUG registry:128 Loading service class in module 'cm/services/apps/htcondor.py'
2015-11-02 12:49:48,306 DEBUG registry:145 Importing service name HTCondorService as module cm.services.apps.htcondor
2015-11-02 12:49:48,310 DEBUG htcondor:24 Condor is preparing
2015-11-02 12:49:48,310 DEBUG registry:170 Loaded service HTCondor
2015-11-02 12:49:48,310 DEBUG registry:128 Loading service class in module 'cm/services/apps/nginx.py'
2015-11-02 12:49:48,323 DEBUG registry:145 Importing service name NginxService as module cm.services.apps.nginx
2015-11-02 12:49:48,327 DEBUG registry:170 Loaded service Nginx
2015-11-02 12:49:48,327 DEBUG registry:128 Loading service class in module 'cm/services/apps/nodejsproxy.py'
2015-11-02 12:49:48,332 DEBUG registry:145 Importing service name NodejsProxyService as module cm.services.apps.nodejsproxy
2015-11-02 12:49:48,334 DEBUG registry:170 Loaded service NodeJSProxy
2015-11-02 12:49:48,334 DEBUG registry:128 Loading service class in module 'cm/services/apps/supervisor.py'
2015-11-02 12:49:48,457 DEBUG registry:145 Importing service name SupervisorService as module cm.services.apps.supervisor
2015-11-02 12:49:48,461 DEBUG registry:170 Loaded service Supervisor
2015-11-02 12:49:48,461 DEBUG registry:128 Loading service class in module 'cm/services/apps/pss.py'
2015-11-02 12:49:48,470 DEBUG registry:145 Importing service name PSSService as module cm.services.apps.pss
2015-11-02 12:49:48,473 DEBUG pss:29 Configured PSS as master
2015-11-02 12:49:48,473 DEBUG registry:170 Loaded service PSS
2015-11-02 12:49:48,473 DEBUG registry:128 Loading service class in module 'cm/services/apps/postgres.py'
2015-11-02 12:49:48,487 DEBUG registry:145 Importing service name PostgresService as module cm.services.apps.postgres
2015-11-02 12:49:48,516 DEBUG registry:170 Loaded service Postgres
2015-11-02 12:49:48,517 DEBUG registry:128 Loading service class in module 'cm/services/apps/hadoop.py'
2015-11-02 12:49:48,721 DEBUG registry:145 Importing service name HadoopService as module cm.services.apps.hadoop
2015-11-02 12:49:48,737 DEBUG registry:170 Loaded service Hadoop
2015-11-02 12:49:48,738 DEBUG registry:128 Loading service class in module 'cm/services/apps/galaxy.py'
2015-11-02 12:49:48,750 DEBUG registry:145 Importing service name GalaxyService as module cm.services.apps.galaxy
2015-11-02 12:49:48,755 DEBUG registry:170 Loaded service Galaxy
2015-11-02 12:49:48,755 DEBUG registry:128 Loading service class in module 'cm/services/apps/galaxyreports.py'
2015-11-02 12:49:48,761 DEBUG registry:145 Importing service name GalaxyReportsService as module cm.services.apps.galaxyreports
2015-11-02 12:49:48,783 DEBUG registry:170 Loaded service GalaxyReports
2015-11-02 12:49:48,784 DEBUG registry:128 Loading service class in module 'cm/services/apps/migration.py'
2015-11-02 12:49:48,802 DEBUG registry:145 Importing service name MigrationService as module cm.services.apps.migration
2015-11-02 12:49:48,808 DEBUG registry:170 Loaded service Migration
2015-11-02 12:49:48,808 DEBUG registry:128 Loading service class in module 'cm/services/apps/proftpd.py'
2015-11-02 12:49:48,813 DEBUG registry:145 Importing service name ProFTPdService as module cm.services.apps.proftpd
2015-11-02 12:49:48,815 DEBUG proftpd:26 Initializing ProFTPdService
2015-11-02 12:49:48,815 DEBUG registry:170 Loaded service ProFTPd
2015-11-02 12:49:48,816 DEBUG registry:128 Loading service class in module 'cm/services/apps/cloudgene.py'
2015-11-02 12:49:48,821 DEBUG registry:145 Importing service name CloudgeneService as module cm.services.apps.cloudgene
2015-11-02 12:49:48,822 DEBUG registry:170 Loaded service Cloudgene
2015-11-02 12:49:48,823 DEBUG registry:128 Loading service class in module 'cm/services/apps/jobmanagers/sge.py'
2015-11-02 12:49:48,861 DEBUG registry:145 Importing service name SGEService as module cm.services.apps.jobmanagers.sge
2015-11-02 12:49:48,879 DEBUG registry:170 Loaded service SGE
2015-11-02 12:49:48,880 DEBUG registry:128 Loading service class in module 'cm/services/apps/jobmanagers/slurmd.py'
2015-11-02 12:49:48,884 DEBUG registry:145 Importing service name SlurmdService as module cm.services.apps.jobmanagers.slurmd
2015-11-02 12:49:48,885 DEBUG registry:170 Loaded service Slurmd
2015-11-02 12:49:48,886 DEBUG registry:128 Loading service class in module 'cm/services/apps/jobmanagers/slurmctld.py'
2015-11-02 12:49:49,225 DEBUG registry:145 Importing service name SlurmctldService as module cm.services.apps.jobmanagers.slurmctld
2015-11-02 12:49:49,229 DEBUG paths:228 Warning: Returning default transient file system path
2015-11-02 12:49:49,230 DEBUG registry:170 Loaded service Slurmctld
2015-11-02 12:49:49,230 DEBUG master:2401 Loaded services: {'NodeJSProxy': NodeJSProxy, 'Postgres': Postgres, 'ProFTPd': ProFTPd, 'Autoscale': Autoscale, 'GalaxyReports': Galaxy Reports service on port 9001, 'Hadoop': Hadoop, 'ClouderaManager': ClouderaManager, 'Slurmd': Slurmd, 'Nginx': Nginx, 'HTCondor': HTCondor, 'Supervisor': Supervisor, 'PSS': PSS, 'Pulsar': Pulsar, 'Slurmctld': Slurmctld, 'SGE': SGE, 'Cloudgene': Cloudgene, 'Galaxy': Galaxy, 'Migration': Migration}
2015-11-02 12:49:49,230 DEBUG master:2721 Starting __monitor thread
2015-11-02 12:49:49,233 DEBUG master:360 Config Data at manager start (with the following keys filtered out ['password', 'freenxpass', 'access_key', 'secret_key']): {'static_images_dir': 'static/images', 'storage_type': u'volume', 'is_secure': True, 's3_port': None, 'cloud_type': u'ec2', 'static_cache_time': '360', 'debug': 'true', 'initial_cluster_type': u'Galaxy', 'static_scripts_dir': 'static/scripts', 'cluster_name': u'cloudlaunch-test-launch-v5', 'role': 'master', 'bucket_cluster': 'bmsrd-ngs-galaxy', 'boot_script_path': '/opt/cloudman/boot', 'ec2_conn_path': u'/', 'region_name': u'us-east-1', 'region_endpoint': u'ec2.amazonaws.com', 'ec2_port': None, 'static_favicon_dir': 'static/favicon.ico', 'storage_size': u'10', 'use_translogger': 'False', 'boot_script_name': 'cm_boot.py', 'log_level': 'DEBUG', 'custom_image_id': u'ami-1234567', 'use_lint': 'false', 'cloudman_file_name': 'cm.tar.gz', 'global_conf': {'__file__': '/mnt/cm/cm_wsgi.ini', 'here': '/mnt/cm'}, 'template_path': 'templates', 'cloud_name': u'Amazon', 'static_dir': 'static', 'static_style_dir': 'static/style', 'cloudman_home': '/mnt/cm', 'bucket_default': 'cloudman', 'custom_instance_type': u'', 's3_host': u's3.amazonaws.com', 's3_conn_path': u'/', 'static_enabled': 'True'}
2015-11-02 12:49:49,233 DEBUG master:2254 Generating root user's public key...
2015-11-02 12:49:49,247 DEBUG base:57 Enabling 'root' controller, class: CM
2015-11-02 12:49:49,268 DEBUG buildapp:93 Enabling 'httpexceptions' middleware
2015-11-02 12:49:49,270 DEBUG buildapp:99 Enabling 'recursive' middleware
2015-11-02 12:49:49,273 DEBUG buildapp:119 Enabling 'print debug' middleware
2015-11-02 12:49:49,281 DEBUG buildapp:133 Enabling 'error' middleware
2015-11-02 12:49:49,281 DEBUG buildapp:143 Enabling 'config' middleware
2015-11-02 12:49:49,283 DEBUG buildapp:147 Enabling 'x-forwarded-host' middleware
Starting server in PID 23047.
serving on 0.0.0.0:42284 view at http://127.0.0.1:42284
2015-11-02 12:49:49,387 DEBUG master:2257 Successfully generated root user's public key.
2015-11-02 12:49:49,387 DEBUG master:2265 Successfully retrieved root user's public key from file.
2015-11-02 12:49:49,388 DEBUG master:113 Activating service Migration
2015-11-02 12:49:49,388 DEBUG master:157 ADD dependencies for service Migration
2015-11-02 12:49:49,388 DEBUG master:113 Activating service Nginx
2015-11-02 12:49:49,388 DEBUG master:157 ADD dependencies for service Nginx
2015-11-02 12:49:49,388 DEBUG master:113 Activating service Supervisor
2015-11-02 12:49:49,388 DEBUG master:157 ADD dependencies for service Supervisor
2015-11-02 12:49:49,413 DEBUG misc:840 'cat /etc/lsb-release | grep DISTRIB_RELEASE | cut -f2 -d'='' command OK
2015-11-02 12:49:49,414 DEBUG master:391 Running on Ubuntu True; using SGE as the cluster job manager
2015-11-02 12:49:49,415 DEBUG master:113 Activating service SGE
2015-11-02 12:49:49,415 DEBUG master:157 ADD dependencies for service SGE
2015-11-02 12:49:49,415 DEBUG registry:99 Removing service Slurmctld from the registry
2015-11-02 12:49:49,415 DEBUG registry:99 Removing service Slurmd from the registry
2015-11-02 12:49:49,416 DEBUG filesystem:32 Instantiating Filesystem object transient_nfs with service roles: 'TransientNFS'
2015-11-02 12:49:49,416 DEBUG filesystem:599 Configuring instance transient storage at /mnt/transient_nfs with NFS.
2015-11-02 12:49:49,416 DEBUG master:107 Adding a new file system service into the registry: transient_nfs
2015-11-02 12:49:49,416 DEBUG registry:111 Registering service transient_nfs with the registry
2015-11-02 12:49:49,416 DEBUG master:113 Activating service transient_nfs
2015-11-02 12:49:49,416 DEBUG master:157 ADD dependencies for service transient_nfs
2015-11-02 12:49:49,417 DEBUG master:113 Activating service PSS
2015-11-02 12:49:49,417 DEBUG master:157 ADD dependencies for service PSS
2015-11-02 12:49:49,417 DEBUG registry:99 Removing service HTCondor from the registry
2015-11-02 12:49:49,417 DEBUG master:483 Checking for and adding any previously defined cluster services
2015-11-02 12:49:49,417 DEBUG master:489 Processing filesystems in an existing cluster config
2015-11-02 12:49:49,417 DEBUG master:960 Trying to discover any volumes attached to this instance...
2015-11-02 12:49:49,686 DEBUG master:977 Attached volumes: [Volume:vol-123456]
2015-11-02 12:49:49,687 DEBUG ec2:384 Adding tag 'clusterName:cloudlaunch-test-launch-v5' to resource 'vol-123456'
2015-11-02 12:49:49,921 DEBUG master:559 Activating previously-available application services from an existing cluster config.
2015-11-02 12:49:49,921 DEBUG master:1335 initialize_cluster_with_custom_settings: cluster_type=Galaxy, storage_type=volume, storage_size=10
2015-11-02 12:49:49,921 INFO master:1378 Initializing 'Galaxy' cluster type with storage type 'volume'. Please wait...
2015-11-02 12:49:49,922 DEBUG master:1435 Checking for cluster template definitions.
2015-11-02 12:49:50,829 DEBUG misc:591 Retrieved file 'snaps.yaml' from bucket 'cloudman' on host 's3.amazonaws.com' to 'cm_snaps.yaml'.
2015-11-02 12:49:50,847 DEBUG ec2:309 Got region name as 'us-east-1'
2015-11-02 12:49:50,847 DEBUG master:310 Loaded default snapshot data for cloud amazon: [{'snap_id': 'snap-e6e1c04a', 'name': 'galaxy', 'roles': 'galaxyTools,galaxyData'}, {'snap_id': 'snap-4b20f451', 'name': 'galaxyIndices', 'roles': 'galaxyIndices'}]
2015-11-02 12:49:50,848 DEBUG master:1461 Processing file system templates: [{'snap_id': 'snap-e6e1c04a', 'name': 'galaxy', 'roles': 'galaxyTools,galaxyData'}, {'snap_id': 'snap-4b20f451', 'name': 'galaxyIndices', 'roles': 'galaxyIndices'}]
2015-11-02 12:49:50,848 DEBUG master:960 Trying to discover any volumes attached to this instance...
2015-11-02 12:49:51,020 DEBUG master:977 Attached volumes: [Volume:vol-123456]
2015-11-02 12:49:51,020 DEBUG ec2:384 Adding tag 'clusterName:cloudlaunch-test-launch-v5' to resource 'vol-123456'
2015-11-02 12:49:51,226 DEBUG master:1464 Processing file system template: galaxy
2015-11-02 12:49:51,226 DEBUG filesystem:32 Instantiating Filesystem object galaxy with service roles: 'galaxyTools,galaxyData'
2015-11-02 12:49:51,226 DEBUG master:584 Checking if vol 'vol-167c56ec' is file system 'galaxy'
2015-11-02 12:49:51,226 DEBUG ec2:409 Getting tag 'clusterName' on resource 'vol-167c56ec'
2015-11-02 12:49:51,226 DEBUG ec2:409 Getting tag 'filesystem' on resource 'vol-167c56ec'
2015-11-02 12:49:51,226 DEBUG master:1476 There are no volumes already attached for file system galaxy
2015-11-02 12:49:51,226 DEBUG filesystem:576 Adding Volume (id=None, size=10, snap=snap-e6e1c04a) into Filesystem galaxy FS
2015-11-02 12:49:51,227 DEBUG volume:64 Volume service object created; creating the actual volume.
2015-11-02 12:49:51,353 DEBUG volume:349 Creating a new volume of size '10' in zone 'us-east-1a' from snapshot 'snap-e6e1c04a' for galaxy FS.
2015-11-02 12:49:51,637 DEBUG volume:360 Created a new volume of size '10' from snapshot 'snap-e6e1c04a' with ID 'vol-4b5c76b1' in zone 'us-east-1a' for galaxy FS.
2015-11-02 12:49:51,638 DEBUG ec2:384 Adding tag 'clusterName:cloudlaunch-test-launch-v5' to resource 'vol-4b5c76b1'
2015-11-02 12:49:51,829 DEBUG ec2:384 Adding tag 'bucketName:my_company_bucket' to resource 'vol-4b5c76b1'
2015-11-02 12:49:52,040 DEBUG volume:148 Getting snaps derived from volume vol-4b5c76b1.
2015-11-02 12:50:33,275 DEBUG volume:159 Got snaps derived from volume vol-4b5c76b1 in 41.2352890968 seconds: []
2015-11-02 12:50:33,278 DEBUG master:107 Adding a new file system service into the registry: galaxy
2015-11-02 12:50:33,278 DEBUG registry:111 Registering service galaxy with the registry
2015-11-02 12:50:33,278 DEBUG master:113 Activating service galaxy
2015-11-02 12:50:33,279 DEBUG master:157 ADD dependencies for service galaxy
2015-11-02 12:50:33,279 DEBUG master:1464 Processing file system template: galaxyIndices
2015-11-02 12:50:33,279 DEBUG filesystem:32 Instantiating Filesystem object galaxyIndices with service roles: 'galaxyIndices'
2015-11-02 12:50:33,279 DEBUG master:584 Checking if vol 'vol-167c56ec' is file system 'galaxyIndices'
2015-11-02 12:50:33,279 DEBUG ec2:409 Getting tag 'clusterName' on resource 'vol-167c56ec'
2015-11-02 12:50:33,279 DEBUG ec2:409 Getting tag 'filesystem' on resource 'vol-167c56ec'
2015-11-02 12:50:33,279 DEBUG master:1476 There are no volumes already attached for file system galaxyIndices
2015-11-02 12:50:33,280 DEBUG filesystem:576 Adding Volume (id=None, size=0, snap=snap-4b20f451) into Filesystem galaxyIndices FS
2015-11-02 12:50:33,280 DEBUG volume:64 Volume service object created; creating the actual volume.
2015-11-02 12:50:34,088 DEBUG volume:349 Creating a new volume of size '700' in zone 'us-east-1a' from snapshot 'snap-4b20f451' for galaxyIndices FS.
2015-11-02 12:50:34,289 DEBUG volume:360 Created a new volume of size '700' from snapshot 'snap-4b20f451' with ID 'vol-455d77bf' in zone 'us-east-1a' for galaxyIndices FS.
2015-11-02 12:50:34,289 DEBUG ec2:384 Adding tag 'clusterName:cloudlaunch-test-launch-v5' to resource 'vol-455d77bf'
2015-11-02 12:50:34,475 DEBUG ec2:384 Adding tag 'bucketName:my_company_bucket' to resource 'vol-455d77bf'
2015-11-02 12:50:34,908 DEBUG volume:148 Getting snaps derived from volume vol-455d77bf.
2015-11-02 12:51:02,842 DEBUG volume:159 Got snaps derived from volume vol-455d77bf in 27.9343369007 seconds: []
2015-11-02 12:51:02,844 DEBUG master:107 Adding a new file system service into the registry: galaxyIndices
2015-11-02 12:51:02,844 DEBUG registry:111 Registering service galaxyIndices with the registry
2015-11-02 12:51:02,844 DEBUG master:113 Activating service galaxyIndices
2015-11-02 12:51:02,844 DEBUG master:157 ADD dependencies for service galaxyIndices
2015-11-02 12:51:02,844 DEBUG master:113 Activating service Postgres
2015-11-02 12:51:02,844 DEBUG master:157 ADD dependencies for service Postgres
2015-11-02 12:51:02,844 DEBUG master:113 Activating service ProFTPd
2015-11-02 12:51:02,845 DEBUG master:157 ADD dependencies for service ProFTPd
2015-11-02 12:51:02,845 DEBUG master:113 Activating service Galaxy
2015-11-02 12:51:02,845 DEBUG master:157 ADD dependencies for service Galaxy
2015-11-02 12:51:02,845 DEBUG master:113 Activating service GalaxyReports
2015-11-02 12:51:02,845 DEBUG master:157 ADD dependencies for service GalaxyReports
2015-11-02 12:51:02,845 DEBUG master:113 Activating service NodeJSProxy
2015-11-02 12:51:02,846 DEBUG master:157 ADD dependencies for service NodeJSProxy
2015-11-02 12:51:02,846 DEBUG ec2:223 Gathering instance private IP, attempt 0
2015-11-02 12:51:02,849 DEBUG ec2:242 Gathering instance local hostname, attempt 0
2015-11-02 12:51:02,905 DEBUG misc:840 'cp /etc/hosts /etc/hosts.orig' command OK
2015-11-02 12:51:02,933 DEBUG misc:840 'cp /tmp/tmpZ2hwc2 /etc/hosts' command OK
2015-11-02 12:51:02,958 DEBUG misc:840 'chmod 644 /etc/hosts' command OK
2015-11-02 12:51:02,958 DEBUG misc:1081 Added the following line to /etc/hosts: 1.3.3.7 ip-1-3-3-7.cloud.domain.com ip-1-3-3-7 master
2015-11-02 12:51:03,006 ERROR misc:849 ---> PROBLEM, running command 'service hostname restart' returned code '1', the following stderr: 'hostname: unrecognized service
' and stdout: ''
2015-11-02 12:51:03,007 INFO master:460 Completed the initial cluster startup process. Configuring a predefined cluster of type Galaxy.
2015-11-02 12:51:03,007 DEBUG master:2726 Monitor started; manager started
2015-11-02 12:51:08,008 DEBUG master:2737 Trying to setup AMQP connection; conn = '<cm.util.comm.CMMasterComm object at 0x7f55a4c53110>'
2015-11-02 12:51:08,010 DEBUG comm:30 Setting up a new AMQP connection
2015-11-02 12:51:08,011 DEBUG comm:49 AMQP Connection Failure: [Errno 111] Connection refused
2015-11-02 12:51:13,013 DEBUG master:2737 Trying to setup AMQP connection; conn = '<cm.util.comm.CMMasterComm object at 0x7f55a4c53110>'
2015-11-02 12:51:13,014 DEBUG comm:30 Setting up a new AMQP connection
2015-11-02 12:51:13,015 DEBUG comm:49 AMQP Connection Failure: [Errno 111] Connection refused
2015-11-02 12:51:18,015 DEBUG master:2737 Trying to setup AMQP connection; conn = '<cm.util.comm.CMMasterComm object at 0x7f55a4c53110>'
2015-11-02 12:51:18,016 DEBUG comm:30 Setting up a new AMQP connection
2015-11-02 12:51:18,016 DEBUG comm:49 AMQP Connection Failure: [Errno 111] Connection refused
2015-11-02 12:51:23,018 DEBUG master:2737 Trying to setup AMQP connection; conn = '<cm.util.comm.CMMasterComm object at 0x7f55a4c53110>'
2015-11-02 12:51:23,019 DEBUG comm:30 Setting up a new AMQP connection
2015-11-02 12:51:23,020 DEBUG comm:49 AMQP Connection Failure: [Errno 111] Connection refused
2015-11-02 12:51:28,021 DEBUG master:2737 Trying to setup AMQP connection; conn = '<cm.util.comm.CMMasterComm object at 0x7f55a4c53110>'
2015-11-02 12:51:28,022 DEBUG comm:30 Setting up a new AMQP connection
2015-11-02 12:51:28,022 DEBUG comm:49 AMQP Connection Failure: [Errno 111] Connection refused