mg-rast / awe Goto Github PK

workflow and resource management system for bioinformatics data analysis

License: BSD 2-Clause "Simplified" License

Go 80.18% HTML 0.28% JavaScript 9.07% Shell 0.62% Perl 6.61% Python 1.72% CSS 0.83% Dockerfile 0.46% Common Workflow Language 0.22%

awe shock workflow multi-cloud mg-rast

awe's Introduction

MG-RAST source code

This is the repository for the MG-RAST metagenome analysis system. Take a look at MG-RAST.

WARNING

Don't try this at home.

LICENSE

MG-RAST is made available under a BSD type LICENSE, see the LICENSE file for details.

Please note: The MG-RAST team is dedicated to supporting the

server at http://www.mg-rast.org, we are not resourced to help with local installations. So as much as we'd like to we can't help with local installations of this software.

REQUIREMENTS

Hardware

MG-RAST is a pipeline, an archive, a complex web interface and several other tools. The entire systems was designed for a Linux/Unix system. We run it on a dedicated small cluster for the server infrastructure and heavily utilize CLOUD computing resources.

Systems-Software

MySQL
Cassandra
Perl
Python
R
Apache
NGINX

For the bioinformatics software and databases used in MG-RAST please see our manual: http://help.mg-rast.org

INSTRUCTIONS

type make

API server

Build image and push to dockerhub:

git clone https://github.com/MG-RAST/MG-RAST.git
cd MG-RAST
docker build -t mgrast/api-server:dev .

docker push mgrast/api-server:dev

Get config: (private mcs git repo, for details see fleet unit)

if cd /home/core/mgrast-config; then git pull; else cd /home/core/ ; git clone [email protected]:mgrast-config.git ; fi

Download data

docker run -t -i --name api -v /media/ephemeral/api-server-data:/m5nr mgrast/api /MG-RAST/bin/download_m5nr_blast.sh
docker rm api

Start container:

docker run -t -i --name api  -v /home/core/mgrast-config/services/api-server:/api-server-conf -v /media/ephemeral/api-server-data:/m5nr -p 80:80 mgrast/api-server /usr/local/apache2/bin/httpd -DFOREGROUND -f /MG-RAST/conf/httpd.conf

awe's People

Contributors

Stargazers

Watchers

awe's Issues

delete Shock nodes that are not needed after job completion

[Feature Request] Job.Info.service or service_name queryable field

Add field like 'service' or 'service_name' to Job.Info. 'service' would be set by the service running the AWE job and would therefore allow the user to filter the displayed jobs by the service that ran the job on their behalf. For example, the user could choose to see only FBA jobs or only assembly jobs, depending on which they were interested in. AWE itself would not use this field other than for queries.

More specific JSON errors

Would it be possible to give the location of the error in the JSON when JSON parse errors occur? For example:

crusherofheads@icrushdeheads:~/awe_sec$ curl -X POST -F [email protected] http://localhost:7080/job
{"status":400,"data":null,"error":["json: cannot unmarshal number into Go value of type string"]}

That's really hard to debug. In this case it turns out that the dependsOn fields in tasks must be strings, not numbers, which was the cause of the error.

better error message for error shock url in output node

Shock server url without protocol scheme does not work, AWE return no useful message to user

[EROR] qmgr.taskEnQueue createOutputNode:unsupported protocol scheme ""

Need to check size of documents before saving to mongo (16MB limit)

Awe server crash on job submit

Versions:

crusherofheads@icrushdeheads:/kb/dev_container/modules/awe_service$ git log
commit 244e569c35375e638d39fc0f2667c976070da92f
Merge: b48dc60 3c32286
Author: jaredbischof <[email protected]>
Date:   Tue Sep 23 12:59:26 2014 -0500

    Merge pull request #28 from jaredbischof/master

    Edited kill command to kill all processes that are child processes of th...

crusherofheads@icrushdeheads:/kb/dev_container/modules/awe_service/AWE$ git log
commit 5cfd7c29cd80b62a3382307175333f0ffb6849c7
Merge: e66cd55 8a00a9d
Author: Travis Harrison <[email protected]>
Date:   Mon Sep 8 11:36:58 2014 -0500

    Merge pull request #294 from teharrison/master

    fix pointer derefence

Setup:

crusherofheads@icrushdeheads:~/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926$ tree
.
├── awe
│   ├── awfs
│   ├── data
*snip*
│   │   ├── maxjid
│   │   ├── pidfile
│   │   └── temp
│   ├── logs
│   │   └── server
│   │       ├── access.log
│   │       ├── debug.log
│   │       ├── error.log
│   │       ├── event.log
│   │       └── perf.log
│   └── site
├── awec
│   ├── data
│   │   ├── clientprofile.json
│   │   └── pidfile
│   ├── logs
│   │   └── client
│   │       ├── access.log
│   │       ├── debug.log
│   │       ├── error.log
│   │       ├── event.log
│   │       └── perf.log
│   ├── site
│   └── work
├── awec.cfg
├── awe.cfg
├── awe_client.log
├── awe_server.log
└── client_script.py

17 directories, 21 files

crusherofheads@icrushdeheads:~/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926$ cat awe.cfg
[Admin]
[email protected]
secretkey=supersecretkey
users=

[Anonymous]
# Controls weither an anonymous user can read/write
# values: true/false
read=false
write=false
delete=false
cg_read=false
cg_write=false
cg_delete=false

[Args]
debuglevel=0

[Auth]
globus_token_url=https://nexus.api.globusonline.org/goauth/token?grant_type=client_credentials
globus_profile_url=https://nexus.api.globusonline.org/users
client_auth_required=false

[Directories]
# See documentation for details of deploying Shock
site=/home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926/awe/site
data=/home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926/awe/data
logs=/home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926/awe/logs
awf=/home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926/awe/awfs

[External]
site-url=
api-url=http://localhost:60961

[Mongodb]
# Mongodb configuration:
# Hostnames and ports hosts=host1[,host2:port,...,hostN]
hosts=localhost
database=AweClientTests_AweDB
user=foo
password=foo

[Mongodb-Node-Indices]
# See http://www.mongodb.org/display/DOCS/Indexes#Indexes-CreationOptions for more info on mongodb index options.
# key=unique:true/false[,dropDups:true/false][,sparse:true/false]
id=unique:true

[Ports]
# Ports for site/api
# Note: use of port 80 may require root access
site-port=
api-port=60961

[Server]
perf_log_workunit=true

crusherofheads@icrushdeheads:~/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926$ curl http://localhost:7044
{"attribute_indexes":[""],"contact":"[email protected]","documentation":"http://localhost:7044/wiki/","id":"Shock","resources":["node"],"type":"Shock","url":"http://localhost:7044/","version":"0.8.23"}

Run the server:

crusherofheads@icrushdeheads:~/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926$ /kb/deployment/bin/awe-server --conf awe.cfg
 +--------------+  +----+   +----+   +----+  +--------------+
 |              |  |    |   |    |   |    |  |              |
 |    +----+    |  |    |   |    |   |    |  |    +---------+
 |    |    |    |  |    |   |    |   |    |  |    |          
 |    +----+    |  |    |   |    |   |    |  |    +---------+
 |              |  |    |   |    |   |    |  |              |
 |    +----+    |  |    |   |    |   |    |  |    +---------+
 |    |    |    |  |    \---/    \---/    |  |    |          
 |    |    |    |  |                      |  |    +---------+
 |    |    |    |   \        /---\       /   |              |
 +----+    +----+     \-----/     \-----/    +--------------+

##### Admin #####
email:  [email protected]
secretkey:  supersecretkey

####### Anonymous ######
read:   false
write:  false
delete: false
clientgroup read:   false
clientgroup write:  false
clientgroup delete: false

##### Auth #####
basic_auth: true
globus_token_url:   https://nexus.api.globusonline.org/goauth/token?grant_type=client_credentials
globus_profile_url: https://nexus.api.globusonline.org/users
admin_auth: true
admin_users:     

##### Directories #####
site:   /home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926/awe/site
data:   /home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926/awe/data
logs:   /home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926/awe/logs
awf:    /home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926/awe/awfs

##### SSL #####
enabled:    false

##### Mongodb #####
host(s):    localhost
database:   AweClientTests_AweDB


##### Ports #####
site:   0
api:    60961

##### pidfile #####
pid: 8136 saved to file: /home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926/awe/data/pidfile

##### Procs #####
Number of available CPUs = 7
Running AWE server with GOMAXPROCS = 5

Submit the job (note I ran this job successfully many times on earlier versions of AWE):

crusherofheads@icrushdeheads:~/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926$ cat job.json
{
    "info": {
        "clientgroups": "kbase-fake-group", 
        "description": "res desc", 
        "noretry": true, 
        "service": "results"
    }, 
    "tasks": [
        {
            "cmd": {
                "args": "--outfiles foo bar", 
                "name": "client_script.py"
            }, 
            "outputs": {
                "bar": {
                    "host": "http://localhost:7044", 
                    "temporary": false
                }, 
                "foo": {
                    "host": "http://localhost:7044", 
                    "temporary": true
                }
            }, 
            "taskid": "0"
        }, 
        {
            "cmd": {
                "args": "--infiles @bar --outfiles baz boo", 
                "name": "client_script.py"
            }, 
            "dependsOn": [
                "0"
            ], 
            "inputs": {
                "bar": {
                    "host": "http://localhost:7044", 
                    "origin": "0"
                }
            }, 
            "outputs": {
                "baz": {
                    "host": "http://localhost:7044", 
                    "temporary": false
                }, 
                "boo": {
                    "host": "http://localhost:7044", 
                    "temporary": true
                }
            }, 
            "taskid": "1"
        }, 
        {
            "cmd": {
                "args": "--infiles @baz @boo --outfiles wugga", 
                "name": "client_script.py"
            }, 
            "dependsOn": [
                "1"
            ], 
            "inputs": {
                "baz": {
                    "host": "http://localhost:7044", 
                    "origin": "1"
                }, 
                "boo": {
                    "host": "http://localhost:7044", 
                    "origin": "1"
                }
            }, 
            "outputs": {
                "wugga": {
                    "host": "http://localhost:7044", 
                    "temporary": false
                }
            }, 
            "taskid": "2"
        }
    ]
}

crusherofheads@icrushdeheads:~/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926$ curl -X POST -H "Datatoken: $TOKEN" -H "Authorization: OAuth $TOKEN" -F [email protected] http://localhost:60961/job
{"status":200,"data":{"id":"4a0f50ca-bd8a-420c-921b-fab7f3bd12c4","jid":"10002","info":{"name":"","xref":"","service":"results","project":"","user":"","pipeline":"","clientgroups":"kbase-"inputs":
*snip*
khost":""},"error":null}

crusherofheads@icrushdeheads:~/localgit/user_and_job_state/ujs_test_temp/AweController-2713733733772064926$ curl -X GET http://localhost:60961/
curl: (7) couldn't connect to host

Server crashed:

##### Procs #####
Number of available CPUs = 7
Running AWE server with GOMAXPROCS = 5

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x461b9e]

goroutine 17 [running]:
github.com/MG-RAST/AWE/lib/core.createOrUpdate(0xc2002e5d00, 0xc2002f9880, 0x15, 0x78ad00, 0x0, ...)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/core/core.go:740 +0x39e
github.com/MG-RAST/AWE/lib/core.PostNodeWithToken(0xc2001d4140, 0x1, 0xc2001b6600, 0x1d9, 0x0, ...)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/core/core.go:195 +0xc4
github.com/MG-RAST/AWE/lib/core.(*ServerMgr).createOutputNode(0xc2000e6b80, 0xc20017f200, 0x0, 0x0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/core/servermgr.go:661 +0x687
github.com/MG-RAST/AWE/lib/core.(*ServerMgr).taskEnQueue(0xc2000e6b80, 0xc20017f200, 0x0, 0x0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/core/servermgr.go:567 +0x2e6
github.com/MG-RAST/AWE/lib/core.(*ServerMgr).addTask(0xc2000e6b80, 0xc20017f200, 0x0, 0x0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/core/servermgr.go:485 +0x265
github.com/MG-RAST/AWE/lib/core.(*ServerMgr).Handle(0xc2000e6b80)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/core/servermgr.go:65 +0x2c2
created by main.main
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/awe-server/awe-server.go:185 +0x14b8

goroutine 1 [chan receive]:
main.main()
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/awe-server/awe-server.go:259 +0x2467

goroutine 2 [syscall]:

goroutine 4 [chan receive]:
github.com/MG-RAST/golib/log4go.ConsoleLogWriter.run(0xc200103000, 0xc2000a61e0, 0xc200000008)
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/termlog.go:27 +0x64
created by github.com/MG-RAST/golib/log4go.NewConsoleLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/termlog.go:19 +0x65

goroutine 5 [select]:
github.com/MG-RAST/golib/log4go.func·002()
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:84 +0x7da
created by github.com/MG-RAST/golib/log4go.NewFileLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:116 +0x29c

goroutine 6 [select]:
github.com/MG-RAST/golib/log4go.func·002()
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:84 +0x7da
created by github.com/MG-RAST/golib/log4go.NewFileLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:116 +0x29c

goroutine 7 [select]:
github.com/MG-RAST/golib/log4go.func·002()
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:84 +0x7da
created by github.com/MG-RAST/golib/log4go.NewFileLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:116 +0x29c

goroutine 8 [select]:
github.com/MG-RAST/golib/log4go.func·002()
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:84 +0x7da
created by github.com/MG-RAST/golib/log4go.NewFileLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:116 +0x29c

goroutine 9 [select]:
github.com/MG-RAST/golib/log4go.func·002()
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:84 +0x7da
created by github.com/MG-RAST/golib/log4go.NewFileLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:116 +0x29c

goroutine 10 [chan receive]:
github.com/MG-RAST/AWE/lib/logger.(*Logger).Handle(0xc2000d2d40)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/logger/logger.go:117 +0x3c
created by github.com/MG-RAST/AWE/lib/logger.Initialize
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/logger/logger.go:30 +0x4f

goroutine 11 [select]:
github.com/MG-RAST/golib/mgo.(*mongoCluster).syncServersLoop(0xc2000eb0f0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/mgo/cluster.go:366 +0x477
created by github.com/MG-RAST/golib/mgo.newCluster
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/mgo/cluster.go:72 +0x116

goroutine 16 [IO wait]:
net.runtime_pollWait(0x7f3010d40e60, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2000dc6b0, 0xb, 0xc200106060)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc2000dc630, 0xc200106b70, 0x24, 0x24, 0x0, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc200000920, 0xc200106b70, 0x24, 0x24, 0x0, ...)
    /usr/local/go/src/pkg/net/net.go:123 +0xc3
github.com/MG-RAST/golib/mgo.fill(0xc2000e56c0, 0xc200000920, 0xc200106b70, 0x24, 0x24, ...)
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/mgo/socket.go:489 +0x59
github.com/MG-RAST/golib/mgo.(*mongoSocket).readLoop(0xc2000d4d20)
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/mgo/socket.go:506 +0x113
created by github.com/MG-RAST/golib/mgo.newSocket
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/mgo/socket.go:163 +0x27d

goroutine 14 [IO wait]:
net.runtime_pollWait(0x7f3010d40f00, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2000dc620, 0xb, 0xc200106060)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc2000dc5a0, 0xc2001064b0, 0x24, 0x24, 0x0, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc200000650, 0xc2001064b0, 0x24, 0x24, 0x0, ...)
    /usr/local/go/src/pkg/net/net.go:123 +0xc3
github.com/MG-RAST/golib/mgo.fill(0xc2000e56c0, 0xc200000650, 0xc2001064b0, 0x24, 0x24, ...)
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/mgo/socket.go:489 +0x59
github.com/MG-RAST/golib/mgo.(*mongoSocket).readLoop(0xc2000d4a80)
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/mgo/socket.go:506 +0x113
created by github.com/MG-RAST/golib/mgo.newSocket
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/mgo/socket.go:163 +0x27d

goroutine 15 [sleep]:
time.Sleep(0x12a05f200)
    /usr/local/go/src/pkg/runtime/ztime_linux_amd64.c:19 +0x2f
github.com/MG-RAST/golib/mgo.(*mongoServer).pinger(0xc2000d49a0, 0xc2000d4901)
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/mgo/server.go:284 +0xfd
created by github.com/MG-RAST/golib/mgo.newServer
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/mgo/server.go:87 +0xea

goroutine 18 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/go/src/pkg/runtime/ztime_linux_amd64.c:19 +0x2f
github.com/MG-RAST/AWE/lib/core.(*ServerMgr).Timer(0xc2000e6b80)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/core/servermgr.go:94 +0x2a
created by main.main
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/awe-server/awe-server.go:186 +0x14dd

goroutine 19 [sleep]:
time.Sleep(0x6fc23ac00)
    /usr/local/go/src/pkg/runtime/ztime_linux_amd64.c:19 +0x2f
github.com/MG-RAST/AWE/lib/core.(*CQMgr).ClientChecker(0xc2000e6b80)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/core/cqmgr.go:84 +0x35
created by main.main
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/awe-server/awe-server.go:187 +0x14ff

goroutine 20 [IO wait]:
net.runtime_pollWait(0x7f3010d40dc0, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc200167110, 0xb, 0xc200106060)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc200167090, 0x823c38, 0x0, 0xc200106060, 0xb, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*TCPListener).AcceptTCP(0xc200148350, 0x18, 0xc200182010, 0x501987)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:229 +0x45
net.(*TCPListener).Accept(0xc200148350, 0x0, 0x0, 0x0, 0xc2001062a0, ...)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:239 +0x25
net/http.(*Server).Serve(0xc200171050, 0xc2001800c0, 0xc200148350, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/net/http/server.go:1542 +0x85
net/http.(*Server).ListenAndServe(0xc200171050, 0xc200171050, 0xc200166030)
    /usr/local/go/src/pkg/net/http/server.go:1532 +0x9e
net/http.ListenAndServe(0xc200148310, 0x2, 0xc200166030, 0xc200148318, 0xc2001155b0, ...)
    /usr/local/go/src/pkg/net/http/server.go:1597 +0x65
github.com/MG-RAST/golib/goweb.ListenAndServeRoutes(0xc200148310, 0x2, 0xc20013f800, 0x1, 0x1, ...)
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/goweb/http_handler.go:125 +0x68
main.launchSite(0xc20013d420, 0x0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/awe-server/awe-server.go:51 +0x715
created by main.main
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/awe-server/awe-server.go:188 +0x1526

goroutine 21 [IO wait]:
net.runtime_pollWait(0x7f3010d40d20, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc200187080, 0xb, 0xc200106060)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc200187000, 0x823c38, 0x0, 0xc200106060, 0xb, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*TCPListener).AcceptTCP(0xc200148440, 0x501b96, 0x7f3010d3ccb8, 0x501b96)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:229 +0x45
net.(*TCPListener).Accept(0xc200148440, 0xc2000e56c0, 0xc2000005d0, 0xc2000dc090, 0x0, ...)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:239 +0x25
net/http.(*Server).Serve(0xc200170050, 0xc2001800c0, 0xc200148440, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/net/http/server.go:1542 +0x85
net/http.(*Server).ListenAndServe(0xc200170050, 0xc200170050, 0x1155b0)
    /usr/local/go/src/pkg/net/http/server.go:1532 +0x9e
net/http.ListenAndServe(0xc200148400, 0x6, 0xc200166030, 0xc200148408, 0xc2001155b0, ...)
    /usr/local/go/src/pkg/net/http/server.go:1597 +0x65
github.com/MG-RAST/golib/goweb.ListenAndServeRoutes(0xc200148400, 0x6, 0xc20013fc20, 0x1, 0x1, ...)
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/goweb/http_handler.go:125 +0x68
main.launchAPI(0xc20013d420, 0xee21)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/awe-server/awe-server.go:83 +0x849
created by main.main
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/awe-server/awe-server.go:189 +0x154d

goroutine 54 [select]:
net/http.(*persistConn).writeLoop(0xc200145200)
    /usr/local/go/src/pkg/net/http/transport.go:774 +0x26f
created by net/http.(*Transport).dialConn
    /usr/local/go/src/pkg/net/http/transport.go:512 +0x58b

goroutine 53 [IO wait]:
net.runtime_pollWait(0x7f3010d40be0, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2000dc860, 0xb, 0xc200106060)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc2000dc7e0, 0xc2002f6000, 0x800, 0x800, 0x0, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc200000580, 0xc2002f6000, 0x800, 0x800, 0x504871, ...)
    /usr/local/go/src/pkg/net/net.go:123 +0xc3
crypto/tls.(*block).readFromUntil(0xc20013b5a0, 0xc200166210, 0xc200000580, 0x5, 0xc200000580, ...)
    /usr/local/go/src/pkg/crypto/tls/conn.go:401 +0xbd
crypto/tls.(*Conn).readRecord(0xc2001d7000, 0x17, 0x0, 0x41a610)
    /usr/local/go/src/pkg/crypto/tls/conn.go:481 +0xfa
crypto/tls.(*Conn).Read(0xc2001d7000, 0xc2002e8000, 0x1000, 0x1000, 0x0, ...)
    /usr/local/go/src/pkg/crypto/tls/conn.go:796 +0x102
bufio.(*Reader).fill(0xc2002e4e40)
    /usr/local/go/src/pkg/bufio/bufio.go:79 +0x10c
bufio.(*Reader).Peek(0xc2002e4e40, 0x1, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/bufio/bufio.go:107 +0xc9
net/http.(*persistConn).readLoop(0xc200145200)
    /usr/local/go/src/pkg/net/http/transport.go:670 +0xc4
created by net/http.(*Transport).dialConn
    /usr/local/go/src/pkg/net/http/transport.go:511 +0x574

predata check

Check that predata exists when enqueuing a task, same as checking that inputs exist. Currently missing predata causes the workunit to fail when downloading, should fail at server level before checkout.

upload attributes to Shock outputs

Need to modify mongo timeouts.

AWE should try connecting to mongo several times (with 10 second timeout) at startup time. If it continues to fail, AWE should exit with error. If it is successful, AWE should start a new new mongo connection with a longer timeout (~5 minutes). This is necessary because the mgo library does not distinguish between a dial timeout and a rw timeout.

add Info, IOmap, Command to data structures wiki

Status Monitor never finishes updating / TypeError: tdata[0] is undefined

Both on my locally installed AWE server and the server at http://140.221.84.148:8080, the 'updating' popup on the Status Monitor never completes, and I see this in the JS console:

TypeError: tdata[0] is undefined
if (typeof(tdata[0][header[i]].replace) != 'function') {

I see the same error at https://kbase.us/services/api-api/ but the 'updating' popup does close.

support fileless output

allow a task to upload just attributes and no file to output node (not use 'upload' form option). an output name would still be required, but this would not map to a file, instead there would be a flag in the output struct (ie. nofile: true) that indicated this.

possible bug with admin users

When starting the server with one or more admins listed the 'users' field of the 'Admin' section of the config file I get this message and awe does not start:

DEBUG_LEVEL > 0
logger.Initialize...
init db...
init db collection for user...
ERROR initializing user database: not found

When starting the server with the 'users' field left empty it starts up fine.

Debug and error messages not very unified

Here a list of possible locations for important debug or error messages. This huge list makes it difficult for users and developers to debug workflows and AWE itself. We should try to decide on a smaller set of log files/API calls and slowly remove the others.

AWE client

awe/logs/client-default_client/access.log
awe/logs/client-default_client/error.log
awe/logs/client-default_client/perf.log
awe/logs/client-default_client/debug.log
awe/logs/client-default_client/event.log
stdout of awe-client binary
stderr of awe-client binary
awe_worknotes.txt file in work directory (or via API)
Notes-field in workflow document
API error response on submission
(stdout and stderr of the actual task is not listed here)

AWE server

awe/logs/server/access.log
awe/logs/server/debug.log
awe/logs/server/event.log
awe/logs/server/error.log
awe/logs/server/perf.log
stdout of awe-server binary
stderr of awe-server binary

explain use and purpose of at-sign in args option of workflow document

DELETE vs. PUT for deleting jobs

The wiki says you should use PUT for deleting suspended and zombie jobs, but I think this should be a DELETE.

automatic retrieval of base image from shock repo

awe-clients report available disk space

Failures in order/limit to get all

I'm trying to get all jobs using a large limit on the /job query. However, I get an error from the GET:

{"status":400,"data":null,"error":["Bad Request"]}

and find this in the logs:

[2014/10/31 16:22:12 CDT] [EROR] err too much data for sort() with no index
[2014/10/31 16:22:15 CDT] [EROR] err too much data for sort() with no index
[2014/10/31 16:22:18 CDT] [EROR] err too much data for sort() with no index
[2014/10/31 16:23:18 CDT] [EROR] err too much data for sort() with no index. add an index or specify a smaller limit
[2014/10/31 16:23:24 CDT] [EROR] err too much data for sort() with no index

Is this a configuration problem somewhere?

--bob

report peak memory usage for each task

replace goweb with go-swagger for API

see https://github.com/go-swagger/go-swagger

download of a directory

maybe as tarball ?

[Feature Request] Mark output data for service/user consumption

Could AWE allow the job submitter to mark which task outputs are final results that are to be consumed by, say, another service, as opposed to intermediate data that is just passed to the next task and never used again? For example, say I have the following tasks:

A -> B -> C

...and A, B and C all have one or more outputs saved in shock. AFAICT, there's no way for me to tell whether the output of A and B should be shown to the end user because they're useful files, or if they're just temporary data and the only data the user cares about was produced by task C.

Something like Task.Outputs.[*].finalData (boolean) or something along those lines. AWE itself wouldn't use these fields; they'd just be user-settable.

substitions fail in awe_submit and awe_submit_auth

The substitution strings in awe_submit and awe_submit_auth generate the following errors....

Having no space between pattern and following word is deprecated at -e line 1.
Bareword found where operator expected at -e line 1, near "s/#shockurl/kbase.us/services"
syntax error at -e line 1, near "s/#shockurl/kbase.us/services"

This can be fixed by replacing slashes with another separator. However, it seems odd to invoke system perl commands inside a perl script. Why not just read the jobscript template in to memory, do the substitutions, then dump the result? In fact all of the system commands and back-tick lines could be removed and native perl functions used.

support additional shock form options

allow output of task to invoke additional shock form options when uploading to shock node. currently only 'upload' and 'attributes' are used. this could be free-form to allow more flexibility in supporting new shock features.
ie. add a 'formoption' field to output struct that takes a string-string mapping and applies the values as " -F key1=value1 -F key2=value2" etc. to shock call

On modifying attributes

Is it possible to modify the attributes of a task while the task is running? I am running the RASTtk pipeline under AWE, and would like to be able to use the attributes in AWE as a mechanism for stashing status of the individual pieces of the pipeline. If it were possible to update the userattr at runtime that would probably suit. I didn't see anything in the API docs that seemed relevant.

Thanks,
--bob

paginated job overviews returned by GET /job

AWE monitor auth

AWE monitor needs to support more than MG-RAST auth. Ideally token retrieval is done by AWE server and the AWE monitor only talks to AWE server.

support UserAttr

Have a UserAttr (simple string map) optional field in the workflow Info and in each task. If used, the value is written to disk (file called userattr.json). This allows the command running in the task to have access to global and task specific info for possible upload as node attributes.

[Feature Request] Job.Info.description non-queryble field

Add a field called something like 'description' to Job.Info. This would allow a job submitter such as a service to provide a short, arbitrary description of the job that could be displayed to users. This field most likely would not need to be queryable.

Recompute requires subsequent "resume" call to resume a job. This should be done all in one step.

Login Widget

Support Casbin as the authorization module

Hi, Casbin is an authorization library that supports models like ACL, RBAC, ABAC.

Related to RBAC, Casbin has several advantages:

roles can be cascaded, aka roles can have roles.
support resource roles, so users have their roles and resource have their roles too. role = group here.
the permission assignments (or policy in Casbin's language) can be persisted in files or database (MySQL and Cassandra).

And you can even customize your own access control model, for example, mix RBAC and ABAC together by using roles and attributes at the same time. It's very flexible.

Casbin can provide more flexibility and security than the current ACL. Let me know if there's any question:) Thanks.

data structures specifications missing info

In the specifications for basic data structures (https://github.com/MG-RAST/AWE/wiki/Specifications-for-basic-data-structures), it looks like

Task.dependsOn is missing
Workunit never appears to be embedded in the job document anywhere (like Task, Command, etc)
Should IOmap in Task be IO? There's no IOmap structure defined.
Task.PartInfo isn't defined

attribute upload with parts

if the output of a task requests an attribute file be uploaded, and the task is split into multiple workunits - then just one of the workunits (first?) should upload the attribute file

[Feature request] DISTINCT query

For the purposes of, say, a UI that shows users which services are currently running jobs on their behalf and then allows them to select a service to view jobs for that service, a DISTINCT query would be useful to return just the list of services rather than returning the entire job set for the user. The former strategy would reduce the load significantly for both the client and the AWE server, e.g.:

> db.Jobs.distinct("info.pipeline", {"info.clientgroups": "kbase-superfake"})
[ "check-token-pipeline", "wait-job-pipeline" ]

I personally just need this for info.service, but one could imagine querying other fields as well, such as pipeline, project, clientgroups, etc.

You could generalize this like:

curl -X GET http://<awe_api_url>/job?query&state=<in-progress|completed>&info.project=xxx&info.user=xxx&...[?limit=25&offset=0&order=updatetime&direction=desc&distinct=yyy]

where yyy is the field on which to run the DISTINCT query.

Perhaps DISTINCT queries only work on a subset of fields if you don't wish to allow DISTINCT queries on any field in the job document.

create index on output

have in option in 'output' struct to create the given index type after upload.

test bug

foo

Workunits getting killed

I have a workflow where I have 292 tasks, all independent. I run this on a service with 12 clients on the same host. Lots of memory and cores available.

Each time I run it I get around 130 or so tasks complete, then start seeing "process killed" and the tasks all go to suspended state. Does this sound familiar?

go get fails

Can't currently install AWE (using go v1.10). Is it still being maintained?

$ go get github.com/MG-RAST/AWE/...
# github.com/MG-RAST/AWE/awe-proxy
../../../MG-RAST/AWE/awe-proxy/awe-proxy.go:116:12: assignment mismatch: 2 variables but 1 values
../../../MG-RAST/AWE/awe-proxy/awe-proxy.go:127:30: worker.InitWorkers() used as value

Submission error when providing environ field

When I try to pass an environ as part of a command in a template, I get the following error:

JSONRPC error code: -32603
JSONRPC error data:submit failed: {"status":400,"data":null,"error":["json: cannot unmarshal object into Go value of type string"]}

Here is the command I'm currently submitting. If the only change I make in the code is to not generate the environ object, the submission succeeds.

     "cmd" : {
        "name" : "awe_preprocess_MIseq_10bp",
        "args" : "@fastq barseq.txt",
        "description" : "preprocess",
        "environ" : {
           "private" : {
              "FOO" : "bar"
           },
           "public" : {
              "KB_AUTH_TOKEN" : "xx"
           }
        }
     },

What does the AWE acronym stand for?

What does the AWE acronym stand for?

Asks the copy editor of CWLProv which has been accepted for publication :-)

For Globus server auth UI login contacts metagenomics.gov

run server in Admin mode with Globus based auth
Navigate to the Status Monitor
Updating will hang 1/2 way through since host/job is not accessible
Click login and enter credentials for admin user (kbasetest in this case)
Click login
Dialog will say you have successfully logged in
However, in the JS console you'll see:

GET http://api.metagenomics.anl.gov?auth=kbgo4711a2Jhc2V0ZXN0OkBTdWl0ZTUyNQ== 200 OK 1.06s  
jquery.min.js (line 2)
in here
jquery.min.js (line 226)
GET http://api.metagenomics.anl.gov/user/kbasetest 404 Not Found 795ms  
"NetworkError: 404 Not Found - http://api.metagenomics.anl.gov/user/kbasetest"
kbasetest
GET http://localhost:7080/queue 200 OK 20ms 
GET http://localhost:7080/job?query&limit=20&offset=0 401 Unauthorized 20ms 
GET http://localhost:7080/job?active 401 Unauthorized   20ms    
GET http://localhost:7080/job?suspend 401 Unauthorized 20ms 
GET http://localhost:7080/job?query&state=completed&limit=20&offset=0 401 Unauthorized 19ms

This is because no token is passed to the host/job calls presumably because metagenomics cannot authenticate the Globus user.

Updating will hang at ~50%.

copy of awe job on client

provide a json dump to file (in working dir) of the current awe job on the client, possibly called 'workflow.json'

[Feature Request] task failure indicator field

Could AWE mark, in some way, the task that caused a job to be suspended? Currently the only way to determine this as far as I can tell is to parse the Notes field which is fragile. Something like a Job.failed field with the task/workunit id or a Task.failed boolean field.

fix unknown timestamps

timestamps that are not yet calculated / known should probably be null and not Jan 1, 1 AD (0001-01-01T00:00:00Z)

Syntax error in config.js.tt

var RetinaConfig = {
    "awe_ip": "[% api_ip %]:[% api_port %]",
    "workflow_ip": "[% api_ip %]:[% api_port %]/awf"  <--- missing comma
    "authResources": { "default": "KBase",
        "KBase": { "icon": "KBase_favicon.ico",
        "prefix": "kbgo4711" } },
    "mgrast_api": "http://api.metagenomics.anl.gov",
    "authentication": true
}

using exit status with perl

This is a helpful tip when using perl to create apps for AWE client. If your perl script is calling a child script using system() and tries to return the child scripts exit status if it fails, make sure you 8 bit shift the return value of system() to get the real exist code. If this is not done you may inadvertently return a value that is a modulus of 256 (ie. 512), causing the AWE client to think it was successful.

http://stackoverflow.com/questions/7080434/getting-perl-to-return-the-correct-exit-code
http://perldoc.perl.org/functions/system.html

delete least recently used images when docker becomes too big

AWE client crashes on relative paths

Adding a relative path to AWE client config makes the client very unhappy:

crusherofheads@icrushdeheads:~/localgit/user_and_job_state$ tree ujs_test_temp/AweController-7213147803773719090/
ujs_test_temp/AweController-7213147803773719090/
├── awe
│   ├── awfs
│   ├── data
*snip*
│   │   ├── maxjid
│   │   ├── pidfile
│   │   └── temp
│   ├── logs
│   │   └── server
│   │       ├── access.log
│   │       ├── debug.log
│   │       ├── error.log
│   │       ├── event.log
│   │       └── perf.log
│   └── site
├── awec
│   ├── data
│   │   ├── clientprofile.json
│   │   └── pidfile
│   ├── logs
│   │   └── client
│   │       ├── access.log
│   │       ├── debug.log
│   │       ├── error.log
│   │       ├── event.log
│   │       └── perf.log
│   ├── site
│   └── work
│       └── 75
*snip*
├── awec.cfg
├── awe.cfg
├── awe_client.log
├── awe_server.log
└── client_script.py

25 directories, 23 files

crusherofheads@icrushdeheads:~/localgit/user_and_job_state$ cat ujs_test_temp/AweController-7213147803773719090/awec.cfg
[Directories]
site=ujs_test_temp/AweController-7213147803773719090/awec/site
data=ujs_test_temp/AweController-7213147803773719090/awec/data
logs=ujs_test_temp/AweController-7213147803773719090/awec/logs

[Args]
debuglevel=0

[Client]
workpath=ujs_test_temp/AweController-7213147803773719090/awec/work
supported_apps=client_script.py
serverurl=http://localhost:36301
group=kbase-fake-group
name=kbase-fakeclient
auto_clean_dir=false
worker_overlap=false
print_app_msg=true
clientgroup_token=

crusherofheads@icrushdeheads:~/localgit/user_and_job_state$ cat ujs_test_temp/AweController-7213147803773719090/awe_client.log 
##### pidfile #####
pid: 6140 saved to file: ujs_test_temp/AweController-7213147803773719090/awec/data/pidfile

clientgroup token not set, register as a public client (can only access public data)
Client registered, name=kbase-fakeclient, id=61fe93e4-4d7a-4d19-8a4b-bf433270652e
dataMover launched, client=61fe93e4-4d7a-4d19-8a4b-bf433270652e
processor launched, client=61fe93e4-4d7a-4d19-8a4b-bf433270652e
deliverer launched, client=61fe93e4-4d7a-4d19-8a4b-bf433270652e
worker: start cmd=client_script.py, args=[]
processor exiting...
panic: runtime error: invalid memory address or nil pointer dereference
    panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x46f7cc]

goroutine 17 [running]:
os.(*File).Close(0x0, 0x7a, 0x0)
    /usr/local/go/src/pkg/os/file_unix.go:99 +0x1c
os.(*File).Close(0x0, 0x7a, 0x0)
    /usr/local/go/src/pkg/os/file_unix.go:99 +0x1c
github.com/MG-RAST/AWE/lib/worker.RunWorkunitDirect(0xc200111210, 0x0, 0xc2000a0030, 0xc20013ce50)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/processor.go:567 +0xce
github.com/MG-RAST/AWE/lib/worker.RunWorkunit(0xc200111210, 0x1ea3984f, 0x0, 0x0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/processor.go:113 +0x45
github.com/MG-RAST/AWE/lib/worker.processor(0xc2000e0cc0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/processor.go:79 +0x55e
created by github.com/MG-RAST/AWE/lib/worker.StartClientWorkers
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/workermgr.go:51 +0x8f

goroutine 1 [chan receive]:
github.com/MG-RAST/AWE/lib/worker.StartClientWorkers()
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/workermgr.go:54 +0xbf
main.main()
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/awe-client/awe-client.go:89 +0x1124

goroutine 2 [syscall]:

goroutine 4 [chan receive]:
github.com/MG-RAST/golib/log4go.ConsoleLogWriter.run(0xc2000f9000, 0xc2000a01b0, 0xc200000008)
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/termlog.go:27 +0x64
created by github.com/MG-RAST/golib/log4go.NewConsoleLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/termlog.go:19 +0x65

goroutine 5 [select]:
github.com/MG-RAST/golib/log4go.func·002()
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:84 +0x7da
created by github.com/MG-RAST/golib/log4go.NewFileLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:116 +0x29c

goroutine 6 [select]:
github.com/MG-RAST/golib/log4go.func·002()
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:84 +0x7da
created by github.com/MG-RAST/golib/log4go.NewFileLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:116 +0x29c

goroutine 7 [select]:
github.com/MG-RAST/golib/log4go.func·002()
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:84 +0x7da
created by github.com/MG-RAST/golib/log4go.NewFileLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:116 +0x29c

goroutine 8 [select]:
github.com/MG-RAST/golib/log4go.func·002()
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:84 +0x7da
created by github.com/MG-RAST/golib/log4go.NewFileLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:116 +0x29c

goroutine 9 [select]:
github.com/MG-RAST/golib/log4go.func·002()
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:84 +0x7da
created by github.com/MG-RAST/golib/log4go.NewFileLogWriter
    /tmp/go_build.tmp/src/github.com/MG-RAST/golib/log4go/filelog.go:116 +0x29c

goroutine 10 [runnable]:
github.com/MG-RAST/AWE/lib/logger.(*Logger).Handle(0xc2000ccbd0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/logger/logger.go:117 +0x3c
created by github.com/MG-RAST/AWE/lib/logger.Initialize
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/logger/logger.go:30 +0x4f

goroutine 14 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/go/src/pkg/runtime/ztime_linux_amd64.c:19 +0x2f
github.com/MG-RAST/AWE/lib/worker.heartBeater(0xc2000e0cc0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/heartbeater.go:40 +0x2a
created by github.com/MG-RAST/AWE/lib/worker.StartClientWorkers
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/workermgr.go:48 +0x4a

goroutine 32 [runnable]:
io.Copy(0xc20010e8d0, 0xc20016f480, 0xc2000a02d0, 0xc200000f40, 0xc200000f40, ...)
    /usr/local/go/src/pkg/io/io.go:336
created by github.com/MG-RAST/AWE/lib/worker.RunWorkunitDirect
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/processor.go:604 +0xad9

goroutine 15 [chan send]:
github.com/MG-RAST/AWE/lib/worker.workStealer(0xc2000e0cc0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/workStealer.go:94 +0x85b
created by github.com/MG-RAST/AWE/lib/worker.StartClientWorkers
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/workermgr.go:49 +0x61

goroutine 16 [chan receive]:
github.com/MG-RAST/AWE/lib/worker.dataMover(0xc2000e0cc0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/dataMover.go:144 +0x138
created by github.com/MG-RAST/AWE/lib/worker.StartClientWorkers
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/workermgr.go:50 +0x78

goroutine 18 [chan receive]:
github.com/MG-RAST/AWE/lib/worker.deliverer(0xc2000e0cc0)
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/deliverer.go:18 +0x138
created by github.com/MG-RAST/AWE/lib/worker.StartClientWorkers
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/workermgr.go:52 +0xa6

goroutine 33 [runnable]:
io.Copy(0xc20010e8d0, 0xc20016f4c0, 0xc2000a02d0, 0xc200000f58, 0xc200000f58, ...)
    /usr/local/go/src/pkg/io/io.go:336
created by github.com/MG-RAST/AWE/lib/worker.RunWorkunitDirect
    /tmp/go_build.tmp/src/github.com/MG-RAST/AWE/lib/worker/processor.go:605 +0xb5a

Absolute paths are fine:

crusherofheads@icrushdeheads:~/localgit/user_and_job_state$ tree ujs_test_temp/AweController-2126706462533561211/
ujs_test_temp/AweController-2126706462533561211/
├── awe
│   ├── awfs
│   ├── data
*snip*
│   │   ├── maxjid
│   │   ├── pidfile
│   │   └── temp
│   ├── logs
│   │   └── server
│   │       ├── access.log
│   │       ├── debug.log
│   │       ├── error.log
│   │       ├── event.log
│   │       └── perf.log
│   └── site
├── awec
│   ├── data
│   │   ├── clientprofile.json
│   │   └── pidfile
│   ├── logs
│   │   └── client
│   │       ├── access.log
│   │       ├── debug.log
│   │       ├── error.log
│   │       ├── event.log
│   │       └── perf.log
│   ├── site
│   └── work
*snip*
├── awec.cfg
├── awe.cfg
├── awe_client.log
├── awe_server.log
└── client_script.py

29 directories, 29 files

crusherofheads@icrushdeheads:~/localgit/user_and_job_state$ cat ujs_test_temp/AweController-2126706462533561211/awec.cfg
[Directories]
site=/home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2126706462533561211/awec/site
data=/home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2126706462533561211/awec/data
logs=/home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2126706462533561211/awec/logs

[Args]
debuglevel=0

[Client]
workpath=/home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2126706462533561211/awec/work
supported_apps=client_script.py
serverurl=http://localhost:56983
group=kbase-fake-group
name=kbase-fakeclient
auto_clean_dir=false
worker_overlap=false
print_app_msg=true
clientgroup_token=

crusherofheads@icrushdeheads:~/localgit/user_and_job_state$ cat ujs_test_temp/AweController-2126706462533561211/awe_client.log 
##### pidfile #####
pid: 6298 saved to file: /home/crusherofheads/localgit/user_and_job_state/ujs_test_temp/AweController-2126706462533561211/awec/data/pidfile

clientgroup token not set, register as a public client (can only access public data)
Client registered, name=kbase-fakeclient, id=fb7e3462-826f-4b5c-bbdd-4a613915e3f0
dataMover launched, client=fb7e3462-826f-4b5c-bbdd-4a613915e3f0
processor launched, client=fb7e3462-826f-4b5c-bbdd-4a613915e3f0
deliverer launched, client=fb7e3462-826f-4b5c-bbdd-4a613915e3f0
worker: start cmd=client_script.py, args=[]
worker: start cmd=client_script.py, args=[]