Permission issue in tutorial

[little-peter-pan]

Permission issue in tutorial

[onuryukselen]

It looks like running singularity inside of the docker container triggered that error. Here, it was reported before: apptainer/singularity#5408
They suggested to updating singularity version. We will update singularity version in our docker image, but for now you can follow these steps in your docker container to test this:

1. Remove existing singularity: https://sylabs.io/guides/3.0/user-guide/installation.html#remove-an-old-version
2. Remove old Go: rm -rvf /usr/local/go/
3. Install latest go v1.3:
   export VERSION=1.13 OS=linux ARCH=amd64 && \ wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \ sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \ rm go$VERSION.$OS-$ARCH.tar.gz && export PATH=$PATH:/usr/local/go/bin
4. apt-get install -y cryptsetup
5. Then install new singularity 3.6: https://sylabs.io/guides/3.6/user-guide/quick_start.html#download-singularity-from-a-release

[onuryukselen]

Hello, could you please enter docker terminal and run following commands and send me the output:
chmod -R 777 /export
ls -lha /

Also may I learn your docker command to initialize your docker.

[little-peter-pan]

Here is the output:
total 84K
drwxr-xr-x 1 root root 4.0K Feb 10 14:53 .
drwxr-xr-x 1 root root 4.0K Feb 10 14:53 ..
-rwxr-xr-x 1 root root 0 Feb 10 14:53 .dockerenv
drwxrwxrwx 1 root root 4.0K Nov 3 18:12 .nextflow
drwxr-xr-x 1 root root 4.0K Nov 3 18:07 bin
drwxr-xr-x 2 root root 4.0K Apr 12 2016 boot
drwxrwxrwx 1 root root 4.0K Nov 3 18:59 data
drwxr-xr-x 9 root root 2.8K Feb 10 14:53 dev
drwxr-xr-x 1 root root 4.0K Feb 10 14:53 etc
drwxrwxrwx 6 opencpu 1000 4.0K Feb 10 14:17 export
drwxr-xr-x 1 root root 4.0K Nov 3 18:49 home
drwxr-xr-x 1 root root 4.0K Nov 3 18:09 lib
drwxr-xr-x 2 root root 4.0K Oct 14 11:01 lib64
drwxr-xr-x 2 root root 4.0K Oct 14 11:00 media
drwxr-xr-x 2 root root 4.0K Oct 14 11:00 mnt
drwxr-xr-x 2 root root 4.0K Oct 14 11:00 opt
dr-xr-xr-x 185 root root 0 Feb 10 14:53 proc
drwx------ 1 root root 4.0K Nov 3 18:58 root
drwxr-xr-x 1 root root 4.0K Nov 3 18:49 run
drwxr-xr-x 1 root root 4.0K Nov 3 18:07 sbin
drwxr-xr-x 2 root root 4.0K Oct 14 11:00 srv
dr-xr-xr-x 11 root root 0 Feb 10 14:53 sys
drwxrwxrwt 1 root root 4.0K Nov 3 18:58 tmp
drwxr-xr-x 1 root root 4.0K Oct 14 11:00 usr
drwxr-xr-x 1 root root 4.0K Nov 3 18:06 var

The docker command I used is: docker run --rm --name test --privileged -m 10G -p 8080:80 -v ~/export:/export -ti ummsbiocore/dolphinnext-studio /bin/bash

[onuryukselen]

Could you please change the ownership of /export folder inside docker.
chown root: root /export
Also you might try changing permission of ~/export in windows.

[little-peter-pan]

I have changed /export ownership to root and tried again, but still the same error message:
ssh -oStrictHostKeyChecking=no -q -oChallengeResponseAuthentication=no -oBatchMode=yes -oPasswordAuthentication=no -oConnectTimeout=3 -i /export/.dolphinnext/.ssh/1_1_ssh_pri.pky docker@localhost "mkdir -p /export/run3 && echo 'INFO: Run directory created.'" 2>&1
mkdir: cannot create directory ‘/export/run3’: Permission denied
ERROR: Run directory cannot be created.\nLOG: mkdir: cannot create directory ‘/export/run3’: Permission denied

Here is what is inside /export:
total 20K
drwxr-xr-x 5 root root 4.0K Feb 10 15:15 .
drwxr-xr-x 1 root root 4.0K Feb 10 15:20 ..
drwxr-xr-x 3 www-data www-data 4.0K Feb 10 15:15 .dolphinnext
drwxr-xr-x 11 www-data www-data 4.0K Nov 3 18:12 dolphinnext
drwxr-xr-x 7 mysql mysql 4.0K Feb 10 15:21 mysql

And it is exactly the same in my Ubuntu /export folder.
I tried to run "mkdir -p /export/run3" directly in docker and the folder can be created, but further analysis is still not working. Not sure if this is due to ownership or something else...

[nephantes]

Looks like you are using docker@localhost user. You need to give the write permission to that user. Why don't you change the ownership of /export folder to docker? Or create a folder /export/test and change the ownership to "docker" user for that folder and use it

[little-peter-pan]

Yes, I just figured out to change to docker user, and it worked, thanks.

But here comes the new problem:

1. When I tried to add fastq files to input_reads, it shows the error message like "Query error, please check your connection...", and I have to restart docker run and then was able to see those files (page refresh not helpful). Not sure what caused this, but at least there is a workaround.

2. Then the run failed in both docker image and singularity image:
   --- The docker version log output:
   ssh -oStrictHostKeyChecking=no -q -oChallengeResponseAuthentication=no -oBatchMode=yes -oPasswordAuthentication=no -oConnectTimeout=3 -i /export/.dolphinnext/.ssh/1_1_ssh_pri.pky docker@localhost "mkdir -p /export/run1 && echo 'INFO: Run directory created.'" 2>&1
   INFO: Run directory created.
   rsync -e 'ssh -oStrictHostKeyChecking=no -q -oChallengeResponseAuthentication=no -oBatchMode=yes -oPasswordAuthentication=no -oConnectTimeout=3 -i /export/.dolphinnext/.ssh/1_1_ssh_pri.pky' ../tmp/pub/ssRieZBHw8e9MPFdbuEehwLmdUCXTU/run.tar.gz docker@localhost:/export/run1 2>&1
   ssh -oStrictHostKeyChecking=no -q -oChallengeResponseAuthentication=no -oBatchMode=yes -oPasswordAuthentication=no -oConnectTimeout=3 -i /export/.dolphinnext/.ssh/1_1_ssh_pri.pky docker@localhost " test -f '/export/run1/run.tar.gz' && echo 'INFO: Run package exists.'" 2>&1
   INFO: Run package exists.
   ssh -oStrictHostKeyChecking=no -q -oChallengeResponseAuthentication=no -oBatchMode=yes -oPasswordAuthentication=no -oConnectTimeout=3 -i /export/.dolphinnext/.ssh/1_1_ssh_pri.pky docker@localhost "source /etc/profile && tar xf /export/run1/run.tar.gz -C /export/run1 && rm /export/run1/run.tar.gz && bash /export/run1/.dolphinnext.init" >> ../tmp/pub/ssRieZBHw8e9MPFdbuEehwLmdUCXTU/run/serverlog.txt 2>&1 & echo $! &
   cp /export/run1/log.txt /export/run1/log.txt.1 2>/dev/null || true && >/export/run1/log.txt && mv /export/run1/timeline.html /export/run1/timeline.html.1 2>/dev/null || true && mv /export/run1/trace.txt /export/run1/trace.txt.1 2>/dev/null || true && mv /export/run1/dag.html /export/run1/dag.html.1 2>/dev/null || true && mv /export/run1/report.html /export/run1/report.html.1 2>/dev/null || true && mv /export/run1/.nextflow.log /export/run1/.nextflow.log.1 2>/dev/null || true && mv /export/run1/err.log /export/run1/err.log.1 2>/dev/null || true && mv /export/run1/out.log /export/run1/out.log.1 2>/dev/null || true && cp /export/run1/initialrun/initial.log /export/run1/initialrun/initial.log.1 2>/dev/null || true && >/export/run1/initialrun/initial.log && mv /export/run1/initialrun/timeline.html /export/run1/initialrun/timeline.html.1 2>/dev/null || true && mv /export/run1/initialrun/trace.txt /export/run1/initialrun/trace.txt.1 2>/dev/null || true && mv /export/run1/initialrun/dag.html /export/run1/initialrun/dag.html.1 2>/dev/null || true && mv /export/run1/initialrun/report.html /export/run1/initialrun/report.html.1 2>/dev/null || true && mv /export/run1/initialrun/.nextflow.log /export/run1/initialrun/.nextflow.log.1 2>/dev/null || true && mv /export/run1/initialrun/err.log /export/run1/initialrun/err.log.1 2>/dev/null || true && mv /export/run1/initialrun/out.log /export/run1/initialrun/out.log.1 2>/dev/null || true && mkdir -p /export/run1/.dolphinnext/uuid && touch /export/run1/.dolphinnext/uuid/ssRieZBHw8e9MPFdbuEehwLmdUCXTU && export NXF_VER=19.10.0 && export NXF_ANSI_LOG=false && source /etc/profile && NXF_SINGULARITY_CACHEDIR="${NXF_SINGULARITY_CACHEDIR:-$HOME/.dolphinnext/singularity}" && export NXF_SINGULARITY_CACHEDIR=$NXF_SINGULARITY_CACHEDIR && rm -rf /export/report1 && cd /export/run1/initialrun && nextflow /export/run1/initialrun/nextflow.nf -with-report -with-trace -with-timeline > /export/run1/initialrun/initial.log && cd /export/run1 && nextflow /export/run1/nextflow.nf -with-report -with-trace -with-timeline > /export/run1/log.txt && rm -rf /export/run1/work
   N E X T F L O W ~ version 19.10.0
   Launching /export/run1/initialrun/nextflow.nf [loving_dubinsky] - revision: 7c01fc7e5b
   [4b/fb4ebd] Submitted process > initialCheck (1)
   [7e/4beb8c] Submitted process > createCollection (3)
   [0b/4c37be] Submitted process > createCollection (1)
   [53/83a105] Submitted process > createCollection (5)
   [b1/4e6135] Submitted process > createCollection (2)
   [0d/9c7af1] Submitted process > createCollection (4)
   [09/bfa6a7] Submitted process > createCollection (6)
   [99/96e553] Submitted process > cleanUp (1)
   ##Initial run summary##
   ##Completed at: 2021-02-10T21:41:51.717Z
   ##Duration: 3.8s
   ##Success: PASSED
   ##Exit status: 0
   ##Waiting for the Next Run..
   N E X T F L O W ~ version 19.10.0
   Launching /export/run1/nextflow.nf [thirsty_ptolemy] - revision: 31a51a5ae2
   [11/1df8d3] Submitted process > FastQC (6)
   [d1/4b276b] Submitted process > FastQC (5)
   Error executing process > 'FastQC (6)'

Caused by:
Process FastQC (6) terminated with an error exit status (127)

Command executed:

fastqc control_rep3.fastq

Command exit status:
127

Command output:
(empty)

Command error:
.command.run: line 240: docker: command not found

Work dir:
/export/run1/work/11/1df8d33f3bafdef99ebea9a4de33a9

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

##Pipeline execution summary##

##Completed at: 2021-02-10T21:41:55.621Z
##Duration: 1s
##Success: failed
##Exit status: 127
WARN: Killing pending tasks (1)

--- The singularity version log output is even longer, but something like this:
ssh -oStrictHostKeyChecking=no -q -oChallengeResponseAuthentication=no -oBatchMode=yes -oPasswordAuthentication=no -oConnectTimeout=3 -i /export/.dolphinnext/.ssh/1_1_ssh_pri.pky docker@localhost "mkdir -p /export/run3 && echo 'INFO: Run directory created.'" 2>&1
INFO: Run directory created.
rsync -e 'ssh -oStrictHostKeyChecking=no -q -oChallengeResponseAuthentication=no -oBatchMode=yes -oPasswordAuthentication=no -oConnectTimeout=3 -i /export/.dolphinnext/.ssh/1_1_ssh_pri.pky' ../tmp/pub/WeJhHrRrJ8lSgDD4rTqoOmGgcNi2Z3/run.tar.gz docker@localhost:/export/run3 2>&1
ssh -oStrictHostKeyChecking=no -q -oChallengeResponseAuthentication=no -oBatchMode=yes -oPasswordAuthentication=no -oConnectTimeout=3 -i /export/.dolphinnext/.ssh/1_1_ssh_pri.pky docker@localhost " test -f '/export/run3/run.tar.gz' && echo 'INFO: Run package exists.'" 2>&1
INFO: Run package exists.
ssh -oStrictHostKeyChecking=no -q -oChallengeResponseAuthentication=no -oBatchMode=yes -oPasswordAuthentication=no -oConnectTimeout=3 -i /export/.dolphinnext/.ssh/1_1_ssh_pri.pky docker@localhost "source /etc/profile && tar xf /export/run3/run.tar.gz -C /export/run3 && rm /export/run3/run.tar.gz && bash /export/run3/.dolphinnext.init" >> ../tmp/pub/WeJhHrRrJ8lSgDD4rTqoOmGgcNi2Z3/run/serverlog.txt 2>&1 & echo $! &
cp /export/run3/log.txt /export/run3/log.txt.2 2>/dev/null || true && >/export/run3/log.txt && mv /export/run3/timeline.html /export/run3/timeline.html.2 2>/dev/null || true && mv /export/run3/trace.txt /export/run3/trace.txt.2 2>/dev/null || true && mv /export/run3/dag.html /export/run3/dag.html.2 2>/dev/null || true && mv /export/run3/report.html /export/run3/report.html.2 2>/dev/null || true && mv /export/run3/.nextflow.log /export/run3/.nextflow.log.2 2>/dev/null || true && mv /export/run3/err.log /export/run3/err.log.2 2>/dev/null || true && mv /export/run3/out.log /export/run3/out.log.2 2>/dev/null || true && cp /export/run3/initialrun/initial.log /export/run3/initialrun/initial.log.2 2>/dev/null || true && >/export/run3/initialrun/initial.log && mv /export/run3/initialrun/timeline.html /export/run3/initialrun/timeline.html.2 2>/dev/null || true && mv /export/run3/initialrun/trace.txt /export/run3/initialrun/trace.txt.2 2>/dev/null || true && mv /export/run3/initialrun/dag.html /export/run3/initialrun/dag.html.2 2>/dev/null || true && mv /export/run3/initialrun/report.html /export/run3/initialrun/report.html.2 2>/dev/null || true && mv /export/run3/initialrun/.nextflow.log /export/run3/initialrun/.nextflow.log.2 2>/dev/null || true && mv /export/run3/initialrun/err.log /export/run3/initialrun/err.log.2 2>/dev/null || true && mv /export/run3/initialrun/out.log /export/run3/initialrun/out.log.2 2>/dev/null || true && mkdir -p /export/run3/.dolphinnext/uuid && touch /export/run3/.dolphinnext/uuid/WeJhHrRrJ8lSgDD4rTqoOmGgcNi2Z3 && export NXF_VER=19.10.0 && export NXF_ANSI_LOG=false && source /etc/profile && NXF_SINGULARITY_CACHEDIR="${NXF_SINGULARITY_CACHEDIR:-$HOME/.dolphinnext/singularity}" && export NXF_SINGULARITY_CACHEDIR=$NXF_SINGULARITY_CACHEDIR && mkdir -p $NXF_SINGULARITY_CACHEDIR && cd $NXF_SINGULARITY_CACHEDIR && if [ ! -f $NXF_SINGULARITY_CACHEDIR/dolphinnext-rnaseq-1.0.simg ]; then singularity pull --name dolphinnext-rnaseq-1.0.simg docker://dolphinnext/rnaseq:1.0; fi && mkdir -p $NXF_SINGULARITY_CACHEDIR && cd $NXF_SINGULARITY_CACHEDIR && if [ ! -f $NXF_SINGULARITY_CACHEDIR/UMMS-Biocore-initialrun-09.06.2020.simg ]; then wget https://galaxyweb.umassmed.edu/pub/dolphinnext_singularity/UMMS-Biocore-initialrun-09.06.2020.simg; fi && find $NXF_SINGULARITY_CACHEDIR -type f -regex '.*UMMS-Biocore-initialrun.*img' -not -name 'UMMS-Biocore-initialrun-09.06.2020.simg' -delete && rm -rf /export/report3 && cd /export/run3/initialrun && nextflow /export/run3/initialrun/nextflow.nf -with-report -with-trace -with-timeline > /export/run3/initialrun/initial.log && cd /export/run3 && nextflow /export/run3/nextflow.nf -with-report -with-trace -with-timeline > /export/run3/log.txt && rm -rf /export/run3/work
--2021-02-10 21:22:36-- https://galaxyweb.umassmed.edu/pub/dolphinnext_singularity/UMMS-Biocore-initialrun-09.06.2020.simg
Resolving galaxyweb.umassmed.edu (galaxyweb.umassmed.edu)... 205.172.168.17
Connecting to galaxyweb.umassmed.edu (galaxyweb.umassmed.edu)|205.172.168.17|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 614080543 (586M) [text/plain]
Saving to: ‘UMMS-Biocore-initialrun-09.06.2020.simg’

 0K .......... .......... .......... .......... ..........  0%  555K 18m1s
 ......

599650K .......... .......... .......... ........ 100% 24.2M=88s

2021-02-10 21:24:05 (6.62 MB/s) - ‘UMMS-Biocore-initialrun-09.06.2020.simg’ saved [614080543/614080543]
N E X T F L O W ~ version 19.10.0
Launching /export/run3/initialrun/nextflow.nf [stupefied_perlman] - revision: 7c01fc7e5b
[ce/0d6c55] Submitted process > initialCheck (1)
Error executing process > 'initialCheck (1)'

Caused by:
Process initialCheck (1) terminated with an error exit status (255)

Command executed:

#!/usr/bin/env perl
......
Command exit status:
255

Command output:
(empty)

Command error:
�[31mFATAL: �[0m container creation failed: mount /proc/self/fd/6->/usr/local/var/singularity/mnt/session/rootfs error: can't mount image /proc/self/fd/6: failed to mount squashfs filesystem: invalid argument

Work dir:
/export/run3/initialrun/work/ce/0d6c55aaf812ea36d5ff3b5e86d335

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

##Initial run summary##
##Completed at: 2021-02-10T21:24:10.064Z
##Duration: 2s
##Success: failed
##Exit status: 255
##Waiting for the Next Run..

I just tried to run fastqc, and not doing other two, and it seems they can not run fastqc or even initial check...

[little-peter-pan]

Hi, following your instruction, the issue was successfully resolved. Thank you!