Comments (38)
A workaround might be just running this thing as root
to use your personal machine owner privilege.
from pgap.
Do I have to install another instance of docker on root?
Probably. Typically software of this kinds is installed by "admin" so more than user can use it. Since this is your personal station, this rule seems to be less obvious, but it makes sense to install it as "root".
from pgap.
Thanks for your report, Ray!
Could you please post a listing of your mg37_results/
directory?
from pgap.
Thanks for yr quick response! I sent mg37_results/cwltool.log. The only other dir is mg37_results/debug/log, but it's completely empty.
from pgap.
Thanks, but all I need is a listing, as in ls -alR
from pgap.
Sorry. Here it is:
(base) gene@precision6:~/mg37_results$ ls -alR
.:
total 20
drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 .
drwxr-xr-x 81 gene gene 4096 Dec 13 10:51 ..
-rw-rw-r-- 1 gene gene 3108 Dec 13 10:51 cwltool.log
drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 debug
-rw-rw-r-- 1 gene gene 74 Dec 13 10:51 .~lock.cwltool.log#
./debug:
total 12
drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 .
drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 ..
drwxrwxr-x 2 gene gene 4096 Dec 13 10:51 log
./debug/log:
total 8
drwxrwxr-x 2 gene gene 4096 Dec 13 10:51 .
drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 ..
(base) gene@precision6:~/mg37_results$
from pgap.
Thanks!
Looks like the permissions are fine. Could you please try to run this command and post the response?
/usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
--volume /home/gene:/pgap/user_input:z \
--volume /home/gene/pgap_input__o732xtc.yaml:/pgap/user_input/pgap_input.yaml:ro,z \
--volume /home/gene/mytmp:/tmp:rw,z \
--volume /home/gene/mg37_results:/pgap/output:rw,z \
--volume /home/gene/mg37_results/debug/log:/log/srv:z \
ncbi/pgap:2023-10-03.build7061 \
mkdir /pgap/output/debug/tmpdir
from pgap.
My bad, command line fixed now.
from pgap.
(base) gene@precision6:~/mg37_results$ /usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z
--volume /home/gene:/pgap/user_input:z
--volume /home/gene/pgap_input__o732xtc.yaml:/pgap/user_input/pgap_input.yaml:ro,z
--volume /home/gene/mytmp:/tmp:rw,z
--volume /home/gene/mg37_results:/pgap/output:rw,z
--volume /home/gene/mg37_results/debug/log:/log/srv:z
ncbi/pgap:2023-10-03.build7061
mkdir /pgap/output/debug/tmpdir
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/host_mnt/home/gene/pgap_input__o732xtc.yaml" to rootfs at "/pgap/user_input/pgap_input.yaml": mount /host_mnt/home/gene/pgap_input__o732xtc.yaml:/pgap/user_input/pgap_input.yaml (via /proc/self/fd/9), flags: 0x5001: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
from pgap.
Thanks, how about this one:
/usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
--volume /home/gene:/pgap/user_input:z \
--volume /home/gene/mytmp:/tmp:rw,z \
--volume /home/gene/mg37_results:/pgap/output:rw,z \
--volume /home/gene/mg37_results/debug/log:/log/srv:z \
ncbi/pgap:2023-10-03.build7061 \
mkdir /pgap/output/debug/tmpdir
from pgap.
(base) gene@precision6:~/mg37_results$ /usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z
--volume /home/gene:/pgap/user_input:z
--volume /home/gene/mytmp:/tmp:rw,z
--volume /home/gene/mg37_results:/pgap/output:rw,z
--volume /home/gene/mg37_results/debug/log:/log/srv:z
ncbi/pgap:2023-10-03.build7061
mkdir /pgap/output/debug/tmpdir
mkdir: cannot create directory '/pgap/output/debug/tmpdir': Permission denied
from pgap.
Thanks. Could you please run these commands?
/usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
--volume /home/gene:/pgap/user_input:z \
--volume /home/gene/mytmp:/tmp:rw,z \
--volume /home/gene/mg37_results:/pgap/output:rw,z \
--volume /home/gene/mg37_results/debug/log:/log/srv:z \
ncbi/pgap:2023-10-03.build7061 \
ls -lanR /pgap/output/debug/
this:
/usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
--volume /home/gene:/pgap/user_input:z \
--volume /home/gene/mytmp:/tmp:rw,z \
--volume /home/gene/mg37_results:/pgap/output:rw,z \
--volume /home/gene/mg37_results/debug/log:/log/srv:z \
ncbi/pgap:2023-10-03.build7061 \
id -u
and
/usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
--volume /home/gene:/pgap/user_input:z \
--volume /home/gene/mytmp:/tmp:rw,z \
--volume /home/gene/mg37_results:/pgap/output:rw,z \
--volume /home/gene/mg37_results/debug/log:/log/srv:z \
ncbi/pgap:2023-10-03.build7061 \
id -g
from pgap.
(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z
--volume /home/gene:/pgap/user_input:z
--volume /home/gene/mytmp:/tmp:rw,z
--volume /home/gene/mg37_results:/pgap/output:rw,z
--volume /home/gene/mg37_results/debug/log:/log/srv:z
ncbi/pgap:2023-10-03.build7061
ls -lanR /pgap/output/debug/
/pgap/output/debug/:
total 12
drwxrwxr-x 3 0 0 4096 Dec 13 15:51 .
drwxrwxr-x 3 0 0 4096 Dec 13 15:51 ..
drwxrwxr-x 2 0 0 4096 Dec 13 15:51 log
/pgap/output/debug/log:
total 8
drwxrwxr-x 2 0 0 4096 Dec 13 15:51 .
drwxrwxr-x 3 0 0 4096 Dec 13 15:51 ..
(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z
--volume /home/gene:/pgap/user_input:z
--volume /home/gene/mytmp:/tmp:rw,z
--volume /home/gene/mg37_results:/pgap/output:rw,z
--volume /home/gene/mg37_results/debug/log:/log/srv:z
ncbi/pgap:2023-10-03.build7061
id -u
1000
(base) gene@precision6:$ /usr/local/bin/docker run -i --rm --user 1000:1000$
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z
--volume /home/gene:/pgap/user_input:z
--volume /home/gene/mytmp:/tmp:rw,z
--volume /home/gene/mg37_results:/pgap/output:rw,z
--volume /home/gene/mg37_results/debug/log:/log/srv:z
ncbi/pgap:2023-10-03.build7061
id -g
1000
(base) gene@precision6:
from pgap.
Thanks, Ray. Now we are talking.
Now I see what the problem is. docker
for some reason does not recognize --user 1000:1000
setting on your machine for user and group ids and uses straight root/root (0/0) settings.
This is peculiar. Normally --user
settings are recognized. I am going to research this on the Internet and I suggest that maybe you can request help of your local system folks to clarify what is going on with your docker
installation
from pgap.
Thanks for your help. Unfortunately, It's my own machine and I installed docker myself so you're my only lifeline for this and I'm a novice at this.
from pgap.
No problem. I opened an internal ticket, now that we identified the specific problem with your installation. The information that it is your personal home computer is useful.
from pgap.
Thanks Azat! I phoned a friend about this and he noticed the docker id was not showing up and had me restart. It's now showing up:
(base) gene@precision6:~$ id
uid=1000(gene) gid=1000(gene) groups=1000(gene),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),132(lxd),133(sambashare),997(docker),998(minknow)
However, when I reran the M. genitalium test it still errors out, bottom lines follow:
"sqn": {
"location": "file:///var/tmp/t7v5wzep/annot.sqn",
"basename": "annot.sqn",
"nameroot": "annot",
"nameext": ".sqn",
"class": "File",
"checksum": "sha1$92a7bf56b7e6b66069d38c76b84197c7e97777e2",
"size": 5411919,
"http://commonwl.org/cwltool#generation": 0
}
}
[2023-12-13 23:31:40] DEBUG [job Generate_Annotation_Reports_gff_enhanced] Removing input staging directory /var/tmp/ijibnx6m
[2023-12-13 23:31:40] DEBUG [job Generate_Annotation_Reports_gff_enhanced] Removing temporary directory /var/tmp/pgx8bx5_
[2023-12-13 23:31:40] DEBUG Moving /var/tmp/6ifwcjq2/calls.tab to /pgap/output/calls.tab
[2023-12-13 23:31:40] ERROR Unhandled error:
[Errno 13] Permission denied: '/pgap/output/calls.tab'
Traceback (most recent call last):
File "/opt/python-3.9/lib/python3.9/shutil.py", line 825, in move
os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/var/tmp/6ifwcjq2/calls.tab' -> '/pgap/output/calls.tab'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/venv/lib/python3.9/site-packages/cwltool/main.py", line 1295, in main
(out, status) = real_executor(
File "/root/venv/lib/python3.9/site-packages/cwltool/executors.py", line 61, in call
return self.execute(process, job_order_object, runtime_context, logger)
File "/root/venv/lib/python3.9/site-packages/cwltool/executors.py", line 147, in execute
self.final_output[0] = relocateOutputs(
File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 383, in relocateOutputs
stage_files(pm, stage_func=_relocate, symlink=False, fix_conflicts=True)
File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 281, in stage_files
stage_func(entry.resolved, entry.target)
File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 356, in _relocate
shutil.move(src, dst)
File "/opt/python-3.9/lib/python3.9/shutil.py", line 845, in move
copy_function(src, real_dst)
File "/opt/python-3.9/lib/python3.9/shutil.py", line 444, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/opt/python-3.9/lib/python3.9/shutil.py", line 266, in copyfile
with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/pgap/output/calls.tab'
full .zip attached:
cwltool20231213.zip
The --debug is the same as before: cwltool.log
from pgap.
I am glad that docker problem was resolved so easily, Ray!
Now to this beast:
OSError: [Errno 18] Invalid cross-device link: '/var/tmp/6ifwcjq2/calls.tab' -> '/pgap/output/calls.tab'
I would suggest to point TMPDIR envar to a directory in the same disk as your work directory.
For example:
mkdir -p tmpdir
export TMPDIR=$(readlink -f tmpdir)
and then try to run MG37 example again.
from pgap.
Azat, reran MG37 AND output is exactly the same as the my post 3 days ago.
from pgap.
Hmm. "Exactly" immediately makes me think of the same cause: Docker shenanigans. Could you please do the id
test for Docker you did before you found necessity to restart it?
from pgap.
It's trying to reinstall everything at /root/. I stopped it. I can't afford the space of another instance of pgap.
from pgap.
I can't afford the space of another instance of pgap.
You have a massive amount of memory, Ray, seems that this should come with analogously massive amount of space. But you have only
"work disk space (GiB)": 571.5
Anyway, I would suggest to ditch the first instance then. But this is actually Plan B. Plan A is still to fix your docker installation. Could you please test what I asked you earlier?
Could you please do the
id
test for Docker you did before you found necessity to restart it?
from pgap.
Thanks, Azat. Installng on root now and will test.
from pgap.
I see you are going with Plan B, which seems to be more straightforward.
from pgap.
Trying to figure out how to get docker running on root. Installation of pgap failed because docker was only running on gene (my user account). Do I have to install another instance of docker on root?
from pgap.
Tried to install docker at root, could not get it to install properly. The desktop came up, but could not get it to turn on at root. Tried sharing dir /root in docker, but it made no difference. Reinstalled docker as user (gene). Reran MG37 AND output is exactly the same as the my post 4 days ago.
(base) gene@precision6:~$ id
uid=1000(gene) gid=1000(gene) groups=1000(gene),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),132(lxd),133(sambashare),997(docker),998(minknow)
from pgap.
Another option to try is make the top of your directory "rw" for everybody: chmod -R a+rw .
from pgap.
Still doesn't seem to be able to make dir pgap?
(base) gene@precision6:$ sudo chmod -R a+rw /home$ ./pgap.py -r -o mg37_results -g $HOME/.pgap/test_genomes/MG37/ASM2732v1.annotation.nucleotide.1.fasta -s 'Mycoplasmoides genitalium' --debug
[sudo] password for gene:
(base) gene@precision6:
PGAP version 2023-10-03.build7061 is up to date.
Output will be placed in: /home/gene/mg37_results
PGAP failed, docker exited with rc = 1
Unable to find error in log file.
(base) gene@precision6:~$
from pgap.
Could you please post output of this commands again?
from pgap.
(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z
--volume /home/gene:/pgap/user_input:z
--volume /home/gene/mytmp:/tmp:rw,z
--volume /home/gene/mg37_results:/pgap/output:rw,z
--volume /home/gene/mg37_results/debug/log:/log/srv:z
ncbi/pgap:2023-10-03.build7061
ls -lanR /pgap/output/debug/
/pgap/output/debug/:
total 12
drwxrwxr-x 3 0 0 4096 Dec 19 15:35 .
drwxrwxr-x 3 0 0 4096 Dec 19 15:49 ..
drwxrwxr-x 2 0 0 4096 Dec 19 15:35 log
/pgap/output/debug/log:
total 8
drwxrwxr-x 2 0 0 4096 Dec 19 15:35 .
drwxrwxr-x 3 0 0 4096 Dec 19 15:35 ..
(base) gene@precision6:~$
(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z
--volume /home/gene:/pgap/user_input:z
--volume /home/gene/mytmp:/tmp:rw,z
--volume /home/gene/mg37_results:/pgap/output:rw,z
--volume /home/gene/mg37_results/debug/log:/log/srv:z
ncbi/pgap:2023-10-03.build7061
id -u
1000
(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \
--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z
--volume /home/gene:/pgap/user_input:z
--volume /home/gene/mytmp:/tmp:rw,z
--volume /home/gene/mg37_results:/pgap/output:rw,z
--volume /home/gene/mg37_results/debug/log:/log/srv:z
ncbi/pgap:2023-10-03.build7061
id -g
1000
from pgap.
Thanks.
Notice that for "all" the permissions are still read-only under docker. Let's double check if chmod -R ...
you did worked.
could you please run ls -lanR mg37_results
(that is, sans docker)?
from pgap.
(base) gene@precision6:~$ ls -lanR mg37_results
mg37_results:
total 16
drwxrwxr-x 3 1000 1000 4096 Dec 19 10:49 .
drwxrwxrwx 88 1000 1000 4096 Dec 19 10:35 ..
-rw-rw-r-- 1 1000 1000 3107 Dec 19 10:35 cwltool.log
drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 debug
mg37_results/debug:
total 12
drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 .
drwxrwxr-x 3 1000 1000 4096 Dec 19 10:49 ..
drwxrwxr-x 2 1000 1000 4096 Dec 19 10:35 log
mg37_results/debug/log:
total 8
drwxrwxr-x 2 1000 1000 4096 Dec 19 10:35 .
drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 ..
(base) gene@precision6:~$
from pgap.
Thanks. I know what is happening. chmod -R a...
worked, but we also need to set this for the future file creations. Could you please execute umask 000
and run PGAP again?
Thanks!
from pgap.
YOU'VE DONE IT! Thanks very much, Azat!
(base) gene@precision6:$ ./pgap.py -r -o mg37_results -g $HOME/.pgap/test_genomes/MG37/ASM2732v1.annotation.nucleotide.1.fasta -s 'Mycoplasmoides genitalium' --debug$
PGAP version 2023-10-03.build7061 is up to date.
Output will be placed in: /home/gene/mg37_results
PGAP completed successfully.
(base) gene@precision6:
from pgap.
You are welcome, Ray. But please have in mind that this was a hacky way, a correct way is to set up docker so it will allow you to recognize user id/group settings from a user outside docker container.
from pgap.
OK thanks. I'll have to read up on docker.
from pgap.
Thanks for your help Azat!
from pgap.
You are welcome, Ray!
from pgap.
Related Issues (20)
- [BUG] SIGBUS error on HPC using singularity HOT 9
- [FEATURE REQUEST] ARM64 support HOT 2
- [FEATURE REQUEST] Add a prefix to the output files of PGAP. HOT 1
- pgap.py not found with docker container ncbi/pgap HOT 4
- Failed to get rootless runtime dir HOT 2
- [BUG] Podman: container name is not respected HOT 3
- [BUG] pgap.py --update HOT 3
- [BUG] PGAP fails install when running on singularity. HOT 2
- [BUG] When two input genomes have same basename, the first genome is mistakenly used for the second run. HOT 3
- [FEATURE REQUEST] direct S3 access to input datasets HOT 5
- Running PGAP with Metagenomic Assembled Genome HOT 11
- [BUG] PGAP analysis generates all files except .aa and .gbk HOT 35
- [FEATURE REQUEST] Unable to retrieve additional reference data from https://s3.amazonaws.com/pgap/input-[version].tgz. HOT 2
- Problem with running pgap annotation HOT 1
- [FEATURE REQUEST] Any plans about EGAP (Eukaryotic Genome Annotation Pipeline)? HOT 2
- [FEATURE REQUEST] <Turn Off GO annotation> HOT 7
- [BUG] Binding non-existent file HOT 6
- [BUG] <title>The length of the gene exceeds the length of the conitg HOT 10
- [BUG] <title>How to run pgap.py with qsub file. HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pgap.