Comments (15)
Hi Randy,
I don't have a clue so far. It looks like some kind of memory leak in the writer for the binary pileups format. Here are some questions to get some light into the dark:
- Is the memory consumption rising over time?
- How big is the input
.las
files?numAlignmentChains
is just 3186 so I guess it's rather small. - Is it possible to share the input files? If yes, please drop them into my ownCloud so I can try reproducing the bug.
- Can you please share the full command that is being executed (you may remove sensitive names from the paths)? You can make snakemake report the command by running it with options
-np
.
from dentist.
from dentist.
from dentist.
Sure, perfect!
from dentist.
from dentist.
Hey Randy,
got the files and they are OK. I will write to you once I have news.
from dentist.
So, in my setup it worked. It took 08:55 hours with 2 CPUs and consumed max. RSS of 86.5 GB. Most of the time (6.7h) was spent reading the large alignment file. The routine is not very optimized, I have to admit.
Now that you know how many resources are required, can you try again? I ran the job with 24h time limit and max allowed RSS of 128G.
from dentist.
from dentist.
from dentist.
from dentist.
Hi Randy,
the write-protection is likely not the cause of your issue because it is made by snakemake after dentist finished successfully. You may double-check this by verifying if dentist is still active.
I guess it might be file-locking: dentist tries to lock files it reads or writes via flockfile
. If an error occurs (like it does on our cluster because file locking is not implemented) it will just open the file without locking and continue. In contrast, it will just get stuck if the file lock cannot be acquired for some reason.
I created another version of dentist that allows skipping the locking step entirely. I hope it just works because I did not test it at all.
dentist.v1.0.0-beta.1-6-gce64df8.x86_64.tar.gz
from dentist.
from dentist.
from dentist.
Sorry, I forgot to mention: SKIP_FILE_LOCKING=1
should do the trick.
from dentist.
from dentist.
Related Issues (20)
- Missing Snakefile from example test dataset HOT 8
- Failed to run with test data set: TypeError in line 45 of Snakefile HOT 6
- MissingOutputException error HOT 1
- md5checksum differences between nodes HOT 9
- Pacbio header line format error HOT 16
- Math error in Snakefile HOT 2
- rule mask_self fails on Docker container HOT 3
- Singularity image is lacking a Python installation HOT 4
- Could not create pipe to check startup of child (Too many open files) HOT 2
- example failing due to computeintrinsicqv HOT 3
- no gap-filling in "scaffold" mode? HOT 11
- Conda-install-error HOT 9
- Can gaps be filled with just extension reads? HOT 5
- `subprocess.run` got an unexpected keyword argument 'text' HOT 4
- No joined contigs with greedy configuration HOT 2
- Error in rule validate_regions_block HOT 2
- one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode! HOT 3
- Error: File stdout.fasta, Line 1: First header in fasta file is missing HOT 2
- Rerun after stop due to time limit HOT 4
- Dentist parameters HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dentist.