Comments (6)
Possibly. I'll keep this open and probably rename the issue, add detail so we might have a roadmap on how to do that. However, there's no simple switch that we can use to make this happen right away.
from dnmtools.
Hi @andrewdavidsmith
Thanks for your response.
I have run DNMTOOLS states on sam file (44GB). It has been running for >100 hours and have generated 3 MB of epireads.
A snippet from "top"
Seems like "states" could be made capable of using more memory and cores. The input sam could be split the temporary samlets and those samlets can be read in parallel.
Thanks.
from dnmtools.
That's not the expected behavior. If the reads are not sorted in the expected order, there's a chance it could turn from a linear time computation into a quadratic time one. If you could find a way to share the data with me I can check. Although I know you might not want to share all the data, the problem might not be easily revealed on just a small part. Let me know and feel free to email me.
from dnmtools.
The right thing for us to do is have the code verify the sorting of reads, but sometimes the code attempts to just proceed and compensate if it gets unexpected input.
I also notice from your screen capture that the program is using 17.2g of vmem, but only 3.2g of pmem, which suggests something else is going on and the program is likely thrashing. Are you sure you have sufficient available physical memory for that process?
from dnmtools.
@andrewdavidsmith You were right, the algorithm was compensating for unsorted inputs. I have now been successfully ran the conversion to epiread pretty quickly.
In terms of accelerating the program, I agree that the code should verify the sorting first.
Run details- ~15 mins to generate 1.5G epireads from 35G deduped sorted sam inputs (HG38).
Adding CPU info and meminfo here in case it's helpful to you-
CPU(s): 96 Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz
Memory: 790 Gigs
I don't understand the thrashing part. Sharing another snippet here-
from dnmtools.
@hchetia I'm closing this because I think the issue has been solved. It should not continue if the input is unsorted in a way that will cause slowdown. Specifically, all reads from the same chromosome need to be consecutive.
from dnmtools.
Related Issues (20)
- dnmtools diff Error HOT 4
- Calling HMRs on the DNA Methylation Atlas Data HOT 4
- roi "cannot sort data unless all data is loaded" HOT 3
- mlml "error: chrom or pos mismatch in both files" HOT 3
- Keep non-CpG sites after sym command
- Broken progress bar
- roi -M return different values for "number of CpGs in the region" for different count files HOT 3
- roi should check for input files before reporting any problems with arguments
- pmd crash if insufficient data unless verbose mode used HOT 1
- -v flag opposite in guessprotocol
- A question for 'dnmtools allelic' HOT 2
- format: max distance when merging mates seems ignored
- merge -t header seems to be not properly formatted HOT 3
- HMR crash when nothing identified
- bad count line HOT 1
- resolved.
- unknown vectorizer cost model ‘very-cheap’ HOT 6
- summary hmr_count is non-zero but hmr output file is empty HOT 5
- Building dnmtools using gcc 8.5.0
- Building dnmtools using gcc 8.5.0 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dnmtools.