Comments (1)
Hi, thanks for your interest in wgd v2
! I did a quick test, which showed that the size of Ks datafile is the determinant of the (maximum) occupied memory while neither the number of EM iterations nor initializations would impact the maximum memory usage. I used two Ks datafile, of Eriobotrya japonica and Vigna angularis respectively, the Ks distribution of which were embedded in the plots. I used the memory_profiler
module to sample the memory usage every 0.1s. As you can see below, the run of Eriobotrya japonica reached the maximum occupied memory as 1400MiB and cost around 480s, while the run of Vigna angularis reached the maximum occupied memory as around 500 MiB and cost around 90s. The second run of Vigna angularis was under the setting of --em_iter 10
and --n_init 10
, which reached the same maximum occupied memory as around 500 MiB, suggesting that the number of EM iterations or initializations have trivial impact on the maximum occupied memory. If you were dealing with very recent WGD events (which I suppose might be your case), of which the number of anchor pairs would be over, say 20000, it would indeed lead to a very high demand of memory due to factors including the size of pandas
dataframe, perhaps a few GB. One possible solution is that you may delete those unused columns in the Ks dataframe, for instance "N", "S", "t", "g1", "g2", "dN/dS", such that the initial occupied memory (and later of the function fit_apgmm_guide
) could be declined to some extents. But if the number of anchor pairs is very big, the maximum occupied memory will be accordingly high anyway, because of factors for instance the imported EM algorithm, one of the largest memory consumers. May you try again with the less heavy Ks datafile and see if how it works?
from wgd.
Related Issues (20)
- viz running problem HOT 2
- error in Ks correction analysis HOT 34
- ks.svg of ksd output make some mistake HOT 3
- Questions about input file sequence HOT 4
- Can Orthologous Isoforms be identified? HOT 4
- Error running wgd dmd with the globalmrbh flag HOT 3
- Issue with the spair flag HOT 8
- ERROR All families are singleton families, No Ks can be calculated - RBH HOT 6
- Differentiation between true wgd and artifact HOT 19
- wgd dmd HOT 34
- Unable to install wgd HOT 3
- Plot that combines paired comparisons and focal species plot
- ksd .tsv out HOT 8
- syn -gff3 file HOT 25
- RE: dating tree HOT 13
- peak HOT 24
- --globalmrbh HOT 19
- ksd not finishing HOT 12
- wgd dmd - struct.error: 'i' format requires -2147483648 <= number <= 2147483647 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wgd.