Comments (4)
I've thought about this strategy. Here are my reasons for hesitation:
- Some clusters have an enormous amount of latency between job submission and job execution. I've done a lot of my past research on clusters where the wait between job submission and the beginning of job execution tends to run in the 1-4 day range. Quadrupling such latency would be painful.
- I know for some fields (e.g. in quantitative finance; Rick's experiences seem to agree with this) that a really slow "brute force" methodology involving some pedantic Monte Carlo simulation is often the benchmark for the more clever methodology. This might also be true for our genomic work but I'm not sure. A single extremely slow method in a dsc would hold up all of the faster methods.
- As you have mentioned in other issues, it is likely that input parsers and the use of multiple pre/post processing steps could make the maximum number of global barriers (
waitForJobs()
) even larger.
If you feel these aren't important, we can definitely do it this way. Your suggestion is probably the simplest implementation.
from dscr.
I think 1 is presumably going to depend on the cluster environment, but it isn't a problem I have come across in practice with the clusters we are using.
For 2 this scenario is indeed not out of the question, but easily dealt with: first run your dsc for all the fast methods. Then add the slow method and run that.
For 3, I agree, but actually suspect that in most use cases it will be
the methods that are the rate-determining step, not the waitForJobs() on parsers etc.
I think the issue is urgent enough, and this approach simple enough, that we would be best off implementing it first, and seeing what our next bottleneck turns out to be.
from dscr.
Sounds good.
from dscr.
Probably addresses #23 as well.
from dscr.
Related Issues (20)
- Warning on load HOT 4
- Show which trials went wrong when "running Scores" or "running outputParser"? HOT 5
- Big if-else datamaker in one_sample_location HOT 3
- we're being schizophrenic about naming of scores HOT 1
- browseVignette names are not very informative HOT 3
- Clean build HOT 1
- input parser
- add reset scores, reset parser HOT 7
- Naming conventions HOT 3
- vignette does not illustrate reset_dsc HOT 1
- vignette does not provide good overview of work flow
- Make dscr minimally ready for broader collaboration HOT 2
- Confusion with tests/one_sample_location_longoutput HOT 1
- Revisit structure of output parsers HOT 4
- Seed subsets
- Spirit of dscr HOT 5
- create gold standard methods HOT 5
- problem with way path to output files is stored
- problem with single score? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dscr.