Comments (8)
Sure, I personally don't mind additional dependencies. We can think about having parallel execution behind a feature gate because the only thing I think is important, is that whether something runs in parallel or not remains the choice of the user. If I want to run thousands of optimizations on a cluster or if my cost function implementation is parallelized, I often don't want the evaluations to run in parallel. This doesn't neccessarly need to be handled by a feature gate though, it could also be solved in another way.
At which level would you perform the parallelization? Maybe extending the ArgminOp
trait with the methods apply_par
, gradient_par
and so on makes sense? They could be automatically implemented like this (I'm not sure if this compiles):
fn apply_par(&mut self, params: &[Self::Param]) -> Result<Vec<Self::Output>, Error> {
params.par_iter().map(|p| self.apply(p)?).collect()
}
This would give automatic parallelization, but it would also leave the implementer of the ArgminOp
trait the choice of changing it if necessary.
It could look like this behind a feature gate:
fn apply_par(&mut self, params: &[Self::Param]) -> Result<Vec<Self::Output>, Error> {
#[cfg(feature = "rayon")]
{
params.par_iter().map(|p| self.apply(p)?).collect()
}
#[cfg(not(feature = "rayon"))]
{
params.iter().map(|p| self.apply(p)?).collect()
}
}
Similarly we could have another compute_all_the_things
(name is open for suggestions ;)) method which computes cost, gradient and Hessian in parallel (or a subset of them, depending on what is needed for a particular solver).
Automatic decision of whether parallelization makes sense or not also sounds like a good idea, given that we already measure the time of each iteration. But as a user, I'd also like to be able to opt out of this.
What do you think about this? I have to admit I haven't thought too much about this yet and my experience with rayon is rather limited.
from argmin.
I'm trying to implement Parallel Tempering with Argmin and something like this would make it much easier
from argmin.
I was playing with this myself and it seems really easy to implement thanks to the way collect
handles Result
#[cfg(feature = "rayon")]
fn apply_par(&self, params: &[Self::Param]) -> Result<Vec<Self::Output>, Error> {
params.par_iter().map(|p| self.apply(p)).collect()
}
#[cfg(feature = "rayon")]
fn modify_par(
&self,
params: &[Self::Param],
extents: &[Self::Float],
) -> Result<Vec<Self::Param>, Error> {
params
.par_iter()
.zip_eq(extents)
.map(|(p, e)| self.modify(p, *e))
.collect()
}
And solvers can use a default sequential trait version if the user didn't implement it
from argmin.
Thanks for having a look at this! I like this approach; however, if we extend the ArgminOp
trait with apply_par
and modify_par
then I think the rayon
feature gate should be moved inside the default implementation of the methods:
fn par_apply(&self, params: &[Self::Param]) -> Result<Vec<Self::Output>, Error> {
#[cfg(feature = "rayon")]
params.par_iter().map(|p| self.apply(p)).collect()
#[cfg(not(feature = "rayon"))]
params.iter().map(|p| self.apply(p)).collect()
}
fn par_modify(
&self,
params: &[Self::Param],
extents: &[Self::Float],
) -> Result<Vec<Self::Param>, Error> {
#[cfg(feature = "rayon")]
params
.par_iter()
.zip_eq(extents)
.map(|(p, e)| self.modify(p, *e))
.collect()
#[cfg(not(feature = "rayon"))]
params
.iter()
.zip_eq(extents)
.map(|(p, e)| self.modify(p, *e))
.collect()
}
Then it is not necessary to deal with the feature gate in the solvers. Also, in contrast to my previous suggestion, I would prefer par_{apply,modify}
over {apply,modify}_par
since it is more in line with iter
and par_iter
.
@TheIronBorn what do you think? Would you be willing to provide a PR for this?
from argmin.
I looked into that but it seemed it adds more trait requirements like Send/Sync
. I don't know how reasonable that is for sequential projects. Maybe there's some trait magic we could do.
Also it might be good to let the user choose which to parallelize since it might be only the cost function or the jacobian/etc which is expensive and so parallelizing everything might slow things down.
If the final version doesn't end up parallel by default perhaps we should reduce the name to {apply/etc}_many
or bulk_{apply/etc}
, though that might make its use-case less obvious.
from argmin.
Those are all very good points. How do you feel about implementing the sequential (=non parallelized) version by default, and leaving the parallel implementation to the user if needed (for now)? Then users can choose how they want to parallelize their code. In that case, bulk_{...}
is probably the better choice for the method names.
I feel this would be a rather temporary solution until we come up with something more convenient, but it would at least allow users to provide a parallelized version if they want to.
from argmin.
Yeah I'll do that
from argmin.
Implemented in #219 .
The rayon
feature enables parallel evaluation of cost function, gradient, operator, Jacobian and Hessian via the bulk_*
methods. One can still opt out of parallelization for individual problem related traits. Say you don't want to compute the Hessian in parallel: by implementing the parallelize
method of the Hessian
trait to return false
, it will be computed sequentially.
The bulk_*
methods can of course also be overwritten.
The only solver which uses this so far is Particle Swarm Optimization.
I'm happy to hear feedback on the design, in particular if anyone is using this for their own solvers.
from argmin.
Related Issues (20)
- How best to share computation between cost and gradient computations in a solver-agnostic way? HOT 2
- LBFGS example, trait bounds HOT 2
- Development on Windows HOT 2
- How to inject custom termination criteria to existing solvers? HOT 2
- Inconvenience when using Executor inside a function with generics HOT 3
- Missing proper documentation on how to run examples
- Create inplace variants of math traits
- ParticleSwarm is not reproducible HOT 3
- More-Thuente Linesearch Algorithm Bug: "Search direction must be a descent direction" HOT 7
- BrentRoot is missing tests
- Add `faer-rs` as math backend
- Add stopping criteria based on the number of function evaluations
- Add LaTeX support to docs
- Add derivatives and Hessians to test functions in argmin_testfunctions
- Add more test functions to `argmin_testfunctions`
- Create visualization of all test functions in `argmin_testfunctions`
- `owl-qn` example does not converge HOT 4
- Observers and checkpointing are missing tests
- Enable observers to create final report
- Thoughts on future state handling
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from argmin.