scaling well in mlr 3

[mlr-org]

scaling well in mlr 3

[berndbischl]

hi!
would you maybe add a few lines about how you would envision that exactly? maybe with a concrete use case / example?

[berndbischl]

or just some extra context

[AtharKharal]

hi,
pbdR has packages like MPI, ZeroMQ, ScaLAPACK, NetCDF4 and PAPI which are highly scalable. The parallel and distributed computing capability of mlr may be enhanced by these packages, specially ScaLAPACK. Somthing from Socket to MPI.

[mllg]

I don't think that these packages and mlr3 team up well.

For parallelization, we rely on the future package which supports (besides many others) MPI/socket clusters.

There is a dbplyr backend to connect to out-of-memory data, e.g. SQL databases, Spark or bigquery. See https://github.com/mlr-org/mlr3db.

Finally, we are not doing any matrix operations ourselves, so we have no use for ScaLAPACK. We call learning algorithms from third party packages, and these are usually linked against the system BLAS/LAPACK.

[mllg]

If there are any interesting learning algorithms provided by the packages, these can of course be connected as learners.