Comments (7)
Hi @KeAWang ,
I know that @benoitmartin88 and @bcharlier worked on it for the Deformetrica software ~6 months ago and got it to work (I think). They may be busy at the moment (as we're working on the R backends with deadlines on Tuesday + Wesnesday), but will probably be able to answer your question.
Best regards,
Jean
from keops.
Hello,
@bcharlier told me about the issue sometimes ago and I forgot to write down the answer I got. I think there is a confusion about the parallelization/asynchronous features at the language level. It is true that pytorch is completely capable to distribute asynchronously workloads across multiple GPU.
On the other hand, the pykeops' kernel computations are launched at the python level and there is no way python could manage this asynchronously across multiple gpu by itself.
The solution is to instantiate tensors directly on multiple gpu and to use the asychronous/multiprocessing python facilities like multiprocess/async.
I'll provide a minimal example.
from keops.
Hi @KeAWang,
@bcharlier and I took a look at your issue.
One easy and quick way to solve this is to use python's multiprocessing module.
Here is a working example:
import torch
from pykeops.torch import Genred
def work(d):
_my_conv = Genred('SqNorm2(x-y)', ['x = Vi(3)', 'y = Vj(3)'])
_x = torch.randn(1000000, 3).to(d) #, device="cuda:" + str(d))
_y = torch.randn(2000000, 3).to(d)
return _my_conv(_x, _y, device_id=d).cpu()
if __name__ == '__main__':
# an intempt of asynchroneous call to keops
torch.multiprocessing.set_start_method("spawn", force=True)
pool = mp.Pool(processes=3)
res = pool.map(work, range(3))
Please note that using python's multiprocessing will cost you the instantiation of spawed python processes.
from keops.
Thank you so much @benoitmartin88 and @fradav! I will try this out.
from keops.
The code you provided works great! However, it seems that PyTorch is unable to maintain the autograd graph across processes.
Sadly, this means that using multiple GPUs with PyKeops through multiprocessing means you won't be able to use the automatic differentiation in PyTorch...
from keops.
Hi @KeAWang,
The multiprocessing package is used in Deformetrica to dispatch the computation load on several GPUs. Deformetrica needs the gradient to perform the estimation process... Well, I am not sure, but I would bet that it is still possible (with some work) to keep the autograd graph alive with the map function.
@benoitmartin88 can you confirm ?
Best,
b.
from keops.
Hi @KeAWang,
As @bcharlier mentioned we do use Pytorch tensors in a multiprocessing context within Deformetrica.
That having been said, we compute the gradients and explicitly return them within each sub-process.
You might have to use a similar technique.
I hope this helps.
from keops.
Related Issues (20)
- Memory leak in Pykeops 2.2 not present in 2.1.2 HOT 18
- Weighted sums reduction HOT 2
- Cannot import pykeops v2.2.1 HOT 2
- sort operation HOT 2
- CUDA out of memory despite trying to run on CPU HOT 2
- Cuda error: cuModuleLoadDataEx(&module, target, 0, NULL, NULL) failed with error CUDA_ERROR_INVALID_IMAGE HOT 6
- sum_reduce along channel dimension HOT 3
- invalid syntax "from pykeops.torch import LazyTensor" HOT 1
- cuCtxSynchronize() failed with error CUDA_ERROR_ILLEGAL_ADDRESS (argKmin) HOT 2
- Max-plus matrix multiplication HOT 6
- Wrong output with batch dimensions on parameters HOT 1
- Support for lazy indexing / lookup? HOT 10
- KeyError: 'nvrtc' HOT 1
- Float vs. int issue with computations? HOT 2
- pip PyKeops==2.2.3 but cannot work well HOT 3
- Feature request: symbolic random matrices HOT 1
- ImportError: dynamic module does not define module export function (PyInit_libKeOpstorchc331696620)
- Eigendecomposition of torch LazyTensor HOT 1
- LazyTensor.ndim not giving correct value HOT 2
- How to solve the problem of OsError? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from keops.