Heterogenous stencil formulations

[omlins]

Heterogenous stencil formulations

[omlins]

Is it possible to apply other, more complicated kernels here?

No, only kernels that act on the boundaries and do not depend on the inner point computations of the primary compute statement (update_T here). These are requirements to guarantee correct results. If we allowed for a second full computation kernel, then we would get wrong results if, for example, its boundary point computations depended on the inner computations of the first kernel. The thing is that the order of execution within the hide communication is not the expected if you had multiple full kernels.

That said, the rules could be weakened a bit in certain cases. However, it is key that the rules are very easily understandable as else users might get wrong results without even notifying...

Now in your case, you should be able to do all the you describe (using the @parallel_ async construct), but outside of a hide communication statement. This might be just fine if you can still apply hide communication on one of your heavier kernels.

[smartalecH]

Thanks for the quick reply, @omlins

These are requirements to guarantee correct results. If we allowed for a second full computation kernel, then we would get wrong results if, for example, its boundary point computations depended on the inner computations of the first kernel.

What if we were to (1) assume all kernel domains are disjoint (such that only boundaries of kernels talk) and (2) explicitly apply a halo around each kernel domain, even if that kernel was on the same device? This way we don't have to worry about "clobbering" each other's boundary points.

If we did this, we could place all "kernels" within the same halo update block, right?

The @parallel_async option looks promising. I'm just looking if we can leverage the best of both worlds (as asynchronous halo updates seem to be what helps ensure consistent weak scaling).

[omlins]

What if we were to (1) assume all kernel domains are disjoint (such that only boundaries of kernels talk) and (2) explicitly apply a halo around each kernel domain, even if that kernel was on the same device? This way we don't have to worry about "clobbering" each other's boundary points.
If we did this, we could place all "kernels" within the same halo update block, right?

Sorry, I'm not sure I get what you mean.

The @parallel_async option looks promising. I'm just looking if we can leverage the best of both worlds (as asynchronous halo updates seem to be what helps ensure consistent weak scaling).

If you split your domain yourself in multiple kernels, then you might also be able to split off the boundary regions yourself explicitly; if so, you can yourself overlap the communication with computation using @parallel_async to launch the kernels.

[smartalecH]

If you split your domain yourself in multiple kernels, then you might also be able to split off the boundary regions yourself explicitly; if so, you can yourself overlap the communication with computation using @parallel_async to launch the kernels.

Got it, thanks 🙂