Question about performance using "generic"

[ManuelCostanzo]

Question about performance using "generic"

[illuhad]

There is nothing in generic that causes perf to be inherently lower. Indeed, there are also cases where it performs notably better:

It is however simply a far newer compiler compared to the other compilation flows. I'm aware of multiple optimizations that could still be implemented on the SSCP side, and which I didn't yet get to implement. We also have plans to implement optimizations that are impossible in the other flows. So the expectation is that in the future performance should improve further, and eventually surpass the other compilation flow substantially on average. If you see a perf delta, it is likely not because of just-in-time compilation, it not knowing the target at compile time or the design of the compiler, but simply due to fewer development hours have so far gone into optimizing it. Optimization for generic targets primarily happens at runtime, when the target device is just as well known as when specifying cuda:sm_86 - perhaps it is even better known then, because we can also take runtime information into account.

If you observe very large performance differences (say >20%) I'd be interested in the code as performance differences are not expected to be that large.

If you believe that there is performance hit simply because to the additional latency due to the time it takes to JIT-compile, my advice is to design the application such that the first kernel invocation in a translation unit (which will trigger JIT) is outside the performance-critical path. This is in general sound application design, since every SYCL implementation targeting SPIR-V or PTX already inherently does just-in-time compilation, since GPU drivers JIT-compile those formats. So the same behavior can also be obverved with other SYCL compilers that output some form form of IR (like PTX or SPIR-V).