tbg10101/dotnet-burst-comparison

Benchmarks to compare .NET and and Unity's Burst compiler.

dotnet-burst-comparison's Issues

.NET 8 Benchmarks are likely testing "cold"

The readme notes:

Each benchmark is run twice to let each runtime warm up. These results are from the second run. Smaller values are better.

However, modern .NET (.NET Core 2.1+) uses "Tiered Compilation" and that defaults to needing approx. 30 invocations and a delay of 1ms where no other Tier-0 code is jitted before methods can start being promoted to Tier-1

The very latest versions (.NET 7+) do have some support for OSR (On Stack Replacement) in which a hot loop may be replaced even when the rest of the method body hasn't been optimized and I expect that's what's happening for these benchmarks (as many do have loops that execute far more than 30 invocations). However, that is still going to miss some opts and pessimize the overall codegen.

In general, I'd recommend either:

• disabling TieredCompilation (https://learn.microsoft.com/en-us/dotnet/core/runtime-config/compilation)
• ensuring that a minimum of 30-50 invocations occur and a sufficient pause time (1-5ms) exists before measuring to allow rejit to kick in
• just use Benchmark.NET which will handle all of this for you, ensuring enough invocations occur and the relevant statistical analysis is done to find and account for noise, outlier results, etc.

If you go with the first approach (disabling TC), it's worth noting that this also opts you out of things like DynamicPGO and optimizations that can only occur via a TieredJIT where you know some code has executed at least once.