.NET 8 Benchmarks are likely testing "cold"

[tannergooding]

.NET 8 Benchmarks are likely testing "cold"

[tannergooding]

Separately, it's worth noting that there are some pessimizations that exist here which could be accounted for...

A simple example is in SieveOfEratosthenes where instead of:

byte* flags = stackalloc byte[size];

// ....

for (b = 0; b < size; b++) {
    flags[b] = 1; // True
}

the code should probably just do:

Span<byte> flags = stackalloc byte[size];

// ...

flags.Fill(1);

Doing cleanup like this will improve code quality and legibility/maintainability. It's really the same kind of scenarios where you'd use memset or similar built-in functions in C/C++.

[tbg10101]

I'll see about using Benchmark.NET today. (for both projects, for consistency)

For the code changes, could you create a separate issue for that? (if not, that's fine, I can) Sadly I don't have the background for why things were done the way they were in the predecessor project so I mostly left things alone. But I agree that the changes are warranted. I'd like each version to be the best it can be for the framework being tested.

[tbg10101]

The migration for the vanilla .NET project was painless but I didn't know it doesn't work for Unity. So I parked that work on a branch called benchmark.net.

I'll implement my own cheap version as outlined here: https://benchmarkdotnet.org/articles/guides/how-it-works.html

[tbg10101]

I've run into a strange issue with the Burst Sieve Of Eratosthenes after changing it to stackalloc a byte[] and use Fill(). It seems to hang the runtime. Unfortunately this is hard to debug because it works fine if I drop a breakpoint into the job.

For example, this works:

const int size = 1024;

Span<byte> flags = stackalloc byte[size];
flags.Fill(1); // True

return (uint)flags.Length;

But this hangs forever:

const int size = 1024;

Span<byte> flags = stackalloc byte[size];
flags.Fill(1); // True

return flags[index];

It is either the fill or the array lookup causing the issue. (the fill gets optimized out of the first one)

This post from 2021 says Burst should have full Span support but perhaps it is an issue with the backing byte array: https://forum.unity.com/threads/burstcompile-span-t-support.603310/#post-7544536

I'll switch it to use a NativeArray<byte> which can also be Fill()-ed.

[tbg10101]

I just pushed a big change to address your comments. Let me know if there is anything else I should do.

I have re-run the benchmark but some things don't match my expectations. For example IL2CPP without Burst is worse than Mono (also without Burst) in the Fibonacci benchmark.

I need to investigate irregularities but I'm going on vacation tomorrow so I figured I'd push what I had.

[tbg10101]

I think I figured out the issue - I found that a background task would start if I left my computer idle while the benchmarks ran. This wasn't an issue in the previous iteration since they only ran once and were quite fast. In the new version the tests are run multiple times so the overall testing time has expanded, leaving time for tasks that detect idleness to kick in.

I have re-run the benchmarks without letting my computer become idle and will publish the results later today. This issue will be closed after that.

[tbg10101]

ee47b79