DotNext.Net.Cluster crash in production since I think version 5.4.0

[guillaume-chervet]

DotNext.Net.Cluster crash in production since I think version 5.4.0

[sakno]

It's hard to say what's the root cause of the problem because there is no stable repro. I can only guess. Possibly it happens because of network timeouts leading to cancellation of the token used by WAL internally to perform I/O. Some I/O were done in a way not safe for cancellation, I've prepared the potential fix. I can't release it right now.

[sakno]

Have you set UseLegacyBinaryFormat property to true?

[guillaume-chervet]

Yes, It is working when I set it back to the old mode. I always errase the previous data storage when I test.
@sakno

[sakno]

Do you mean that it crashes on empty WAL with a new format?

[guillaume-chervet]

It happen with empty WAL sometime and sometime after an amount of time with the existing WAL @sakno .

[sakno]

Do you have a stable repro? I see that the second stack trace is from the tests in your repository.

[guillaume-chervet]

It is a kind of random behavior.
But when it start to happen it does not stop.

First logs comes from our production.
Second comes from dev environment from one of 3 nodes at startup.
@sakno

[sakno]

It could happen if you trying to open WAL produced by version < 5.4.0 with a new version >=5.4.0 without UseLegacyBinaryFormat set to true. Are you sure that dev environment starts with clean environment without older WAL files?

[guillaume-chervet]

Yes I am sure. My store for testing was completly errased.
Idem for the updated production. @sakno

[guillaume-chervet]

SlimFaas is compiled in AOT.

[sakno]

The second stack trace indicates that WAL is trying to read existing files:

at System.IO.RandomAccess.ValidateInput(SafeFileHandle, Int64, Boolean) + 0x5f
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Table.Initialize() + 0x2a2
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<.ctor>g__CreateTables|28_1(SortedSet`1, DirectoryInfo, Int32, Int32, PersistentState.BufferManager&, Int32, PersistentState.WriteMode, Int64) + 0x14f

There is a code for Initialize:

dotNext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/PersistentState.Partition.cs

Lines 507 to 552 in cacf3e5

	internal override void Initialize()
	{
	using var handle = File.OpenHandle(FileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, FileOptions.SequentialScan);
	// read header
	if (RandomAccess.Read(Handle, header.Span, fileOffset: 0L) < HeaderSize)
	{
	header.Span.Clear();
	}
	else if (IsSealed)
	{
	// partition is completed, read table
	var tableStart = RandomAccess.GetLength(Handle);
	RandomAccess.Read(Handle, footer.Span, tableStart - footer.Length);
	}
	else
	{
	// read sequentially every log entry
	int footerOffset;
	long fileOffset;
	if (PartitionNumber is 0L)
	{
	footerOffset = LogEntryMetadata.Size;
	fileOffset = HeaderSize + LogEntryMetadata.Size;
	}
	else
	{
	footerOffset = 0;
	fileOffset = HeaderSize;
	}
	for (Span<byte> metadataBuffer = this.metadataBuffer.Span, metadataTable = footer.Span; ; footerOffset += LogEntryMetadata.Size)
	{
	var count = RandomAccess.Read(Handle, metadataBuffer, fileOffset);
	if (count < LogEntryMetadata.Size)
	break;
	fileOffset = LogEntryMetadata.GetEndOfLogEntry(metadataBuffer);
	if (fileOffset <= 0L)
	break;
	metadataBuffer.CopyTo(metadataTable.Slice(footerOffset, LogEntryMetadata.Size));
	}
	}
	}

To get an exception like in your stack trace the program needs to go to the second or third if branch. It is possible only if there is a file in the file system.

[guillaume-chervet]

forgot the latest logs @sakno I may made a mistake in our dev kubernetes environment.

Here the logs my collegues sent to me from the crash in production. Occur with the new protocol (in random laps of time near 48 hours and do not happen with the old one). I think it manage near 400 000 writes operation by day.
slimfaas-1-slimfaas.log
slimfaas-2-slimfaas.log
slimfaas-0-slimfaas.log

I do no kown where can come from the negative number.

[sakno]

How WAL is configured? How many records per partition, parallel IO, etc? What's the target architecture, x86_64?

[guillaume-chervet]

Target architecture is x86 64.
The other options I do not know what it is. Here is the SlimData persistent constructor https://github.com/AxaFrance/SlimFaas/blob/2ca3a8c7589b87dcd560164d7ed643f8f17aa89b/src/SlimData/SlimPersistentState.cs#L19

Thank you @sakno for your help

[sakno]

Did you have a chance to check the fix?

[guillaume-chervet]

Hi @sakno do you have a way to publish an alpha?

[guillaume-chervet]

My level in c# is not the best 😜
It my favorite one but i do not code a lot with (unfortunately).

[sakno]

You can reference a project explicitly from your csproj file without published alpha.

[sakno]

Release 5.7.0 has been published.

[guillaume-chervet]

Thank you @sakno I test it today and tell you if it fix the problem