dotnet/dotnext

Next generation API for .NET

.NEXT

.NEXT (dotNext) is a set of powerful libraries aimed to improve development productivity and extend .NET API with unique features. Some of these features are planned in future releases of .NET platform but already implemented in the library:

Proposal	Implementation
Interop between function pointer and delegate	DelegateHelpers factory methods
Check if an instance of T is default(T)	IsDefault() method
Expression Trees covering additional language constructs, e.g. foreach, await, patterns, multi-line lambda expressions	Metaprogramming
Async Locks	Documentation
High-performance general purpose Write-Ahead Log	Persistent Log
Memory-mapped file as Memory<byte>	MemoryMappedFileExtensions
Memory-mapped file as ReadOnlySequence<byte>	ReadOnlySequenceAccessor
A dictionary where the keys are represented by generic arguments	Documentation
Process asynchronous tasks as they complete	Documentation
Soft References	Documentation

Quick overview of additional features:

• Attachment of user data to an arbitrary objects
• Extended set of atomic operations
• Fast conversion of bytes to hexadecimal representation and vice versa with Hex class
• ManualResetEvent, ReaderWriterLockSlim and other synchronization primitives now have their asynchronous versions
• Atomic memory access operations for arbitrary value types
• PipeExtensions provides high-level I/O operations for pipelines such as string encoding and decoding
• A rich set of high-performance memory buffers for efficient I/O
• String formatting, encoding and decoding with low GC pressure: dynamic char buffers
• Fully-featured Raft implementation
• Fully-featured HyParView implementation

All these things are implemented in 100% managed code on top of existing .NET API.

Quick Links

• Features
• API documentation
• Benchmarks
• NuGet Packages

What's new

Release Date: 04-20-2024

DotNext.IO 5.4.0

• Added FileWriter.WrittenBuffer property

DotNext.Net.Cluster 5.4.0

• Changed binary file format for WAL for more efficient I/O. A new format is incompatible with all previous versions. To enable legacy format, set PersistentState.Options.UseLegacyBinaryFormat property to true
• Introduced a new experimental binary format for WAL based on sparse files. Can be enabled with PersistentState.Options.MaxLogEntrySize property

DotNext.AspNetCore.Cluster 5.4.0

• Updated dependencies

Changelog for previous versions located here.

Release & Support Policy

The libraries are versioned according with Semantic Versioning 2.0.

Version	.NET compatibility	Support Level
0.x	.NET Standard 2.0	❌
1.x	.NET Standard 2.0	❌
2.x	.NET Standard 2.1	❌
3.x	.NET Standard 2.1, .NET 5	❌
4.x	.NET 6	✅
5.x	.NET 8	✔️

❌ - unsupported, ✅ - bug and security fixes only, ✔️ - active development

Development Process

Philosophy of development process:

1. All libraries in .NEXT family are available for various .NET form factors: Mono, WASM, NativeAOT
2. Minimal set of dependencies
3. Provide high-quality documentation
4. Stay cross-platform
5. Provide benchmarks

Users

.NEXT is used by several companies in their projects:

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the .NET Foundation Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

dotnext's Issues

Possible problem with redirect logic

If my cluster is configured to have the following members:
localhost:1000, localhost:1001, localhost:1002

The following code will not redirect to the leader properly since the port that is used for the redirect is not the leader port:

internal Task Redirect(HttpContext context)
{
    if (context.Request.Path.StartsWithSegments(pathMatch, StringComparison.OrdinalIgnoreCase))
    {
        var leader = cluster.Leader;
        if (leader is null)
        {
            context.Response.StatusCode = StatusCodes.Status503ServiceUnavailable;
            return Task.CompletedTask;
        }
        else if (leader.IsRemote)
        {
            return redirection(context.Response, new UriBuilder(context.Request.GetEncodedUrl()) { Host = leader.Endpoint.Address.ToString(), Port = applicationPortHint ?? context.Connection.LocalPort }.Uri);
        }
    }

    return next(context);
}

should this not use something like the following?

new UriBuilder(context.Request.GetEncodedUrl()) { Host = leader.Endpoint.Address.ToString(), Port = leader.Endpoint.Port }.Uri)

Raft - Usage and Production Readiness

Hello,
Thank you for sharing your work with the community.
The documentation of .NEXT is very detailed - going through it right now.

I'd be lying if I said that I understand all the complexities involved in what you've built.
That being said, I'm currently evaluating options to fulfill the following use case:

• Several instances of the same webhost (might vary at runtime due to autoscaling)
• Each instance has a BackgroundService that should perform some work
• Only one of said instances should be processing a backlog of tasks at a time.

So I thought that some kind of leader election would get me there.
I've used distributed locks (via a DB) in the past, but I thought I could do better.
Would you say I'm on the right path with .NEXT Raft?

The other question is: are you currently using .NEXT Raft in production or know of case histories I might share with the higher-ups, so I can promote the usage of this library?

Thank you again for your contribution to the community and for your time.

Best,
Rob

Inaccurate sample code in documentation for CodeGenerator.ForEach

This is a minor documentation issue.

According to the docs, CodeGenerator.ForEach can be used as follows:

Lambda<Action<string>>(fun => 
{
    ForEach(fun[0], ch =>
    {
        WriteLine(ch);
    });
});

But this code throws System.ArgumentException because CodeGenerator.WriteLine does not work for System.Char. In fact it appears it doesn't work for any primitive value type. Perhaps I can just update the docs from string to List<string> instead?

RaftCluster - AllowPartitioning option

I'm trying to set up a system where things function more or less normally no matter how many members are online and no matter how many members are part of the cluster. Looking at the documentation I thought AllowPartitioning would provide that functionality, but it wasn't working and looking at the test cases (SingleNodeWithoutConsensus) it appears it isn't meant to work the way I want... but I'm a little bit confused about why the current behavior is desired.

As best as I can tell, the only real reference to the option is when in the leader state, you won't bump out of leader state upon discovering that a quorum is not present. The state machine cannot make commits, and non-leaders will not become leaders.

I understand that strictly speaking Raft doesn't allow commits without consensus... so maybe what I want isn't really Raft, but it feels like what you've got is so close to being what I need. What do you think?

Documentation missing

The documentation guide is missing at this link: https://sakno.github.io/dotNext/migration/2.html

Optimization

For best performance, we use TcpConfiguration and create a simple k/v Store, that uses a simple dictionary. And not a configuration snapshot in PersistentState. we found that the optimistic way to create log entries is using fixed struct size like example. but I get 136ms avg latency that is very high for me.
SqlServer instance give me 40ms avg latency for the same job! and we need 1-3ms avg latency.
I wanna know we could reach 3ms with this library or should use other libraries?

unexpected election timeouts when rejecting vote requests in raft

When handling vote requests in the following method, the election timeout when rejecting vote requests (when line 544 is false):

dotNext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs

Lines 516 to 552 in 6b01f63

	protected async Task<Result<bool>> VoteAsync(ClusterMemberId sender, long senderTerm, long lastLogIndex, long lastLogTerm, CancellationToken token)
	{
	var currentTerm = auditTrail.Term;
	if (currentTerm > senderTerm)
	return new(currentTerm, false);
	using var tokenSource = token.LinkTo(LifecycleToken);
	using var transitionLock = await transitionSync.AcquireAsync(token).ConfigureAwait(false);
	var result = false;
	if (currentTerm != senderTerm)
	{
	Leader = null;
	await StepDown(senderTerm).ConfigureAwait(false);
	}
	else if (state is FollowerState follower)
	{
	follower.Refresh();
	}
	else if (state is StandbyState)
	{
	Metrics?.ReportHeartbeat();
	}
	else
	{
	goto exit;
	}
	if (auditTrail.IsVotedFor(sender) && await auditTrail.IsUpToDateAsync(lastLogIndex, lastLogTerm, token).ConfigureAwait(false))
	{
	await auditTrail.UpdateVotedForAsync(sender).ConfigureAwait(false);
	result = true;
	}
	exit:
	return new(currentTerm, result);
	}

Note that this happens both explicitely in line 533, and indirectly in line 529. It seems to be problematic in both cases, although in the later when it is a leader it does need to start a new election timeout (it is supposed to change though, instead of remaining the same which seems to be the case following the implementation of StepDown).

Some of this is not explicit at all in the raft papers, not even the phd dissertation. However, one can simulate various scenarios in the raft visualization by sending requests/stopping/resuming/timing out nodes and seeing how exactly it deals with those cases https://raft.github.io/

More info on some of the scenarios related to the votes being rejected with the checks in line 529:

• (low impact) we already voted for a node and a second node timed out (goes to line 533). The node first saw the election when it first voted, so it resets its election timeout accordingly (so it can propose itself as a leader if the candidate fails to be established as a leader). By resetting it on the second vote, we are extending our election timeout a second time potentially beyond a single election time window. It reduces our chances of becoming a candidate if the election fails.
• (high impact) we have committed entries the vote requester does not have. Without the reset it is now more likely to time out first before the sender (which will reset its timeout after losing the election) + it is also more likely to time out before other nodes that don't have the committed entries (as the accepted vote would reset their election timeout). By resetting here we are removing the advante of nodes with committed entries, so more rounds are needed until by chance one of the nodes with the committed entries becomes the leader.

Consider how the later scenario can play when there are only 2 out of 3 nodes remaining. One of them has an entry the other one does not. If one is very unlucky with timings then the node without the extra entries keeps becoming a candidate first on many rounds. This alone does not match the many rounds I have seen it take, but this might combine with other implementation details in dotnext raft, as maybe an unlucky election timeout might keep getting reused.

Problem using Mvc controller in embedded http cluster

Hello, I have setup my cluster in embedded http mode (using DotNext.Net.Cluster.Consensus.Raft.Http.Embedding;) and am able to achieve consensus of all the nodes. However, when I try to setup my own controller to respond to requests on the cluster, All calls to it come back as "The requested URL can't be reached". There is no response code.

Here is my cluster configuration:

{
  "Kestrel": {
    "EndpointDefaults": {
      "Protocols": "Http2"
    }
  },
  "partitioning": false,
  "lowerElectionTimeout": 500,
  "upperElectionTimeout": 1000,
  "members": [ "https://localhost:3262", "https://localhost:3263", "https://localhost:3264" ],
  "protocolVersion": "Http2",
  "metadata": {
    "key": "value"
  },
  "logLocation": "c:\\locks",
  "recordsPerPartition": 10000,
  "allowedNetworks": [ "::1", "127.0.0.0", "255.255.0.0/16", "2001:0db9::1/64" ],
  "hostAddressHint": "192.168.0.1",
  "requestJournal": {
    "memoryLimit": 5,
    "expiration": "00:00:10",
    "pollingInterval": "00:01:00"
  },
  "resourcePath": "/cluster-consensus/raft",
  "port": 3262,
  "heartbeatThreshold": 0.5,
  "requestTimeout": "00:01:00",
  "keepAliveTimeout": "00:02:00",
  "requestHeadersTimeout": "00:00:30"
}

Here is my project startup:

public static void Main(string[] args)
{
    CreateHostBuilder(args).Build().Run();
}

public static IHostBuilder CreateHostBuilder(string[] args) =>
       Host.CreateDefaultBuilder(args)
           .ConfigureWebHostDefaults(webBuilder => {
               webBuilder.UseStartup<Startup>();
            })
           .JoinCluster();

In the app configuration I call app.UseConsensusProtocolHandler()

Finally, here is what my controller looks like:

using DotNext.Net.Cluster.Consensus.Raft;
using DotNext.Net.Cluster.Replication;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;
using System;

namespace CRB.DistributedServices.LockServer
{
    [Route("[controller]")]
    public class MyController : ControllerBase
    {
        private readonly IPersistentState _persistentState;
        private readonly IReplicationCluster<LockMessageLogEntry> _replicationCluster;
        private readonly ILogger<MyController> _logger;

        public MyController(IPersistentState persistentState, IReplicationCluster<LockMessageLogEntry> replicationCluster, ILogger<MyController> logger)
        {
            _persistentState = persistentState ?? throw new ArgumentNullException(nameof(persistentState));
            _replicationCluster = replicationCluster ?? throw new ArgumentNullException(nameof(replicationCluster));
            _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        }


        [HttpGet]
        public ActionResult Get()
        {
            _logger.LogInformation("Got");
            return Ok();
        }
    }
}

Do you have any idea if the cluster might be blocking my controller?

Metaprogramming: `NullReferenceException` when exception is thrown in nested `AwaitForEach`

When an exception is thrown in CodeGenerator.AwaitForEach which contains another CodeGenerator.AwaitForEach (nested), you'll receive an NullReferenceException instead of the thrown exception.

Note: with a single CodeGenerator.AwaitForEach the exception that was being thrown in the body is being thrown, as expected.

Information

.NET 5
DotNext.Metaprogramming 2.12.1

Fiddle

Nested: https://dotnetfiddle.net/yfswYy (invalid, throws NullReferenceException)
Single: https://dotnetfiddle.net/DPnQrZ (valid, throws user exception)

Example

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using static DotNext.Metaprogramming.CodeGenerator;
					
public class Program
{
	public static async Task Main()
	{
		var result = AsyncLambda<Func<IAsyncEnumerable<char>, IAsyncEnumerable<char>, Task>>(fun => 
		{
			AwaitForEach(fun[0], ch =>
			{
				WriteLine(ch);
				AwaitForEach(fun[1], ch => WriteLine(ch));
			});
		}).Compile();
		
		await result(GetData('1'), GetData('2'));
	}
	
	public static async IAsyncEnumerable<char> GetData(char data)
	{
		await Task.Yield();
		yield return data;
		throw new Exception("Expected exception");
	}
}

Expected

1
2
Unhandled exception. System.Exception: Expected exception

Actual

1
2
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at lambda_method2(Closure , AsyncStateMachine`1& )
   at DotNext.Runtime.CompilerServices.AsyncStateMachine`1.System.Runtime.CompilerServices.IAsyncStateMachine.MoveNext() in /_/src/DotNext.Metaprogramming/Runtime/CompilerServices/AsyncStateMachine.cs:line 124
--- End of stack trace from previous location ---
   at Program.Main()
   at Program.<Main>()

Network.ToEndPoint method does not seem to support a dns based endpoint

Hello,

I am trying to change my cluster to run in containers and am having a problem with configuring the members. Since docker assigns them their own IPs and makes the nodes available through a dns system (eg https://mydockerraftnodeservice), I have configured the members collection to use the service name. However, when I attempt to bring up the cluster, the constructor of the RaftClusterMember class tries to convert the member to an endpoint and fails since it is not a loopback endpoint:

internal static IPEndPoint? ToEndPoint(this Uri memberUri)
{
    switch (memberUri.HostNameType)
    {
        case UriHostNameType.IPv4:
        case UriHostNameType.IPv6:
            return new IPEndPoint(IPAddress.Parse(memberUri.Host), memberUri.Port);
        case UriHostNameType.Dns:
            return memberUri.IsLoopback ? new IPEndPoint(IPAddress.Loopback, memberUri.Port) : null;
        default:
            return null;
    }
}

This is a problem since I do not know the IPs of the containers ahead of time and since the standard way to refer to the containers is through dns.

Do you have any ideas?

IClusterMember::GetMetadataAsync throwing exceptions when used with IRaftHttpCluster

Hi,

I am currently trying to use the Raft feature, and got members to join a cluster. However when I try to get their metadata via GetMetadataAsync the receiving instance (who is also the leader) just throws a middleware exception about not being able to properly handle the json data that is used internally to communicate.

I have tried setting no meta data, I have tried setting a simple "hello" -> "world" pair, nothing seems to make a difference.

Can someone please check if this function is currently working for them? I am using the latest stable release 4.1.3

Thank you!

This is all the information given to me. Visual studio is unable to catch the error, there is no more output besides this:

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.InvalidCastException: Unable to cast object of type 'System.Text.Json.Serialization.Converters.ObjectConverter' to type 'System.Text.Json.Serialization.JsonConverter`1[System.String]'.
         at System.Text.Json.Serialization.JsonDictionaryConverter`3.GetConverter[T](JsonTypeInfo typeInfo)
         at System.Text.Json.Serialization.Converters.DictionaryDefaultConverter`3.OnWriteResume(Utf8JsonWriter writer, TDictionary value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonDictionaryConverter`3.OnTryWrite(Utf8JsonWriter writer, TDictionary dictionary, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Converters.JsonMetadataServicesConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.JsonSerializer.WriteCore[TValue](JsonConverter jsonConverter, Utf8JsonWriter writer, TValue& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.JsonSerializer.WriteStreamAsync[TValue](Stream utf8Json, TValue value, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
         at System.Text.Json.JsonSerializer.WriteStreamAsync[TValue](Stream utf8Json, TValue value, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
         at System.Text.Json.JsonSerializer.WriteStreamAsync[TValue](Stream utf8Json, TValue value, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

IMemberDiscoveryService

Hi,

Do you have a example to use IMemberDiscoveryService? I would like to add members dynamically in the cluster...

Best regards,

Luis

Microsoft.CodeAnalysis.FxCopAnalyzers Is both outdated and deprecated

Microsoft.CodeAnalysis.FxCopAnalyzers used throughout NEXT is 3.3.1, 3.3.2 is the latest version and it is marked as deprecated. Consider removing/replacing this library.

Some nodes stops participating in leader elections with tcp transport

Hi,
I try to start example/RaftNode and saw what some nodes can stop participating in leader elections.
What can lead to this behavior?

.NET Core SDK (reflecting any global.json):
Version: 3.1.401
Commit: 5b6f5e5005

Runtime Environment:
OS Name: Windows
OS Version: 10.0.17763
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\3.1.401\

Host (useful for support):
Version: 3.1.7
Commit: fcfdef8d6b

The host also has the Hyper-V role installed, if relevant.

node1:
cmd: RaftNode.exe tcp 3262

New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 911. Election timeout 00:00:00.2750000
Consensus cannot be reached
Term of local cluster member is 911. Election timeout 00:00:00.2750000
New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 913. Election timeout 00:00:00.2330000
Consensus cannot be reached
Term of local cluster member is 913. Election timeout 00:00:00.2330000
New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 919. Election timeout 00:00:00.2880000

node2:
cmd: RaftNode.exe tcp 3263

New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 911. Election timeout 00:00:00.2720000
Consensus cannot be reached
Term of local cluster member is 911. Election timeout 00:00:00.2720000
New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 913. Election timeout 00:00:00.2720000
Consensus cannot be reached
Term of local cluster member is 913. Election timeout 00:00:00.2720000
New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 919. Election timeout 00:00:00.1820000

node3: Problem node
cmd: RaftNode.exe tcp 3264

Consensus cannot be reached
Term of local cluster member is 465. Election timeout 00:00:00.1930000
New cluster leader is elected. Leader address is 127.0.0.1:3263
Term of local cluster member is 469. Election timeout 00:00:00.2030000
Consensus cannot be reached
Term of local cluster member is 469. Election timeout 00:00:00.2030000

In fact, I have seen the same behavior when trying to use RaftCluster with TCP transport in our project and cannot find the problem. So I tried to reproduce the problem with the example project.

Another problem is that after a while, the worker node stops with an assertion error:

Process terminated. Assertion failed.
   at DotNext.Net.Cluster.Consensus.Raft.TransportServices.ClientExchange.ProcessInboundMessageAsync(PacketHeaders headers, ReadOnlyMemory`1 payload, EndPoint sender, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\TransportServices\ClientExchange.cs:line 70
   at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpClient.ClientNetworkStream.Exchange(IExchange exchange, Memory`1 buffer, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\Tcp\TcpClient.cs:line 59
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpClient.ClientNetworkStream.Exchange(IExchange exchange, Memory`1 buffer, CancellationToken token)
   at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpClient.Enqueue(IExchange exchange, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\Tcp\TcpClient.cs:line 139
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpClient.Enqueue(IExchange exchange, CancellationToken token)
   at DotNext.Net.Cluster.Consensus.Raft.TransportServices.ExchangePeer.SendAsync[TResult,TExchange](TExchange exchange, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\TransportServices\ExchangePeer.cs:line 48
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at DotNext.Net.Cluster.Consensus.Raft.TransportServices.ExchangePeer.SendAsync[TResult,TExchange](TExchange exchange, CancellationToken token)
   at DotNext.Net.Cluster.Consensus.Raft.TransportServices.ExchangePeer.VoteAsync(Int64 term, Int64 lastLogIndex, Int64 lastLogTerm, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\TransportServices\ExchangePeer.cs:line 68
   at DotNext.Net.Cluster.Consensus.Raft.RaftClusterMember.DotNext.Net.Cluster.Consensus.Raft.IRaftClusterMember.VoteAsync(Int64 term, Int64 lastLogIndex, Int64 lastLogTerm, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\RaftClusterMember.cs:line 78
   at DotNext.Net.Cluster.Consensus.Raft.CandidateState.VotingState.VoteAsync(IRaftClusterMember voter, Int64 term, IAuditTrail`1 auditTrail, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\CandidateState.cs:line 35
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at DotNext.Net.Cluster.Consensus.Raft.CandidateState.VotingState.VoteAsync(IRaftClusterMember voter, Int64 term, IAuditTrail`1 auditTrail, CancellationToken token)
   at DotNext.Net.Cluster.Consensus.Raft.CandidateState.VotingState..ctor(IRaftClusterMember voter, Int64 term, IAuditTrail`1 auditTrail, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\CandidateState.cs:line 55
   at DotNext.Net.Cluster.Consensus.Raft.CandidateState.StartVoting(Int32 timeout, IAuditTrail`1 auditTrail) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\CandidateState.cs:line 131
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState() in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\RaftCluster.cs:line 661
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState()
   at DotNext.Net.Cluster.Consensus.Raft.FollowerState.Track(TimeSpan timeout, IAsyncEvent refreshEvent, Action candidateState, CancellationToken[] tokens) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\FollowerState.cs:line 34
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at DotNext.Threading.QueuedSynchronizer.WaitAsync(WaitNode node, TimeSpan timeout, CancellationToken token) in g:\work\VSNET\github\dotNext\src\DotNext.Threading\Threading\QueuedSynchronizer.cs:line 121
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Threading.Tasks.TaskFactory.CompleteOnInvokePromise.Invoke(Task completingTask)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.TrySetResult()
   at System.Threading.Tasks.Task.DelayPromise.CompleteTimedOut()
   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()

Update to .NET5

Are there any plans to update everything to .NET5?

Exceptions thrown in async lambda seem to be supressed

I had to do quite a bit of work to get my code running, but so far it does generate perfectly well working asynchronous methods for my automatic serialization of entities. But while there is no compilation error and it does execute like expected, I realized that exceptions that are thrown inside the generated delegate don't propagate. I catch an exception in the async method I call via the AsyncLambda and rethrow it so it bubbles up my pipeline. But it never reaches the code calling the "AsyncLambda". It seems like the exception just get lost and execution stops. This is a serious issue as I can't recover from this state and don't get feedback when any exception occurs.

If this is a known problem, I'd like to hear if there's any progress to fixing that. If you need an example or more thorough explanation I'd give that preferrably in private.

How to use raft to build a kv system？

Thank you very much for providing a raft C # implementation. I now want to build a distributed kv system. Can I use your framework to achieve this?

Cluster unstable when running tcp example

Discussed in #83

^{Originally posted by AntonWerenberg November 8, 2021}
I've been experimenting with the raft node example code for some time, and one issue keeps being present: Instability in the cluster.

The nodes will often run fine in the beginning, but sooner or later they will start to go out of sync and fail. One node will show the tcp error warning:
"warn: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74022]
Request has timed out"
and others will start running elections, but will not reach consensus.

the behaviour can be seen in the attached image.

I'm running with default election timeout settings from the example.

I'm running with default election timeout settings from the example.

I'm not a great programmer and I'm really having trouble seeing which direction to go to get to the bottom of this?

My main concern is currently if this could be due to my own setup, something not configured correctly, or something like that.

I measured Broadcasttime using Metrics collector. It is showing broadcast times of around 3 ms.

Cluster: Always getting exception: Configured list of members in cluster doesn't contain address of entire application

I am working on an implementation of Raft using this library. I have 2 implementations of my own that give me the exception: 'Configured list of members in cluster doesn't contain address of entire application'.

Thinking about it, I thought I probably made a mistake on my implementation and tried to run the RaftNode example that is part of the repository. However it still throws the same exception in the same place.

I hope anyone can give me a hand with this, I am stuck not knowing what I am missing and/or doing wrong.

Thanks

Hosted host not disposed

Hi @sakno,

Starting with version 3, the hosted host is not being disposed. I'm including screenshot that will explain everything:

AsyncForEach

Currently it's possible to create a foreach-statement with CodeGenerator.ForEach, but it's not possible to create an async foreach-statement.

Suggestion

Create a new method in CodeGenerator named AsyncForEach that does the same as ForEach but for IAsyncEnumerable.

Example

C#

IAsyncEnumerable<string> elements;

async foreach (var element in elements)
{
	// use element
}

DotNext.Linq.Expressions

Expression elements;

CodeGenerator.AsyncForEach(elements, element =>
{
	// use element
});

running RaftNode example in tcp mode. All nodes become leader

Discussed in #80

^{Originally posted by AntonWerenberg October 28, 2021}
When attempting to run the unmodified RaftNode example for tcp nodes I encounter problems.
all running are in the leader state and appear not to interact with each other.

Upon closer inspection, I have found that the memberlist of each node only contains itself. I suspect that the problem may be because the node is configured with coldstart = true. Could this be the case?

Upon reading the documentation I am in doubt whether only a single node should be configured with coldstart = true.

When running the example with http configuration everything runs flawlessly.

commands run in seperate teminals when starting example in tcp mode
~/dotNext/src/examples/RaftNode$ dotnet run --framework net6.0 tcp 3262 node1
~/dotNext/src/examples/RaftNode$ dotnet run --framework net6.0 tcp 3263 node2
~/dotNext/src/examples/RaftNode$ dotnet run --framework net6.0 tcp 3264 node3

Untyped async lambda expression?

Hi, is there support here for an untyped async lambda expression (akin to LambdaExpression in the standard library)? I see there's AsyncLambdaExpression<TDelegate> but I don't know the input/return types at the time of constructing the expression in my use case. Thanks!

AsyncLambda throws System.ArgumentException when invoked within another AsyncLambda with Assign expressions

I encountered this issue when trying to build some fairly complicated async expressions. A minimal reproducible example is here:

class Program
{
    private static readonly PropertyInfo _propertyInfo = typeof(TestClass).GetProperties().First();

    static async Task Main(string[] args)
    {
        var innerExp = GetTestExpression(false);

        var outerExp = CodeGenerator.AsyncLambda<Func<TestClass, Task<TestClass>>>(context =>
        {
            var output = innerExp.Invoke(context[0]).Await();
            CodeGenerator.Assign(output, _propertyInfo, Expression.Constant("updated", typeof(string)));
            CodeGenerator.Return(output);
        });

        var dlg = outerExp.Compile();
        var result = await dlg(new TestClass("original"));
    }

    private static Expression<Func<TestClass, Task<TestClass>>> GetTestExpression(bool useCompilerGeneratedExpression)
    {
        if (useCompilerGeneratedExpression)
        {
            return v => Task.FromResult(v);
        }

        return CodeGenerator.AsyncLambda<Func<TestClass, Task<TestClass>>>(context =>
        {
            CodeGenerator.Return(context[0]);
        });
    }

    public class TestClass
    {
        public TestClass(string testString)
        {
            TestString = testString;
        }

        public string TestString { get; set; }
    }
}

This code crashes with a System.ArgumentException on .NET Core 3.1. The exception message is as follows:

Expression of type 'System.Void' cannot be used for return type 'System.Threading.Tasks.Task`1[MetaprogrammingBug.Program+TestClass]

If I remove the CodeGenerator.Assign call in the outer expression, then the code runs with no problem. Also, if I set useCompilerGeneratedExpression to true, it runs with no problem as well. I believe the compiler-generated expression should be semantically identical to the one generated with CodeGenerator.AsyncLambda, so the latter fact most certainty indicates that there is a bug somewhere.

I have not read the source code for CodeGenerator.AsyncLambda so I don't have the slightest idea what could have gone wrong... Would greatly appreciate if you can help with this issue!

Async suffix for Wait method

This is more like a question.
Shouldn't the Wait method of the Synchronizer class have the Async suffix? I learnt somewhere that every async method or every method returning a task should have that suffix. Are there logical exceptions?

Transfer to .NET Foundation

This project is transferring to .NET Foundation. Location of this repository will be changed. Due to transfer, we need to do the following tasks:

• Transfer to dnfadmin
• Change address of GH pages and notify users about new address
• Fix all possible links and references to a new location
• Change Azure Pipelines configuration
• Reconfigure Gitter integration
• Reconfigure LGTM integration
• Fix badges
• Add Code of Conduct
• Fix GH pages
• Update copyrights
• Fix link to repo on raft.github.io website
• Obtain code signing cert
• Publish information about the project on .NET Foundation website

Cluster node hosted via embedded can't find address of the application

Hi,

I ran into some issue while configuring Kestrel through the application config file using the server.urls parameter.
If the StartUp class takes longer than RaftHttCluster to start, it cannot find the server address and the application stops with the message:

g:\work\VSNET\github\dotNext\src\examples\RaftNode\bin\Debug\netcoreapp3.1>RaftNode.exe http 3262
Unhandled exception. DotNext.Net.Cluster.Consensus.Raft.RaftProtocolException: Configured list of members in cluster doesn't contain address of entire application
   at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.StartAsync(CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.AspNetCore.Cluster\Net\Cluster\Consensus\Raft\Http\RaftHttpCluster.cs:line 138
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at RaftNode.Program.Main(String[] args) in g:\work\VSNET\github\dotNext\src\examples\RaftNode\Program.cs:line 143
   at RaftNode.Program.<Main>(String[] args)

I think this is related to the issue aspnet/Hosting#1390 which says that server.Features.Get<IServerAddresses>() gets populated after StartUp is done.

It's easy to reproduce:

1. add the parameter to the configuration dictionary {"server.urls", "http: // localhost: 3262"}
2. configure Kestrel without specifying the port UseKestrel()
3. add a delay at the end of the Startup.ConfigureServices (..) method System.Threading.Thread.Sleep (10000)
4. compile and run the application with http protocol

The only workaround is setting up a Kestrel listener address and port while building the host, isn't it?

DotNext.Net.Cluster.Consensus.Raft: Seed members and joining members?

Is the DotNext.Net.Cluster.Consensus.Raft cluster capable of allowing non-specified members to join the cluster?
e.g. cluster boots with 3 seed nodes, then additional nodes join in against one of those, and the cluster become aware of the newly joined nodes?

Or is it purely capable of consensus between the pre-specified members?

In https://github.com/AsynkronIT/protoactor-dotnet , we have externalized all cluster management.
e.g. we use Consul or Kubernetes API to get the cluster topology.

I would very much like to give it a try to use dotNext Raft as an alternative cluster provider, as it wouldn't require users to have some specific infrastructure in place.

CodeGenerator.AsyncLambda does not contain overload with result variable

According to the docs on Async Lambda:

AsyncLambda factory method is overloaded by two versions of this method:
AsyncLambda((fun, result) => { }) introduces special variable result that can be used to assign result of the function. This approach is similar to Result variable in Pascal programming language.
AsyncLambda(fun => { }) doesn't introduce special variable for the function result and control transfer to the caller is provided by CodeGenerator.Return method.

However, it appears that the first overload (the one with result) does not exist in the source code. The only overload I can find is the second one here. It appears that to return values from an AsyncLambda, one needs to explicitly call CodeGenerator.Return instead.

Perhaps the discrepancy is because the overload with result is yet to be implemented?

Support of arbitrary async state machine

At this moment, async lambda function compiler implemented by Metaprogramming library supports only Task and Task<R> as valid return types. ValueTask is not supported. To do that, any custom async state machine builder should be recognizable by compiler with help of AsyncMethodBuilderAttribute

Sample for transport-agnostic DotNext.Net.Cluster package

Hi,

I would like to implement a gRPC version of RaftCluster in the DotNext.Net.Cluster package. However, I can't find any samples or detailed information on how to go about this. I have looked at the HTTP version but it seems very different than the transport-agnostic implementation. Do you have a working sample using RaftCluster?

Thanks.

DotNext.Threading.Synchronizer is not thread safe, is this intended?

I noticed that Synchronizer uses internal state quite liberally and would fail in a multithreaded environment. Code like this

return node is null ? CompletedTask<bool, BooleanConst.True>.Task : node.Task.WaitAsync(timeout, token);

will fail with NullReferenceException if other thread calls AsyncManualResetEvent.Set(false). With a small modification example code here would trigger this error.

Create Snapshot with CommandInterpreter pattern?

In docs not enough information about Snapshot. How I understand snapshot is simple command (implementing serialization/desirialization in CommandFormatter etc). How concatinate WriteToAsync in inherit from SnapshotBuilder and Command?

DotNext.AspNetCore.Cluster Messaging Timeouts

Background:
I'm trying to use your Raft implementation to implement a distributed lock service in an Asp.Net Core application. I'm using the IMessageBus.LeaderRouter to send messages to the leader where the lock is entered into the consensus store and, if needed, it waits for the lock to become available (up to a supplied timeout value).

Seems to work reasonably well as long as there is no contention... but if there were no contention I wouldn't need a lock!

I think the issue I'm running into is that the timeout is specified for all requests at the HttpClient level, using a timeout value that is only appropriate for the built-in election messages.

Reading the documentation led me to this blurb in the Microsoft docs for HttpClient:

The same timeout will apply for all requests using this HttpClient instance. You may also set different timeouts for individual requests using a CancellationTokenSource on a task. Note that only the shorter of the two timeouts will apply.

Before I go in and hack up the code to try to make this work for me, am I missing something?

Possible race condition between snapshot and logentry

Hello, I am seeing a strange behavior. My state (and therefore my snapshot) is a dictionary object. The last logentry before a snapshot builder is requested enters a particular key into the dictionary. The snapshotbuilder then creates a snapshot with the entire dictionary.

Now, when I bring down the nodes and bring them up again, a replay occurs in which the snapshot is (obviously) applied first, but then the logentry which was seen previous to the building of the snapshot is applied after which causes a duplicate key exception. It seems as if this is a race condition between the last entry before the snapshot and the snapshot record itself.

Any ideas?

(DotNext.Net.Cluster) Leader does not receive IRaftCluster.LeaderChanged event when downgrading to follower

Hello,

I am currently trying to run the 4.2.0-beta.1 version and I have come across a specific issue regarding the log entry writing with Raft and the events associated with it

1. Have a single node start a standalone cluster (first node)
2. Add a second member with AddMember (second node)
3. Kill the process of node 2
4. Start the process of node 2 again

After step 3 and inspecting my logs, it seems like the leader steps down to follower but does not fire the IRaftCluster.LeaderChanged event. IRaftCluster.Members will still report the local (Remote == false) node 1 as leader on node 1, but attempting to write to the log now will result in

      System.InvalidOperationException: The local cluster member is not a leader
         at DotNext.Threading.Tasks.ValueTaskCompletionSource`1.GetResult(Int16 token) in /_/src/DotNext.Threading/Threading/Tasks/ValueTaskCompletionSource.T.cs:line 272
         at DotNext.Threading.Tasks.ValueTaskCompletionSource`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/DotNext.Threading/Threading/Tasks/ValueTaskCompletionSource.T.cs:line 279
         at DotNext.Net.Cluster.Consensus.Raft.LeaderState.ReplicationCallback.Invoke() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/LeaderState.Replication.cs:line 177
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 818

IClusterMember.MemberStatusChanged will correctly fire and set node 2 to Unavailable during step 3.
This behavior was not present before the upgrade to 4.2.0-beta.1.

After step 4 is done, (on node 1) IClusterMember.MemberStatusChanged will correctly fire and mark node 2 as available again. However, node 1 is still unable to write to the log as it seemingly isn't leader anymore. Only some time after this step, node 1 will correctly fire IRaftCluster.LeaderChanged to NO LEADER, and then fire it again but WITH A LEADER.

Issues after members are reconfigured

I'm testing your Raft implementation and I observe issues when cluster configuration changes in runtime. The point is to simulate behavior in a Kubernetes cluster where containers are always recreated with new IP addresses.

I have modified the RaftNode project to be able to update the configuration. I introduced appsettings.json and moved the in-memory configuration from Program.cs into it. This allows me to update the config file at any time (the members collection) and that change is picked up by all three running instances of RaftNode.

When there are no configuration changes, the cluster is very stable - I can restart any RaftNode instance (both the leader or any of the followers) and it immediately joins the cluster - if a new leader election is needed it usually happens within one term. However, when I stop a node and change its port number in the config file and then start it again with the new port, I sometimes observe these issues:

1. The cluster has hard time electing a new leader - there can be many terms until the leader is established and both followers receive values. Typically after the new node joins the cluster the following repeats multiple times: there is a new election, the leader becomes one of the original two nodes, those two nodes exchange couple of values, but the new node doesn't receive anything, it times out and calls for new election.
2. This occurred only once and it may not be related to the configuration changes. Plus, it may have happened before I got the latest changes from you (I don't remember when it happened anymore). But anyway, this is what I observed: when I once stopped the leader, it could not be restarted - every time I attempted to start it again the whole process crashed immediately in PersistentState.ApplyAsync(long startIndex, CancellationToken token) on the line with Debug.Fail.

I'm running this on Windows, using HTTP for communication and I use persistent storage - the node with the modified port reuses the storage.

AsyncBarrier does not work

I can't get AsyncBarrier to work at all, even under trivial usage.

The following code never completes - the Console.WriteLine is never reached. Putting breakpoints on the return statements reveals that one of them is reached, but the other never is.

[Fact]
public async Task Test1()
{
    var barrier = new AsyncBarrier(2);

    var results = await Task.WhenAll(
        Task.Run(async () =>
        {
            await barrier.SignalAndWaitAsync();
            return 24;
        }),
        Task.Run(async () =>
        {
            await barrier.SignalAndWaitAsync();
            return 42;
        })
    );

    Console.WriteLine(string.Join(',', results));
}

Cluster consensus on more complex objects

I wanted to try and use the cluster library for a project, using the provided playground as an example, but I hit a roadblock.

Multiple methods in the example have type of content that can be used for consensus limited to unmanaged types. I wish to send a more complex object, or at least a string. What am I missing? And if I am not asking for too much, can a minimal example be provided?

Thank you

An attempt was made to transition a task to a final state when it had already completed

Partially related to my other issue (#14), when I fixed the issue in my code the state machine seems to become invalid at some point (I guess the return might not be valid). I get this exception + stack trace:

at void DotNext.Runtime.CompilerServices.AsyncStateMachine<TState, TResult>.System.Runtime.CompilerServices.IAsyncStateMachine.MoveNext() in /_/src/DotNext.Metaprogramming/Runtime/CompilerServices/AsyncStateMachine.cs:line 380
at void System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox.MoveNext(Thread threadPoolThread)
at Action System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents(Task task, Action continuation)+(Action innerContinuation, Task innerTask) => { }
at void System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, object state, ref Task currentTask)
at void System.Threading.Tasks.Task.ThrowAsync(Exception exception, SynchronizationContext targetContext)+(object state) => { }
at void System.Threading.QueueUserWorkItemCallback.s_executionContextShim(QueueUserWorkItemCallback quwi)
at void System.Threading.QueueUserWorkItemCallback.Execute()
at bool System.Threading.ThreadPoolWorkQueue.Dispatch()

This is my code for generating the lambda:

var funcSignature = typeof(Func<,,,>).MakeGenericType(writerType, typeof(EntityBase),
    typeof(CancellationToken), typeof(Task<>).MakeGenericType(typeof(byte[])));

var method = typeof(CodeGenerator).GetMethod(nameof(CodeGenerator.AsyncLambda))
    .MakeGenericMethod(funcSignature).Unreflect();
var compileMethod = typeof(Expression<>).MakeGenericType(funcSignature)
    .GetMethod(nameof(LambdaExpression.Compile), BindingFlags.Public | BindingFlags.Instance, null,
        CallingConventions.Any, Type.EmptyTypes, null).Unreflect();

finalExpression = method(null, (Action<LambdaContext>)GenerateAsyncLambda);
finalExpression = compileMethod(finalExpression);

void GenerateAsyncLambda(LambdaContext c)
{
    var localWriter = DeclareVariable("writer", c[0]);
    var localEntity = DeclareVariable("e", c[1]);
    var localToken = DeclareVariable("token", c[2]);

    var actualEntity = DeclareVariable(entityType, "actualEntity");
    Assign(actualEntity, Expression.Convert(localEntity, entityType));
    
    ParameterReplacer replacer = new ParameterReplacer(localWriter, actualEntity, localToken, entityType);
    foreach (var expr in writerExpressions)
        Await(replacer.Visit(expr), true);
}

EDIT: I tried adding Return() at the end, but that's not working either.

Improve performance of persistent write-ahead log

See suggestions in #56 .
Tasks:

• Add SkipAsync to IAsyncBinaryReader
• Move identifier of the log record to its metadata.
• Rewrite lock mechanism of PersistentState in a way that allows to do the compaction in parallel with writes or reads
• Provides buffering API for log entries
• Reuse buffering API at transport-level
• Split file buffer size of regular log entries and snapshot entry
• fsync for every write must be disabled by default (FileOptions.WriteThrough flag)
• Introduce in-memory cached log entries

Improve performance of WAL part 2

This task is the next step of improvements introduced in #56 :

• Improve locking mechanism: writes while reading are allowed
• Reduce lock time when compacting log

Example raft cluster consensus

Hi,

I read your documentation and try to enter in this web link: https://github.com/sakno/dotNext/tree/develop/src/examples/RaftNode , but is show 404 error. I would like to know if you have some simple example project to use consensus for some data.

Greetings,

Luis

Make LexicalScope public

I understand wanting to keep LexicalScope an internal implementation detail.

But allowing adding statements manually instead of using CodeGenerator would be really helpful when there is a lot of existing code using standard System.Linq.Expressions patterns.

If I could manually call LexicalScope.Current.AddStatement, it would save me a lot of code rewriting. And make gradual migrations to the dotNext LinqEx handling.

Preferably I would like to be able to do the following:

1. Create a LexicalScope/AsyncLambdaExpression using manually specified ParameterExpression[], instead of just System.Type[]
2. Be able to manually construct the LexicalScope, specifically using ILexicalScope.AddStatement

DotNext.Metaprogramming. Regression: worked in v 2.12.2, hangs in higher versions.

public class Test
{
    [Fact]
    public async Task AsyncLambdaDoesntHangInTryBlock()
    {
        Expression<Func<Task<string>>> exprThrowException = () => ThrowException();
        Expression<Func<Task<string>>> exprReprocess = () => Reprocess();

        var asyncTryCatchExpression = AsyncLambda<Func<Task<string>>>(
            (lambdaContext, result) =>
            {
                Try(() =>
                    {

                        Assign(result, Expression.Invoke(exprThrowException).Await());
                        Return(result);
                    })
                    .Catch<Exception>(() =>
                    {
                        Assign(result, Expression.Invoke(exprReprocess).Await());
                        Return(result);
                    })
                    .End();
            });

        var asyncTryCatchFunc = asyncTryCatchExpression.Compile();
        var asyncTryCatchTask = asyncTryCatchFunc();

        var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(5));
        await await Task.WhenAny(asyncTryCatchTask, Task.Delay(-1, cancellationTokenSource.Token));
    }


    private static async Task<string> ThrowException()
    {
        await Task.Delay(TimeSpan.FromSeconds(1));
        throw new Exception();
    }

    private static Task<string> Reprocess()
    {
        return Task.FromResult(string.Empty);
    }
}

Compile time codegen?

Via ltrzesniewski/InlineIL.Fody#6 ?
Or msbuild task with roslyn code generator?

[bug] AsyncAutoResetEvent deadlocks

Run this test. On my computer, it deadlocks after about 6 seconds

	[TestMethod]
	public async Task DotNextAsyncAutoResetEvent()
	{
		DotNext.Threading.AsyncAutoResetEvent autoResetEvent = new(false);

		var loopCount = 0;
		var setCount = 0;
		var consumedCount = 0;
		var timer =Stopwatch.StartNew();
		var lastSecondReported = 0;
		var producerTask = Task.Run(() => {

			while (true)
			{
				loopCount++;
				var didSet = autoResetEvent.Set();
				if (didSet)
				{
					setCount++;
				}

				if (timer.Elapsed > TimeSpan.FromSeconds(lastSecondReported))
				{
					var tup1 = new { loopCount };
					var tup = new { loopCount, setCount, consumedCount };
					Console.WriteLine($"t={lastSecondReported}sec:  {new { loopCount, setCount, consumedCount }}");
					lastSecondReported++;
				}

				if (timer.Elapsed > TimeSpan.FromSeconds(30))
				{
					break;
				}
			}
		});


		var consumerTask = Task.Run(async () => {

			while (true)
			{
				var success = await autoResetEvent.WaitAsync(TimeSpan.FromMilliseconds(1));
				if (success)
				{
					consumedCount++;
				}
				if (producerTask.IsCompleted)
				{
					break;
				}
			}
		});


		await producerTask;
		autoResetEvent.Set();
		await consumerTask;
	}

using DotNext 4.0.0-beta.8 from Nuget

Unexpected exception thrown when services are not yet up

Hello,

I am seeing a strange behavior on some of my nodes but not on others. My first node comes up first and successfully polls for the other members. The other two members, however, come up, try to connect, get a 403 resonse from the first node and crash with the following error:

info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://0.0.0.0:443
info: CRB.DistributedServices.LockServer.LockServerNodeLifeTime[0]
      Cluster member initialized.
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: QA
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app
dbug: DotNext.Net.Cluster.Consensus.Raft.Http.Embedding.RaftEmbeddedCluster[0]
      Sending request of type PreVote to member Unspecified/lockserver01-ingress-svc.lockserver-ns.svc.cluster.local:443
dbug: DotNext.Net.Cluster.Consensus.Raft.Http.Embedding.RaftEmbeddedCluster[0]
      Sending request of type PreVote to member Unspecified/lockserver03-ingress-svc.lockserver-ns.svc.cluster.local:443
warn: DotNext.Net.Cluster.Consensus.Raft.Http.Embedding.RaftEmbeddedCluster[0]
      Cluster member Unspecified/lockserver03-ingress-svc.lockserver-ns.svc.cluster.local:443 is unavailable
System.Net.Http.HttpRequestException: Name or service not known
 ---> System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at DotNext.Net.Cluster.Consensus.Raft.Http.RaftClusterMember.SendAsync[TResult,TMessage](TMessage message, CancellationToken token) in /_/src/cluster/DotNext.AspNetCore.Cluster/Net/Cluster/Consensus/Raft/Http/RaftClusterMember.cs:line 96
dbug: CRB.DistributedServices.LockServer.LockServerMetricsCollector[0]
      ReportResponseTime: 00:00:00.0433675
dbug: CRB.DistributedServices.LockServer.LockServerMetricsCollector[0]
      ReportResponseTime: 00:00:00.1943564
Unhandled exception. DotNext.Net.Cluster.Consensus.Raft.Http.UnexpectedStatusCodeException: Response status code does not indicate success: 403 (Forbidden).
   at DotNext.Net.Cluster.Consensus.Raft.Http.RaftClusterMember.SendAsync[TResult,TMessage](TMessage message, CancellationToken token) in /_/src/cluster/DotNext.AspNetCore.Cluster/Net/Cluster/Consensus/Raft/Http/RaftClusterMember.cs:line 115
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState>g__PreVoteAsync|74_0(Int64 currentTerm) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 740
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 694
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

I am not sure why the first node is returning a 403 (I never configure any authorization) or why such a response would cause a crash of the node.

The CopyTo method not supported

Hey! I use the PooledBufferWriter class from the DotNext v 3.2.1 package, after converting to Stream, then call the CopyTo method and get an exception NotSupportedException
I wanted to know if it was so conceived or am I doing something wrong?

using var writer = new PooledBufferWriter<byte>(ArrayPool<byte>.Shared.ToAllocator());
using Stream writeStream = StreamSource.AsStream(writer);
using MemoryStream memoryStream = new MemoryStream();
memoryStream.Write(new byte[1024]);
			
writeStream.CopyTo(memoryStream); // thow exception 

have problem with PersistentClusterConfigurationStorage

Hi,
I found that if I want to persist the config on file should use the class PersistentClusterConfigurationStorage. I don't know is correct or not but from tracing the source I create the class
`
internal class PersistentClusterConfigurationStorage : PersistentClusterConfigurationStorage
{
public PersistentClusterConfigurationStorage(string path, int fileBufferSize = 512, MemoryAllocator allocator = null) : base(path, fileBufferSize, allocator)
{
}
protected override void Encode(IPEndPoint address, ref BufferWriterSlim output)
{
output.WriteEndPoint(address);
}

    protected override IPEndPoint Decode(ref SequenceReader reader)
    {
        return (IPEndPoint)reader.ReadEndPoint();
    }
}

and using for persisting data
static async ValueTask AddMembersToCluster(AppSettingConfig serverConfig, PersistentClusterConfigurationStorage storage)
{
if (serverConfig.Members == null) return;
for (int i = 0; i < serverConfig.Members.Length; i++)
{
var t = Dns.GetHostAddresses(serverConfig.Members[i].EndPoint);
var ipEndPint = new IPEndPoint(t[0], serverConfig.Members[i]?.Port ?? serverConfig.PeerCommunicationPort);
await storage.AddMemberAsync(ClusterMemberId.FromEndPoint(ipEndPint), ipEndPint);
}
}
`
but when the running application LeaderChanged is not called and can't understand what is false.