Comments (6)
Done. Discussion #27
from dotnext.
You're welcome, @squarezx! In your case Raft in just a beginning of the way.
Part 1: Database backend
- You need
DotNext.AspNetCore.Cluster
dependency from NuGet. It is required because it contains implementation of HTTP-based communication between Raft nodes on top of ASP.NET Core infrastructure with all its benefits: DI, logging etc. However, you're able to provide your own network transport (e.g. TCP) without this library. In this case you needDotNext.Net.Cluster
library and RaftCluster class from it. This class provides transport-agnosic implementation of Raft and actively used by HTTP-specific implementation fromDotNext.AspNetCore.Cluster
library. But I think that HTTP is OK for you because this implementation supports HTTP 1 and 2 as well as SSL and other things. - Any reliable database engine constructed on top .NEXT framework should use persistent Write Ahead Log. You're lucky and it is shipped with library. Read this article carefully.
- Now you have two key components for your K/V database. However, some level of customization is required. You need to derive from PersistentState and override
CreateSnapshot
method as described in the article above. This method is responsible for reducing size of Write Ahead Log through compaction of log entries into single one. This process knowns as Snapshotting. I hope you're aware about Write Ahead Log role in database engines. If not then this is different thread for discussion. - The next important method is
ApplyAsync
which is virtual by default and do nothing. In your derived class this method should be responsible for decoding command for K/V database from log entry. Decoded command can be applied to underlying K/V database engine to reflect all necessary changes (CRUD operations). Here you need to design your own binary format for each command. LogEntry value type contains all necessary methods for decoding binary data in stream-like format with low overhead.CreateSnapshot
andApplyAsync
methods are called automatically in synchronized context so you don't need to care about concurrency. - Now you need to design K/V engine which can handle commands decoded from log entries and maintain K/V storage on disk. Let's assume that your engine can be described by interface
IKeyValueDatabaseEngine
and appropriate implementation registed using DI container as singleton. - Parameter of type
IKeyValueDatabaseEngine
should be added to constructor of your type derived fromPersistentState
. This approach allows to inject database engine using DI.
Now the backend for K/V database in finished. However, we need a fronted in the form of REST API which allows clients to operate with this database.
Part 2: Database frontend
- Design your REST API. Read operations can be performed on every Raft node even if it is not a leader. Therefore, such API can call some methods from
IKeyValueDatabaseEngine
for reading data - All writes can be handled by leader node only. However, the request can be received by follower node due to load balancing and HTTP reverse proxy. Read this chapter carefully to solve this issue using automatic routing to leader node.
- Every write produces a set of changes. The changeset can be represented in the form of log entries. These log entries should be submitted to Write Ahead Log and committed.
- Use IReplicatedCluster interface that can be injected using DI for writing changes into Write Ahead Log. You need to call
WriteAsync
method withWriteConcern.LeaderOnly
orWriteConcern.None
depending on chosen consistency model. WriteAsync
will write uncommited entries intoPersistentState
and wait when all of them will be committed. When entries are marked as commited thenApplyAsync
overridden method will be called automatically. Note thatWriteAsync
method can be called by leader node only. However, automatic redirection helps you with that.
Additionally, you can examine RaftNode example which demonstrates most of concepts described in this post.
DotNext.Threading
library can help you to organize concurrent asynchronous access to the database and parallelize reads. Take a look at this for more info,
from dotnext.
Thank you very much
from dotnext.
@squarezx adoption for 2.0 is improved and published here. I recommend you to use latest version of library because now you don't need to implement persistent state of your database. State reconstruction provided by persistent WAL implementation shipped with the library can do this for you.
from dotnext.
Hello, regarding the following:
"Use IReplicatedCluster interface that can be injected using DI for writing changes into Write Ahead Log. You need to call WriteAsync method"
I can not find this method on the interface nor can I find an extension method for this. Has this been refactored since this issue? If so, what is the current correct way to initiate a write to the audit log?
from dotnext.
@natilivni , correct. 2.x branch doesn't contain WriteAsync
method of IReplicationCluster
interface anymore. Could you please open a new topic on Discussions tab?
from dotnext.
Related Issues (20)
- [API Proposoal]: Additional methods for "DotNext.Span" HOT 57
- Update to .NET 8 and AOT compatibility
- Raft Cluster Asp.net Core Dynamically AddMemberAsync HOT 5
- [API Proposal]: Add .ctor(Span<T> span) to MemoryOwner<T> HOT 10
- API Ref: Universally Broken Links (Wrong Repo Target) HOT 4
- ASP.net Cluster Raft question to configure full in memory mode HOT 1
- Quorum node for a 2-node cluster
- Using Optional<T> with data validation attributes HOT 5
- `AsyncBarrier.AddParticipant()` does not work
- Cluster node status HOT 1
- DotNext vs Community toolkit
- How to use Optional<string> as query parameter? HOT 5
- Support for non-LTS .NET releases HOT 1
- Various trimming warnings in DotNext.Metaprogramming HOT 15
- Potential addition of an `OrderedDictionary<TKey, TValue>` type HOT 6
- raft leader loses leadership and the node gets stuck HOT 19
- Metaprogramming: Try-catch not catching exception HOT 2
- ask for set up workaround to make AOT works HOT 4
- Directly reference algorithm used in int sqrt HOT 3
- Not seeing expected improvement in throughput of RaftCluster.ReplicateAsync method when cluster minority is inaccessible HOT 20
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dotnext.