jille/raft-grpc-example

Example code for how to get hashicorp/raft running with gRPC

raft-grpc-example

This is some example code for how to use Hashicorp's Raft implementation with gRPC.

Start your own cluster

$ mkdir /tmp/my-raft-cluster
$ mkdir /tmp/my-raft-cluster/node{A,B,C}
$ ./raft-grpc-example --raft_bootstrap --raft_id=nodeA --address=localhost:50051 --raft_data_dir /tmp/my-raft-cluster
$ ./raft-grpc-example --raft_id=nodeB --address=localhost:50052 --raft_data_dir /tmp/my-raft-cluster
$ ./raft-grpc-example --raft_id=nodeC --address=localhost:50053 --raft_data_dir /tmp/my-raft-cluster
$ go install github.com/Jille/raftadmin/cmd/raftadmin@latest
$ raftadmin localhost:50051 add_voter nodeB localhost:50052 0
$ raftadmin --leader multi:///localhost:50051,localhost:50052 add_voter nodeC localhost:50053 0
$ go run cmd/hammer/hammer.go &
$ raftadmin --leader multi:///localhost:50051,localhost:50052,localhost:50053 leadership_transfer
$ wait

You start up three nodes, and bootstrap one of them. Then you tell the bootstrapped node where to find peers. Those peers sync up to the state of the bootstrapped node and become members of the cluster. Once your cluster is running, you never need to pass --raft_bootstrap again.

raftadmin is used to communicate with the cluster and add the other nodes.

This example uses Jille/raft-grpc-transport to communicate between nodes using gRPC.

This example uses Jille/raft-grpc-leader-rpc to send RPCs to the leader.

Hammer is a client that connects to your raft cluster and sends a bunch of requests. Trigger some leadership failovers to show that it's unaffected.

What's what

Raft uses logs to synchronize changes. Every change submitted to a Raft cluster is a log entry, which gets stored and replicated to the followers in the cluster. In this example, we use raft-boltdb to store these logs. Once in a while Raft decides the logs have grown too large, and makes a snapshot. Your code is asked to write out its state. That state captures all previous logs. Now Raft can delete all the old logs and just use the snapshot. These snapshots are stored using the FileSnapshotStore, which means they'll just be files in your disk. Raft also needs a way to talk to other nodes, that's called a Transport. This example uses Jille/raft-grpc-transport to communicate between nodes using gRPC.

You can see all this happening in NewRaft() in main.go.

Your application

See application.go. You'll need to implement a raft.FSM, and you probably want a gRPC RPC interface.

raft-grpc-example's Issues

Confusing remark about reading ReadAtIndex from AppliedIndex

First of all, I really like this very concrete example! Really helps to get a head start.

See remark:

raft-grpc-example/application.go

Line 96 in f5868cd

// TODO: These two should be read under a mutex to handle concurrent Apply() calls.

The comment from AppliedIndex specifically mentions that it can lag behind the application state. How would one implement the TODO locking? I have no idea.

It would be really great if you could share your insights.

unknown field Scheme in struct literal of type resolver.Target

I followed the steps in README and run
go install github.com/Jille/raftadmin/cmd/raftadmin@latest
then I met this error:_

github.com/Jille/grpc-multi-resolver

/go/pkg/mod/github.com/!jille/[email protected]/target.go:19:3: unknown field Scheme in struct literal of type resolver.Target
/go/pkg/mod/github.com/!jille/[email protected]/target.go:20:3: unknown field Authority in struct literal of type resolver.Target_