truechain/truechain-consensus-core

TrueChain Consensus Protocol: Minerva

truechain-consensus-core's Issues

Hi，is this project still ongoing?

why make the `hosts` file only in the home directory?

IMO user should be able to create hosts file anywhere not specifically in the home directory, unless there is a reason for doing it like this.
I'll be very happy to send a PR for this as I already have one.

pbft stage completion step - for when leader is corrupted, re-send request from client

Harden local folders permissions

pbft-core/utils.go

func MakeDirIfNot(dir string) {
	_, err := os.Stat(dir)
	if os.IsNotExist(err) {
		err := os.Mkdir(dir, 0777) // HERE
		CheckErr(err)
	}
}

0777 is too permissive. Should be 0700 or 0750 or even 0755, but not allow everyone to read/write/exec the new directory as this could enable any local users on the system to have full control over the new directories created.

commit vendor libs / 3rd party libs and freeze/audit versions

review, freeze and commit specific versions of all ethereum/google/ go dependencies.

ref: #39 #27

Data sync strategy for fastchain with snailchain

Brainstorm our data sync strategy for hybrid consensus here... Data includes syncing of LOG for every new BFT Node, access of accounts from off/on-chain database, access to common ledgers (if needed) etc..

this highly affects truechain-consensus-core because we need to design & implement relevant functions for this, while validating transactions as well as orchestrating committee.

build test failure: unable to find yaml config while it exists

[test]              /src:/src/..
[test]              warning: "./trueconsensus/dapps/..." matched no packages
[test]              warning: "./trueconsensus/db/..." matched no packages
[test]              warning: "./trueconsensus/evm/..." matched no packages
[test]              2018/08/01 01:35:30 Unable to read config file. Error:open /etc/truechain/tunables_bft.yaml: no such file or directory 
[test]              --- FAIL: TestConfig (0.00s)
[test]              	assertions.go:256: 
[test]              			Error Trace:	config_test.go:32
[test]              			Error:      	Expected nil, but got: &os.PathError{Op:"open", Path:"/etc/truechain/tunables_bft.yaml", Err:0x2}
[test]              			Test:       	TestConfig
[test]              	assertions.go:256: 
[test]              			Error Trace:	config_test.go:33
[test]              			Error:      	Should not be: 0
[test]              			Test:       	TestConfig
[test]              FAIL
[test]              FAIL	trueconsensus/fastchain	0.024s
[test]              ?   	trueconsensus/fastchain/proto	[no test files]
[test]              ?   	trueconsensus/minerva	[no test files]
[test]              ?   	trueconsensus/client	[no test files]
[test]              ?   	trueconsensus/common	[no test files]
[test]              ?   	trueconsensus/logging	[no test files]
[test]              ?   	trueconsensus/snailchain	[no test files]

while expected output should be: PASS

I've tested this locally, abstracting the logic away from the codebase and it works.

So there's something that has not caught my attention yet.

pemkey save in clear text is insecure

in trueconsensus/fastchain/genkeys.go
WriteNewKeys function. (which called in pbftserverengine.go -> main() -> cfg.GenerateKeysToFile() -> WriteNewKeys(cfg.Network.NumKeys, cfg.Logistics.KD))

func WriteNewKeys(kcount int, kdir string) {
	for k := 0; k < kcount; k++ {
		privateKey, _ := ecdsa.GenerateKey(ethcrypto.S256(), rand.Reader)
		publicKey := &privateKey.PublicKey

		pemEncoded := hex.EncodeToString(ethcrypto.FromECDSA(privateKey))
		pemEncodedPub := hex.EncodeToString(ethcrypto.FromECDSAPub(publicKey))

		pemkeyFname := fmt.Sprintf("sign%v.pem", k)
		err1 := ioutil.WriteFile(path.Join(kdir, pemkeyFname), []byte(pemEncoded), 0600)
		common.CheckErr(err1)
		pubkeyFname := fmt.Sprintf("sign%v.pub", k)
		err2 := ioutil.WriteFile(path.Join(kdir, pubkeyFname), []byte(pemEncodedPub), 0644)
		common.CheckErr(err2)
	}
}

ethcrypto.FromECDSA just make privatekey to byte type.
function save the pemkey as pemEncoded which Unencrypted.

If some node be attacked, the attacker can steal pemkey of users with a malicious software.

gvt restore issue in hmake build (rebuilds entire dep on change in vendor/manifest)

either sort this in gvt restore or shift to go dep.

ref: https://gitter.im/truechain-net/engg-foss-global?at=5b45b9413d8f71623d6e9357

Align structure of truechain-consensus-core with py-trueconsensus

For the sake of verbosity, here's the salvation of discussion from #10 (comment)

---

The current aim for this repo is to have a modularized consensus, which would very soon be integrated into go-ethereum's fork of truechain as per https://github.com/truechain/truechain-consensus-core/milestone/2

Eventually the bft committee in hybrid consensus would need PBFT code in a separate package and HyperMake should modified to build the entire modularized consensus itself, rather than just PBFT.

---

Here's what I have in my head:

.
├── docs/
│   ├── DEV.md
│   └── TODO.md
├── HyperMake
├── LICENSE
├── trueconsensus/ (code for hybrid consensus)
│   ├── cmds
│   │   └── main.go [calls both fast and snailchain]
│   ├── fastchain/ *
│   ├── slowchain/ **
│   ├── config.yaml ***
│   ├── bft_committee.cfg ****
│   └── vendor/
│       └── manifest
├── README.md
├── support/
│   ├── scripts/
│   └── toolchain/
└── tests
    ├── core-consensus

• * fastchain/ (contains current PBFT codebase as currently under src/pbft/)
• ** slowchain/ is the snailchain in our hybrid consensus, a POW variant called fruitchain, which for this development purpose, is going to be a dummy block producer.
• *** config.yaml contains node environment settings, tunables, etc.. just like #11
• **** bft_committee.cfg contains log directory path, network config paths and so on..

---

FYI

This derives design decisions from https://github.com/truechain/py-trueconsensus (parallel effort, initial faster prototyping but supposed to be maintained separately)

This slowchain is responsible for electing BFT committee, so essentially for single node testing purpose, we'd have multiple instances of slowchain and fastchain, all interacting through gRPC interfaces. The number of fastchain instances would be controlled by a tunable specified under config.yaml

obfuscated committee election

WIP

todo: update from internal

cc/ @hzh1982 let's hold another meeting with you, @panghalamit and Elon (add his github?) asap, then I'll update it here after common agreement.

Make sure duplicate transaction requests are rejected/ignored from being added to txpool

Right now duplicate transaction requests are added to the transaction pool and become part of the block as well. That needs to be fixed.

live orchestration of truechain-engine | event response daemon

Use capabilities of cli for controlling truechain-engine and make it run as a daemon.

This really should be a geth like integration, but for fastchain. For example: https://github.com/ethereum/go-ethereum/wiki/Command-Line-Options

Currently feasible options:

For client (currently simulated with the binary :

Usage of ./bin/linux/truechain-engine:
  -numquest int
    	number of requests (default 10)

For server, something like:

..
dump
checkview/sequence
...

..basically all things pbft, maybe specific to this tunable (as found under config.yaml):

---
basePort: 49500
grpcBasePort: 10000
maxFail: 1
bftcommittee:
  actualDelta: 0
  csize: 0
  delta: 1
  lambda: 1
  tbft: 1
  th: 10
  timeout: 300
slowchain:
  csize: 0
testbedconfig:
  clientID: 5
  initServerID: 4
  requests:
    batchSize: 32
    max: 256
  threadingEnabled: true
  total: 5

cc/ @hixichen

Support for Config files

Adding support for config file will help making the project more modular.
Right now, the hosts file lies in the $HOME, truechain-engine creates two additional files in the same directory, having a separate, user specified location of the files created by truechain-engine will help user to manage their files easily.
User create a config file like.

[keys]
pem_keys = /tmp/truechain/keys
[log]
logs = /tmp/truechain/logs

and truechain-engine will parse the config and create the logs and keys files accordingly. These config files can be made per project.

similar to this.

@arcolife

Add support for a sample transaction dapp

An example is that of bank.py under https://github.com/truechain/py-trueconsensus/blob/master/trueconsensus/dapps/bank.py

which gives something to this consensus to execute, as a PoC for payload execution, in future, with Truechain Virtual Machine (~EVM)

At the moment, the codebase doesn't execute anything

func (nd *Node) execute(am ApplyMsg) {
	// TODO: add msg to applyCh, should be executed in a separate go routine
	// TODO: add we probably want to keep a log for this
	nd.applyCh <- am
}

Missing interfaces for node.go and everything else

To use this consensus by an orchestration tool, we need to expose it using interfaces.

Ongoing work under arcolife@2762a44

Related to #26

server crash logging/reporting and crash queue maintenance / rollup

if server crashes, we need (WAL) to notify and dump env vars/tracebacks and report it. And then use that dump for patched releases.

think: open tracing / airbrake / honeybadger

cc/ @miamiruby

Also, add airlog explicit log level handing flag/config param; obfuscate sensitive info in dump.

documentation for dev workflow (log, console, testing)

add a wait threshold timer instead of timer.sleep(1)

TODO comments in src/pbft-core/pbft-sim-engine/pbftserverengine.go

	// TODO: change listenready to switch case assertion and kill-timer clock
	for i := 0; i < cfg.N; i++ {
		<-svList[i].Nd.ListenReady
	}

	time.Sleep(1 * time.Second) // wait for the servers to accept incoming connections
	for i := 0; i < cfg.N; i++ {
		svList[i].Nd.SetupReady <- true // make them to dial each other's RPCs
	}

	// TODO: change listenready to switch case assertion and kill-timer clock
	//fmt.Println("[!!!] Please allow the program to accept incoming connections if you are using Mac OS.")
	time.Sleep(1 * time.Second) // wait for the servers to accept incoming connections

done

user abuse

missing deps while running hmake test

related: #48 #29 #39

can't run hmake test unless we add assertify et al to vendor/manifest

gvt fetch/update cycle review

here's the scenario, that is hurtful with gvt right now:

1. Bob adds 2 packages which have N deps. Gets updated into vendor/manifest
2. one of the packages from group N gets an update on its github. In this case it's testdata folder inside grpc library. So when Bob runs a test. It fails, because the update is missing from the git version.

[test]              /src:/src/..
[test]              src/vendor/google.golang.org/grpc/benchmark/worker/benchmark_client.go:38:2: cannot find package "google.golang.org/grpc/testdata" in any of:
[test]              	/src/src/vendor/google.golang.org/grpc/testdata (vendor tree)
[test]              	/src/src/vendor/google.golang.org/grpc/testdata
[test]              	/usr/local/go/src/google.golang.org/grpc/testdata (from $GOROOT)
[test]              	/src/src/google.golang.org/grpc/testdata (from $GOPATH)
[test]              	/src/google.golang.org/grpc/testdata

4. . Bob performs a gvt update for that specific package. He sees a change reflected in manifest for grpc-go.

                        "vcs": "git",
-                       "revision": "b519e3d28d377e55a0d44c9b83e6aee4d02d731d",
+                       "revision": "445634bdcc9393d2681e504aafd3efc9b28c4bf2",
                        "branch": "master",

5. And so determined that all that's needed is done, he performs another test with hmake test. But he still gets:

[test]              /src:/src/..
[test]              src/vendor/google.golang.org/grpc/benchmark/worker/benchmark_client.go:38:2: cannot find package "google.golang.org/grpc/testdata" in any of:
[test]              	/src/src/vendor/google.golang.org/grpc/testdata (vendor tree)
[test]              	/src/src/vendor/google.golang.org/grpc/testdata
[test]              	/usr/local/go/src/google.golang.org/grpc/testdata (from $GOROOT)
[test]              	/src/src/google.golang.org/grpc/testdata (from $GOPATH)
[test]              	/src/google.golang.org/grpc/testdata

6. So he checks the folder for latest from https://github.com/grpc/grpc-go/tree/master/testdata

ls vendor/google.golang.org/grpc/testdata                        ✔  5579  18:43:36
ls: vendor/google.golang.org/grpc/testdata: No such file or directory

7. So to solve this for now, Bob performs a go get -u google.golang.org/grpc and solves this with cp -r $GOPATH/src/google.golang.org/grpc/testdata src/vendor/google.golang.org/grpc/

But bob is guilty as charged.

And tired of this game of deps, he wants to get rid of this problem, along with this #39 and FiloSottile/gvt#16 (gvt delete doesn't clear all deps, or does it?)

Also, if impossible, Bob would probably go for golang/dep#679

use xgo for cross compile

Failed to produce darwin binary with hmake build, because a vendor dep uses libsecp256k1 which in turn depends on import "C" in golang from under secp256.go

screenshot courtesy: @hixichen

So now, we have 2 options:

1. Cross compile from linux to darwin https://github.com/karalabe/xgo
2. Compile natively

Cross check sender pubkey / nonce from accounts db access

Need to replace the functionality "verifySender for client request" with accountNonce cross-check logic from db, in future, when we would have implemented access to db. Ref #55

update .proto rpc calls / data structures in line with engg codebase

in line with HybridConsensusHelp.proto from truechain-engineering-code

ref: https://github.com/truechain/truechain-engineering-code/blob/master/eth/truechain/HybridConsensusHelp.proto

no way to specify number of requests for client at the moment through cmdline

add flag module to client.go

checks for prevention of transaction re-orderin by leader?

also,

1. How do replicas track corruption ? Theoretically they should be saving last request received regardless of whether they're primary nodes or not
2. When they can indeed track corruption in committee, everybody becomes a proposer according to paper (todo: recheck theory). So how does price sortition of transactions in block work out for all nodes?

after >10 iterations of pbft-client requests to nodes, the server stops dumping tx req details

Normal case

The last log line from server side (./bin/linux/truechain-engine) is the one containing New Sequence Item

[?]Transacion count is 1
[.]Adding transaction request 49 to block 14

[?]Received new transacion request 49 from client on node 1
fetching file:  sign3.pub
[ ]Txn verified
[?]Added request 49 to transaction pool on node 1.
[!][0] Leader acked: {{3 0 0 4  Header:<Number:15 GasLimit:5000 GasUsed:445 Timestamp:1532292697 ParentHash:"*\004I\301\027\242\227\232\256f\255A\315^\377\240w\367\371WL\233\236\0029\260\231\200\257F\017\233" TxnsHash:"\355\017c]\177\214\362ig\213\267\265k \302\027\251\203s{\312,#kA\336\307\r\324\252p\213" > Txns:<data:<AccountNonce:40 Price:40 Hash:"An'\200\2130=\007\200W\2639F\222\351\316\323\032!\216\376n\001H9e!%\253c\324J" > >  1532292697} fa34666614cacfa0e20899b695ecbdf42b8368c3b04eca7d0ac18a16d5772ea38fa0aa87f89e51634cfb021df6ada67039ffbd3a94ed885e8af531d9f99dc925 {114228187525300136869126589949946702411048996284305879532910232289786292870326 80221517476341951832338308655862238779378846682929138267853255451220187384345}}.

[.]digest 3fe70fad55c21dbacf402e8e063ae15b6f1c44944f1313da05b64de62fbfc021d0110f5f12df170621575e227e6d93fc561f361cd3fc766e6c3b912a3ed165f5.

[?]Received new transacion request 49 from client on node 2
[!][2] ProxyProcessPrePrepare {{{0 15 0 0 fa34666614cacfa0e20899b695ecbdf42b8368c3b04eca7d0ac18a16d5772ea38fa0aa87f89e51634cfb021df6ada67039ffbd3a94ed885e8af531d9f99dc925 Header:<Number:15 GasLimit:5000 GasUsed:445 Timestamp:1532292697 ParentHash:"*\004I\301\027\242\227\232\256f\255A\315^\377\240w\367\371WL\233\236\0029\260\231\200\257F\017\233" TxnsHash:"\355\017c]\177\214\362ig\213\267\265k \302\027\251\203s{\312,#kA\336\307\r\324\252p\213" > Txns:<data:<AccountNonce:40 Price:40 quest 43" Signature:"z\231\261E\262\0212t\213\376\307\3340\224|G;p\212,\314\262\203\332\202o\375\003i\211\366\252su\215 '\177\"\337o,*A]\277l*k4\360e\370\023\262\224Ua\221\312\336\246f\243\000" 
...
[.]digest f6a1aa277415fee887f573aa32f63d56b7edbcc0cb27402d53592fdb47831e42945442b2e1c48ad0e380a0f3b6b3281073f45801128566c473cb222968016be0.

fetching file:  sign3.pub
[.]digest 749e5e025e256a79c6f66c7d9d73a8e33b0fc215e6f3bdb919ebdf467f227603877719b63287677e1f5e316d1738b03eb4e23cba7337ad8a4b773a4368e4b4fc.

[ ]Txn verified
[?]Added request 49 to transaction pool on node 2.
[!][1] Broadcasting to Node.ProxyProcessPrepare, {{{1 15 0 1 fa34666614cacfa0e20899b695ecbdf42b8368c3b04eca7d0ac18a16d5772ea38fa0aa87f89e51634cfb021df6ada67039ffbd3a94ed885e8af531d9f99dc925 Header:<Number:15 GasLimit:5000 GasUsed:445 Timestamp:1532292697 ParentHash:"*\004I\301\027\242\227\232\256f\255A\315^\377\240w\367\371WL\233\236\0029\260\231\200\257F\017\233" TxnsHash:"\355\017c]\177\214\362ig\213\267\265k \302\027\251\203s{\312,#kA\336\307\r\324\252p\213" > Txns:<data:<AccountNonce:40 Price:40 Payload:"Request 40" Signature:"x\275)\322X\265\014\030\373\307\030/\307-\250\206\362e!%\253c\324J" > >  -1} 749e5e025e256a79c6f66c7d9d73a8e33b0fc215e6f3bdb919ebdf467f227603877719b63287677e1f5e316c {79268607248763335051141996654494652372639595856080284853135830670826484330562 29523290159598255139086912673083037644138473982758858904072562300196137560401}} 1}
...
...

[!][1] Committed 15

[!][0] ProxyProcessCommit 15

[!][1] ProxyProcessCommit 15

[.][0.0.0.0:40542] [2] New Sequence Item: {15 {{1 15 0 2 fa34666614cacfa0e20899b695ecbdf42b8368c3b04eca7d0ac18a16d5772ea38fa0aa87f89e51634cfb021df6ada67039ffbd3a94ed885e8af531d9f99dc925 Header:<Number:15 GasLimit:5000 GasUsed:445 Timestamp:1532292697 652bfaa50785070a9bf7c76320828871e97472ee945eea6dcae00490 {18704172220060137031230359153475805489348593663570609262442406299954186161642 114691425572839107254689997849255936013092033288100089168574433829570816936729}}}

<output ends here>

---

after 11 iterations of ./bin/linux/pbft-client

last line doesn't have [.][0.0.0.0:40542] [2] New Sequence Item: and stops dumping debugging info abruptly

[?]Added request 48 to transaction pool on node 2.
[?]Received new transacion request 49 from client on node 0
fetching file:  sign3.pub
[ ]Txn verified
[?]Added request 49 to transaction pool on node 0.
[?]Received new transacion request 49 from client on node 1
fetching file:  sign3.pub
[ ]Txn verified
[?]Added request 49 to transaction pool on node 1.
[?]Received new transacion request 49 from client on node 2
fetching file:  sign3.pub
[ ]Txn verified
[?]Added request 49 to transaction pool on node 2.

<abrupt stop after last log line containing [?] >

issue with channel/buffers?

Validate transactions in block during pbft phases

When a proposed block goes through pbft phases, transactions in the block need to be validated:

1. Check if the transaction is present in its transaction pool
2. Check if the transaction account nonce matches with that proposed in the block
3. Validate transaction sender

Queuing mechanism for txpool

At the moment, txn pool is a channel and can't be re-read to check for consistency / transaction execution during the round validations of PBFT.

Need to add a queue for this, and also attach multiple channels for handoff during errors, ensuring high availability.

cc/ @hixichen

Trie + leveldb integration for on-chain logging/storage

Txn pool / blocks exchanged are all in memory at the moment. This should follow the paradigm of file storage drivers such as leveldb

test cases for pbft-core funcs

Need to integrate with Travis CI / Jenkins.

A highly disrupted dev cycle led us to a dark dark place, one without test cases.

Need to push for *_test.go files, just like https://github.com/truechain/truechain-consensus-core/blob/devel/pbft/src/pbft-core/config/config_test.go

cc/ @samikshan @hixichen

verify sender from VRS, not hardcoded client ID

the client could just be anyone.
So verify the sender of the block by obtaining the VRS in block, deconstructing public key from that and comparing the obtained key with stored public key of the sender.

Also fix the 3 modes, 2 out of which are currently dummy, as per https://github.com/truechain/truechain-consensus-core/blob/master/README.md#step-3

4. `TRUE_SIMULATION` is as follows:
  - if set to 0 (default) - should tell the project to pickup testbed configurations. 
  - if set to 1 - staging, meaning all CI/CD tests are run before draft run. (dummy functionality at the moment)
  - if set to 2 - production. Will try to connect to boot nodes. (dummy functionality at the moment)

.env file to segregate development/staging v/s production

evm integration for transaction execution

to facilitate payload execution, we need compiler integration. Related to #31

cc/ @samikshan

Logging for TrueChain

enhance logging for truechain, just like in https://github.com/truechain/py-trueconsensus (truechain's py based alt-ego)

for orchestrator, referred to as minverva.py:

py-trueconsensus  ./minerva.py 
loaded local_config.py from /etc/truechain
Storing engine logs to file: /var/log/truechain/engine.log
Storing client logs to file: /var/log/truechain/client.log
Start time:  2018-07-07 17:17:01.603656
Threading enabled:  True
run started
run started
run started
run started
Node: [3], Msg: [Firing up gRPC channel], Address: [127.0.0.1:49503]
Node: [0], Msg: [Firing up gRPC channel], Address: [127.0.0.1:49500]
Node: [2], Msg: [Firing up gRPC channel], Address: [127.0.0.1:49502]
Node: [1], Msg: [Firing up gRPC channel], Address: [127.0.0.1:49501]
Node: [3], Msg: [INIT SERVER LOOP]
Node: [1], Msg: [INIT SERVER LOOP]
Node: [0], Msg: [INIT SERVER LOOP]
Node: [2], Msg: [INIT SERVER LOOP]

for client:

py-trueconsensus  tail -f /var/log/truechain/client.log 
...
[2018-07-03 02:57:28,891] [DEBUG ] [bank.py:118:open_db_conn()] - Msg: [Opened Database connection..], ForNode: [None]
[2018-07-03 02:57:28,891] [DEBUG ] [bank.py:140:gen_accounts()] - Generating 4 accounts
[2018-07-03 02:57:28,897] [ERROR ] [client.py:99:send_requests()] - Msg: [failed to send], Target: [127.0.0.1:49500], Error => {Value out of range: 77600048550433786810477053529344464877384026879796736630228681607545426539267}

for server:

py-trueconsensus  tail -f /var/log/truechain/engine.log
...
[2018-07-07 17:17:01,612] [DEBUG ] [engine.py:171:init_grpc_server()] - Node: [3], Msg: [Firing up gRPC channel], Address: [127.0.0.1:49503]
[2018-07-07 17:17:01,614] [DEBUG ] [engine.py:171:init_grpc_server()] - Node: [0], Msg: [Firing up gRPC channel], Address: [127.0.0.1:49500]
[2018-07-07 17:17:01,614] [DEBUG ] [engine.py:171:init_grpc_server()] - Node: [2], Msg: [Firing up gRPC channel], Address: [127.0.0.1:49502]
[2018-07-07 17:17:01,616] [DEBUG ] [engine.py:171:init_grpc_server()] - Node: [1], Msg: [Firing up gRPC channel], Address: [127.0.0.1:49501]
[2018-07-07 17:17:01,810] [INFO ] [node.py:934:parse_request()] - Node: [0], Phase: [PARSE REQUEST], RequestInnerID: [2]
[2018-07-07 17:17:01,830] [INFO ] [node.py:61:record_pbft()] - Node: [0], Req Type: [INIT], Seq: [0], Received From: [2], In View: [0]
[2018-07-07 17:17:01,937] [INFO ] [node.py:934:parse_request()] - Node: [0], Phase: [PARSE REQUEST], RequestInnerID: [3]
[2018-07-07 17:17:01,937] [INFO ] [node.py:61:record_pbft()] - Node: [0], Req Type: [INIT], Seq: [0], Received From: [3], In View: [0]
...
[2018-07-07 17:17:03,281] [INFO ] [node.py:211:unicast_message()] - Node [2], Msg: [FastChainStub.Send()], Status: [200], Target Replica: [3]
[2018-07-07 17:17:03,281] [INFO ] [engine.py:224:run()] - Node: [2], Current Primary: [0]
...
[2018-07-07 17:17:22,128] [DEBUG ] [engine.py:131:NewTxnRequest()] - Node: [0], Msg: [Received Client Request for Acc Nonce 254 for Recipient b'0xa593094cebb06bf34df7311845c2a34996b52324']
[2018-07-07 17:17:23,081] [DEBUG ] [engine.py:274:run()] - Node: [3], Waiting for next batch..
[2018-07-07 17:17:23,198] [DEBUG ] [engine.py:274:run()] - Node: [1], Waiting for next batch..

hmake deps fails because it requires golang.org/x/net/context which is natively available

was changed in pbft-server.go but not client.go.

either reasons:

• traces of rebasing without tests.
• skipped

but got caught when building deps on darwin, for some reason. fixing it asap.

done

abuse