phoreproject / graphene Goto Github PK
View Code? Open in Web Editor NEWPhore Synapse working repository
License: MIT License
Phore Synapse working repository
License: MIT License
The Synapse module should tie all current module together. It will multiple modules together and allow them to be run from a single process with a YAML config file.
First, all of the configs for apps should be YAML serializable/deserializable. Then, we need to implement a common app interface. Finally, we write the synapse module to tie all of the modules together.
If the beacon chain lags, re-request epoch data to make sure valid block can be created.
Check the state root to make sure it's equal to the previous state root after processing the block.
There should be a GetHash method for each of these takes the hash of all of the elements of each to validate state. We also possibly want this to be able to be updated without the entire tree, so possibly a sparse merkle tree would be the best route.
i.e. setting both finalized state and finalized block node
This will include both a gRPC interface and a JSON RPC interface for easy interaction with the old Phore API.
This involves adding an interface for checking/sending broadcasts to the beacon chain and then updating the shard code to use those broadcasts.
First, we need a relayer module so that the shard module can get transactions to send. Dependent on #77
https://goreportcard.com/report/github.com/phoreproject/synapse
Split up some state functions to decrease cyclomatic complexity.
test
Tests sometimes randomly fail with this.
The command "golangci-lint run --disable megacheck" exited with 4.
Ask peers for other peers.
Reproduce:
The Beacon node will return success.
Expect: Beacon should report error.
Tracking issue for Phore Synapse main network
Synapse should close the database cleanly when CTRL-C is run.
State manager to hold state from shards including full vs. partial state.
Process block signature validation in parallel for the current epoch while syncing and then set a flag to skip signature validation on processing. Then, we can sync 8 blocks at a time and process them each as they become ready.
Validate signatures 1-8 in parallel (8/numcores)
Process blocks 1-8 skipping signature validation (8)
--------
Total Runtime = 8 + 8/numcores
vs.
Validate block 1 (1)
Process block 1 (1)
Validate block 2 (1)
Process block 2 (1)
Validate block 3 (1)
Process block 3 (1)
Validate block 4 (1)
Process block 4 (1)
Validate block 5 (1)
Process block 5 (1)
Validate block 6 (1)
Process block 6 (1)
Validate block 7 (1)
Process block 7 (1)
Validate block 8 (1)
Process block 8 (1)
--------
Total Runtime = 16
In a simple test (not many slots, maybe just one or two days), a lastSlotReceipts
can hold more than 5M receipts, which is more than 100M RAM. And if there are more slots, I ever observed it uses too much RAM and crashes the OS.
I'm not sure if it can be eliminated, if not, we may need to use external database for it instead of residenting in memory. If we are going to use database, I can do it.
For the choice of database, current database for the chain uses very much disk. I think SQLite maybe better. The disadvantage of using SQLite is that we introduce a new kind of database to Synapse.
Currently the node addresses look like /ip4/0.0.0.0/tcp/11781/ipfs/12D3KooWSzvx4mxBSGhLzqS3gLp4wrGqivLuNghZadTgH1R1fbVR
, which uses IPFS protocol. There are several evidence showing that IPFS is deprecated and it's replaced by P2P.
Evidence 1, in multiformats/go-multiaddr/protocols.go source code,
P_P2P = 0x01A5
P_IPFS = 0x01A5 // alias for backwards compatability
Evidence 2, Python version libp2p has nothing about IPFS, and all its example and test code use p2p.
Current problem is that if we continue using IPFS, I'm not sure if py-libp2p can parse the nodes' addresses.
When the beacon chain is syncing or isn't connected to the network, it should tell the validator module not to propose any blocks and not accept any blocks until the chain is fully synced and the client is connected.
Branch: master
Code (in validatormanager.go):
// NewSlot is run when a new slot starts.
func (vm *Manager) NewSlot(slotNumber uint64) error {
earliestSlot := vm.latestEpochInformation.earliestSlot
logrus.WithField("slot", slotNumber).Debug("heard new slot")
///////////////////////// here caused index out of range
proposerSlotCommittees := vm.latestEpochInformation.slots[int64(slotNumber-1)-earliestSlot]
Stack trace:
panic: runtime error: index out of range
goroutine 1 [running]:
github.com/phoreproject/synapse/validator.(*Manager).NewSlot(0xc0000c82c0, 0x21749, 0xc000083ad8, 0x1)
E:/projects/wqking/go/src/github.com/phoreproject/synapse/validator/validatormanager.go:267 +0xef8
github.com/phoreproject/synapse/validator.(*Manager).ListenForBlockAndCycle(0xc0000c82c0, 0xc000062e40, 0xada960)
E:/projects/wqking/go/src/github.com/phoreproject/synapse/validator/validatormanager.go:422 +0x69a
github.com/phoreproject/synapse/validator.(*Manager).Start(...)
E:/projects/wqking/go/src/github.com/phoreproject/synapse/validator/validatormanager.go:437
github.com/phoreproject/synapse/validator/module.(*ValidatorApp).Run(0xc00009ab40, 0x18, 0xc0000640e0)
E:/projects/wqking/go/src/github.com/phoreproject/synapse/validator/module/app.go:105 +0x661
main.main()
E:/projects/wqking/go/src/github.com/phoreproject/synapse/cmd/validator/synapsevalidator.go:28 +0x1d3
Sorry to be the bearer of bad news, but: I just stumbled across this project and I'm concerned that the name collides badly with https://github.com/matrix-org/synapse - the reference Matrix server implementation. Given Matrix is in a similar space (decentralised communication) and bears a lot of similarities with blockchains (we actually use replicated Merkle trees for decentralisation), I'm worried that users might get badly confused between the two. Matrix's Synapse even has sharding work for scalability, for instance - https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-scalability.
So... given this project hasn't had that much visibility yet (e.g. only 9 github stars, relative to matrix-synapse's 6.8K) and is relatively new (matrix-synapse has been around and accelerating since 2014), I was wondering if you might consider renaming at this point to avoid future confusion? thank you! :)
panic: rpc error: code = Unknown desc = error processing block submitted through RPC: block signature was not valid
goroutine 1381 [running]:
github.com/phoreproject/synapse/validator.(*Manager).NewSlot.func1(0xc0018b1ea0, 0x3, 0xab, 0x11, 0xc005f3e690)
/home/meyer9/development/synapse/validator/validatormanager.go:303 +0xbe
created by github.com/phoreproject/synapse/validator.(*Manager).NewSlot
/home/meyer9/development/synapse/validator/validatormanager.go:300 +0xeb6
exit status 2
The log15
library doesn't allow global level setting. Testing is currently printing out a ton of logging which it shouldn't. We can't set the log level globally so we can't stop this behavior.
The wallet should have the following commands:
getbalance <addr>
- gets the balance of an addresssendtoaddress <fromaddr> <toaddr> <amount>
- sends money to a certain addressredeem <toaddr>
- redeems premineThe BLS library was recently updated to allow public keys in G1. Update our wrapper to use this implementation.
Also, data types need to be updated to reflect that the public key now takes 48 bytes and the signature takes 96 bytes.
The shard module will execute transactions and generate a new state root based on witness/transaction packages received from relayers.
This should be done similar to the beacon chain and we'll probably want to split the folders up so that we can run the shard and beacon chains more separately.
Currently, the max key is calculated on the fly in CSMT, but it really should be calculated when inserting.
On recursing back, on the way up all the hashes and max-key values along the minimum distance path are readjusted.
This way, we can do one single verify instead of three per-epoch.
Reproduce:
The Beacon node will eat up all RAM and freeze the OS.
Reproduce:
The Beacon node will return success.
Expect: Beacon should report error on wrong epoch.
Add latest block to BlockMessage. When a client receives a block message where the last block isn't the latest block, request more blocks by sending a GetBlockMessage with the current chain locator.
These data types are for shard storage.
Tree.set(key: uint256, value: uint256)
Tree.setWithWitness(key: uint256, value: uint256) → Witness
Tree.get(key: uint256) → value: uint256
Tree.delete(key: uint256)
Tree.deleteWithWitness(key: uint256) → Witness
Tree.proveKey(key: uint256) → Witness
Tree.hash() → uint256
PartialTrees are not really trees. They're simply the state root and witnesses. The witnesses allow certain parts of the tree to be updated or verified.
PartialTree.new(stateRoot: uint256, witnesses: \[Witness\])
PartialTree.set(key: uint256, value: uint256) → Error
PartialTree.delete(key: uint256) → Error
PartialTree.verify(key: uint256, value: uint256) → Error
PartialTree.hash() → (uint256, Error)
set
or verify
and calculates a new state root using the witnesses provided.Witnesses are trees that detail certain parts of the tree. Witnesses do not have to contain the whole tree, but just certain branches needed to update information about the tree or verify information about the tree. Partial trees only contain leaf nodes (values) if the verifier needs to verify a key, but not if they just need to update the key.
Witness.compress(witnesses: \[Witnesses\]) → Witness
tree = Tree()
tree.set(1, 1)
tree.set(2, 2)
tree.set(3, 3)
tree.set(4, 4)
hashTree0 = tree.hash()
witnesses = []
witnesses.append(tree.setWithWitness(1, 2))
witnesses.append(tree.setWithWitness(2, 3))
witnesses.append(tree.deleteWithWitness(3))
witnesses.append(tree.proveKey(4))
hashTree1 = tree.hash()
partialTree = PartialTree(hashTree0, witnesses)
partialTree.set(1, 2)
partialTree.set(2, 3)
partialTree.delete(3)
assert partialTree.verify(4, 4)
assert partialTree.hash() == hashTree1
partialTree.set(3, 4) # error: no witness for setting key 3
partialTree.delete(4) # error: no witness for deleting key 4
partialTree.verify(4, 5) # error: no witness proving key 4 is set to 5
This is a diagram to explain how updating/verifying keys should work.
Currently, the tree root can be calculated by calculating: (using x = 1
)
l0 = hash(x || 2)
l1 = hash(l0 || l0_sibling)
root = hash(l1 || l2_sibling)
We can prove 1 is in the root by verifying that when 1 is plugged in for x, the state root is returned.
We can also update the tree by changing 1 to another value, yielding a new state root.
The PartialTree
should also keep track of changed values which should take priority over proving in the tree. Because the state root will be different after modifying the tree at all, proving will have to use the original state root. Proving a value that was not modified will obviously verify. Proving a value that was modified will verify because the modified value is cached.
In the diagram above, the yellow node is the node to update/verify and the red nodes are the witnesses needed.
The witness tree should store witnesses on the opposite branch to where they actually are in the tree. In the example above, the witnesses are all on the right branch on the tree. The witness tree should actually store them on the left branches of the tree. (Hopefully this doesn't conflict with how a CSMT is defined).
Currently the folder 'blockchain' and the package 'blockchain' are used by beacon chain, and 'database' is used by beacon chain too.
If the shard chain is under the same name, it will be confusing.
So we may consider to use 'beacon' for beacon chain related stuff, and 'shard' for shard chain?
On the shard chain, we include proofs in each block proving the position and public key of a validator with a beacon chain block. The following conditions have to be verified in order to enforce validity of shard blocks:
ValidatorProof
.If an attacker wants to fake a shard chain block, they would have to fake the validity of the public key included in the proof. To do that, they would have to have a finalized block in the beacon chain with a fake public key included. To do that, they would have to finalize a beacon chain. So, any attack that passes these 3 checks must have faked a finalized beacon block and we assume that is secure.
To find the proposer of a shard block, we need an algorithm that ensures that we pick a proposer in the committee of the shard block and picks a different, randomized proposer for each block. The randomness is already taken care of because of the shuffling that happens with validators, so we'll just choose the n
'th validator in the committee to propose the n + EPOCH_LENGTH * m
where m
is the EpochIndex
block. So, the 0th validator proposes the block corresponding to EPOCH_LENGTH * m
, 1th to EPOCH_LENGTH * m + 1
, etc.
In the beacon validator proofs, we only need to keep track of EPOCH_LENGTH
validators per shard. So, for the validator proofs, we'll set the key as hash(shard + slot)
where shard
is the committee the validator is assigned and slot
is the slot the validator is assigned.
For syncing/propogation, the peers will send the client a block hash and the client will download the full block/transaction from IPFS. This improves interoperability.
This will prevent many types of DoS attacks that would be possible using Protobuf.
Reproduce:
The Beacon node will return success.
Expect: Beacon should report error because the request type is wrong (it should be GetValidatorRequest, but here is MempoolRequest).
This can be reproduced in Python since Python is dynamic type.
Currently, to run a local testnet, you run with regtest.json
with rootkey testnet
even though there is already a testnet.json
, but that's for a different purpose.
This all needs to be organized and documented.
Update github.com/btcsuite/btcutil to be Synapse compatible.
Specifically, do the following:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.