webb-tools/dkg-substrate

Multy-party threshold ECDSA (GG20) Substrate node

dkg-substrate's Issues

[TASK] Test offline timeout reporting when a node drops off the network.

[TASK] Test persistence and define the scope of the persistent storage data.

Overview

Persistence isn't working quite correctly as the system doesn't continue to refresh properly once the network has stopped and then started up again (locally at least). We haven't tested this with a single node directly also. Nonetheless, we should ensure that the persistence system spins down the right data in order to spin itself back up entirely to resume from its last point.

[SPEC] Encryption of relevant P2P messages

[BUG] Sign / keygen cache does not work

Description

Persistent DKG key material doesn't work as expected when using 3-node testnet with --tmp DBs (and likely when nodes are separated, which is untested). We should identify how to test this more deeply and decide on the expected behavior.

Steps to reproduce

1. Run the 3 node testnet, finish keygen.
2. Shut off a node and wait.
3. Turn the node back on.
4. Initiate anchor proposal sign script, proposal succeeds but never gets signed.

[TASK] E2E test that threshold-signature verifies against EVM contract (using docker)

[SPEC] Slashing conditions in multi-party-ecdsa

Overview

In order to report a misbehaviour we need to obtain and utilise proofs of this misbehaviour from underlying multi-party-ecdsa library (further MPE).

Research:

• Mapping between MPE parties and node authority ids.
  • Parties in MPE are identified by numbers from 1 to N inclusive. This identifiers are assigned at the start of Keygen process.
  • For party id we use authority index in authorities set.
    • Need to make sure the set has consistent order
• Proof that party i has violated the protocol
  • The MPE seems to only return a set of violating party ids, so we need to construct a proof that certain party indeed has violated the protocol.
  • One possible way to do so is by voting, similar to what we have in proposals right now. For this solution to work, the MPE’s state machine misbehaviour report must be consistent across all the nodes.
    • Check if MPE broadcasts misbehaviour with proofs to other parties, so that eventually every good party will have the same information about malicious parties.
    • Check if there is a possibility for malicious parties to manipulate others so good parties never agree reach voting threshold for slashing. E.g. malicious party sends corrupted messages to only a subset of parties while sending correct messages to others

[CHECKLIST] Slashing condition implementations

Checklist

• Slashing on keygen failure
• Slashing on sign failure

[TASK] Test misbehavior reporting when a signer drops off the network.

Overview

We need to be able to adequately report misbehaving peers. Some instances include but not limited to:

• If a peer is not participating we should produce a misbehaviour report
• If a peer does not sign a message another peer may report that peer's misbehaviour
• Node drops off, we should see misbehaviour of some capacity (e.g keygen timeout)

[TASK] Update readme to use dkg-standalone-node as the testing system

[SPEC] Continuous offline stage restarts

Since we intend to use the CompletedOfflineStage for one signature, we need a strategy to continuously recreate the offline stage. Also this is needed so that when a node that is part of the current dkg is restarted, it can rejoin the protocol and sign proposals.

Options

1. Creating a pool for CompletedofflineStage
   The difficult part here is how to make sure all nodes are using the exact same CompletedOfflineStage to sign
   proposals, especially when a node restarts thereby loosing all the previous CompletedOfflineStage they had stored in
   memory, if a node uses a different CompletedOfflineStage to sign, different from the one been used by majority nodes, this
   will result in a bad signature.

   A potential solution would be to create a new pool at intervals, so if a node drops off and comes back, they can wait until this
   interval before creating new pool of CompletedOfflineStage data, this way we might be able to maintain the same order of
   CompletedOfflineStage in the pool for each node.

2. Creating the CompletedofflineStage each time we need to sign a proposal
   This would be a very slow approach to siging proposals.

[TASK] Make DKG Signature EVM compatible

Related: #81

[TASK] Add the respective public key (next public key) to `NextPublicKeySignatureSubmitted` event.

[TASK] Test token (add, remove, set fee) call proposals to the DKG for Substrate.

Write a test in the dkg-test-suite for TokenAdd and TokenRemove call proposals. Review testAnchorProposal.ts for initial setup.

[TASK] Add substrate format config

Task

We want to add formatting script to the repo

• Add the format config from protocol-substrate to this repo.
• Add the format to the CI.
• Format the repo.

[TASK] QA the token update and wrapping fee update integration tests

Overview

The new integration tests need QA. The proposals are submitted but the response of the signed proposal wasn't always showing.

[BUG] `swap_remove index (is 4) should be < len (is 3)` while running offchain workers

Full stack-trace:

2022-01-12 16:30:42 Error running offchain workers at BlockId::Hash(0xb7e91242bc8f6114720f9f3eaaf2cb007103f86f33bd7e88ac855355d1bc9e0f): Application(Execution(RuntimePanicked("swap_remove index (is 4) should be < len (is 3)")))

====================

Version: 3.0.0-0eba6c4-x86_64-macos

   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: sp_panic_handler::set::{{closure}}
   3: std::panicking::rust_panic_with_hook
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/std/src/panicking.rs:610:17
   4: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/std/src/panicking.rs:502:13
   5: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/std/src/sys_common/backtrace.rs:139:18
   6: rust_begin_unwind
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/std/src/panicking.rs:498:5
   7: core::panicking::panic_fmt
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/core/src/panicking.rs:107:14
   8: alloc::vec::Vec<T,A>::swap_remove::assert_failed
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/alloc/src/vec/mod.rs:1300:13
   9: sp_runtime::offchain::storage::StorageValueRef::mutate
  10: pallet_dkg_proposal_handler::<impl pallet_dkg_proposal_handler::pallet::Pallet<T>>::get_next_offchain_signed_proposal
  11: pallet_dkg_proposal_handler::<impl pallet_dkg_proposal_handler::pallet::Pallet<T>>::submit_signed_proposal_onchain
  12: <(TupleElement0,TupleElement1) as frame_support::traits::misc::OffchainWorker<BlockNumber>>::offchain_worker
  13: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::offchain_worker
  14: sp_offchain::runtime_decl_for_OffchainWorkerApi::offchain_worker_native_call_generator::{{closure}}
  15: std::thread::local::LocalKey<T>::with
  16: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
  17: sc_executor::wasm_runtime::RuntimeCache::with_instance
  18: <sc_executor::native_executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  19: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
  20: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
  21: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  22: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  23: sp_offchain::runtime_decl_for_OffchainWorkerApi::offchain_worker_call_api_at
  24: <dkg_standalone_runtime::RuntimeApiImpl<__SR_API_BLOCK__,RuntimeApiImplCall> as sp_offchain::OffchainWorkerApi<__SR_API_BLOCK__>>::OffchainWorkerApi_offchain_worker_runtime_api_impl
  25: sp_offchain::OffchainWorkerApi::offchain_worker_with_context
  26: <F as threadpool::FnBox>::call_box
  27: std::sys_common::backtrace::__rust_begin_short_backtrace
  28: core::ops::function::FnOnce::call_once{{vtable.shim}}
  29: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/alloc/src/boxed.rs:1694:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/alloc/src/boxed.rs:1694:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/std/src/sys/unix/thread.rs:106:17
  30: __pthread_start


Thread 'offchain-worker' panicked at 'swap_remove index (is 4) should be < len (is 3)', library/alloc/src/vec/mod.rs:1300

This is a bug. Please report it at:

	support.anonymous.an

Full Debug data:

offchain-worker dkg_proposal_handler: Offchain signed proposals: [([AnchorUpdateSigned { data: [0, 0, 0, 0, 0, 0, 0, 0, 211, 12, 136, 57, 193, 20, 86, 9, 229, 100, 185, 134, 246, 103, 178, 115, 221, 203, 132, 150, 0, 0, 19, 138, 68, 52, 123, 169, 0, 0, 0, 0, 0, 0, 19, 137, 0, 0, 0, 0, 41, 14, 28, 191, 232, 139, 89, 149, 194, 218, 185, 239, 57, 237, 238, 49, 152, 128, 199, 203, 40, 43, 36, 29, 153, 110, 255, 232, 87, 51, 106, 67], signature: [208, 249, 87, 79, 26, 192, 146, 117, 91, 64, 201, 113, 170, 229, 206, 219, 177, 11, 243, 219, 86, 255, 112, 10, 121, 31, 128, 116, 16, 178, 44, 6, 89, 131, 151, 108, 140, 253, 93, 61, 43, 74, 214, 117, 58, 3, 135, 133, 34, 201, 156, 245, 111, 58, 169, 27, 40, 45, 140, 198, 245, 141, 225, 131, 1] }], 31), ([AnchorUpdateSigned { data: [0, 0, 0, 0, 0, 0, 0, 0, 211, 12, 136, 57, 193, 20, 86, 9, 229, 100, 185, 134, 246, 103, 178, 115, 221, 203, 132, 150, 0, 0, 19, 138, 68, 52, 123, 169, 0, 0, 0, 1, 0, 0, 19, 137, 0, 0, 0, 1, 41, 14, 28, 191, 232, 139, 89, 149, 194, 218, 185, 239, 57, 237, 238, 49, 152, 128, 199, 203, 40, 43, 36, 29, 153, 110, 255, 232, 87, 51, 106, 67], signature: [208, 249, 87, 79, 26, 192, 146, 117, 91, 64, 201, 113, 170, 229, 206, 219, 177, 11, 243, 219, 86, 255, 112, 10, 121, 31, 128, 116, 16, 178, 44, 6, 62, 153, 26, 30, 93, 18, 141, 211, 34, 198, 168, 121, 131, 232, 171, 39, 9, 78, 195, 170, 98, 160, 204, 197, 58, 172, 44, 215, 41, 154, 108, 191, 0] }], 31), ([AnchorUpdateSigned { data: [0, 0, 0, 0, 0, 0, 0, 0, 211, 12, 136, 57, 193, 20, 86, 9, 229, 100, 185, 134, 246, 103, 178, 115, 221, 203, 132, 150, 0, 0, 19, 138, 68, 52, 123, 169, 0, 0, 0, 0, 0, 0, 19, 137, 0, 0, 0, 0, 41, 14, 28, 191, 232, 139, 89, 149, 194, 218, 185, 239, 57, 237, 238, 49, 152, 128, 199, 203, 40, 43, 36, 29, 153, 110, 255, 232, 87, 51, 106, 67], signature: [208, 249, 87, 79, 26, 192, 146, 117, 91, 64, 201, 113, 170, 229, 206, 219, 177, 11, 243, 219, 86, 255, 112, 10, 121, 31, 128, 116, 16, 178, 44, 6, 89, 131, 151, 108, 140, 253, 93, 61, 43, 74, 214, 117, 58, 3, 135, 133, 34, 201, 156, 245, 111, 58, 169, 27, 40, 45, 140, 198, 245, 141, 225, 131, 1] }], 42), ([AnchorUpdateSigned { data: [0, 0, 0, 0, 0, 0, 0, 0, 211, 12, 136, 57, 193, 20, 86, 9, 229, 100, 185, 134, 246, 103, 178, 115, 221, 203, 132, 150, 0, 0, 19, 138, 68, 52, 123, 169, 0, 0, 0, 1, 0, 0, 19, 137, 0, 0, 0, 1, 41, 14, 28, 191, 232, 139, 89, 149, 194, 218, 185, 239, 57, 237, 238, 49, 152, 128, 199, 203, 40, 43, 36, 29, 153, 110, 255, 232, 87, 51, 106, 67], signature: [208, 249, 87, 79, 26, 192, 146, 117, 91, 64, 201, 113, 170, 229, 206, 219, 177, 11, 243, 219, 86, 255, 112, 10, 121, 31, 128, 116, 16, 178, 44, 6, 62, 153, 26, 30, 93, 18, 141, 211, 34, 198, 168, 121, 131, 232, 171, 39, 9, 78, 195, 170, 98, 160, 204, 197, 58, 172, 44, 215, 41, 154, 108, 191, 0] }], 42), ([AnchorUpdateSigned { data: [0, 0, 0, 0, 0, 0, 0, 0, 211, 12, 136, 57, 193, 20, 86, 9, 229, 100, 185, 134, 246, 103, 178, 115, 221, 203, 132, 150, 0, 0, 19, 138, 68, 52, 123, 169, 0, 0, 0, 0, 0, 0, 19, 137, 0, 0, 0, 0, 41, 14, 28, 191, 232, 139, 89, 149, 194, 218, 185, 239, 57, 237, 238, 49, 152, 128, 199, 203, 40, 43, 36, 29, 153, 110, 255, 232, 87, 51, 106, 67], signature: [208, 249, 87, 79, 26, 192, 146, 117, 91, 64, 201, 113, 170, 229, 206, 219, 177, 11, 243, 219, 86, 255, 112, 10, 121, 31, 128, 116, 16, 178, 44, 6, 89, 131, 151, 108, 140, 253, 93, 61, 43, 74, 214, 117, 58, 3, 135, 133, 34, 201, 156, 245, 111, 58, 169, 27, 40, 45, 140, 198, 245, 141, 225, 131, 1] }], 53), ([AnchorUpdateSigned { data: [0, 0, 0, 0, 0, 0, 0, 0, 211, 12, 136, 57, 193, 20, 86, 9, 229, 100, 185, 134, 246, 103, 178, 115, 221, 203, 132, 150, 0, 0, 19, 138, 68, 52, 123, 169, 0, 0, 0, 0, 0, 0, 19, 137, 0, 0, 0, 0, 41, 14, 28, 191, 232, 139, 89, 149, 194, 218, 185, 239, 57, 237, 238, 49, 152, 128, 199, 203, 40, 43, 36, 29, 153, 110, 255, 232, 87, 51, 106, 67], signature: [208, 249, 87, 79, 26, 192, 146, 117, 91, 64, 201, 113, 170, 229, 206, 219, 177, 11, 243, 219, 86, 255, 112, 10, 121, 31, 128, 116, 16, 178, 44, 6, 89, 131, 151, 108, 140, 253, 93, 61, 43, 74, 214, 117, 58, 3, 135, 133, 34, 201, 156, 245, 111, 58, 169, 27, 40, 45, 140, 198, 245, 141, 225, 131, 1] }], 64), ([AnchorUpdateSigned { data: [0, 0, 0, 0, 0, 0, 0, 0, 211, 12, 136, 57, 193, 20, 86, 9, 229, 100, 185, 134, 246, 103, 178, 115, 221, 203, 132, 150, 0, 0, 19, 138, 68, 52, 123, 169, 0, 0, 0, 0, 0, 0, 19, 137, 0, 0, 0, 0, 41, 14, 28, 191, 232, 139, 89, 149, 194, 218, 185, 239, 57, 237, 238, 49, 152, 128, 199, 203, 40, 43, 36, 29, 153, 110, 255, 232, 87, 51, 106, 67], signature: [208, 249, 87, 79, 26, 192, 146, 117, 91, 64, 201, 113, 170, 229, 206, 219, 177, 11, 243, 219, 86, 255, 112, 10, 121, 31, 128, 116, 16, 178, 44, 6, 89, 131, 151, 108, 140, 253, 93, 61, 43, 74, 214, 117, 58, 3, 135, 133, 34, 201, 156, 245, 111, 58, 169, 27, 40, 45, 140, 198, 245, 141, 225, 131, 1] }], 75)]
2022-01-12 16:30:42.107 DEBUG      offchain-worker dkg_proposal_handler: Offchain signed proposals left: 7

To trigger this bug, I guess you have to add a lot of unsigned proposals really quickly to the dkg system, and it triggers this error when it try to submit the signed proposals back on-chain.

[BUG] Chain Nonces never gets updated after a Proposal Succeed

Bug Description

ChainNonces which is a Map between ChainId to ProposalNonces is used in two cases:

1. To check if chain_id is whitelisted or not, by checking that chain_id exists in this map.
2. To track how many transactions are executed for this chain.

Right now, the first one is good, but the second one is not implemented anywhere when the proposal gets executed here:

dkg-substrate/pallets/dkg-proposals/src/lib.rs

Lines 642 to 655 in 16ea8dc

	/// Execute the proposal and signals the result as an event
	fn finalize_execution(
	src_id: T::ChainId,
	nonce: ProposalNonce,
	prop: T::Proposal,
	) -> DispatchResultWithPostInfo {
	Self::deposit_event(Event::ProposalApproved { chain_id: src_id, proposal_nonce: nonce });
	T::ProposalHandler::handle_unsigned_proposal(
	prop.into(),
	dkg_runtime_primitives::ProposalAction::Sign(0),
	)?;
	Self::deposit_event(Event::ProposalSucceeded { chain_id: src_id, proposal_nonce: nonce });
	Ok(().into())
	}

Suggested Solution
Before Emitting the final event here:

+		ChainNonces::<T>::insert(&src_id, nonce);

Motivation and use-cases

Right now the Relayer needs this to check if we shall send this proposal to the DKG System or not by checking the proposal nonce on the DKG side and the new proposal we have in hand right now and compare the nonces.

[TASK] Test that signers rotate each session.

Overview

• Run the node, output the current signing set, fast-forward to next session, output the next signing set
• Repeat and ensure they change often
• Include in dkg-test-suite

[TASK] E2E test that anchor update verifies on Signature bridge

Overview

For the E2E setup we should have a docker system set up for both DKGs and a local EVM chain. The EVM chain should have the right smart contracts deployed to it and can leverage the libraries we've written on protocol-solidity side.

[TASK] Test anchor update call proposal to the DKG for Substrate.

Write a test in the dkg-test-suite.

[BUG] Standalone node polkadotjs apps UI types issue

Overview

This issue was found while working on a testnet deployment on branch nathan/dkg-testnet.

3 Validator nodes were spun up and are able to communicate and produce blocks at endpoints:

• wss://testnet1.webb.tools
• wss://testnet2.webb.tools
• wss://testnet3.webb.tools

Observed Behavior

While attempting some queries on the dkg pallet through the apps UI, the UI will hit an error. The below error is shown when attempting to select the dkgPublicKey() option under dkg pallet for chain state queries:

Note: some chain state queries such as authorities() work as expected, but the above dkgPublicKey() errors.

Expected Behavior

The DKG public key should be queried and presented on the apps UI.

[TASK] Test set resource ID proposal to the DKG for Substrate.

[TASK] E2E test that add token update verifies on Signature bridge

[BUG] Standalone non-validator node requires keystore to sync blocks

Overview

This issue was found while working on a testnet deployment on branch nathan/dkg-testnet.

Validator nodes were added to the chainspec in /chainspecs/arana_testnet_raw.json. These nodes were populated with the appropriate Aura, Grandpa, and DKG keys. These validator nodes are able to spin up and communicate with each other to produce blocks. These nodes were started like so:

• Webb-dkg-1:

./target/release/dkg-standalone-node --chain=chainspecs/arana_testnet_raw.json --port 30333 --ws-port 9944 --rpc-port 9933 --node-key <key> --validator --rpc-cors all

• Webb-dkg-2:

./target/release/dkg-standalone-node --chain=chainspecs/arana_testnet_raw.json --port 30333 --ws-port 9944 --rpc-port 9933 --node-key <key> --validator --rpc-cors all

• Webb-dkg-3:

./target/release/dkg-standalone-node --chain=chainspecs/arana_testnet_raw.json --port 30333 --ws-port 9944 --rpc-port 9933 --node-key <key> --validator --rpc-cors all

On another machine, I attempt to spin up a non-validator node with the command:

./target/release/dkg-standalone-node --tmp --chain=chainspecs/arana_testnet_raw.json

Observed behavior

The non-validator node fails to startup:

2022-01-10 09:11:09 Substrate Node
2022-01-10 09:11:09 ✌️  version 3.0.0-c9a8690-aarch64-macos
2022-01-10 09:11:09 ❤️  by Webb tools, 2017-2022
2022-01-10 09:11:09 📋 Chain specification: Arana
2022-01-10 09:11:09 🏷  Node name: clear-jump-0122
2022-01-10 09:11:09 👤 Role: FULL
2022-01-10 09:11:09 💾 Database: RocksDb at /var/folders/4h/l9xvh27j7456vgxr6kzq5cxc0000gn/T/substratepx6NdG/chains/arana/db/full
2022-01-10 09:11:09 ⛓  Native runtime: dkg-standalone-node-100 (dkg-standalone-node-1.tx1.au1)
2022-01-10 09:11:09 🔨 Initializing Genesis block/state (state: 0x107f…d534, header-hash: 0xa80e…df65)
2022-01-10 09:11:09 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
2022-01-10 09:11:09 Using default protocol ID "sup" because none is configured in the chain specs
2022-01-10 09:11:09 🏷  Local node identity is: 12D3KooWGLDF7eo2Remk2Vyp1PYnKpgiuv9VCQkEQQ72ZzeXgw5j
2022-01-10 09:11:09 📦 Highest known block at #0
2022-01-10 09:11:09 〽️ Prometheus exporter started at 127.0.0.1:9615
2022-01-10 09:11:09 Listening for new connections on 127.0.0.1:9944.
2022-01-10 09:11:10 🔍 Discovered new external address for our node: /ip4/73.244.238.57/tcp/30333/ws/p2p/12D3KooWGLDF7eo2Remk2Vyp1PYnKpgiuv9VCQkEQQ72ZzeXgw5j
2022-01-10 09:11:11 [150] 💸 generated 3 npos voters, 3 from validators and 0 nominators
2022-01-10 09:11:11 [#150] 🗳  creating a snapshot with metadata SolutionOrSnapshotSize { voters: 3, targets: 3 }
2022-01-10 09:11:11 [#150] 🗳  Starting signed phase round 1.
2022-01-10 09:11:11 [#200] 🗳  Starting unsigned phase round 1 enabled true.
2022-01-10 09:11:11 [#201] 🗳  queued unsigned solution with score [3333333333333333, 9999999999999999, 33333333333333326666666666666667]
2022-01-10 09:11:11 [250] 💸 new validator set of size 3 has been processed for era 1

====================

Version: 3.0.0-c9a8690-aarch64-macos

   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: sp_panic_handler::set::{{closure}}
   3: std::panicking::rust_panic_with_hook
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/std/src/panicking.rs:610:17
   4: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/std/src/panicking.rs:502:13
   5: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/std/src/sys_common/backtrace.rs:139:18
   6: rust_begin_unwind
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/std/src/panicking.rs:498:5
   7: core::panicking::panic_fmt
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/core/src/panicking.rs:107:14
   8: core::result::unwrap_failed
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/core/src/result.rs:1613:5
   9: dkg_gadget::worker::DKGWorker<B,C,BE>::process_block_notification
  10: dkg_gadget::worker::DKGWorker<B,C,BE>::handle_finality_notification
  11: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  12: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  13: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  14: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  15: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  16: std::thread::local::LocalKey<T>::with
  17: tokio::park::thread::CachedParkThread::block_on
  18: tokio::runtime::handle::Handle::block_on
  19: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  20: std::panicking::try
  21: tokio::runtime::task::harness::Harness<T,S>::poll
  22: tokio::runtime::blocking::pool::Inner::run
  23: std::sys_common::backtrace::__rust_begin_short_backtrace
  24: core::ops::function::FnOnce::call_once{{vtable.shim}}
  25: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/alloc/src/boxed.rs:1694:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/alloc/src/boxed.rs:1694:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/a77da2d454e6caa227a85b16410b95f93495e7e0/library/std/src/sys/unix/thread.rs:106:17
  26: _pthread_from_mach_thread_np


Thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Keystore("no Keystore")', /Users/nathanbarnavon/workspace/dkg-substrate/dkg-gadget/src/persistence.rs:63

This is a bug. Please report it at:

	support.anonymous.an

Expected Behavior

Non-validator nodes should not require a keystore to sync blocks.

[CHECKLIST] Error and misbehavior reporting

Overview

The goal of this checklist is to document all tasks relevant for implementing and testing error misbehaviors.

Checklist

• #94
• #95

[TASK] Submitting a batch of signed proposals per block

Overview

We want to submit a batch of signed proposals if there are more than 1. We do however want to ensure that this submission is properly weighted.

// choose weight dependent on `props.len()`
pub fn submit_signed_proposal(
  origin: OriginFor<T>,
  props: Vec<ProposalType>,
) -> DispatchResultWithPostInfo {

Checklist

• Implement batched submissions
• Implement benchmarking that is dependent on the # of submissions.

[TASK] E2E test that change wrapping fee update verifies on Signature bridge

[TASK] Fix deserialization of structs from mp-ecdsa update

Consider using protocol buffers / JSON and define the set of tasks necessary for completion @temld4.

[BUG] Signature Generated from DKG Fails to verify on EVM

Example Data

DKG PublicKey: 0x0232ff45c7166e958b73f00fd6519ddc598751cd5227efdf37be2aa74686b11030 (Compressed)
DKG PublicKey: 0x0432ff45c7166e958b73f00fd6519ddc598751cd5227efdf37be2aa74686b1103077cf0f24be635e453409745b51eb28bd4cc6247621cd4ecb7726a7ee2b333e6e (Uncompressed)
Governer Owner: 0x85157Bc12E1dc99E628796a468b61F450720BdC5 (it is drived from the publickey).

Proposal Data: 0x00000000000000006d5a4d246617d711595a1657c55b17b97e20bdda0000138a44347ba90000000000001389000000001d60a4821084001cc6bc8614cd04c14c5dc6dfcb4ec77d404029da58ba31ad0f

Signature: 0xfd3ea1f634196d1cbfc5e8b8b90875d3ffa60645d3c84c7fc4cd5bc8c1cc3a824c737ae4e59d8dbdf1001f15f62856922a6731f18f4c0cd45b0f659bb3a43fbd00

Try it online: Remix IDE

[TASK] Add a proposal for setting new resource IDs on Solidity / external contracting systems

[BUG] Refresh vote fails once ready to refresh

I've added extra logs in the branch: drew/rotate-on-session. Currently, the refresh will attempt to start but it fails. We are likely not preparing offline stages for refreshes and should do so immediately on genesis (store an offline stage for the next refresh) as well as after a successful refresh.

2022-01-18 01:53:21.036 DEBUG tokio-runtime-worker dkg: Check refresh (in progress: false, rounds: false)    
2022-01-18 01:53:21.037 DEBUG tokio-runtime-worker dkg: Refresh state (delay: Permill(900000), progress: Permill(900000))    
2022-01-18 01:53:21.037 DEBUG tokio-runtime-worker dkg: Should refresh key for at block_num 94    
2022-01-18 01:53:21.037 ERROR tokio-runtime-worker dkg: 🕸️  error creating new vote: Not ready to vote

[CHECKLIST] E2E SignatureBridge Integration

Overview

The goal of this checklist is to document tasks needed to complete to take us to an E2E SignatureBridge integration.

Checklist

Functionality

• #41
• #79
• #96
• #111
• [SPEC] Validation for Substrate signed call proposals on the pallet-signature-bridge.
• #114

Testing

• #80
• #97
• #98
• #99
• #100
• #101
• #102
• #103
• #104
• [TASK] Write test utilities for creating Substrate proposal structure.
• #108

Process

• #90
• #91
• #105

Relayer

• webb-tools/relayer#92
• webb-tools/relayer#93
• webb-tools/relayer#94
• webb-tools/relayer#137

[TASK] Add logging to see the signing process in action.

We should have better logging of keygen and signing for testing purposes and readability about different actions taking place. Some of these are:

• Seeing the active group public key generated from keygen of current authorities
• Seeing the next group public key generated from keygen of next authorities
• Seeing the refresh process in action.

Ideally we have this possible on a separate logging target so that the large DKG message spam doesn't get shown.

[TASK] Implement testing for refresh keys

[TASK] Test replaying txes on EVM to ensure failure.

Overview

• Ensure values tampered with are not executable

[TASK] Add a proposal for removing a token

[SPEC] Investigate best line of action if key refresh fails

If next public key signature is not found on chain at session change, Some availabe options that would ensure the protocol keeps running is to

1. Retain current authorities and current public key and do not signal an authority set change.
   This way we can ensure the dkg system can still sign proposals.
   Slashing the incoming set whose key was not found onchain
2. Have a set of emergency DKG authorities that have an emergency publickey and signature on chain and rotate the
   authorities to that instead.

[SPEC] Persistent DKG node restarts

Overview

We want the node to be able to continue to participate in DKG after restart. Currently generated keys are stored in memory. We want to store generated keys on disk and read them upon node startup. The file contents are updated during state transitions.

Questions

• Should we save current votes data or just discard it in case of the node restart?

Checklist

• When the node starts:
  • Check if there is a file with DKG data.
    1. If the file is missing
       • Try to start a new DKG process
    2. If the file is present
       • Re-create the DKG state from that file
• When Keygen process completes:
  • Check if there is a file with DKG data.
    1. If the file is missing
       • Create a new file
    2. If the file is present
       • Delete old file (since we have generated new keys, we assume the old DKG epoch is over)
       • Create a new file
• When OfflineStage process completes:
  • Check if there is a file with DKG data.
    1. If the file is missing
       • Report the error
       • Try to start a new DKG process
    2. If the file is present and does not contain OfflineStage
       • Update file with OfflineStage info
    3. If the file is present and contains OfflineStage
       • Update file with OfflineStage overriding the old OfflineStage data (we assume the signers set changed, so we discard the old OfflineStage data)
• If at any read attempt the file is corrupted:
  • Report the error
  • Delete the file
  • Try start a new DKG process

[SPEC] Message buffering in DKG to prevent initialisation bugs

Overview

Upon a new authorities set notification the nodes start a new DKG process. We cannot guarantee a perfect simultaneous startup so in the current implementation some nodes might miss initial messages and be unable to further participate in the protocol.

Checklist

On every new DKG message:

• Iterate through the buffer and perform an action for each message. An action can be:
  • Feed the message to the corresponding rounds object if possible
  • Do nothing if the message is not considered expired yet.
  • Discard the message if it is expired
• Try to feed newly received message to the corresponding rounds object
• If there is no rounds object available, append the message to the buffer
• Define en expiration policy for buffered messages.

Issues

Linked tasks

[TASK] Update docker for DKG standalone node

[TASK] Rework the DKG submission delay

The offchain submission delay is defined to be:

let delay = (block_number + 1u32.into()) + (max_delay % party_inx.into());

Problem with this delay:

• At high block numbers this just continues to increase
• Party index of 1 gets larger delay than party index > 1 with increasing difference.

What we should expect from the delay:

• What values do we want the delay to take?
• What is reasonable for offchain submission?

[BUG] No `DKGPublicKeySignature` value stored unless the authorities get changed

Description

When you run the node for the first time and wait until the DKGPublicKey gets submitted on-chain, and now when you query for the DKGPublicKeySignature corresponds to this public key you will find it empty.

We need this Signature to be available too on-chain, so other tools (like the relayer) on anyone actually be able to update the SignatureBridge governer through calling transferOwnershipWithSignaturePubKey(bytes memory publicKey, uint32 nonce, bytes memory sig) and without the signature being on-chain then there is no way we can update it.

[TASK] E2E test that remove token update verifies on Signature bridge

[TASK] Ensure on each session a random set of signers is selected.

Overview

We want the signers to update after each session who may take part in signing arbitrary proposals and also key refresh proposals (public key signature submissions). This will impact rewards when we have them and allow different parties to partake in the incentive game across sessions even if the authorities don't change. We will choose t out-of n signers and t<n, so that there will be a different t ideally after each key refresh.

Tasks

• Ensure that signers are rotating after session/era/authority changes.
• Write an integration test to verify this across sessions/eras.

[TASK] Add round # to refresh proposal (public key signature) and verify increasing nonce on-chain

[TASK] Update to the latest Substrate commit (Standalone & Parachain Node)

This is so important since the relayer (uses subxt) expects the node to be running on the updated substrate.

[SPEC] Integrate stake based relayer selection

The task should be modeled similarly to the collator selection pallet.

[TASK] Add code coverage to testing and CI.

[TASK] Improve logging of the DKG

Overview

We should have better logging of keygen and signing for testing purposes and readability about different actions taking place. Some of these are:

• Seeing the active group public key generated from keygen of current authorities
• Seeing the next group public key generated from keygen of next authorities
• Seeing the refresh process in action.
• #89

Ideally we have this possible on a separate logging target so that the large DKG message spam doesn't get shown.

[SPEC] DKG Gadgets Refactoring

Rounds (dkg-primitives/src/rounds.rs)

• Split into smaller modules
  • Traits for Keygen and Sign
  • Move Keygen and Sign impl into separate file
• Improve the code readability and add comments when possible.
  • User Builder Patteren for the MultiPartyECDSARounds, see the typed_builder crate.
• Refactor and run Clippy over the code.