paritytech/polkadot-sdk

The Parity Polkadot Blockchain SDK

Polkadot SDK

  

    

  

The Polkadot SDK repository provides all the components needed to start building on the Polkadot network, a multi-chain blockchain platform that enables different blockchains to interoperate and share information in a secure and scalable way.

📚 Documentation

• 🦀 rust-docs
• Introduction to each component of the Polkadot SDK: Substrate, FRAME, Cumulus, and XCM
• Other Resources:
  • Polkadot Wiki -> Build

🚀 Releases

Note

Our release process is still Work-In-Progress and may not yet reflect the aspired outline here.

The Polkadot-SDK has two release channels: stable and nightly. Production software is advised to only use stable. nightly is meant for tinkerers to try out the latest features. The detailed release process is described in RELEASE.md.

😌 Stable

stable releases have a support duration of three months. In this period, the release will not have any breaking changes. It will receive bug fixes, security fixes, performance fixes and new non-breaking features on a two week cadence.

🤠 Nightly

nightly releases are released every night from the master branch, potentially with breaking changes. They have pre-release version numbers in the format major.0.0-nightlyYYMMDD.

🔐 Security

The security policy and procedures can be found in docs/contributor/SECURITY.md.

🤍 Contributing & Code of Conduct

Ensure you follow our contribution guidelines. In every interaction and contribution, this project adheres to the Contributor Covenant Code of Conduct.

👾 Ready to Contribute?

Take a look at the issues labeled with mentor (or alternatively this page, created by one of the maintainers) label to get started! We always recognize valuable contributions by proposing an on-chain tip to the Polkadot network as a token of our appreciation.

Polkadot Fellowship

Development in this repo usually goes hand in hand with the fellowship organization. In short, this repository provides all the SDK pieces needed to build both Polkadot and its parachains. But, the actual Polkadot runtime lives in the fellowship/runtimes repository. Read more about the fellowship, this separation, the RFC process here.

History

This repository is the amalgamation of 3 separate repositories that used to make up Polkadot SDK, namely Substrate, Polkadot and Cumulus. Read more about the merge and its history here.

polkadot-sdk's Issues

Identity: Remove registrar

There currently doesn't seem to be a way to remove a registrar in charge of verifying identities. If a registrar turns out to be malicious, gets compromised, or simply stops providing this service, there needs to be a way to both:

1. Remove the registrar from the list, probably solvable with a simple council vote
2. Treat the identities that registrar verified in a certain way.

Specifically, does the registrar's behavior influence past actions somehow, like a tainted judge throwing shade on all cases they presided over? Do identities stop being valid if the registrar the verified them is found of having approved Sybil accounts? Do they get flagged as having been validated by a now-defunct registrar and it's up to the community to get further information themselves? Do the identities keep their full status as if nothing has changed? Do they lose that verification?

CLI for disabling the gossip service

Depends on paritytech/substrate#459

Gossip includes all full nodes, but some should be able to opt-out and accept no messages. Could include an extension of the network protocol to broadcast a "non-gossip" flag to peers to signal that they shouldn't send messages, rather than simply ignoring the ones peers send.

Unused `Proxy` storage item in Pallet Elections

At the moment there is an unused Proxy storage item in pallet-elections and as a result, an "uncallable" extrinsic: proxy_set_approvals.

It seems we are missing a set_proxy and remove_proxy extrinsic.

As the storage item currently exists, it seems that a single account can have multiple proxies set for it (map from proxy => voter :: many => one). So a user should have to pay a deposit for each proxy they set for their account. They can reclaim their deposit by calling remove_proxy.

@gavofyork to clarify if this is the right direction, or if we have other plans.

[Feature request] pruning mode: "since"

Currently, two pruning modes are available: archive and "keep X recent blocks."

I propose a third which would sync from block X onwards. Useful for dapps running their own nodes, but which know they've only existed since block X and as such only need archive data from that point on.

polkadot --pruning 120350+

The addition of a + at the end of the block number would indicate "from this block onwards".

Verify that addresses reported by identify are correct before inserting them in the DHT

Right now nodes tell us the addresses they are reachable from, and we immediately insert them in the DHT without any verification. This leads to a lot of unreachable addresses in the DHT, such as 127.0.0.1 or IPs behind NATs.
Instead, we should first attempt to connect to these addresses to see if they're reachable.

Note that connectivity is not transitive. For example if a node A is listening on a given address, this address might be reachable by B but not by C. If we implement this change, then C will not tell B about that address that A has, even though it would have been desirable. This is considered an acceptable trade-off.

On the implementation side, the good news is that Kademlia should automatically detect us reaching the node and add the address we connected to to the DHT, without having to call any method (although I'm not sure that this is the case, but I think that it should be the right logic).

In other words, all we have to do is call Swarm::dial for each address that a node reports through identity.

However, this is complicated by the "one connection per node" policy. If we connect to a node we're already connected to, we're going to drop the existing connection.
In other words, we first have to land libp2p/rust-libp2p#1440

cc @romanb @arkpar

Track potential untracked removals in Offences

I don't see where those prune removal are added into weight:
https://github.com/paritytech/substrate/blob/a67dcfb281ee54f3bd9a4fc217134d7f7da0e9ee/frame/staking/src/slashing.rs#L529-L542

Should we consider them negligible? or do I miss something ?

Originally posted by @thiolliere in paritytech/substrate#5961 (comment)

Write a test to demonstrate multisig reentrancy attack

Basically a test for paritytech/substrate#6445; will require a pallet with interior dispatch mutability in order to call the cancel_as_multi from the as_multi dispatcher.

Exploit

When starting a multisig transaction with threshold >= 2, a certain amount of your balance will be reserved as a deposit (and unreserved once the multisig transaction is executed or cancelled). However, the implementation of as_multi and cancel_as_multi has no protection against re-entrance. So if the call argument to as_multi somehow triggers a cancel_as_multi for the same multisig transaction, the deposit will be unreserved twice. With normal pallets this is not possible since the cancel_as_multi call will need the call_hash argument and this call_hash must be a hash of an encoded call to cancel_as_multi with the same call_hash, which is computationally infeasible to find assuming that the hash function (blake2_256) is cryptographically secure.

However, the multisig call could call a smart contract, which will load the call_hash and timepoint arguments from its storage and then call cancel_as_multi via env.dispatch_call.

Exploit outline:

• Create two account ids (account1 and account2) with sufficient funds to pay transaction fees as required (account2 may also be a sub-account of account1 so that only one funded account is required)
• Calculate multisig account id for these two accounts with threshold=2
• As account1: Do some operations (in other pallets) which legitimately require reserving a deposit
• Give the multisig account proxy permission (ProxyType::Any) to access account1
• As account1: Create a smart contract which allows calling cancel_as_multi with the parameters timepoint and call_hash stored in the storage of the smart contract
• As account1: Start a multisig transaction via approve_as_multi. This will reserve some balance of account1 as a deposit. The call_hash corresponds to the following call: as_proxy(account1) => contracts.call.
• Get the timepoint and call_hash parameters from the multisig transaction and store these values in the storage of the smart contract.
• As account2 trigger the multisig transaction by calling as_multi
• During the call from as_multi to call.dispatch, the smart contract is executed (via as_proxy(account1)). The smart contract takes the timepoint and call_hash parameters from storage and uses that to call cancel_as_multi for the very same multisig transaction, which will unreserve the deposit reserved for the multisig transaction
• After dispatching the call, as_multi will unreserve the same deposit a second time. By repeating the process over and over again, an attacker can unreserve an arbitrary amount of funds reserved by other pallets for a different reason.

Make the notification protocols handshake message configurable through the API

At the moment, the handshake is hard-coded to be empty. After paritytech/substrate#5665, it is hard-coded to be a single byte containing the Role of the node.

The intent, however, is to make this configurable by whatever code has registered a notifications protocol.

The difficult part in this is to figure out a good API for this.
I also don't think it's possible before paritytech/substrate#5670, because it is possible for the legacy substream to be opened before the notifications substream.

Make Origin accessible in weight calculation

Currently, inside the weight calculation attribute #[weight = { ... }] one has access to the arguments of the annotated function with the exception of Origin. It is conceivable that having that available comes in handy.

Sync should support fetching and importing long forks

(outdated, kept for posterity)

For GRANDPA, Casper CBC, and other chain-based consensus algorithms, we may occasionally need to download many forks which are typically small and typically not the best.

Ideally this will be via an API where it will be possible to tell the sync service which forks need downloading.

---

For 1.0 Gamma we added an announce_block function to NetworkService for GRANDPA nodes to announce blocks that they voted on to peers. GRANDPA voters additionally will announce periodically until the current round completes. The NetworkService announcement logic was changed to download and repropagate announcements that are for small (32 block) forks.

---

Further down the line we will need synchronization for arbitrarily long forks, hopefully with state pruning allowing us to keep both heads' states. Forks that connect before the finality point should be ignored.

sc_network::NetworkWorker::event_streams should be a bounded channel

Right now if for example a sc_network_gossip::GossipEngine doesn't process notifications quickly enough, we will continue buffering notifications forever in its events stream.

Instead, we should considering limiting the events stream to a certain size. If the channel is full, we would freeze the network worker itself, with a certain timeout after which the events stream is dropped and an error printed in the logs.

The first step to do so would be to turn impl Future for NetworkWorker into an async method.

PVF determinism

As the time of writing, polkadot validation function execution is performed using the same executor that is used for executing Substrate runtimes. Substrate executor doesn't guarantee non-determinism of execution, or more precise it doesn't provide more than base guarantees of wasm execution. If the chain-writer manages to exploit these then it's their fault.

We would like to eradicate these effects for the execution of PVF.

We should come up with a way on how to avoid that. (See this discussion for more details)

There are following sources of non-determinism:

1. Syntactic limits. Things like, the maximum size of a function body, the maximum number of imports, etc. This can be checked at the loading time.
2. Binary limits. For example, the size of a module as a whole or one of its sections. This can be checked at the loading time.
3. Validation limits. Validation of wasm binary can be performed eagerly (i.e. at loading time) or lazy (i.e. a function validated just before execution). So far, in our practice, we employed eager validation.
4. Execution. This consists of execution limits and permissible non-determinism of individual instructions.

Let's break down the point about execution in detail.

Execution limits are specified here. These limits are specified in abstract terms of the specification. Practically, they could be divided on two groups: memory-limited and stack-limited. The size of linear memory and tables can be trivially limited, even statically. The stack-limited parameters, at least in general situation (e.g. without limiting recursion, etc), require us dynamic counting.

There are some instructions that are non-deterministic. This is a local non-determinism, meaning that there are some instructions that have not a single outcome but rather a finite set of outcomes for the same arguments. Theoretically, a wasm VM implementation that chooses the result from a set of the possible outcomes is still deemed as a conforming implementation.

The following is the list of such instructions:

1. memory.grow, i.e. trying to increase the size of the linear memory. This is easily mitigated by limiting the maximal size of the memory or by counting.
2. invoking a host function. Host functions are not limited what changes they do to the wasm universe as long as these changes are valid (e.g. host functions cannot change typing of another function).
3. numeric operations. AFAIK, this only concerns floating point numbers.

Scheduler for off-chain workers execution

Offchain workshop follow up task.

There is a common pattern emerging in the way we run offchain workers. Usually we want them to be triggered:

1. After a block number set in the storage
2. Only once after that threshold,
3. But retry after the previous run does not seem to complete.
4. Randomly for different validators, etc

That could be easily made into a re-usable piece of code that would schedule the runs. Rough (and incomplete) proposal from @kianenigma

fn offchain_worker() {
	// simplest case. It basically adds a `should_run` impl that always return true
	let OffchainLimit::default()::build();

	// Run every N block
	let OffchainLimit::default()::with_interval(10)

	// We can also implement maybe your idea of 
	let OffchainLimit::default().with_interval(10).randomise(2)

        /// after some point in time
	let OffchainLimit::after(Storage::get(()).with_interval(10).randomise(2)

	// Any more attributes. I really think the idea of _composability_ can be very neat here, which
	// the builder pattern does very well.

Switch `ValidateUnsigned` over to `SignedExtension`

Currently to validate an unsigned transaction we require to implement ValidateUnsigned and need to register the unsigned validation in construct_runtime!. We should remove all this functionality. The validation of an unsigned transaction should be done by a signed extension. By default, signed extension should deny any unsigned transaction. Only if there is a SignedExtension that validates an unsigned transaction and approves it, we should approve it to the calling context.

As validating an unsigned transaction is in most cases always a very similar operation, we should provide an extendable SignedExtension that can be used by any module to validate its unsigned transactions.

1. Implement the new functionality into SignedExtension to validate unsigned transactions.
2. Remove ValidateUnsigned and all the related functionality and move all the code over to SignedExtension.
3. Write the generic SignedExtension for validating unsigned transactions.

All 3 points should be do-able in separate prs. Mentoring and help can be provided by @bkchr

Runtime restriction of peers

Right now, managing (reserved) peers is entiely client-side with no input/integration from within the runtime. However there are usecases where we might want to allow the runtime to state which node IDs should be connected to (e.g. with a secure private network).

This could be done in one of two ways:

• A well-known storage item that dictates to nodes (at least those that do not opt-out) about which other node IDs they are allowed to peer with. This lets the runtime command the participants of the network very easily, but cannot do anything "adaptive" where its logic changes depending on those participants.

• Two additional OCW APIs, one to get the current set of peers, and another to set the reserved nodes. The chain spec should provide defaults (bootnodes already exists but we probably want a "bootnodes-only" flag). They should persist across restarts for that chain so that at the initial startup (before the OCW runs) the node doesn't try to connect to nodes that it shouldn't. This allows on-chain logic to get exposure to potentially every node's peer connectivity and to manage their peering individually, potentially openning the door to some quite interesting things.

Sanity check consistency of public APIs (best effort)

We should best effort try and patch any inconsistencies of the Substrate public api. This includes, but is not limited to:

• Function Names
• Function Parameters and order
• Variable Names
• Structure and Custom Type names
• Event Names and parameters
• Error Names
• etc...

To be more explicit, we should avoid situations where two different traits expose functions with different names when their behavior is the same. i.e. one trait has remove_account and the other has delete_account.

We should avoid situations where the same variable is referenced with different variable names. For example caller vs sender in a public API.

We should make sure that the order of items in events are logical. i.e. we want to avoid:

• UsersBecameFriends(BlockNumber, User1, User2)

• UsersAreNotFriendsAnymore(User1, User2, BlockNumber)

This task is truly endless and I imagine will be full of opinions, so it is important to note that this is best effort.

Let's just fix up what we can.

"And gate" for EnsureOrigin

It should be possible to easily specify that two (independent) EnsureOrigin implementations must both agree that dispatch should happen as a third EnsureOrigin.

Since the origin cannot be collected together, it would necessitate a module which retained state over dispatch/proposal hashes, allowing one origin to approve a hash first, and the other origin to approve it later causing the dispatch.

An analogous "Or gate" would also be useful.

Fix the event topic list inefficiency

With paritytech/substrate#2491 merged, we maintain a list of topics. Each item is paired with the block number in which the item was deposited. We did that to fix the issue when a block has identical contents for two blocks, the notifications won't be triggered for the second block.

However, this scheme is clumsy, redundant and inefficient, it just was the easiest way to leverage ::append. A better solution would be to put a block number only once instead for each item.

Use `Verify::Signer` for the offchain `SigningTypes` trait definition

As a follow up to the refactoring of paritytech/substrate#5182, Implementing SigningTypes could be simplified to just specify the Signature type while Public can be defined as <T::Signature as traits::Verify>::Signer

https://github.com/paritytech/substrate/blob/4a6d2686e7871137f744f31781ab22e8f65f61d1/frame/system/src/offchain.rs#L471-L495

pallet-session: disabled validator is not disabled from the queued validator set.

in the following scenario:

• validators = [a, b]
• queued_validators = [a,b]
• QueuedChanged = true (for instance one validator changed its session key)
• session = 10

if a got disabled in session 10 then in session 11 the validator set is [a, b].

So if a has been slashed it is still part of the session 11. the forced new era in staking will apply at session 12.

Is this correct ?

Make NAT traversal more user-friendly

During office hours today, we (re) realized that getting nodes to connect from behind routers is a bit of a pain for a user who is just looking to join a network.

One option is to use upnp. I had a good experience with that at rchain (their upnp code)

@folsen mentioned there are other solutions too.

I don't have strong feelings about which solution we use, or that it has to happen fast, but it would be one fewer hurdle for beginner and intermediate users.

dispatch_thunk and ext_ functions use different WASM return convention

Inconsistency: which one uses lower 32 bits, ptr or len?

https://github.com/paritytech/substrate/blob/df338c248aa10413d45965040e0f50960a47f708/core/sr-api-macros/src/impl_runtime_apis.rs#L256

https://github.com/paritytech/substrate/blob/6a0313a9c9198abab7768edba07bdbbaad2e7cfc/core/sr-sandbox/without_std.rs#L249

I feel like there should be a single common function doing &[u8] -> i64

Governance: Approval vote alternatives for referendums

In the current governance framework, referendums are all-or-nothing affairs. Referendums should be multiple-choice affairs with an approval voting mechanism to select which one to proceed with. This allows for much more efficient enactment of a representative outcome.

Specifically, when a referendum for outcome A is suggested (but before it is accepted and reaches voting stage), then other outcomes B and C may be tied to A (noted B -> A, C -> A) as derivative alternatives. Tying an outcome B to a suggestion of a referendum on outcome A has a number of repercussions:

• Unlike a normal suggestion, B is not an independent proposal and can no longer be selected for referendum "alone";
• like a normal suggestion, an initial deposit must be bonded to B and other deposits may be added to it;
• any deposits left with B are, for the purposes of selecting the next referendum, now counted towards A's deposit total;
• whenever A gets accepted for a referendum, B will join it on the ballot as an alternative.

The referendum ceases to be between A and N (no change), and now becomes an approval vote between A, B, C and N. This is specifically designed to allow for compromise motions to be suggested. The price that A pays in suffering a semantic dilution of having B join it on the ballot is covered by B's deposits giving A an advantage to bring the referendum to bear sooner.

The system protects against attackers spamming alternatives through two means:

• the initial deposit for the alternatives is fixed at the same initial deposit amount for the original, meaning alternatives must be willing to pay that amount to have their option listed;
• all deposits except those of the winning motion are burnt;
• there may be a limit to the number of alternatives on the ballot to prevent any serious problems.

Transient storage runtime host function

Right now we abuse storage for intra-block data such as block number, parent hash and block author as well as various housekeeping information and flags like whether we set the uncles Authorship::DidSetUncles.

When initially writing, this incurs an extra trie lookup, which is slow. Instead there should be another host API, which works exactly like set_storage/get_storage but has no trie backing, so it never tries to lookup the value in the trie, nor does it write the value at the end of the block.

Extend network protocol with justification announcements

Just as new blocks are announced and gossiped on the network, new justifications (if produced behind the head) should be as well.

The sync protocol needs to be provided with an object that knows how to check + import justifications (this could go on the BlockImport trait)

Altering the finality notification stream slightly should make issuing such broadcasts easy to do.

Vision: Simplify test setup

They are many duplicated setup code required for any new modules, mostly config system::Trait. There must be a way to avoid duplicate this code all over the places.
It also make any modification to system::Trait tedious as it have to change every single module test setup.

I am not sure what is the best way, but at least a macro can avoid most of the duplications.

All those code are duplicated and irrelevant for most of the tests

https://github.com/paritytech/substrate/blob/9b3e9f32cd5243d2eeff3d5d8337033aa64c091b/srml/aura/src/mock.rs#L35-L63

VISION: Try..catch for substrate runtimes

ℹ️ NOTE 2022-11-07:

This is a classical issue which I wrote with my contracts hat on. However, it is not specific to contracts.

At the moment, a runtime call modifies storage eagerly. I.e. the runtime performed some storage changes there is no way of rolling back these changes. Because of that, substrate modules leverage the first-check-then-modify approach. This means that all conditions should be checked before making any side-effects. This also has a serious ramification for the runtime, the runtime MUST NOT panic under any circumstances. If it were to do so, then it opens up a DoS vector, since all changes including the charging fees for a transaction are rolled back.

In the srml-contracts module we implement a system that emulates checkpoint functionality. It does this by storing all changes performed by a smart-contract in memory, committing them to the main storage if and only if the smart-contract successfully finishes the execution. This however is not ideal, since the rest of the runtime logic acts eagerly and changes the storage as soon as that logic is called and there is no way of rolling these changes back. This really poses a problem when interaction between the smart-contract module and the rest of the runtime comes into a play: the smart-contract should revert all changes in the case of failure, so that means we can't call the runtime directly and we need to delay all calls after the smart-contract finished successfully. That also implies that the smart-contracts can't easily get the result of the execution of such a runtime call.

This makes me think that it might be beneficial to lift the checkpointing logic to substrate itself. If we had substrate-wide support of such pattern, then we could fix the issue of interaction between smart-contracts, make some patterns safer and most importantly make the runtime more tolerable to panics.

There might be different designs of such a mechanism, but here is one:

1. We introduce a new host function, called ext_try. It takes the name or the pointer to the function.
2. It creates a new Ext that is linked with the parent's one.
3. A new instance of the runtime is initialized and the specified function is called.
4. All changes are collected in the nested Ext.
5. If the nested call succeeds, then we commit all the changes to the parent's Ext. If the runtime traps (e.g. panics, overflows its stack or so on), the changes are just discarded.
6. The status is conveyed to the caller of ext_try.

Having this functionality:

1. We could express executive module as just charging the fee and ext_try-ing to dispatch transaction. There is a chance that this would open us a way to use more sanity checks (i.e using assert!s) further improving the security of a chain. It is worth noting that some of the path still have to be panic-free.
2. We could simplify smart-contracts to just execute code directly without involving abstractions like AccountDb. Transfers would be implemented as regular transfers, without trying to replicate what the T::Currency trait does. Calling to the runtime would be as simple as dispatching the function and getting the result.
3. It might open-up other patterns that can leverage this mechanism, where it is really hard to ensure all the invariants up-front (or at least without reingereering the whole runtime).

Update the networkState RPC with the latest networking change

First of all, we can now have multiple active connections to the same node, so the RPC query should be updated to show multiple addresses when that happens.

Additionally, we should also add a lot of things to this query: the list of notification streams open, the ongoing requests, and so on.
Tackling this issue consists in figuring out exactly what can be added, knowing that the point of this query is to show to humans what the current networking state is.

Sync: security review

Syncing code should be reviewed for any potential

• Eclipse or attention stealing attacks.
• DOS attacks on memory. All data collected per peer or globally should be bounded.
• DOS attacks on the CPU. There should not be potentially slow request, and in case there are, there should be a throttling mechanism.

Add Recovery Options for Validators

Let's say that a validator has a catastrophic failure in GRANDPA round n, such that they lose their DB (which keeps track of their votes). They have two options to restart:

1. Conservative: Wait until the next session to come online. This guarantees that GRANDPA is in a round greater than n and they have no risk of equivocating. However, if anyone else misses the Session heartbeat, then there will be a slash and forced era change. The validator will miss one era of rewards, but presumably gets elected in the subsequent era.
2. Aggressive: Come back online immediately. If GRANDPA is still stuck in round n for some reason, there is a risk of equivocating.

Can we add an option, say a --recovery flag, that would ask other voters what the last vote they received from my validator was when you sync?

If we operate on the assumptions that >50% of validators are honest and that our vote would be gossiped to most of the other validators, we should have some confidence that the majority of "your last vote was..." messages would help us avoid equivocation. It doesn't eliminate, but does reduce, the risk if you choose the aggressive path above.

cc @kirushik

DoS proof sync protocol.

• Enable parallel block download to prevent attention stealing attack.
• Tie reputation system to bad block reporting.
• Limit known message hashes collections for extrinsic and consensus gossip.

Report Aura Equivocations to runtime

Following paritytech/substrate#911, we have an Aura consensus engine embedded in substrate. And while its verifier rejects unverified blocks, it does not issue misbehavior checks on the wire nor chain yet. Unlike with other block production mechanisms, these aren't happening in messages between one another but at the level where the new block would be attempted to be imported.

The main type of misbehavior to check for is equivocating, or signing multiple blocks in the same slot. Other types of misbehavior, like signing a bad block or signing a block out-of-turn have intrinsic economic penalties associated with them, either by skipping out on rewards or losing reliability with peers in the p2p network.

A proof-of-equivocation is just two headers (the body may be valid or invalid), both signed by the same voter, with seal mentioning the same slot. A runtime module should be created for checking equivocations.

This can be tracked by keeping track of the last N ( = 1000) slots we've seen blocks for per author. We add to this after checking the header. If there is ever one in that position already, we will issue a report to the runtime with our proof. See paritytech/substrate#2181 for a rough outline on how to do such reports.

Serialize the peerset state on disk and reload it at startup

We used to store a nodes.json file on disk containing the known state of the network. This feature got removed when we introduced the peerset manager, but it has always been intended that it should be restored.

It is fundamentally better to not be based too much on bootnodes, and instead try to restore connections to previously working nodes.

This issue is about restoring this feature.

Problems connecting to node with mDNS

Trying to connect two nodes on the same WLAN with an external IP address where one acts as a boot node:

  "bootNodes": [
    "/ip4/XX.XXX.XXX.XX/tcp/30333/p2p/QmXa53KMgFtQWgwVaLQM9YfNvVSSecYaHpUvyWFsubNFTc"
  ],

Both nodes get the error:

2019-10-01 18:07:07.751 tokio-runtime-worker-2 TRACE sub-libp2p  Libp2p => Reach failure for Some(PeerId("QmbMfWoA5QUQEfwqe8TmeqeWvptLUKakccBw122puoxSbs")) through "/ip4/XX.XXX.XXX.XX/tcp/30333": Transport(Other(Custom { kind: Other, error: Other(A(A(A(B(Underlying(B(Os { code: 61, kind: ConnectionRefused, message: "Connection refused" }))))))) }))

From @tomaka :

Hmmm, there's probably a bug somewhere when it comes to mDNS addresses

In sc-network, isolate accesses to client, txpool, finality proof builder, ... in separate tasks.

Right now the sc-network directly calls the client, tx-pool, finality proof builder, and so on (full list in the config module).

However whenever one of these components takes a long time to respond, the entire networking stack freezes for everyone.
Not only does this happens right now due to some runtime performance issues, but it also makes us quite vulnerable to attacks.

Instead of performing straight calls, we should in my opinion spawn a few background tasks (or even a new background task per call when that is possible) and perform the calls there.

My personal opinion is that isolating the calls to the clients/tx-pool/... should be performed by sc-service, but from a pragmatic perspective let's do it in sc-network.

Instead of passing directly an Arc<dyn Client> to the sync state machine, we should instead pass some sort of Isolated<Arc<dyn Client>> where Isolated exposes an asynchronous API.

Responsible Memory Usage

News: https://hackmd.io/pvj8fsC3QbSc7o54ZH4qJQ

There are memory leaks in the codebase. Not only do we need to fix them, we need a proper attitude to ensure that they do not happen again.

Thus we need to achieve three things:

1. A solid and well-documented approach to heap tracing.

2. To rewrite our optional allocations under an attitude of zero-cost optimisations and, in general, ensure that all dynamic allocations are bounded.

3. To integrate all dynamic allocations into our triad of reporting apparatus: prometheus, telemetry and informant.

Heap tracing

It should not take days of research to figure out how to determine what lines in the codebase are resulting in so much memory being allocated. We need a clear, well-maintained document on how to setup and run Substrate (and derivative clients like Polkadot) in order to get full traces of allocations.

There are several tools that may be used (heaptrack, jemalloc, Massif/Valgrind, ...) and it is not clear which are viable, and/or sensible.

Zero-cost optimisations

This is principle a little like the "pay for what you use" mantra of Zero-Cost Abstractions that Rust gives. Essentially, it should be possible to "turn off" or reduce non-mandatory memory allocations to a large degree. This would come at a potentially large performance cost, but we will assume for now that under some circumstances this is acceptable, and in any case, having the ability to tune how much memory is allowed to be allocated in each of the various dynamic subsystems is useful.

We need to introduce CLI options such that, at the extreme end, a user (or, developer) can minimise expected memory usage. In many cases, this memory usage (given by buffer sizes, cache limits, queue capacities and pre-allocation values) is hard-coded. In other cases, it's unbounded. It should, at all times, be bounded whose value is given by the user to be configured over the CLI.

It should be configurable for all dynamically allocating subsystems. Every buffer, queue and cache. Every time something it allocated and kept around for later use. Every allocation that isn't strictly required for normal operation. If a buffer or queue is automatically grown under some circumstance, it should be instantly shrunk again once the need is gone.

One example would be the 1GB that is given to rocks db's cache size - it should be possible to lower this, all the way to zero, ideally. There are, I'm sure, many examples.

Items that allocate large amounts but that are used transiently (such the memory slab for wasm execution) should be configurable so that they remain allocated only as long as they are being actively used. This will mean that every execution will need to allocate afresh, but we don't care.

The end result of this effort is to be able to lower the expected amount of memory used in stable-state (i.e. when not importing or authoring) to around 250 MB, even with dozens of peers.

Footprint reporting

In addition to this, every dynamically allocated queue, buffer and cache should be continuously reporting its size and footprint out to telemetry, informant and/or prometheus. We should obviously begin with the lower-hanging fruit - those that we suspect of being bad behavers, but eventually we should have an attitude of reporting everything.

We probably want to do this only in a certain mode of operation, to avoid performance costs under normal operation.

Related: paritytech/substrate#4679

Per-pallet child tries

There should be a variant of decl_storage that stores all items in a child-trie rather than the main trie.

Storage format/hashing should be equivalent, except that there should be no module prefix.

Additionally, there should be migration code allowing for modules using the existing system (e.g. Kusama) to migrate their modules' storage over to per-pallet child tries during an upgrade.

Change tries should be made to work on child tries as per the main trie.

Networking metrics to add to Prometheus

To add:

• Number of records in the Kademlia storage. (#5660)
• Number of active Kademlia iterative queries.
• Number of active Kademlia RPC queries for which we're waiting for an answer.
• Number of open connections that are being kept alive but unused at this time.
• Number of open connections that are being used for Kademlia queries.
• Number of open connections that are being used for light client requests.
• Number of open connections that are being used for syncing (new protocol).
• Number of open connections that are being used for notifications (for each notification protocol).
• Sizes of notifications (#5535)
• Sizes of light client requests and responses.
• Sizes of block requests (new protocol).
• On the requesting side, delay between light client request and response.
• On the requesting side, delay between block request and response.

Fuzzing testing

It is important to ensure the substrate codebase is bullet proof and one way to detect potential issues is fuzzing testing.
Error like paritytech/substrate#1649 will be discovered on the first second of the fuzzing process.

All public interface should be fuzzed, including HTTP and WS RPC endpoints as well as the p2p protocol. This ensures malicious input cannot easily exploit / crash the nodes.

Refuse incoming connections before we know a bit about the network

Right now, when the node starts, if someone floods us with incoming TCP connections, we treat them the same way as if they were discoveries made through the random Kademlia process. This could maybe lead to an eclipse-attack if the stars align.

As a solution, we can refuse all incoming connections as long as we didn't ourselves learn a bit about the network.

This should be done a bit carefully. I'm a bit worried about just adding ifs here and there.

I'm tagging this as "security", but it's not really an urgent measure.

Gossip push-pull protocol

The main idea is to treat gossip has a set of messages that we want everyone to converge to, these messages are fed into some protocol (e.g. GRANDPA) which according to its local view allows it to determine what the canonical set of messages should be. Nodes can hash all the messages in their set and use that hash to figure out if they should pull any messages from other peers.

Using GRANDPA has an example: Let's say that according to your local view of the protocol you keep all messages for the last 2 rounds and the last 2 commits. When you're syncing let's say you're at round 1, and the current round is 10. You ask peers for their messages set hash and you pull messages from them if the hashes are different (we can also put some bounds on expected number of messages to download). After importing all the messages, your local view of the protocol is updated which allows you to filter all old messages you had on your set, so you either arrive at the same protocol state and therefore the same messages set hash, or you have a different hash because you're in a further state (presumably because you had some future message that the peer you synced from didn't have yet).

When a new message is broadcast it is still gossiped, everyone up-to-date imports it so they end up with same messages set as everyone else, avoiding the need to pull.

The protocol might also be able to identify future messages (e.g. futures rounds in GRANDPA), since we have the pull fallback we can be more aggressive about ignoring future messages (just using some bounded cache for future messages), avoiding avenues for DoS while still ensuring the protocol will make progress because it will just keep pulling messages as needed.

Make clear which RPC methods are safe/unsafe

It would be nice to have some sort of list that says which RPC methods are safe or unsafe to make it easier for the user.

Maybe this could be exposed using the RPC that lists all available RPC calls?

CC @Xanewok

Better child storage support

Improve decl_storage macro or create a new decl_child_storage macro to provide a proper support of child storage.

Currently child storage can only be used with raw key which requires some boilerplate code to use and easy to make mistake (e.g. pick a bad hasher). Those information also are not exposed on metadata which makes JS SDK implementation hard.

Usage of child storage should be similar to StorageMap / StorageValue for better developer experience.

Previous discussion: paritytech/polkadot#315 (comment)

Transfer gossiping messages from background tasks straight to event listeners

At the moment, each notification received from the network sends a message on a channel whose destination is the network worker. The network worker then reads from that channel and sends out events to various listeners grabbed via NetworkService::event_stream.

The problem is that this channel is already quite overloaded, and we should ideally try to reduce the traffic on this point of contention.

Instead, we should somehow make background tasks directly send notifications to the various listeners.

Generate `StorageValue` key at compile time

For StorageValue we can hash the key at compile time and can omit this at runtime.

Add an UDP listening endpoint to prove reachability

With the intent to solve #564

The idea is to add a small UDP listening endpoint that answers requests. Then we can tackle #564 by sending out these requests.

cc'ing @romanb @arkpar @kirushik

Compute more precise payout.

currently we do payout using this formula:

payout = yearly_inflation(...) * total_tokens_at_start_of_era / era_per_year

this is an approximation of this formula:

payout_for_era = total_tokens_at_start_of_era * (( 1 + yearly_inflation(..)) ^ ( 1 / era_per_year) - 1)

(
justification of the equation:

total_tokens_at_start_of_the_year*(1+c)^N = total_tokens_at_start_of_the_year*(1+yearly_inflation(..))
c = payout_for_era/total_tokens_at_start_of_era

)

but this approximation is not very accurate as reported here: paritytech/substrate#4964

having an approximation with taylor serie on zero can works quite well with small values, I think we should add at least one term of the serie or two.
graph for approximation: https://www.desmos.com/calculator/irdkafljuq

• P0 is exact payout function as defined by payout_for_era = total_tokens_at_start_of_era * (( 1 + yearly_inflation(..)) ^ ( 1 / era_per_year) - 1)
• P1 is current approximation payout = yearly_inflation(...) * total_tokens_at_start_of_era / era_per_year
• P2 is approximation with 2 taylor serie term
• P2s is approximation with 2 taylor serie term but each term is approximated
• P3 is approximation with 3 taylor serie term
• P4 is approximation with 4 taylor serie term
• P4s is approximation with 4 taylor serie term but each term is approximated

EDIT: for context, those computation happens here https://github.com/paritytech/substrate/blob/b9206e1f969c44270c0ca32f0781863e43418839/frame/staking/src/inflation.rs#L24-L46

Add `iter_first_keys()` to `StorageDoubleMap`

The StorageDoubleMapin substrate does not currently allow you to iterate over the first set of keys of the storage item.

It would be nice if in addition to iter_prefix() it also had the function iter() so that a user in the runtime would be able to access all elements through iteration.

cc @thiolliere

Add alternatives to the system_networkState RPC query

It's always been in a kind-of-deprecated-not-supposed-to-be-stable state.
With the new version of libp2p, its output doesn't necessarily make sense anymore (nodes can be connected multiple times), and we would need to change it. Maybe now is the right time to remove this method.

However we first need to add replacements for:

• Getting the local PeerId.
• Getting the local listened multiaddresses.
• more?

Vision: Upgrade code via delta.

Context

instead of just having srml-system::set_code we could introduce a new extrinsic which allows to send a delta to apply to the current code.

Interest

• the extrinsic to change the code can be in some situation way smaller.
• the LLVM compilation can't be really predicted but maybe if code is organized in a way and if delta algorithm is ok we can achieve having the delta very small in case of just changing a constant defined with parameter_type!
• the logic on chain can cost a bit more but seems ok.

Unknown variables:

• finding a binary delta algorithm for our usecase
• maybe we have to tell LLVM to not inline some stuff to actually being to have a small delta for small constant change.
• this feature is quite unpredictable and this unpredictability might make this feature not wanted at all. and a proper predictable mecanism should be prefered