ssb-ngi-pointer to demonstrate the outcome of the work done by the SSB NGI Pointer team during 2020 and 2021. The SSB NGI Pointer team is no longer active because we completed our grant project.
Status: Ready for implementation
In classical SSB an identity is tied to a single feed. All messages for different kinds of applications are posted to this single feed. While it is possible to create multiple feeds, there has been no formal specification for how these feeds relate and what their purposes are.
Meta feeds aim to solve these problems by tying an identity to a meta feed instead. A meta feed references other feeds (or even meta feeds) and contains metadata about the feed including purpose and feed format. This allows for things like feed rotation to a new feed format, splitting data into separate (sub)feeds and to create special indexing feeds for partial replication.
A meta feed is tied to a single identity and thus should only be used on a single device. There is a separate fusion identity protocol that only deals with how to relate multiple devices to a single identity. This spec here is not for that use-case.
Meta feeds will use a specialized feed format known as bendy butt that aims to be very easy to implement. The aim is that this will make it easier for implementations which do not need or want to support the classical SSB format.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.
We use bencode and BFE notations as defined in the bendy butt spec.
Meta feeds MUST use the bendy butt feed format with a few additional constraints.
The content dictionary inside the contentSection of meta feed messages
MUST conform to the following rules:
- Has a
typefield mapping to a BFE string (i.e.<06 00> + data) which can assume only one the following possible values:metafeed/add/existingmetafeed/add/derivedmetafeed/updatemetafeed/tombstone
- Has a
subfeedfield mapping to a BFE "feed ID", i.e.<00> + format + data - Has a
metafeedfield mapping to a BFE "Bendy Butt feed ID", i.e.<00 03> + data - (Only if the
typeismetafeed/add/derived): anoncefield mapping to a BFE "arbitrary bytes" with size 32, i.e.<06 03> + nonce32bytes
The contentSignature field inside a decrypted contentSection MUST use
the subfeed's cryptographic keypair.
Here is an an example of a meta feed with 2 sub feeds: one for main
social data and another one for application-x in a different format.
rankdir=RL node [shape=record];
edge [tailclip=false]; a [label="{ | main }"] b [label="{ | application-x }"]; b:ref:a -> a:data [arrowhead=vee, arrowtail=dot, dir=both]; }
Contents of messages in the meta feed that acts as meta data for feeds:
{
"type" => "metafeed/add/existing",
"feedpurpose" => "main",
"subfeed" => (BFE-encoded feed ID for the 'main' feed),
"metafeed" => (BFE-encoded Bendy Butt feed ID for the meta feed),
"tangles" => {
"metafeed" => {
"root" => null,
"previous" => null
}
},
},
{
"type" => "metafeed/add/existing",
"feedpurpose" => "application-x",
"subfeed" => (BFE-encoded Bamboo feed ID),
"metafeed" => (BFE-encoded Bendy Butt feed ID for the meta feed),
}
Initially the meta feed spec supports three operations: add/existing
add/derived, and tombstone. Note, signatures (see key
management section) are left out in the examples here.
Tombstoning means that the feed is no longer part of the meta feed. Whether or not the sub feed itself is tombstoned is a separate concern.
Example tombstone message:
{
"type" => "metafeed/tombstone",
"subfeed" => (BFE-encoded Bamboo feed ID),
"metafeed" => (BFE-encoded Bendy Butt feed ID for the meta feed),
"reason" => (some BFE string),
"tangles" => {
"metafeed" => {
"root" => (BFE-encoded message ID of the "metafeed/add" message),
"previous" => (BFE-encoded message ID of the "metafeed/add" message),
}
}
}
Updating the metadata on a sub feed which is a member of a meta feed is currently not supported.
Note: while the metafeed: ... field on the add and tombstone messages
seems redundant, it is important to have it and check that the metafeed field
equals the author of the metafeed itself to protect against replay attacks.
An example of the applications meta feed with two different applications.
rankdir=RL nodesep=0.6 node [shape=record];
edge [tailclip=false]; a [label="{ | App1 }"] b [label="{ | App2 }"];
b:ref:a -> a:data [arrowhead=vee, arrowtail=dot, dir=both]; }
{
"type" => "metafeed/add/derived",
"feedpurpose" => "gathering",
"subfeed" => (BFE-encoded feed ID dedicated for the gathering app),
(other fields...)
},
{
"type" => "metafeed/add/derived",
"feedpurpose" => "chess"
"subfeed" => (BFE-encoded feed ID dedicated for the chess app),
(other fields...)
}
As mentioned earlier, in classical SSB the feed identity is the same as the feed. Here instead we want to decouple identity and feeds.
To generate a meta feed and link it to an existing main feed, first
a seed is generated:
const seed = crypto.randomBytes(32)From this seed, a meta feed can be generated using:
const salt = 'ssb'
const prk = hkdf.extract(lhash, hash_len, seed, salt)
const mf_info = "ssb-meta-feed-seed-v1:metafeed"
const mf_seed = hkdf.expand(hash, hash_len, prk, length, mf_info)
const mf_key = ssbKeys.generate("ed25519", mf_seed)Note we use metafeed here in the info. As the top/genesis meta feed is
special we use that string, for all other derived feeds a nonce is used,
which is also published in the corresponding metafeed/add/derived
message.
We also encrypt the seed as a private message from main to main (so
it's a private message to yourself; notice this is JSON, because it's
published on the main):
{
"type": "metafeed/seed",
"metafeed": ssb:feed/bendybutt-v1/bendyButtFeedID,
"seed": seedBytesEncodedAsHexString
}
By doing so we allow the existing feed to reconstruct the meta feed and all sub feeds from this seed.
Then the meta feed is linked with the existing main feed using a new
message on the meta feed signed by both the main feed and the meta
feed. For details this see bendy butt.
{
"type" => "metafeed/add/existing",
"feedpurpose" => "main",
"subfeed" => (BFE-encoded feed ID for the 'main' feed),
"metafeed" => (BFE-encoded Bendy Butt feed ID for the meta feed),
"tangles" => {
"metafeed" => {
"root" => (BFE nil),
"previous" => (BFE nil)
}
}
}
In order for existing applications to know that the existing feed
supports meta feeds, a special message of type metafeed/announce
is created on the main feed (notice this is JSON, because the
main feed is not in Bendy Butt):
{
// ... other msg.value field ...
content: {
type: 'metafeed/announce',
metafeed: 'ssb:feed/bendybutt-v1/-oaWWDs8g73EZFUMfW37R_ULtFEjwKN_DczvdYihjbU=',
subfeed: MAIN_FEED_ID,
tangles: {
metafeed: {
root: null,
previous: null
}
},
signature: SIGNATURE_OF_THE_ABOVE
}
}Note that MAIN_FEED_ID is the ID of the main feed, and that
SIGNATURE_OF_THE_ABOVE is the signature (using the meta feed
keys) of the stringified content without content.signature
itself, in a similar manner to how the message signature
msg.value.signature is constructed relative to msg.value. So
msg.value.signature is signed with the main feed's keys, but
msg.value.content.signature is signed with the meta feed keys.
A feed can only have one meta feed. If for whatever reason an
existing meta feed needs to be superseed, a new message is created
pointing to the previous metafeed/announce message via the tangle.
A new identity also starts by constructing a seed. From this seed both
the meta feed keys and the main feed keys are generated. The main
should use the info: ssb-meta-feed-seed-v1:<base64 encoded nonce>
and the nonce is also published as part of the metafeed/add/derived
message on the meta feed.
{
"type" => "metafeed/add/derived",
"feedpurpose" => "main",
"subfeed" => (BFE-encoded feed ID for the 'main' feed),
"metafeed" => (BFE-encoded Bendy Butt feed ID for the meta feed),
"nonce" => (bencode byte sequence with 32 random bytes),
"tangles" => {
"metafeed" => {
"root" => null,
"previous" => null
}
}
}
The seed will also be encrypted to the main feed and the meta feed linked to the main feed just like for existing feeds.
By building a layer on top of existing feeds we maintain backwards compatible with existing clients. The identity to be used by new applications should be that of the meta feed. For backwards compatibility contact messages forming the follow graph together with secret handshake will continue to use the key of the main feed.
It is worth noting that even though the examples above specify ways to generate new feeds from a single seed, it is perfectly fine and in some cases a better idea to generate a feed not from this seed. Thus in the case the main key being broken or stolen, you don't loose everything.
If a key is reused in another part of the tree it must include a reference to the original sub feed or meta feed it was defined in. The original place is the authorative place for its metadata.
Using BIP32-Ed25519 instead was considered but that method has a weaker security model in the case of a key compromised where keys are shared between devices.
Let us see how we can use the above abstraction to solve several common examples:
Changing to a new feed format could be implemented by adding a new feed to the meta feed state, and by adding a tombstone message to the old feed pointing and assigning the new feed as active in the meta feed.
In case of backwards compability with clients that do not support a newer feed format or in the case of only wanting to support newer feed formats, maintaining muliple feeds with the same content would be an interesting avenue to explore. As the hash of the messages in the two feeds would be different, there could be a way to include the hash of the corresponding message in old feed in the newer feed.
Lower end clients could offload this extra storage requirement to larger peers in the network.
For classical SSB feeds if one would like to replicate a specific part of a feed, such as the contact messages, one could request another peer to generate a feed that only references these messages. Then when exchanging data, the original messages could be included as auxiliary data. This would only act as a claim, never as a proof that some messages were not left out. Naturally this comes down to trust then. Using the friend graph would be natural, as would using trustnet together with audits of these claims.
Similar to claims it would be possible to create sub feeds that would only contain certain messages. This might be useful for specific apps. Another use case for this would be curated content, where specific messages are picked out that might be of particular interest to a certain application or specific people, or say messages within the last year.
Using the metadata it would be possible to attach a lifetime to feeds, meaning honest peers would delete the feeds after a specific time. This would enable applications to generate a short lived feed only for the communication between two parties.
Similar to ephemeral feeds it would be possible to attach an allow list to a feed and only distribute this feed to people on the allow list. As with ephemeral feeds, this cannot be enforced, but assuming honest peers would give piece of mind that the data is only stored on a certain subset of the whole network. This can naturally be combined with private groups to better ensure safety.
- In the case of claims, how are bad actors handled?
- What are the broader consequences of ephemeral feeds. Maybe they can only be used in limited circumstances, and if so which ones?
- For sub feeds and feed rotation what is the best way to handle potentially overlapping messages
CFT suggested the use of meta feeds in
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent