Skip to content
Snippets Groups Projects

WIP: WS2P v2

Open WIP: WS2P v2
ws2p_v2 into master
Open Éloïs requested to merge ws2p_v2 into master
Viewing commit 3f1e7c45
Show latest version
1 file
+ 61
− 61
Compare changes
  • Side-by-side
  • Inline
rfc/0006_ws2p_v2.md
+ 61
− 61
@@ -18,6 +18,12 @@ We are also taking advantage of these changes to address minor issues :
* [3 Main objectives](#3-main-objectives)
* [Contents](#contents)
* [FAQ](#faq)
* [What is WS2P ?](#what-is-ws2p)
* [Why create a synchronization process ?](#why-create-a-synchronization-process)
* [Why remove shared peer records ?](#why-remove-shared-peer-records)
* [Why not a total binarization?](#why-not-a-total-binarization)
* [What consequences for clients softwares ?](#what-consequences-for-clients-softwares)
* [Conventions](#conventions)
* [JSON format](#json-format)
* [Binary format](#binary-format)
@@ -44,12 +50,56 @@ We are also taking advantage of these changes to address minor issues :
* [HEADs v2](#heads-v2)
* [HEADs v3](#heads-v3)
* [Documents messages](#documents-messages)
* [FAQ](#faq)
* [What is WS2P ?](#what-is-ws2p)
* [Why create a synchronization process ?](#why-create-a-synchronization-process)
* [Why remove shared peer records ?](#why-remove-shared-peer-records)
* [Why not a total binarization?](#why-not-a-total-binarization)
* [What consequences for clients softwares ?](#what-consequences-for-clients-softwares)
## FAQ
### What is WS2P
WS2P means "**W**eb **S**ocket **To** **P**eer".
WS2P is the inter-node network layer, it is the only guarantor of synchronization between the different nodes of the network, so its role is critical.
WS2P is the network part of the Duniter protocol, so any Duniter implementation must integrate the WS2P network layer in order to work.
WS2P is exclusively based on websocket technology.
### Why create a synchronization process
The download phase of the synchronization process is essential, it allows a new peer to download the entire blockchain quickly (then indexed in a second phase called "apply").
Historically, this process was managed by the BMA crawler, but since the arrival of WS2P v1 the BMA crawler is no longer in service, and we keep its code only for `sync` command. So for maintainability reasons, it is necessary to migrate this process outside the BMA crawler, then delete the BMA crawler code.
Without its crawling part, BMA has become a Client API, which is to be removed and replaced by a new Client API named GVA (see [GVA RFC](https://git.duniter.org/nodes/common/doc/blob/graphql_api_rfc/rfc/0003%20RFC%20GraphQL%20API%20for%20Duniter%20Clients.md)).
Finally, the synchronization process is more about the inter-node network than the interface with clients, so it makes sense to migrate this process into WS2P.
### Why remove shared peer records
Historically, shared peer files had been set up to allow the BMA crawler to automatically detect the use of multiple member nodes with the same key and automatically assign a different prefix to different nodes of the same member so that they calculate different nonces spaces.
Since Duniter-ts 1.6, the user can now manually assign a different prefix to each of his machines.
In addition, WS2P v1 has provided a new way to uniquely identify nodes (the node id, we will discuss this later). This new way of uniquely identifying nodes opens the way to possible new automatic prefix allocation mechanisms that will not require a shared peer record.
Finally, shared peer cards have a big drawback: a peer cannot prune obsolete endpoints of other peer sharing the same key because it cannot easily evaluate if these other endpoints are obsolete. In absolute terms, it is possible to bypass problems but it is simpler to directly delete shared peer records, and as we no longer need them: Occam's razor.
### Why not a total binarization
I want to limit this RFC to changes that I am able to implement in duniter-ts, and those within a reasonable time, because I want ws2p v2 to be integrated into Duniter-ts 1.7.
That's why I limit the Duniter-ts changes as much as possible: documents and requests will always be sent and received in the same format. Only peer files and heads change.
On the other hand, on the Duniter-Rust side, the binarization will be total right away, because duniter-rust already stores all documents in binary format.
To allow compatibility between different implementations, I introduced in WS2P v2 a powerful concept of features (this concept is detailed later in the RFC).
To summarize, each WS2P Public node will declare in its endpoint the list of features it supports, and the nodes that contact them can adapt their communication format accordingly.
### What consequences for clients softwares
Developers of client software need to study the new format of peer and HEAD records.
They must be able to parse this new format such that BMA 1.7 will provide them.
And above all they must be able to binarise peer records to verify the integrity of their signature.
Finally, users associating external endpoints to their peer cards will have to redeclare their external endpoints in the new generic enpoint format.
## Conventions
@@ -210,7 +260,7 @@ The interpretation of this field depends on the API because it represents API-sp
WS2PFeatures type definition :
| bit | feature |
|:----- ---:|-----------|
|:---------:|-----------|
| 0000_0001 | DEFLATE |
| 0000_0010 | LOW |
| 0000_0100 | RBF |
@@ -258,6 +308,8 @@ Same endpoint in binary format :
## Peer card
Signed document declaring all the endpoints of a peer.
### Peer card binary format
| data name | size in bytes | data type |
@@ -315,8 +367,6 @@ Initially, Duniter is totally endpoint agnostic and don't know any node.
The ws2p module requires that at least one or more peer cards are already stored in the database when launching the node,
it's up to the Duniter `sync` command to handle this (see "WS2P synchronisation" part).
_*peer card : Signed document declaring all the endpoints of a peer._
When duniter starts up, the WS2P module accesses the peer cards stored in a local database following synchronization.
## Priority of WS2P outcoming connections
@@ -379,7 +429,7 @@ flags_queries := See "WS2PFlags type definition" below.
peer_card := This field is optional, if it's not present it must be replaced by 2 bytes filled with zeros. Why two bytes ? Because the first field of any peer card, `peer_card_size`, is 2 bytes long.
_*Fiels `chunk_id` and `chunk_hash` are present only if flag `ASK_SYNC_CHUNK` is present in `flags_queries`._
_*Fields `chunk_id` and `chunk_hash` are present only if flag `ASK_SYNC_CHUNK` is present in `flags_queries`._
chunk_id := Last block number of the chunk.
chunk_hash := Last block hash of the chunk.
@@ -419,7 +469,7 @@ challenge_sig := Signature of the challenge given by the other node.
#### WS2PSecretFlags type definition
| bit | flag |
|:-------- :|-----------------|
|:---------:|-----------------|
| 0000_0001 | LOW_FLOW_DEMAND |
| 0000_0010 | MEMBER_PUBKEY |
| 0000_0100 | MEMBER_PROOF |
@@ -699,53 +749,3 @@ All WS2Pv2 documents are necessarily in RBF (Rust Binary Format). When document
We must therefore wait for the binarization of the blockchain protocol.
In the meantime, Duniter-Rust nodes exchange documents directly as portable binary objects (this is a Rust feature).
## FAQ
### What is WS2P
WS2P means "**W**eb **S**ocket **To** **P**eer".
WS2P is the inter-node network layer, it is the only guarantor of synchronization between the different nodes of the network, so its role is critical.
WS2P is the network part of the Duniter protocol, so any Duniter implementation must integrate the WS2P network layer in order to work.
WS2P is exclusively based on websocket technology.
### Why create a synchronization process
The download phase of the synchronization process is essential, it allows a new peer to download the entire blockchain quickly (then indexed in a second phase called "apply").
Historically, this process was managed by the BMA crawler, but since the arrival of WS2P v1 the BMA crawler is no longer in service, and we keep its code only for `sync` command. So for maintainability reasons, it is necessary to migrate this process outside the BMA crawler, then delete the BMA crawler code.
Without its crawling part, BMA has become a Client API, which is to be removed and replaced by a new Client API named GVA (see [GVA RFC](https://git.duniter.org/nodes/common/doc/blob/graphql_api_rfc/rfc/0003%20RFC%20GraphQL%20API%20for%20Duniter%20Clients.md)).
Finally, the synchronization process is more about the inter-node network than the interface with clients, so it makes sense to migrate this process into WS2P.
### Why remove shared peer records
Historically, shared peer files had been set up to allow the BMA crawler to automatically detect the use of multiple member nodes with the same key and automatically assign a different prefix to different nodes of the same member so that they calculate different nonces spaces.
Since Duniter-ts 1.6, the user can now manually assign a different prefix to each of his machines.
In addition, WS2P v1 has provided a new way to uniquely identify nodes (the node id, we will discuss this later). This new way of uniquely identifying nodes opens the way to possible new automatic prefix allocation mechanisms that will not require a shared peer record.
Finally, shared peer cards have a big drawback: a peer cannot prune obsolete endpoints of other peer sharing the same key because it cannot easily evaluate if these other endpoints are obsolete. In absolute terms, it is possible to bypass problems but it is simpler to directly delete shared peer records, and as we no longer need them: Occam's razor.
### Why not a total binarization
I want to limit this RFC to changes that I am able to implement in duniter-ts, and those within a reasonable time, because I want ws2p v2 to be integrated into Duniter-ts 1.7.
That's why I limit the Duniter-ts changes as much as possible: documents and requests will always be sent and received in the same format. Only peer files and heads change.
On the other hand, on the Duniter-Rust side, the binarization will be total right away, because duniter-rust already stores all documents in binary format.
To allow compatibility between different implementations, I introduced in WS2P v2 a powerful concept of features (this concept is detailed later in the RFC).
To summarize, each WS2P Public node will declare in its endpoint the list of features it supports, and the nodes that contact them can adapt their communication format accordingly.
### What consequences for clients softwares
Developers of client software need to study the new format of peer and HEAD records.
They must be able to parse this new format such that BMA 1.7 will provide them.
And above all they must be able to binarise peer records to verify the integrity of their signature.
Finally, users associating external endpoints to their peer cards will have to redeclare their external endpoints in the new generic enpoint format.