All numbers (integers and floats) are encoded in big endian.
## Binary messages format
All binary WS2P v2 messages are encapsulated in the following format :
| data name | size in bytes | data type |
|:----------------:|--------------:|----------:|
| currency_code | 2 | u16 |
| ws2p_version | 2 | u16 |
| issuer_node_id | 4 | u32 |
| issuer_pubkey | 32 | [u8; 32] |
| message_type | 2 | specific |
| elements_count | 2 | u16 |
| payload_size | 4 | u32 |
| payload | payload_size | ?* |
| message_hash | 32 | [u8; 32] |
| signature | 64 | [u8; 64] |
_* The type of `payload` is determined by the content of `message_type`._
The signature is not always present, it's only present for messages_type values `0x00??`.
The signature is generated from the hash sha 256 of the message. This hash is always given to allow the receiving node to verify that it has not received corrupted datas.
### currency_code
| currency | code |
|:----------------------:|-------:|
| null* | 0x0000 |
| g1 | 0x0001 |
| Others prod currencies | 0x0??? |
| g1-test | 0x1000 |
| Others test currencies | 0x???? |
_*the `null` value is essential, it allows the user to synchronize his node without having to manually enter the currency on which he synchronizes. The node will then adopt the currency that is specified in the CONNECT message it will receive from the selected reference node._
### message_type interpretation
| message type | code | Signed | elements_count interval |
| PENDING_IDENTITIES | 0x2001 | no | [1, u16::MAX] |
| PENDING_MEMBERSHIPS | 0x2002 | no | [1, u16::MAX] |
| PENDING_CERTS | 0x2003 | no | [1, u16::MAX] |
| PENDING_REVOCATIONS | 0x2004 | no | [1, u16::MAX] |
| PENDING_TXS | 0x2005 | no | [1, u16::MAX] |
## Endpoints
### Endpoint binary format
| data name | size in bytes | data type |
|:----------------:|--------------:|----------:|
| api_size | 1 | u8 |
| host_size | 1 | u8 |
| path_size | 1 | u8 |
| api_name | 1/api_size | specific |
| api_version | 2 | u16 |
| network_features | 2 | specific |
| api_features | 2 | specific |
| ip_v4 | 4 | [u8; 4] |
| ip_v6 | 16 | [u16; 8] |
| host | host_size | utf8 |
| port | 2 | u16 |
| path | path_size | utf8 |
#### api_code interpretation
If api_size > 0 then `api_name` is a string utf8.
If api_size == 0 then `api_name` is an 8-bit binary value :
| API | code |
|:--------------------:|----------|
| BASIC_MERKLED_API | 0x00 |
| WS2P | 0x01 |
| GVA | 0x02 |
| DASA | 0x03 |
#### network_features
The 16 bits represent booleans to define the presence or absence of 16 network features. WS2Pv2 defines only 4 features, the remaining 12 are undefined and are in anticipation of future Ğfeatures.
Network features :
| bit | feature |
|:-------------------:|---------|
| 0000_0000 0000_0001 | ip_v4 |
| 0000_0000 0000_0010 | ip_v6 |
| 0000_0000 0000_0100 | S |
| 0000_0000 0000_1000 | TOR |
The first two features are a bit special because they must be read to parse the endpoint, indicating if the ipv4 and ipv6 fields are present or not.
S := This feature indicates that the endpoint should be contacted with an SSL/TLS overlay (HTTPS or WSS).
TOR := This feature indicates that the endpoint must be contacted via the tor network (hidden service).
#### api_features
The interpretation of this field depends on the API because it represents API-specific features. Here is the interpretation for the WS2P API :
WS2P API features :
| bit | feature |
|:-----------------:|-----------|
| 00_0000 0000_0001 | DEFLATE |
| 00_0000 0000_0010 | LOW |
| 00_0000 0000_0100 | RBF |
DEFLATE := Supports permessage-deflate extension
LOW := Accept low speed connection requests
RBF := Accept rust binary format
WS2P v2 uses only 3 of 16 features. The 13 free bits can be used for future versions of WS2P.
### Endpoint utf8 format
The utf8 format is used to display the endpoint in a human-readable format. It is also in this format that the user can manually enter an endpoint.
General utf8 endpoint format :
API_NAME VERSION NF1 .. NFn (AF1 .. AFn) IP4 [IP6] HOST PORT PATH
NF := NETWORK_FEATURE
AF := API_FEATURE
Example:
WS2P 2 S (DEFLATE LOW RBF) g1.durs.ifee.fr 443 ws2p
Initially, Duniter is totally endpoint agnostic and don't know any node.
The ws2p module requires that at least one or more peer cards are already stored in the database when launching the node,
it's up to the Duniter `sync` command to handle this (see "WS2P synchronisation" part).
_*peer card : Signed document declaring all the endpoints of a peer._
When duniter starts up, the WS2P module accesses the peer cards stored in a local database following synchronization.
## Priority of WS2P outcoming connections
The WS2P module attempts to establish outcoming connections at the kick-start of the node and then every 10 minutes.
At each attempt, all known endpoints are contacted until the outcoming quota is reached.
If the outcoming quota is already reached at the beginning of an attempt, the least priority connection is removed in order to keep the network dynamic and scalable.
The WS2P module does not attempt to connect randomly to any node among those of which it knows a WS2P endpoint,
it classifies them according to the following criteria :
### 1st criterion: the network layer
If the node is configured to use an X network layer, then all nodes that accept a connection through this X network layer will have priority.
Currently, only the TOR network layer is implemented.
_Note: the first criterion takes precedence over the second one._
### 2nd criterion: priority score
The priority score of a node depends on its public key and is calculated as follows :
by default the priority score is zero
If the key is a member: +1
If the key is preferred/privileged: +2
If the key is the same as yourself: +4 (for multi-node)
## Establishing a WS2P connection
After sorting all WS2P endpoints according to the criteria described in the previous section,
Duniter tries to connect to the other nodes.
for each connection attempt, the process is as follows:
1. Opening a websocket pointing to the remote endpoint. If the websocket has been successfully opened before the timeout*, send a CONNECT message. Otherwise, log the error and close the websocket.
2. Wait to receive a CONNECT message from the remote node, checking the validity. If the CONNECT message is valid, then sending an ACK message. Otherwise, log the error and close the websocket.
3. Wait to receive an ACK message from the remote node, checking the validity. If the ACK message is valid, then sending a FLAGS message. Otherwise, log the error and close the websocket.
4. Wait to receive a FLAGS message from the remote node, checking the validity. If the FLAGS message is valid, then sending an OK message. Otherwise, log the error and close the websocket.
5. Wait to receive an OK message from the remote node, checking the validity. If the OK message is valid, then consider the ws2p connection completely established. Otherwise, log the error and close the websocket.
*timeout : 15 seconds for connection to a conventional endpoint. 30 seconds for connection to a hidden tor service.
### CONNECT message
| data name | size in bytes | data type |
|:-------------:|---------------|-----------|
| challenge | 32 | [u8; 32] |
| api_features | 2 | u16 |
| flags_queries | 1 | u8 |
| peer_card | ? | Peer card |
| chunk_id* | 4 | u32 |
| chunk_hash* | 32 | [u8; 32] |
challenge := random hash generated by the sending node of the CONNECT message, the receiving node will then have to sign this challenge and then send this signature in its ACK message to prove that it has the corresponding private key to the public key it indicates.
api_features := This is exactly the same type as the field of the same name in the endpoints. But private WS2P nodes do not declare endpoints, so they must be able to indicate in the CONNECT message which features they support. Public WS2P nodes also fill this field, so any changes in the configuration of a public node will be applied on the 1st new connection. (If this was not the case, we would have to wait for the update of the peer record).
flags_queries := See "flags_queries interpretations" below.
peer_card := This field is optional, if it's not present it must be replaced by 2 bytes filled with zeros. Why two bytes ? Because the first field of any peer card, `peer_card_size`, is 2 bytes long.
_*Fiels `chunk_id` and `chunk_hash` are present only if flag `ASK_SYNC_CHUNK` is present in `flags_queries`._
chunk_id := Last block number of the chunk.
chunk_hash := Last block hash of the chunk.
#### flags_queries interpretations
| bit | flag |
|:---------:|-----------------|
| 0000_0001 | SYNC |
| 0000_0010 | ASK_SYNC_CHUNK |
| 0000_0100 | RES_SYNC_CHUNK |
SYNC := Boolean indicating whether the connection corresponds to a synchronization request or not. In case of a synchronization request, the connection will necessarily be accepted but temporary. Whereas in the normal case the connection may or may not be accepted according to classic ws2p rules, but it will be permanent.
ASK_SYNC_CHUNK := So that the synchronization is not too slow, the nodes to which chunk are requested are required to accept the connection and send at least 1 chunk before closing it. They can of course accept the connection for longer and send several chunks.
RES_SYNC_CHUNK := A WS2P Public node that asks to synchronize sends its PeerCard to the network, which can then contact it spontaneously to send chunk, the node in synchronization will accept in priority this type of connections.
### ACK message
| data name | size in bytes | data type |
|:-------------:|---------------|-----------|
| challenge_sig | 64 | [u8; 64] |
Each node must sign the challenge of the other to prove that it's in possession of the private key corresponding to the public key under which it identifies.
challenge_sig := Signature of the challenge given by the other node.
### FLAGS message
| data name | size in bytes | data type |
|:-------------:|---------------|-----------|
| flags | 2 | b16 |
| member_proof | 64 | [u8; 64] |
| member_pubkey | 32 | [u8; 32] |
#### flags interpretation
| bit | flag |
|:-------------------:|-----------------|
| 0000_0000 0000_0001 | LOW_FLOW_DEMAND |
| 0000_0000 0000_0010 | MEMBER_PUBKEY |
| 0000_0000 0000_0100 | MEMBER_PROOF |
LOW_FLOW_DEMAND := The sender node of this message indicates that it's behind a low speed connection and therefore care must be taken to transmit low volumes of data to it.
MEMBER_PUBKEY: Indicates whether or not the member_pubkey field is present in the message.
MEMBER_PROOF := Indicates that the message contains proof that the sender node is a member. If this boolean is false, then the member_proof field must not be present.
If this boolean is true, the "member_proof" must contain a signature must sign the following datas :
| data name | size in bytes | data type |
|:--------------:|---------------|-----------|
| flags | 2 | b16 |
| prefix | 2 | u16 |
| issuer_node_id | 4 | u32 |
| m_pubkey | 32 | [u8; 32] |
| challenge* | 32 | [u8; 32] |
_*This is the remote challenge for the signatory, and the local challenge for the verifier._
### OK message
If payload_size == 0 :
In most cases, the OK message is empty, it simply indicates that the remote node accepts the connection establishment. If the local node has also sent an OK message, then it considers the connection as fully established.
But sometimes the OK message can also transmit additional informations, the `payload_size` value can handle these cases.
prefix := In ws2p v2, member nodes have the option of not publicly communicating their prefix. It will only reveal their prefix to the node of their choice among those with which they establish a connection. For example, a member node may decide to share its prefix only with other member nodes of the same key.
prefix := If this field is zero, it means that the remote node does not want to reveal its prefix (the prefix being necessarily greater than or equal to 1).
target_b_number := Indicates the current block number of the message sender node. This number will be the target to reach for the node being synchronized.
target_b_hash := Indicates the current block hash of the message sender node. This hash will be the target to reach for the node being synchronized.
chunks_hashs_count := `chunks_hashs` array size
chunks_hashs := Hash table of the last block of each chunk.
### KO message
| data name | size in bytes | data type |
|:---------:|---------------|-----------|
| reason | 1 | u8 |
#### reason interpretation
| value | significance |
|:-----:|------------------|
| 0 | TIMEOUT |
| 1 | FULL |
| 2 | LOW_NO_SYNC |
| 3 | DIFFERENT_BRANCH |
## WS2Pv2 Messages
### List of accepted v1 messages
Some WS2Pv1 messages are accepted within WS2Pv2 connections, here is the exhaustive list :
* document messages
* v1 requests
* responses to v1 requests
WARNING : Exception with the RBF api feature : In a WS2P connection between two nodes that both support RBF api feature, any WS2Pv1 message is rejected !
WOT_POOL := For network performance reasons, a Duniter-Rust node never shares its entire wot pool at once. It randomly selects `folders_count` folders among those having received at least `min_cert` certifications. It's the requesting node that sets the values of `min_cert` and `folders_count` according to its connection rate, its configuration and the rate of new folders it has obtained in these previous requests.
_*Duniter-Rust assigns a unique integer to each member according to the order in which the identities are declared in the blocks, so it's guaranteed that all Duniter-Rust nodes will always assign the same integer to any given member._
software_size := Field size `software_name` in number of bytes.
soft_version_size := Field size `soft_version` in number of bytes.
api_outgoing_conf := See "api_outgoing_conf interpretation".
api_incoming_conf := See "api_incoming_conf interpretation".
free_member_rooms := Number of rooms available for incoming ws2p connections.
low_priority_rooms := Number of rooms occupied by nodes with lower priority than a member. The number of rooms available for a member is equal to `free_member_rooms` + `low_priority_rooms`.
node_id := Field randomly drawn by the node during its 1st configuration.
pubkey := Public key of the network keychain of the node.
blockstamp := Current blockstamp of the node.
signature := Direct signature of the byte vector from `software_size` to `soft_version` included.
step := Number of head rebounds (unsigned field).
#### HEAD v3 JSON format
{
"api_outgoing_conf": Number,
"api_incoming_conf": Number,
"free_mirror_rooms": Number,
"low_priority_rooms": Number,
"node_id": Number,
"pubkey": String(pubkey in base58),
"blockstamp": String(BlockId-BlockHash in base16),
"software_name": String,
"software_version": String,
"signature": String(signature in base64),
}
#### api_outgoing_conf interpretation
| bit | signification |
|:---------:|---------------------------|
| 0000_0001 | WS2P_PRIVATE_CLEAR |
| 0000_0010 | WS2P_PRIVATE_TLS |
| 0000_0100 | WS2P_PRIVATE_TOR_TO_CLEAR |
| 0000_1000 | WS2P_PRIVATE_TOR_TO_TOR |
There are still 4 flags for future network layers that could be implemented in the future.
#### api_incoming_conf interpretation
| bit | signification |
|:---------:|-------------------|
| 0000_0001 | WS2P_PUBLIC_CLEAR |
| 0000_0010 | WS2P_PUBLIC_TLS |
| 0000_0100 | WS2P_PUBLIC_TOR |
There are still 5 flags for future network layers that could be implemented in the future.
### Documents messages
All WS2Pv2 documents are necessarily in RBF (Rust Binary Format). When document binarization is coded in Duniter-ts, then it will be necessary to define a common binary format for each document. For performance reasons, we will probably opt for the format in which each document is signed.
We must therefore wait for the binarization of the blockchain protocol.
In the meantime, Duniter-Rust nodes exchange documents directly as portable binary objects (this is a Rust feature).
## FAQ
### What is WS2P
WS2P means "**W**eb **S**ocket **To** **P**eer".
WS2P is the inter-node network layer, it is the only guarantor of synchronization between the different nodes of the network, so its role is critical.
WS2P is the network part of the Duniter protocol, so any Duniter implementation must integrate the WS2P network layer in order to work.
WS2P is exclusively based on websocket technology.
### Why create a synchronization process
The download phase of the synchronization process is essential, it allows a new peer to download the entire blockchain quickly (then indexed in a second phase called "apply").
Historically, this process was managed by the BMA crawler, but since the arrival of WS2P v1 the BMA crawler is no longer in service, and we keep its code only for `sync` command. So for maintainability reasons, it is necessary to migrate this process outside the BMA crawler, then delete the BMA crawler code.
Without its crawling part, BMA has become a Client API, which is to be removed and replaced by a new Client API named GVA (see [GVA RFC](https://git.duniter.org/nodes/common/doc/blob/graphql_api_rfc/rfc/0003%20RFC%20GraphQL%20API%20for%20Duniter%20Clients.md)).
Finally, the synchronization process is more about the inter-node network than the interface with clients, so it makes sense to migrate this process into WS2P.
### Why remove shared peer records
Historically, shared peer files had been set up to allow the BMA crawler to automatically detect the use of multiple member nodes with the same key and automatically assign a different prefix to different nodes of the same member so that they calculate different nonces spaces.
Since Duniter-ts 1.6, the user can now manually assign a different prefix to each of his machines.
In addition, WS2P v1 has provided a new way to uniquely identify nodes (the node id, we will discuss this later). This new way of uniquely identifying nodes opens the way to possible new automatic prefix allocation mechanisms that will not require a shared peer record.
Finally, shared peer cards have a big drawback: a peer cannot prune obsolete endpoints of other peer sharing the same key because it cannot easily evaluate if these other endpoints are obsolete. In absolute terms, it is possible to bypass problems but it is simpler to directly delete shared peer records, and as we no longer need them: Occam's razor.
### Why not a total binarization
I want to limit this RFC to changes that I am able to implement in duniter-ts, and those within a reasonable time, because I want ws2p v2 to be integrated into Duniter-ts 1.7.
That's why I limit the Duniter-ts changes as much as possible: documents and requests will always be sent and received in the same format. Only peer files and heads change.
On the other hand, on the Duniter-Rust side, the binarization will be total right away, because duniter-rust already stores all documents in binary format.
To allow compatibility between different implementations, I introduced in WS2P v2 a powerful concept of features (this concept is detailed later in the RFC).
To summarize, each WS2P Public node will declare in its endpoint the list of features it supports, and the nodes that contact them can adapt their communication format accordingly.
### What consequences for clients softwares
Developers of client software need to study the new format of peer and HEAD records.
They must be able to parse this new format such that BMA 1.7 will provide them.
And above all they must be able to binarise peer records to verify the integrity of their signature.
Finally, users associating external endpoints to their peer cards will have to redeclare their external endpoints in the new generic enpoint format.
