diff --git a/README.md b/README.md index 3da88df6c600e12bcf8c5d770d94002f4a5945bc..26f67596a04f102e64fcb6ae84749ed38c22c4bb 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,10 @@ sh launch-me-once-after-git-clone-to-auto-update-readme.sh - [Dunitrust (Duniter-Rust)](https://librelois.duniter.io/slides/rml14/dunitrust/) - [Public Key Secure Transport Layer](https://librelois.duniter.io/slides/rml14/pkstl/) +### RML 16 (mai 2022) + +- [Duniter-v2s (substrate)](https://librelois.duniter.io/slides/rml16/duniter-v2s/) + ### Rustkathon 1 (mars 2019) - [Le protocole DUP](https://librelois.duniter.io/slides/rustkathon2019_1/dup-protocol/) diff --git a/public/images/Duniter-v2.png b/public/images/Duniter-v2.png new file mode 100644 index 0000000000000000000000000000000000000000..552b6965badae2abf93f97dfaee3306fbfe32f17 Binary files /dev/null and b/public/images/Duniter-v2.png differ diff --git a/public/images/Substrate.png b/public/images/Substrate.png new file mode 100644 index 0000000000000000000000000000000000000000..8993dacf54f4747ec26872de5a58e317a31ae849 Binary files /dev/null and b/public/images/Substrate.png differ diff --git a/public/images/patricia-trie.png b/public/images/patricia-trie.png new file mode 100644 index 0000000000000000000000000000000000000000..48bc9883f52f9059a617b6c6a8bb7d454bdde948 Binary files /dev/null and b/public/images/patricia-trie.png differ diff --git a/public/rml16/duniter-v2s/index.html b/public/rml16/duniter-v2s/index.html new file mode 100644 index 0000000000000000000000000000000000000000..11bcd3f0079737d341f750cfb33c26f571e80d11 --- /dev/null +++ b/public/rml16/duniter-v2s/index.html @@ -0,0 +1,38 @@ +<!DOCTYPE html> +<html> + +<head> + <title>RML16: Duniter-v2s</title> + <meta charset="utf-8"> + <link rel="stylesheet" href="../../common.css"> + <link rel="stylesheet" href="../../lib/mermaid.dark.css"> + <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/tonsky/FiraCode@1.206/distr/fira_code.css"> +</head> + +<body> + <textarea id="source"></textarea> + <script src="../../lib/mermaid.min.js"></script> + <script src="../../lib/remark-latest.min.js"></script> + <script> + var slideshow = remark.create({ + sourceUrl: 'slideshow.md', + highlightStyle: 'tomorrow-night', + countIncrementalSlides: false + }); + // don't let mermaid automatically load on start + mermaid.initialize({ + startOnLoad: false, + cloneCssStyles: false + }); + + slideshow.on("afterShowSlide", s => { + const [slide] = document.getElementsByClassName("remark-visible"); + mermaid.sequenceConfig = { + diagramMarginY: 0 + }; + mermaid.init(Array.from(slide.getElementsByClassName("mermaid"))); + }); + </script> +</body> + +</html> \ No newline at end of file diff --git a/public/rml16/duniter-v2s/slideshow.md b/public/rml16/duniter-v2s/slideshow.md new file mode 100644 index 0000000000000000000000000000000000000000..19d5b5bdcf70257b8d75a1843a803a9c39e0f3ee --- /dev/null +++ b/public/rml16/duniter-v2s/slideshow.md @@ -0,0 +1,584 @@ +class: dark, middle, center + +## Duniter-v2s (substrate) + + + +<!----> + +_21 mai 2022_ +_RML #16_ +_Librelois <c@elo.tf>_ + +Follow the presentation on your screen : + +`librelois.duniter.io/slides/rml16/duniter-v2s` + +--- + +layout: true +class: dark + +.center[Duniter-v2s (substrate)] + +--- + +## Sommaire + +* [Project history](#history) +* [Codebase & Contributors](#codebase) +* [Implemented features](#features) +* [Missing features](#missing-features) +* [Ğ1 migration plan](#g1-migration) +* [Substrate concepts](#substrate) + * [State Transition Function](#stf) + * [Wasm runtime](#wasm) + * [Forkless runtime upgrades](#forkless) + * [Onchain storage](#storage) + * [FRAME and Pallets](#frame) + * [Calls](#calls) + * [Runtime metadata](#metadata) + * [Extrinsic](#extrinsic) + * [SignedExtra](#extra) + * [SCALE](#scale) + * [RPC](#rpc) +* [Recommended libs](#libs) + +--- + +name: history + +## .center[Project history] + +- **July 2021:** Creation of the first PoC + +- **mid July 2021:** video presentation with basic wot and UT creation + +- **January 30, 2022:** video presentation with smiths sub-wot and wire transfers + +- **Feb-May, 2022:** No big step, but a lot of small work on a wide range + +- **May 27, 2022:** launch of the first test currency called `ĞDev` + +--- + +name: codebase + +## .center[Codebase & Contributors] + +### Codebase stats + +- 120 files +- 20 crates (18 lib + 2 bin) +- 43 Merge requests merged +- 13_918 rust lines code (including tests) +- 3_693 comments lines +- ~200h hours of work over 4 months + +### Contributors + +- librelois: 231 commits +- Tuxmain: 8 commits +- Pini: 4 commits +- Vit: 1 commit +- Hugo: 1 commit +- Poka: 1 commit + +--- + +name: features + +## .center[Implemented features] + +* p2p network (provided by substrate) +* forkless protocol upgrade (provided by substrate) +* UD creation and revalution according to the RTM +* Currency transfers (in unit and in UD) +* User actions can be scheduled, which allows for instance automatic transfers (substrate pallet). +* Multisig accounts: shared account requiring the signature of several private keys (substrate pallet). +* Atomic swap (substrate pallet). +* A ğ1-like web of trust (without distance) +* Smiths sub-wot +* On-chain governance for protocol upgrades (2/3 of smiths) +* The possibility to delegate rights on your account to another account (substrate pallet) +* A common treasury managed by on-chain governance (substrate pallet) +* And some other features + +--- + +name: missing-features + +## .center[Missing features] + +* Weighs benchmarking: essential to correctly estimate the cost of execution of each transaction. At the moment, weights are entered manually at overvalued values. +* The management of "offences": i.e. how to sanction the authorities that act badly. The BABE/GRANDPA consensus manages the system of automatic reporting of "offences" and verification of their validity, which allows to prove that an authority has acted badly, but it does not manage the sanctions +* Evaluation of the distance rule: due to the way substrate is designed, and the constraints of the BABE/GRANDPA consensus, we cannot directly compute the distance on-chain. So we will have to develop an oracle system to submit the computation result. +* Automatic renewal of membership +* À better on-chain governance (modalities to be decided with the community) +* Lots of other small features and improvements +* And more tests and documentation + +--- + +name: g1-migration + +## .center[Ğ1 migration plan] + +* Deploy a first test currency without migration (ĞDev) +-- + +* Development of the new technical ecosystem around the ĞDev (indexers, wallets, tools, etc) +-- + +* Fake migration of the ğ1-test + * Convert ğ1-test state to duniter-v2s genesis config + * launch a ğtest whose genesis takes the state of the Ğ1-test, but keep the ğ1-test in parallel + * massively test this ĞTest and fix bugs + * Eventual shutdown of ĞDev to not have too many different networks to manage. + * Redo this fake migration until it's perfectly mastered +-- + +* Fake migration of Ğ1 +-- + +* Official communication to users about the upcoming changes and calls to test the new wallets on this "fake Ğ1" +-- + +* Redo the fake ğ1 migration until it's perfectly mastered +-- + +* Choose dates for the definitive migration of ğ1-test and Ğ1 +-- + +* New duniter-v1 release that will stop producing blocks at the chosen dates +-- + +* Real ğ1-test migration then real Ğ1 migration + +--- + +name: substrate + +## .center[Substrate concepts] + +very generic & customisable blockchain framework with many concepts + +* State Transition Function & storage by block +* WASM runtime +* Forkless runtime upgrades +* Storage trie and proofs +* pallets and FRAME (a framework in the framework) +* + +--- + +name: stf + +## .center[State Transition Function] + +<div class="mermaid"> +graph LR + B[Block_n]-->F + State_n-1-->F("STF (Bn, Sn-1)") + F-->State_n +</div> + +* The STF check all rules and produce the new state. +* The STF shoud be a "pure" function (perfectly reproducible). +* Verifying a block is the same as executing the STF with this block as input + + +--- + +name: wasm + +## .center[Wasm runtime] + +<div class="mermaid"> +graph LR + B["Substrate Node (Host)"]-->|runtime APIs| R(Wasm runtime) + R-->|"Externalities (Host functions)"| B +</div> + +* The host (also called node/binary/client) use runtime APis to call the runtime. +-- + + * Examples of runtime APIs: + * execute_block(block) + * validate_transaction(source, tx, block_hash) + * generate_session_keys(seed) +-- +* The runtime use externalities to call host functions. +-- + + * Examples of externalities: + * get(key) + * set(key, value) + * ed25519_verify(sig, msg, pub_key) + * sr25519_verify(sig, msg, pub_key) + * blake2_256(data) + * twox_128(data) + +--- + +name: forkless + +## .center[Forkless runtime upgrades] + +* The runtime wasm bytecode is written in the onchain storage +* => so the runtime can modify its own code (with externality `set(key, value)`)! +-- + +* And this is necessarily the case, because any change of onchain storage can only be done by the State Transiton Function (the block execution) +-- + + + +=> This implies that any blockchain substrate must define an on-chain governance mechanism for the update of the runtime. + + +--- + +name: storage + +## .center[Onchain storage] + +Each block have an onchain storage. + +The onchain storage should contains only the data that are needed to verify *efficiently* all protocol rules. + +<div class="mermaid"> +graph LR + B[Block_n+1]-->F + State_n-->F("STF (Bn+1, Sn)") + F-->State_n+1 +</div> + +The block itself does not contain the storage, only its merkle root. + +--- + +name: storage2 + +## .center[Onchain storage (2)] + +.center[<div class="mermaid"> +graph TD + E["Storage Externalities"]-->O[Overlay Change Set] + O-->M[Patricia Merkle trie] + M-->D["Key value Database (ParityDB)"] +</div>] + +--- + +name: storage3 + +## .center[Onchain storage (3)] + +### .center[Patricia Trie] + +* Position in the tree defines the associated key. + +* space optimized for elements which share a prefix + +.center[] + +--- + +name: storage4 + +## .center[Onchain storage (4)] + +### .center[Patricia Merkle Trie] + +More detailed presentation here: [Substrate Storage Deep Dive](https://www.shawntabrizi.com/substrate/substrate-storage-deep-dive/) + +To resume: + +* The performance of the storage (and thus the execution time of the blocks), depends strongly on the number of keys, but especially on the good distribution of these. + +* The access time to a key in the tree evolves according to log(N), where N is the number of keys, but only if the keys are well distributed. + +=> All storage keys should be prefixed by a hash ! + +--- + +name: frame + +## .center[FRAME and Pallets] + +FRAME: Framework for Runtime Aggregation of Modularized Entities + +FRAME is divided in "pallets". A pallet looks like a module. Each pallet contains its own call, storage entries, events. + +All storage keys in a pallet are prefixed by the hash `twox128(palletInstanceName)`. +-- + + +Some pallets are instantiable, this means that it is possible to use several instances of this pallet in the same runtime (max 16), most pallets are not instantiable (max 1 instance per runtime). + +For instance, substrate pallets "collective" and "treasury" are instantiable. + +??? + +FRAME is optional, but provides a framework that helps a lot to code our runtime cleanly, and already provides a lot of features that can be reused directly (because coded in a very generic way). In practice, all substrate based projects use FRAME + +--- + +name: calls + +## .center[Calls] + +Each pallet can define a list of calls. + +Calls are "dispachable" functions, that can be called by a user with an extrinsic or by the runtime itself. + +Each user transaction contain one call (and some metadata). + +<div class="mermaid"> +graph LR + U["User app"]-->|"Transaction (1 call)"| N[Substrate node] + N-->|"Extrinsic (1 call)"| R[Runtime] +</div> + +From runtime point of view, transaction are called extrinsics, because they come from the external world. + +Each call expects an exact number of parameters, each of a precise type, indicated in the call definition and in the runtime metadata. + +--- + +name: origin + +## .center[Origin] + +Each Call is dispatched by an origin, it's looks like the "caller". + +<div class="mermaid"> +graph LR + S["Signed extrinsic"]-->|"Origin::Signed(pubkey)"| C("call()") + U["Unsigned extrinsic (aka. Inherent)"]-->|"Origin::None"| C + R[Runtime]-->|"Any origin"| C +</div> + +Every FRAME runtime has at least 3 kind of origin: Signed, None and Root. Each pallet can define additional Origin types. + +--- + +name: origin + +## .center[Origin (2)] + +Origin are very usefull to manage permnissions easily. + +Example: In ĞDev 2/3 of smiths members can act as Root. + +Substrate pallet collective define Origin `Members(p, n)`, where `p` is a number of members of the group and `n` the size of the group. + +In ĞDev runtime, we have an instance of pallet collective named `SmithsCollective`, so we just need to declare that origin `SmithsCollective::Members(x, y)` can act as root if x/y >= 2/3. +-- + + +Thanks to FRAME and Rust declarative style, we can configure this permission behavior in one line of code! + +```rust +UpgradableOrigin = EnsureProportionAtLeast<2, 3, AccountId, SmithsInstance>; +``` + +--- + +name: events + +## .center[Events] + +Substrate events are for external world, and mainly for indexers. +It's the way to say from the runtime "Something happened". + +Each pallet can defined it's how events. Events are emitted by calls execution or by hooks execution. + +The code of a typical call consists of 4 steps: + +```rust +fn call_name(origin, param1, param2) { + // 1. Verify origin and params + // 2. Read storage to ensure all protocol rules are + // 3. Write storage changes + // 4. Emit events +} +``` + +--- + +name: events2 + +## .center[Events (2)] + +Example: events of identity pallet: + +```rust +pub enum Event<T: Config> { + /// A new identity has been created + IdtyCreated { + idty_index: IdtyIndex, + owner_key: AccountId, + }, + /// An identity has been validated + IdtyValidated { idty_index: IdtyIndex }, + /// An identity has been removed + IdtyRemoved { idty_index: IdtyIndex }, +} +``` + +--- + +name: metadata + +## .center[Runtime metadata] + +The runtime metadata define: + +- Extrinsic format +- All calls definition (pallet, callName, params nams and types) +- All events definition (pallet, eventName, params nams and types) +- All storage entries definitions +- All constants definitions +- All errors definitions +- SCALE definition of all types + +Theses metadata can be obtained with the RPC method `state_getMetadata`. + +--- + +name: extrinsic + +## .center[Extrinsic] + +The runtime define also the format of extrinsics, for now duniter-v2s use the defaut format, already very generic: + +```rust +struct UncheckedExtrinsic<Address, Call, Signature, SignedExtra> +{ + /// The signature, address, and signed extra + signature: Option<(Address, Signature, SignedExtra)>, + /// The function that should be called. + function: Call, +} +``` + +`SignedExtra`, define extra metadata that we should append to the signed message. + +All theses data shoud be SCALE encoded. + +--- + +name: extra + +## .center[SignedExtra] + +```rust +pub type SignedExtra = ( + frame_system::CheckSpecVersion<Runtime>, + frame_system::CheckTxVersion<Runtime>, + frame_system::CheckGenesis<Runtime>, + frame_system::CheckEra<Runtime>, + frame_system::CheckNonce<Runtime>, + transaction_payment::ChargeTransactionPayment<Runtime>, +); +``` + +Our SignedExtra contains: + +* Runtime spec version +* Transaction format version +* Genesis hash (to avoid replay attack on another network) +* Mortality +* User nonce +* an optional tip for block author (can be zero) + +--- + +name: scale + +## .center[SCALE] + +The SCALE (Simple Concatenated Aggregate Little-Endian) Codec is a lightweight, efficient, binary serialization and deserialization codec. +In substrate, SCALE is used everywhere: + +- All RPC method parameters must be SCALE encoded +- Extrinsics must be SCALE encoded +- RPC responses payloads are SCALE encoded +- Blocks contents is SCALE encoded +- … + +--- + +name: rpc + +## .center[RPC] + +Substrate RPC API is the way to interact with a substrate chain. +It's a generic API that work with all substrate chains. +This API follow the [JSON-RPC 2.0 specification](https://www.jsonrpc.org/specification). + +Essentials: + +* `state_getMetadata()`: get runtime metadata +* `system_accountNextIndex(accountId)`: get next user nonce +* `author_submitExtrinsic(extrinsic)`: submit an extrinsic +* `chain_getFinalizedHead()`: get head of last finalized block header +* `state_getStorage(key, at?)`: get a value in onchain storage at specified block hash (or best block) +* `state_subscribeStorage(keys?)`: subcribe to storage changes at given keys prefix + + +The method `rpc_methods()` return the list of available RPC methods. + +--- + +name: rpc2 + +## .center[RPC (2)] + +### .center[RPC query example] + +* Get runtime metadata + +``` +curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "state_getMetadata"}' http://localhost:9933 +``` + +* Get current universal dividend amount + +``` +curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "state_getStorage", "params": ["0x235cdca73b1fc4b26dd60639fe5bf213e693e94477a5a91aed0757f3de0b8168"]}' http://localhost:9933 +``` +-- + +=> twox128("UniversalDividend") ++ twox128("CurrentUd") + +--- + +name: libs + +## .center[Recommended libs] + +In practices, it's recommended to use libs that provide more convenient interface for your language in front of RPC API. + +It's not magic, all theses libs use the runtime metadata (queried at startup or provided at compile time) to generate a convenient API for a specific runtime. + +* Javascript: [Polkadot-JS API](https://github.com/polkadot-js/api/) +* Python: [Python Interface](https://github.com/polkascan/py-substrate-interface) +* Rust: [subxt](https://github.com/paritytech/substrate-subxt) +* Go: [Go RPC Client](https://github.com/centrifuge/go-substrate-rpc-client/) + +Find more on [awesome-substrate repo](https://github.com/substrate-developer-hub/awesome-substrate). + + +--- + +## .center[Thanks for your attention] + +Slides made with [remark](https://github.com/gnab/remark). +Graphes created with [mermaid](https://github.com/knsv/mermaid). + +See the sources of this presentation on duniter gitlab: + +.center[[https://git.duniter.org/librelois/slides](https://git.duniter.org/librelois/slides)] diff --git a/public/rml16/event.md b/public/rml16/event.md new file mode 100644 index 0000000000000000000000000000000000000000..430f44c3f54a8f50927934205ea43d449a4ce648 --- /dev/null +++ b/public/rml16/event.md @@ -0,0 +1 @@ +RML 16 (mai 2022) \ No newline at end of file