Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found
Select Git revision
Loading items

Target

Select target project
  • nodes/rust/duniter-v2s
  • llaq/lc-core-substrate
  • pini-gh/duniter-v2s
  • vincentux/duniter-v2s
  • mildred/duniter-v2s
  • d0p1/duniter-v2s
  • bgallois/duniter-v2s
  • Nicolas80/duniter-v2s
8 results
Select Git revision
Loading items
Show changes
Showing
with 484 additions and 267 deletions
docs/dev/img/release-pipeline.png

44.1 KiB

# How to launch a live network # How to launch a live network
## 1. Choose the currency type Launching a new live network is more difficult than spawning a local blockchain. Follow this process if you know what you are doing and you already experimented a bit with local blockchains. Part of this process is automated with Rust scripts, including interaction with GitLab's GraphQL API. Do not hesitate to improve and complete it (see TODOs inside `xtask/**/*.rs` files).
Ensure that the currency type you want has the requirements. ## Requirements
In order to build in a standardized environment, you need Docker.
- see docker docs to [install docker](https://docs.docker.com/engine/install/)
- make sure you can run docker as non-root user with `docker info` or so
## Preparation
When launching a new network, you're likely to use a new runtime. See how to [release a new runtime](./release-new-runtime.md).
### Inject runtime in chainspec
ĞDev runtime is automatically embeded in the raw chainspec with the `include_bytes!` macro. An other way to inject the runtime is to use "inject-runtime-code" xtask:
```bash
cargo xtask inject-runtime-code --runtime runtime/gdev/target/srtool/release/wbuild/gdev-runtime/gdev_runtime.compact.compressed.wasm --raw-spec resources/gdev-raw.json
```
## Bootstraping
### Choose the currency type
For now, only `gdev` is supported. For now, only `gdev` is supported.
In the commands that will be indicated afterwards, you will have to replace `CURRENCY` by the In the commands that will be indicated afterwards, you will have to replace `CURRENCY` by the
currency type you have chosen. currency type you have chosen.
## 2. Choose the docker image ### Choose the docker image
Choose or build the docker image that contains the version of the code thut you want to use. Choose the docker image that contains the version of the code that you want to use.
In the commands that will be indicated afterwards, you will have to replace `TAG` by the tag of the In the commands that will be indicated afterwards, you will have to replace `TAG` by the tag of the
docker image that you have chosen. docker image that you have chosen (example : runtime-400).
## 3. Generate the session keys of genesis authority ### Generate the session keys of genesis authority
Generate a random secret phrase: Generate a random secret phrase:
```bash ```bash
$ docker run --rm -it --entrypoint duniter duniter/duniter-v2s:TAG key generate $ docker run --rm duniter/duniter-v2s:TAG -- key generate
Secret phrase: noble stay fury mean poverty delay stadium organ evil east vague can Secret phrase: noble stay fury mean poverty delay stadium organ evil east vague can
Secret seed: 0xb39c31fb10c5080721738880c2ea45412cb3df33df022bf8d9a51483b3a9b7a6 Secret seed: 0xb39c31fb10c5080721738880c2ea45412cb3df33df022bf8d9a51483b3a9b7a6
Public key (hex): 0x90a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d62 Public key (hex): 0x90a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d62
...@@ -35,42 +56,64 @@ Keep this secret phrase **carefully**, it will be used **several** times later. ...@@ -35,42 +56,64 @@ Keep this secret phrase **carefully**, it will be used **several** times later.
Then, generate the session keys: Then, generate the session keys:
```bash ```bash
$ docker run --rm -it --entrypoint duniter duniter/duniter-v2s:TAG key generate-session-keys --chain CURRENCY_local --suri "<your secret phrase>" $ docker run --rm duniter/duniter-v2s:TAG -- key generate-session-keys --chain CURRENCY_local --suri "<your secret phrase>"
Session Keys: 0x87189d723e1b2826c243bc433c718ac26ba60526932216a09102a254d54462b890a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d6290a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d6290a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d62 Session Keys: 0x87189d723e1b2826c243bc433c718ac26ba60526932216a09102a254d54462b890a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d6290a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d6290a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d62
``` ```
## 4. Paste sessions keys in the genesis configuration file ### Paste sessions keys in the genesis configuration file
An example of genesis configuration file: `resources/gdev.json` An example of genesis configuration file: `resources/gdev.json`. Paste your session keys in your `smith` identity with key `session_keys`.
## 5. Generate raw spec ### Generate raw spec
```docker ```docker
docker run -v $HOME/dev/duniter-v2s/resources:/var/lib/duniter/resources -e DUNITER_GENESIS_CONFIG=/var/lib/duniter/resources/gdev.json --rm -it --entrypoint duniter duniter/duniter-v2s:TAG build-spec -lerror --chain=gdev-gl --raw > name-raw.json docker run -v $HOME/dev/duniter-v2s/resources:/var/lib/duniter/resources -e DUNITER_GENESIS_CONFIG=/var/lib/duniter/resources/gdev.json --rm duniter/duniter-v2s:TAG -- build-spec -lerror --chain=gdev_live --raw > name-raw.json
``` ```
```bash ```bash
./scripts/gen-live-network-raw-spec.sh CURRENCY "<path/to/your/genesis/config/file>" ./scripts/gen-live-network-raw-spec.sh CURRENCY "<path/to/your/genesis/config/file>"
``` ```
## 6. Generate the docker compose and prepare nodes keys This builds the specs using debug version of Duniter.
### Generate the docker compose and prepare nodes keys
```bash ```bash
./scripts/create-live-network.sh "<your secret phrase>" CURRENCY "<path/to/dist/folder>" ./scripts/create-live-network.sh "<your secret phrase>" CURRENCY "<path/to/dist/folder>"
``` ```
The distribution folder can then be copied to a server, then the compose must be launched from the The new distribution folder can be copied to a server
root of the distribution folder:
```bash ```bash
scp -r -P SSH_PORT "<path/to/dist/folder>" user@ip:/remote/dist/path scp -r "<path/to/dist/folder>" <server>:/remote/dist/path
cd "<path/to/dist/folder>"
docker compose up -d
``` ```
Then, on the server: then on the server, launch the compose file from the the distribution folder's root:
```bash ```bash
ssh <server>
cd "<path/to/dist/folder>" cd "<path/to/dist/folder>"
docker compose up -d docker compose up -d
``` ```
This is the first node of the new live network.
## Finalization
The following steps should be completed once you are satisfied with the new live network.
### Rotate session keys
You should rotate session keys for more secured keys produced on the server (the one you used before are still in your develop machine bash history and clipboard).
### Publish image
With these new session keys in the chainspec and the runtime build with srtool, you can release the new runtime again with:
```bash
cargo xtask release-runtime 400
```
### Tell the other smith
Once you completed all these steps, the other smith can pull the docker image with a genesis containing your bootnode with the correct session keys. They can base their `docker-compose.yml` on the `duniter-validator` template.
# Duniter Pallet Conventions
## Call
Custom Duniter pallet calls should adhere to the standard Substrate naming convention:
- `action_` for regular calls (e.g., `create_identity`).
- `force_action_` for calls with a privileged origin (e.g., `force_set_distance_status`).
## Error
In the event of a call failure, it should trigger a pallet error with a self-explanatory name, for instance, `IdtyNotFound`.
## Event
Successful calls should deposit a system event to notify external entities of the change. The event name should be self-explanatory and structured in the form of a Rust struct with named fields, ensuring clarity in autogenerated documentation. An example is:
```rust
IdtyRemoved {
idty_index: T::IdtyIndex,
reason: IdtyRemovalReason<T::IdtyRemovalOtherReason>,
}
```
## Hook
Hooks are inherently infallible, and no errors should be emitted within them. To monitor progression from inside the hook, events can be employed to inform external entities about changes or no-changes.
## Internal Function
Internal functions should adhere to the following naming convention:
- `do_action_` for regular functions executing the base logic of a call (e.g., `do_remove_identity_`). These functions should directly emit events and trigger errors as needed.
- `force_action_` for privileged functions that bypass any checks. This can be useful for specific benchmarking functions.
- `check_` for functions performing checks and triggering errors in case of failure.
# Release a new Runtime
![](img/release-pipeline.png)
> The following instructions have been described in french at: [Créer une release](https://forum.duniter.org/t/industrialiser-le-demarrage-dune-nouvelle-gx/11535/41).
## Process
Example for `runtime-800`.
### New release with new Runtime
* create a `release/runtime-800` branch locally
* update the values:
* update spec version (in `runtime/<currency>/src/lib.rs`)
* eventually update `gdev.yml` (smiths, tech. committee, ...)
* push the `release/runtime-800` branch
* in the CI/CD, wait for `Create release` button to be available and click on it (see above screenshot)
The Runtime is now available on the release page [runtime-800](https://git.duniter.org/nodes/rust/duniter-v2s/-/releases/runtime-800).
### New Client
The Client is published as a Docker image.
You may want to publish a new Client version along with a Runtime update.
#### New raw specs (optional)
For a reboot, you will likely want to update the raw specs:
* in the CI/CD, wait for `release_gdev_specs` button to be available and click on it
* in the CI/CD, wait for `release_gtest_specs` button to be available and click on it
* wait for both jobs to finish
* update the Client raw specs with `cargo xtask update-raw-specs runtime-800`
#### New version (mandatory)
Update Client values:
* update Client version (in `Cargo.toml`)
* update `Cargo.lock` with `cargo build`
#### Publish Docker image
Commit everything and push the branch:
* in the CI/CD, a new pipeline has been launched
* you can stop jobs `create_g1_data`, `gdev_srtool`, `gtest_srtool` (won't be used)
* click on `gdev_docker_deploy` and `gtest_docker_deploy`
The Docker images should now be available at: https://hub.docker.com/r/duniter/duniter-v2s-gdev/tags.
## Runtime tag and spec version
Our runtime tags use `xxyy` version numbers where `x` corresponds to major change and `y` hotfix.
Make sure to move any issue or merge request assigned to the choosen milestone `runtime-xxyy` to the next one. This prevents from forgetting unfinished work.
# How to replay a block # How to replay a block
WARN: try-runtime is not properly implemented
You can use `try-runtime` subcommand to replay a block against a real state from a live network. You can use `try-runtime` subcommand to replay a block against a real state from a live network.
1. Checkout the git tag of the runtime version at the block you want to replay 1. Checkout the git tag of the runtime version at the block you want to replay
...@@ -9,13 +11,13 @@ You can use `try-runtime` subcommand to replay a block against a real state from ...@@ -9,13 +11,13 @@ You can use `try-runtime` subcommand to replay a block against a real state from
5. Replay the block a first time to get the state: 5. Replay the block a first time to get the state:
``` ```
duniter try-runtime --exectuion=Native execute-block --block-at 0x2633026e3e428b010cfe08d215b6253843a9fe54db28748ca56de37e6a83c644 live -s tmp/snapshot1 -u ws://localhost:9944 duniter try-runtime --execution=Native execute-block --block-at 0x2633026e3e428b010cfe08d215b6253843a9fe54db28748ca56de37e6a83c644 live -s tmp/snapshot1 -u ws://localhost:9944
``` ```
6. Then, replay the block as many times as you need against your local snapshot: 6. Then, replay the block as many times as you need against your local snapshot:
``` ```
duniter try-runtime --exectuion=Native execute-block --block-at 0x2633026e3e428b010cfe08d215b6253843a9fe54db28748ca56de37e6a83c644 --block-ws-uri ws://localhost:9944 snap -s tmp/snapshot1 duniter try-runtime --execution=Native execute-block --block-at 0x2633026e3e428b010cfe08d215b6253843a9fe54db28748ca56de37e6a83c644 --block-ws-uri ws://localhost:9944 snap -s tmp/snapshot1
``` ```
try-runtime does not allow (for now) to store the block locally, only the storage can be stored. try-runtime does not allow (for now) to store the block locally, only the storage can be stored.
...@@ -35,7 +35,7 @@ Use a terminal shell to execute the following commands: ...@@ -35,7 +35,7 @@ Use a terminal shell to execute the following commands:
```bash ```bash
sudo apt update sudo apt update
# May prompt for location information # May prompt for location information
sudo apt install -y cmake pkg-config libssl-dev git build-essential clang libclang-dev curl sudo apt install -y cmake pkg-config libssl-dev git build-essential clang libclang-dev curl protobuf-compiler
``` ```
### Arch Linux ### Arch Linux
...@@ -79,3 +79,16 @@ rustup update nightly ...@@ -79,3 +79,16 @@ rustup update nightly
rustup update stable rustup update stable
rustup target add wasm32-unknown-unknown --toolchain nightly rustup target add wasm32-unknown-unknown --toolchain nightly
``` ```
### Installing mold linker to decrease build time
Mold (modern linker) (https://github.com/rui314/mold) decreases the build time. Install it through your system package for example then add the following to your `~/.cargo/config`:
```toml
[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=/usr/bin/mold"]
```
(see https://forum.duniter.org/t/decrease-duniter-build-time/10170 on the forum)
\ No newline at end of file
# Upgrade Substrate # Polkadot Upgrade Guide
We need to keep up to date with Substrate. Here is an empirical guide. ParityTech frequently releases upgrades of the polkadot-sdk. For each upgrade, Duniter should be upgraded following the instructions below. These instructions are based on upgrading from version 1.8.0 to 1.9.0.
Let's say for the example that we want to upgrade from `v0.9.26` to `v0.9.32`. ## 1. Upgrade the duniter-polkadot-sdk
## Upgrade Substrate fork * Clone the repository: `git clone git@github.com:duniter/duniter-polkadot-sdk.git`
* Set the upstream repository: `git remote add upstream git@github.com:paritytech/polkadot-sdk.git`
* Fetch the latest released version: `git fetch --tag polkadot-v1.9.0`
* Create a new branch: `git checkout -b duniter-polkadot-v1.9.0`
* Rebase the branch, keeping only specific commits: "fix treasury benchmarks when no SpendOrigin", "allow manual seal to produce non-empty blocks with BABE", "add custom pallet-balance GenesisConfig", and "remove pallet-balances upgrade_account extrinsic", "remove all paritytech sdk dependencies".
* Push the new branch: `git push`
TBD (only Élois has done this for now) ## 2. Upgrade repository
## Upgrade Subxt fork * In the `Cargo.toml` file of Duniter, change the version number from 1.8.0 to 1.9.0 for all polkadot-sdk dependencies. Also, change the version for Subxt. `find . -type f -name "Cargo.toml" -exec sed -i'' -e 's/polkadot-v1.8.0\/polkadot-v1.9.0/g' {} +`.
* Upgrade the version number of all crateio dependencies to ensure compatibility with those used in the polkadot-sdk, see the node template at: [Node Template](https://github.com/paritytech/polkadot-sdk/blob/master/templates/solochain/node/Cargo.toml) (choose the correct branch/tag).
1. Checkout the currently used branch in [our Subxt fork](https://github.com/duniter/subxt), e.g. `duniter-substrate-v0.9.26` At this point, two cases may arise:
2. Create a new branch `duniter-substrate-v0.9.32`
3. Fetch the [upstream repository](https://github.com/paritytech/subxt)
4. Rebase on an upstream stable branch matching the wanted version
## Upgrade Duniter 1. If the upgrade only adds some types and minor changes, add the types in the pallet configuration, replace the offending `WeightInfo`, and delete the corresponding weights files until they can be regenerated.
1. Replace `duniter-substrate-v0.9.26` with `duniter-substrate-v0.9.32` in `Cargo.toml` 2. If there are many breaking changes, it is recommended to break down the process:
2. Update the `rust-toolchain` file according to [Polkadot release notes](https://github.com/paritytech/polkadot/releases)
* Tip: To save storage space on your machine, do `rm target -r` after changing the rust toolchain version and before re-building the project with the new version. * Start by correcting errors on individual pallets using `cargo check -p my_pallet` to identify and rectify any errors. Then, test using `cargo test -p my_pallet` and benchmark using `cargo test -p my_pallet --feature runtime-benchmark`.
3. While needed, iterate `cargo check`, `cargo update` and upgrading dependencies to match substrate's dependencies * After correcting all pallets, fix the runtimes using the same approach: check for trait declarations added or removed in each pallet configuration, and use `cargo check -p runtime`, `cargo test -p runtime`, and `cargo test -p runtime --feature runtime-benchmark`.
4. Fix errors in Duniter code * Repeat this process with the node part, the distance-oracle, all the tests, xtask, and the client.
* You may need to check how Polkadot is doing by searching in [their repo](https://github.com/paritytech/polkadot). Luckily, the project structure and Substrate patterns are close enough to ours. * Conclude the process by executing all benchmarks using the command `scripts/run_all_benchmarks.sh`.
* Some errors may happen due to two semver-incompatible versions of a same crate being used. To check this, use `cargo tree -i <crate>`. Update the dependency accordingly, then do `cargo update`.
5. As always, don't forget to `clippy` once you're done with the errors. ## 4. Troubleshooting
6. Test benchmarking:
`cargo run --features runtime-benchmarks -- benchmark overhead --chain=dev --execution=wasm --wasm-execution=interpreted-i-know-what-i-do --weight-path=. --warmup=10 --repeat=100` As Duniter may sometimes be the only chain implementing advanced features, such as manual sealing, not many references can be found. However, the following projects may be useful:
\ No newline at end of file
* Node template for general up-to-date implementation: [Node Template](https://github.com/paritytech/polkadot-sdk/tree/master/templates)
* Acala: [Acala](https://github.com/AcalaNetwork/Acala), which also uses manual sealing add a similar node implementation.
...@@ -2,14 +2,18 @@ ...@@ -2,14 +2,18 @@
When voting for a runtime upgrade, you should check that the proposed hash actually corresponds to the published code you reviewed. Otherwise, a malicious runtime upgrade could be advertised as a legitimate one. When voting for a runtime upgrade, you should check that the proposed hash actually corresponds to the published code you reviewed. Otherwise, a malicious runtime upgrade could be advertised as a legitimate one.
```docker ```sh
mkdir runtime/gdev/target
chmod o+w runtime/gdev/target
# Workaround see !239
echo -e "[toolchain]\nchannel = \"1.74.0\"\ncomponents = [ \"rust-std\", \"rust-src\" ]" > runtime/gdev/rust-toolchain.toml
docker run \ docker run \
-i \ -i \
--rm \ --rm \
-e PACKAGE=gdev-runtime \ -e PACKAGE=gdev-runtime \
-e RUNTIME_DIR=runtime/gdev \ -e RUNTIME_DIR=runtime/gdev \
-v $PWD:/build \ -v $PWD:/build \
paritytech/srtool:1.60.0 build --app --json -cM paritytech/srtool:1.74.0-0.13.0 build --app --json -cM
``` ```
Then, the runtime wasm bytecode is generated in this location: Then, the runtime wasm bytecode is generated in this location:
...@@ -19,5 +23,6 @@ runtime/gdev/target/srtool/release/wbuild/gdev-runtime/gdev_runtime.compact.comp ...@@ -19,5 +23,6 @@ runtime/gdev/target/srtool/release/wbuild/gdev-runtime/gdev_runtime.compact.comp
``` ```
To compare it to last official : To compare it to last official :
- download it here : https://git.duniter.org/nodes/rust/duniter-v2s/-/releases - download it here : https://git.duniter.org/nodes/rust/duniter-v2s/-/releases
- compare `sha256sum` of it and of the one you've built - compare `sha256sum` of it and of the one you've built
...@@ -14,21 +14,35 @@ The cross compiled binary is generated here: `target/armv7-unknown-linux-gnueab ...@@ -14,21 +14,35 @@ The cross compiled binary is generated here: `target/armv7-unknown-linux-gnueab
## How to benchmarks weights of a Call/Hook/Pallet ## How to benchmarks weights of a Call/Hook/Pallet
1. Create the benchmarking tests, see commit 31057e37be471e3f27d18c63818d57cc907b4b4f for a 1. Create the benchmarking tests. See commit f5f2ae969ac592ba9957b0e40e18d6e4b0048473 for a
complete real example. complete real example.
2. Run the benchmark test on your local machine: 2. Run the benchmark test on your local machine:
`cargo test -p <pallet> --features runtime-benchmarks`
```
cargo test -p <pallet> --features runtime-benchmarks
```
3. If the benchmark tests compiles and pass, compile the binary with benchmarks on your local 3. If the benchmark tests compiles and pass, compile the binary with benchmarks on your local
machine: `cargo build --release --features runtime-benchmarks` machine:
4. Run the benchmarks on your local machine (to test if it work with a real runtime). The command
is: `duniter benchmark pallet --chain=CURRENCY-dev --steps=50 --repeat=20 --pallet=pallet_universal_dividend --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=.` ```
5. If it worked, use the generated file content to create or update the `WeightInfo` trait and the `()` dummy implementation. Then use the `WeightInfo` tarit in the real code of the pallet. See 79e0fd4bf3b0579279fc957da5e2fdfc6d8a17fa for a cargo build --release --features runtime-benchmarks
complete real example. ```
4. Run the benchmarks on your local machine (to test if it work with a real runtime). See 0d1232cd0d8b5809e1586b48376f8952cebc0d27 for a complete real example. The command is:
```
duniter benchmark pallet --chain=CHAINSPEC --steps=50 --repeat=20 --pallet=<pallet> --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/common/src/weights/
```
5. Use the generated file content to create the `WeightInfo` trait and the `()` dummy implementation in `pallets/<pallet>/src/weights.rs`. Then use the `WeightInfo` trait in the real code of the pallet. See 62dcc17f2c0b922e883fbc6337a9e7da97fc3218 for a complete real example.
6. Redo steps `3.` and `4.` on the reference machine. 6. Redo steps `3.` and `4.` on the reference machine.
7. Put the generated file on `runtime/common/src/weights` and use it in the runtimes configuration.
See cee7c3b2763ba402e807f126534d9cd39a8bd025 for a complete real example.
Note 1: You *must* replace `CURRENCY` by the currency type, or for ĞDev use directly `--chain=dev`. 7. Use the `runtime/common/src/weights/pallet_<pallet>.rs` generated on the reference machine in the runtimes configuration. See af62a3b9cfc42d6653b3a957836f58540c18e65a for a complete real example.
Note 1: Use relevant chainspec for the benchmarks in place of `CHAINSPEC`, for example `--chain=dev`.
Note 2: If the reference machine does not support wasmtime, you should replace `--wasm-execution=compiled` Note 2: If the reference machine does not support wasmtime, you should replace `--wasm-execution=compiled`
by `--wasm-execution=interpreted-i-know-what-i-do`. by `--wasm-execution=interpreted-i-know-what-i-do`.
...@@ -39,11 +53,10 @@ by `--wasm-execution=interpreted-i-know-what-i-do`. ...@@ -39,11 +53,10 @@ by `--wasm-execution=interpreted-i-know-what-i-do`.
2. Run base block benchmarks command: 2. Run base block benchmarks command:
``` ```
./duniter benchmark overhead --chain=gdev --execution=wasm --wasm-execution=interpreted-i-know-what-i-do --weight-path=. --warmup=10 --repeat=100 duniter benchmark overhead --chain=dev --execution=wasm --wasm-execution=compiled --weight-path=./runtime/common/src/weights/ --warmup=10 --repeat=100
``` ```
3. Copy the generated file `block_weights.rs` in the codebase in folder `runtime/common/src/weights/`. 3. Commit changes and open an MR.
4. Commit changes and open an MR.
## Generate storage benchmarking ## Generate storage benchmarking
...@@ -52,8 +65,29 @@ by `--wasm-execution=interpreted-i-know-what-i-do`. ...@@ -52,8 +65,29 @@ by `--wasm-execution=interpreted-i-know-what-i-do`.
3. Run storage benchmarks command, example: 3. Run storage benchmarks command, example:
``` ```
./duniter benchmark storage -d=/mnt/ssd1/duniter-v2s/t1 --chain=gdev --mul=2 --weight-path=. --state-version=1 duniter benchmark storage -d=/mnt/ssd1/duniter-v2s/t1 --chain=gdev --mul=2 --weight-path=. --state-version=1
``` ```
4. Copy the generated file `paritydb_weights.rs` in the codebase in folder `runtime/common/src/weights/`. 4. Copy the generated file `paritydb_weights.rs` in the codebase in folder `runtime/common/src/weights/`.
5. Commit changes and open an MR. 5. Commit changes and open an MR.
## How to Write Benchmarks
### Calls
Ensure that any extrinsic call is benchmarked using the most computationally intensive path, i.e., the worst-case scenario.
### Hooks
Benchmark each hook to determine the weight consumed by it; hence, it is essential to benchmark all possible paths.
### Handlers and Internal Functions
When designing handlers and internal functions, it is advisable to avoid having them return weight for the following reasons:
1. **Simplified Benchmarking**: Writing benchmarks for hooks or calls where handlers and internal functions are utilized becomes more straightforward.
2. **Reduced Benchmarking Complexity**: By directly measuring execution and overhead in a single pass, the number of benchmarks is minimized.
3. **Enhanced Readability**: Understanding that weight accounting occurs at the outermost level improves the overall readability of the code.
One notable exception is the internal functions called in hooks like `on_idle` or `on_initialize` that can be easier to benchmark separately when the hook contains numerous branching.
# How to Build Duniter-V2S Debian Package
Compile packages for native integration for Debian-based systems.
## With Docker (on any system)
1. Install Docker and Docker Buildx.
2. Use the `scripts/build-deb.sh` script.
3. The `.deb` packages will be located in the `target/debian` folder.
## Without Docker (on a Debian-based system)
1. Install the necessary dependencies:
```sh
sudo apt-get install -y clang cmake protobuf-compiler libssl-dev
```
2. Compile the project:
```sh
cargo build --release
```
3. Install `cargo-deb`:
```sh
cargo install cargo-deb
```
4. Build the Duniter node `.deb` package:
```sh
cargo deb --no-build -p duniter
```
5. The `.deb` package will be located in the `target/debian` folder.
# How to build duniter-v2s for arm # How to build duniter-v2s for arm
Cross-compile Duniter to arm (e.g. Raspberry Pi).
## With Docker
1. Create a docker image that contains the build environment 1. Create a docker image that contains the build environment
```bash ```bash
...@@ -13,3 +17,20 @@ docker build -t duniter-v2s-arm-builder -f docker/cross-arm.Dockerfile . ...@@ -13,3 +17,20 @@ docker build -t duniter-v2s-arm-builder -f docker/cross-arm.Dockerfile .
``` ```
then, get the final binary at `target/armv7-unknown-linux-gnueabihf/release/duniter`. then, get the final binary at `target/armv7-unknown-linux-gnueabihf/release/duniter`.
## Without Docker
**Warning**: armv7 (default for Raspberry Pi) is **not** supported. Linux on RPi can be easily switched to aarch64, please search how to do so on the Internet.
This produces a musl build: the resulting executable is static, hence more portable than a dynamic one. It will be compatible with systems older than the compilation host.
```bash
# Install the tools
rustup target add aarch64-unknown-linux-musl --toolchain nightly-2023-08-23-x86_64-unknown-linux-musl
sudo dpkg --add-architecture arm64
sudo apt update
sudo apt install musl-dev:arm64 musl-tools g++-aarch64-linux-gnu gcc-aarch64-linux-gnu
# Cross-compile
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc cargo build --target=aarch64-unknown-linux-musl --release
```
# How to Build the Duniter RPM Package
1. Install dependencies:
```sh
# Fedora
sudo dnf install clang cmake protobuf-compiler openssl-devel
```
2. Compile the project:
```sh
cargo build --release
```
3. Install `cargo-generate-rpm`:
```sh
cargo install cargo-generate-rpm
```
4. Build the package:
```sh
cargo generate-rpm -p node
```
5. The `.rpm` package will be located in the `target/generate-rpm` folder.
# Autocompletion # Autocompletion
One can generate autocompletion for its favorite shell using the following option: You can generate autocompletion for your favorite shell using the following option:
```sh ```sh
cargo run -- completion --generator <GENERATOR> cargo run -- completion --generator <GENERATOR>
......
# Distance rule evaluation
The [distance rule](https://duniter.org/blog/duniter-deep-dive-wot/) is computationally too heavy to be handled by the runtime. Therefore it is computed offchain using the distance oracle.
Distance evaluation is operated on a voluntary basis by individual smiths. Since evaluators can lie or make errors, the result considered for applying the distance rule is the median of results published by the different evaluators.
## Running distance evaluation
Any smith member authoring blocks can run a distance evaluation oracle. It is better to have a machine more powerful than the reference machine.
Create a service from this command line, run by the same user as Duniter, on the same system:
/absolute/path/to/duniter distance-oracle --interval <duration>
The duration is the number of seconds between two evaluations. It should be less than the duration of a distance evaluation period. If it is equal, your node may not have the time to evaluate distance.
The oracle communicates with Duniter using its RPC API and using temporary files. Without additional (unsupported) configuration, both must run on the same filesystem. The node also needs to be forging blocks for the evaluations to be published.
### Additional Duniter configuration
Duniter should keep states at least one distance evaluation period old. If this is more than the default 256 and your node is not already an archive (`--state-pruning archive`), use the option `--state-pruning <blocks>`.
## Weights and Fees Calculation
### Introduction
Transaction weights and fees ensure network efficiency, fairness, and security in Substrate-based blockchains. These concepts are designed to manage resource allocation and incentivize proper usage of the blockchain infrastructure.
### Transaction Weights
Transaction weight measures the computational resources required to process a transaction. It is determined by factors such as the complexity of the transaction logic and the amount of data involved. Transactions with higher weights consume more resources and thus contribute to the overall load on the network.
### Transaction Fees
Transaction fees in Substrate-based blockchains are crucial for efficiently managing network resources and sustaining economic viability. They regulate resource allocation by ensuring transactions consuming more computational resources incur higher fees, discouraging spam, and promoting fair use of network capacity.
The fees are computed as follows:
`fee = base_fee + weight2fee * fee_multiplier + length2fee + tip`
## Fees in Duniter
### Fees Implementation Details
Implementing a zero-fee chain in Duniter involves configuring the blockchain to waive transaction fees when the current block weight is below a specified target. This approach aims to promote accessibility and encourage participation by eliminating fees during periods of lower network activity.
However, transaction fees are applied when the block weight or length surpasses the defined targets to ensure network security and stability during increased usage. Additionally, leveraging the fee multiplier mechanism helps deter potential prolonged network attacks by dynamically adjusting fee levels based on previous network conditions.
Duniter members benefit from the quota system, which refunds transaction fees during high network activity periods.
Fees are computed as follows:
* If `current_weight < 0.25 * max_weight` and `current_length < 0.25 * max_length` and `fee_multiplier = 1`, ie. normal load:
`fee = 0`
* If `current_weight > 0.25 * max_weight` or `current_length > 0.25 * max_length` or `fee_multiplier > 1`, ie. heavy usage (approximately more than 135 transactions per second):
`fee = `5cĞ1 + extrinsic_weight * (5cĞ1/base_extrinsic_weight)* fee_multiplier + extrinsic_length/100 + tip`
The multiplier is updated as follows:
* If `current_weight > 0.25 * max_weight` or `current_length > 0.25 * max_length`:
`Min(fee_multiplier += 1, 10)`
* If `current_weight < 0.25 * max_weight` and `current_length < 0.25 * max_length`:
`Max(fee_multiplier -= 1, 1)`
# Debian Setup Instructions
## Mirror Node
1. Download the Duniter .deb file.
2. Install the package: `dpkg -i duniter_vx.y.z.deb`.
3. Change the default configuration (at least the node name) by modifying `/etc/duniter/env_file`.
4. Start the service: `sudo systemctl start duniter-mirror.service`.
5. Enable the service at startup: `sudo systemctl enable duniter-mirror.service`.
## Smith Node
1. Download the Duniter .deb file.
2. Install the package: `dpkg -i duniter_vx.y.z.deb`.
3. Change the default configuration (at least the node name) by modifying `/etc/duniter/env_file`.
4. Create network keys using the same base path as in the config file: `duniter2 key generate-node-key --base-path <YOUR_BASE_PATH> --chain <YOUR_CHAIN>`.
5. Start the service: `sudo systemctl start duniter-smith.service`.
6. Enable the service at startup: `sudo systemctl enable duniter-smith.service`.
## Distance Oracle
A Smith node needs to be installed.
1. Change the default configuration by modifying `/etc/duniter/env_file`.
2. Start the service: `sudo systemctl start distance-oracle.service`.
3. Enable the service at startup: `sudo systemctl enable distance-oracle.service`.
# How to deploy a permanent rpc node on ĞDev network
## Publish a node
### Duniter part
- Add this docker-compose template on your server:
[docker/compose/gdev-rpc.docker-compose.yml](https://git.duniter.org/nodes/rust/duniter-v2s/-/blob/master/docker/compose/gdev-mirror.docker-compose.yml)
- Rename the file : `mv gdev-mirror.docker-compose.yml docker-compose.yml`
- In the same folder, create a `.env` file that defime environment variables `SERVER_DOMAIN`:
```bash
SERVER_DOMAIN=YOUR_DOMAIN
```
- Do `docker compose up -d` to start your node
### Reverse-proxy part (with Nginx)
In `/etc/nginx/sites-enabled/gdev.YOUR_DOMAIN` put (you can probably do simpler):
```nginx
server {
server_name gdev.YOUR_DOMAIN.fr;
listen 443 ssl http2;
listen [::]:443 ssl http2;
ssl_certificate /etc/nginx/ssl/YOUR_DOMAIN.cert;
ssl_certificate_key /etc/nginx/ssl/YOUR_DOMAIN.key;
root /nowhere;
add_header X-Frame-Options SAMEORIGIN;
add_header X-XSS-Protection "1; mode=block";
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_read_timeout 90;
location /http {
proxy_pass http://localhost:9933;
proxy_http_version 1.1;
}
location /ws {
proxy_pass http://localhost:9944;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_read_timeout 1200s;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
}
}
```
and replace `YOUR_DOMAIN` by your domain each time.
- [generate your ssl certificates](https://github.com/acmesh-official/acme.sh) with let's encrypt
if you don't already have a wildcard certificate.
- `service nginx reload`
Your node is now online as a rpc node. It's fully capable for wallet use.
To go further, read [How to become a (black)smith](./smith.md)
## Upgrade your node with minimal interruption
1. Modify docker image tag on your compose file
2. Run `docker compose pull`, this will pull the new image.
3. Run `docker compose up -d --remove-orphans`, this will recreate the container
4. Verify that your node restarted well `docker compose logs duniter-rpc`
5. Remove the old image `docker images rmi duniter/duniter-v2s:OLD_TAG`
# How to become a (black)smith
## Publish a node
### Duniter part
- Add this docker-compose on your server :
[docker/compose/gdev-validator.docker-compose.yml](https://git.duniter.org/nodes/rust/duniter-v2s/-/blob/master/docker/compose/gdev-validator.docker-compose.yml)
- Create a `.env` file that define environment variable `SERVER_DOMAIN`:
```bash
SERVER_DOMAIN=YOUR_DOMAIN
```
- `docker compose up -d` to start your node
### Reverse-proxy part (with Nginx)
In `/etc/nginx/sites-enabled/gdev.YOUR_DOMAIN` put (you can probably do simpler):
```nginx
server {
server_name gdev.YOUR_DOMAIN.fr;
listen 443 ssl http2;
listen [::]:443 ssl http2;
ssl_certificate /etc/nginx/ssl/YOUR_DOMAIN.cert;
ssl_certificate_key /etc/nginx/ssl/YOUR_DOMAIN.key;
root /nowhere;
add_header X-Frame-Options SAMEORIGIN;
add_header X-XSS-Protection "1; mode=block";
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_read_timeout 90;
location /http {
proxy_pass http://localhost:9933;
proxy_http_version 1.1;
}
location /ws {
proxy_pass http://localhost:9944;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_read_timeout 1200s;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
}
}
```
and replace `YOUR_DOMAIN` by your domain each time.
- [generate your ssl certificates](https://github.com/acmesh-official/acme.sh) with let's encrypt
if you don't already have a wildcard certificate.
- `service nginx reload`
Your node is now online as a rpc node. It's fully capable for wallet use.
## Join the Smith WoT
- add polkadot webextension to be able to authentificate with your account.
- Go to [any node with polkadotjs ui](https://gdev.1000i100.fr/dev-ui/?rpc=wss://gdev.1000i100.fr/ws)
- Ask to join Smith WoT (you need to already be in the main WoT)
- developer > extrinsics > YOUR_SMITH_ACCOUNT > smithMembership > requestMemberShip(metadata)
- add your p2p endpoint (optional)
- add your session key (follow point 1 to 4 from Validate blocks > Generate and publish your session key)
- Send the query
- Await smith certification : developer > extrinsics > CERTIFIER_SMITH_ACCOUNT > smithCert > addCert(receiver)
When you have at least 3 certifications, your'in!
## Validate blocks (blacksmith work)
- Generate and publish your session keys
1. create an ssh bridge from your desktop/laptop to your server : `ssh -L 9945:localhost:9945 SSH_USER@YOUR_SERVER`
2. In your browser go to [polkadotjs : ws://localhost:9945](https://polkadot.js.org/apps/?rpc=ws%3A%2F%2Flocalhost%3A9945#/explorer)
3. In the UI : developer > appel RPC > author > rotateKey() and run
4. copy the result in clipboard
5. In the UI : developer > extrinsics > YOUR_SMITH_ACCOUNT > authorityMembers > setSessionKeys(keys) then copy your session keys and run the query.
6. **wait 48h to verify you keep sync**
- Join
- In the UI : developer > extrinsics > YOUR_SMITH_ACCOUNT > authorityMembers > goOnline()
If you're not able to monitor, reboot, act on your node, goOffline() to avoid penality to the blockchain and to you.
## Upgrade your node with minimal interruption
1. Modify docker image tag on your compose file
2. Run `docker compose pull`, this will pull the new image.
3. Run `docker compose up -d --remove-orphans`, this will recreate the container
4. Verify that your node restarted well `docker compose logs duniter-validator`
5. Remove the old image `docker images rmi duniter/duniter-v2s:OLD_TAG`
[package] [package]
authors = ['Axiom-Team Developers <https://axiom-team.fr>'] authors.workspace = true
description = 'duniter end2end tests.' description = "duniter end2end tests"
edition = "2021" edition.workspace = true
homepage = 'https://duniter.org' homepage.workspace = true
license = 'AGPL-3.0' license.workspace = true
name = 'duniter-end2end-tests' name = "duniter-end2end-tests"
repository = 'https://git.duniter.org/nodes/rust/duniter-v2s' repository.workspace = true
version = '3.0.0' version.workspace = true
[dev-dependencies]
anyhow = "1.0"
async-trait = "0.1"
clap = { version = "3.2.23", features = ["derive"] }
ctrlc = "3.2.2"
cucumber = "0.11"
env_logger = "0.9.0"
hex = "0.4"
notify = "4.0"
parity-scale-codec = "3.1.5"
portpicker = "0.1.1"
serde_json = "1.0.64"
sp-keyring = { git = "https://github.com/duniter/substrate", branch = "duniter-substrate-v0.9.32" }
subxt = { git = 'https://github.com/duniter/subxt.git', branch = 'duniter-substrate-v0.9.32' }
tokio = { version = "1.15.0", features = ["macros"] }
[[test]] [[test]]
name = "cucumber_tests" name = "cucumber_tests"
harness = false # allows Cucumber to print output instead of libtest harness = false # allows Cucumber to print output instead of libtest
[features]
default = ["std"]
std = [
"anyhow/std",
"codec/std",
"distance-oracle/std",
"hex/std",
"serde_json/std",
"sp-core/std",
"sp-runtime/std",
]
standalone = ["distance-oracle/standalone"]
try-runtime = ["distance-oracle/try-runtime", "sp-runtime/try-runtime"]
runtime-benchmarks = []
[dev-dependencies]
anyhow = { workspace = true }
clap = { workspace = true, features = ["derive", "cargo"] }
codec = { workspace = true }
ctrlc = { workspace = true }
cucumber = { workspace = true, features = ["macros"] }
distance-oracle = { workspace = true }
env_logger = { workspace = true }
hex = { workspace = true }
notify = { workspace = true }
notify-debouncer-mini = { workspace = true }
portpicker = { workspace = true }
serde_json = { workspace = true }
sp-core = { workspace = true }
sp-keyring = { workspace = true }
sp-runtime = { workspace = true }
subxt = { workspace = true, features = [
"native",
"jsonrpsee",
] }
tokio = { workspace = true, features = ["macros", "time", "rt-multi-thread"] }
...@@ -125,7 +125,7 @@ For some scenarios, you may need to perform an action (When) that fails voluntar ...@@ -125,7 +125,7 @@ For some scenarios, you may need to perform an action (When) that fails voluntar
### Run cucumber functional tests ### Run cucumber functional tests
The cucumber tests use the last debug binary in your `target` folder. Make sure this binary corresponds to the executable you want to test by running `cargo build` before. The cucumber tests use the last debug binary in your `target` folder. Make sure this binary corresponds to the executable you want to test by running `cargo build --features constant-fees` before.
To run the cucumber tests, you will need to have the rust toolchain installed locally. To run the cucumber tests, you will need to have the rust toolchain installed locally.
...@@ -134,7 +134,7 @@ To run all the scenarios (there are many) use the command: `cargo cucumber` ...@@ -134,7 +134,7 @@ To run all the scenarios (there are many) use the command: `cargo cucumber`
You can filter the `.feature` files to run with the option `i`, for instance: You can filter the `.feature` files to run with the option `i`, for instance:
``` ```
cargo cucumber -i monetary* cargo cucumber -i 'monetary*'
``` ```
will only run `.feature` files that start with `"monetary"`. will only run `.feature` files that start with `"monetary"`.
...@@ -168,6 +168,19 @@ subxt metadata -f bytes > resources/metadata.scale ...@@ -168,6 +168,19 @@ subxt metadata -f bytes > resources/metadata.scale
If you don't have subxt, install it: `cargo install subxt-cli` If you don't have subxt, install it: `cargo install subxt-cli`
### Debug
Cucumber uses carriage returns to pretty-print.
Hence you may need to append an additional newline to your debug printings.
This commandline can be used to enable more debugging:
```bash
RUST_LOG=debug cargo cucumber -i distance_fail.feature -vvv --show-output
```
You can use the `log::debug` macro if the line `env_logger::init()` is uncommented in `cucumber_tests.rs`.
[BDD]: https://en.wikipedia.org/wiki/Behavior-driven_development [BDD]: https://en.wikipedia.org/wiki/Behavior-driven_development
[cucumber]: https://cucumber.io/ [cucumber]: https://cucumber.io/
[Cucumber Rust Book]: https://cucumber-rs.github.io/cucumber/current/writing/index.html [Cucumber Rust Book]: https://cucumber-rs.github.io/cucumber/current/writing/index.html
......