Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found
Select Git revision
  • 1000i100-test
  • 105_gitlab_container_registry
  • cgeek/issue-297-cpu
  • ci_cache
  • debug/podman
  • elois-compose-metrics
  • elois-duniter-storage
  • elois-smoldot
  • feature/dc-dump
  • feature/distance-rule
  • feature/show_milestone
  • fix-252
  • fix_picked_up_file_in_runtime_release
  • gdev-800-tests
  • hugo-release/runtime-701
  • hugo-tmp-dockerfile-cache
  • hugo/195-doc
  • hugo/195-graphql-schema
  • hugo/distance-precompute
  • hugo/endpoint-gossip
  • hugo/tmp-0.9.1
  • master
  • network/gdev-800
  • network/gdev-802
  • network/gdev-803
  • network/gdev-900
  • network/gtest-1000
  • pini-check-password
  • release/client-800.2
  • release/hugo-chainspec-gdev5
  • release/poka-chainspec-gdev5
  • release/poka-chainspec-gdev5-pini-docker
  • release/runtime-100
  • release/runtime-200
  • release/runtime-300
  • release/runtime-400
  • release/runtime-401
  • release/runtime-500
  • release/runtime-600
  • release/runtime-700
  • release/runtime-701
  • release/runtime-800
  • runtime/gtest-1000
  • tests/distance-with-oracle
  • tuxmain/anonymous-tx
  • tuxmain/benchmark-distance
  • tuxmain/fix-change-owner-key
  • update-docker-compose-rpc-squid-names
  • upgradable-multisig
  • gdev-800
  • gdev-800-0.8.0
  • gdev-802
  • gdev-803
  • gdev-900-0.10.0
  • gdev-900-0.10.1
  • gdev-900-0.9.0
  • gdev-900-0.9.1
  • gdev-900-0.9.2
  • gtest-1000
  • gtest-1000-0.11.0
  • gtest-1000-0.11.1
  • runtime-100
  • runtime-101
  • runtime-102
  • runtime-103
  • runtime-104
  • runtime-105
  • runtime-200
  • runtime-201
  • runtime-300
  • runtime-301
  • runtime-302
  • runtime-303
  • runtime-400
  • runtime-401
  • runtime-500
  • runtime-600
  • runtime-700
  • runtime-701
  • runtime-800
  • runtime-800-backup
  • runtime-800-bis
  • runtime-801
  • v0.1.0
  • v0.2.0
  • v0.3.0
  • v0.4.0
  • v0.4.1
88 results

Target

Select target project
  • nodes/rust/duniter-v2s
  • llaq/lc-core-substrate
  • pini-gh/duniter-v2s
  • vincentux/duniter-v2s
  • mildred/duniter-v2s
  • d0p1/duniter-v2s
  • bgallois/duniter-v2s
  • Nicolas80/duniter-v2s
8 results
Select Git revision
  • distance
  • elois-ci-binary-release
  • elois-compose-metrics
  • elois-duniter-storage
  • elois-fix-85
  • elois-opti-cert
  • elois-remove-renewable-period
  • elois-rework-certs
  • elois-smoldot
  • elois-substrate-v0.9.23
  • elois-technical-commitee
  • hugo-cucumber-identity
  • master
  • no-bootnodes
  • poc-oneshot-accounts
  • release/runtime-100
  • release/runtime-200
  • ts-types
  • ud-time-64
  • runtime-100
  • runtime-101
  • runtime-102
  • runtime-103
  • runtime-104
  • runtime-105
  • runtime-200
  • runtime-201
  • v0.1.0
28 results
Show changes
Showing
with 531 additions and 245 deletions
# Compilation
Duniter is compiled using the Rust compiler. For a general overview, refer to the [Rustc Dev Guide](https://rustc-dev-guide.rust-lang.org/overview.html).
Substrate and Duniter provide a set of features enabling or disabling parts of the code using conditional compilation. More information on conditional compilation can be found [here](https://doc.rust-lang.org/reference/conditional-compilation.html), or by enabling or disabling compilation of packages. Below is a list of all available features:
## External
- **runtime-benchmarks**: Compiles the runtime with benchmarks for extrinsics benchmarking.
- **try-runtime**: Compiles the runtime for tests and verifies operations in a simulated environment.
- **std**: Enables the Rust standard library.
## Duniter
- **gdev**: Sets `gdev-runtime` and `std` used to build the development chain.
- **gtest**: Sets `gtest-runtime` and `std` used to build the test chain.
- **g1**: Sets `g1-runtime` and `std` used to build the production chain.
- **constant-fees**: Uses a constant and predictable weight-to-fee conversion only for testing.
- **embed**: Enables hardcoded live chainspecs loaded from "../specs/gtest-raw.json" file.
- **native**: Compiles the runtime into native-platform executable only for debugging purposes.
Note: By default, Duniter will be compiled using the `gdev` feature and including the compilation of the distance oracle. Since the three Duniter chains are mutually exclusive, it is mandatory to disable the default feature to compile `gtest` and `g1` as follows:
- `cargo build --no-default-features --features gtest`
- `cargo build --no-default-features --features g1`
- `cargo build --no-default-features -p distance-oracle --features std`
......@@ -15,10 +15,10 @@ The summary gives an overview of the rules described below. Reading it will help
## Naming commits
Every commit must follow [conventional commit specification v1.0.0].
Every commit must comply with [conventional commit specification v1.0.0].
The commit name hase to be meaningful in the context of commit history reread. It should not make reference to a specific MR or discussion.
Among other, commit history is used in changlogs and to follow the project progress, that's why it has to be self-explanatory.
The commit name has to be meaningful in the context of commit history reread. It should not make reference to a specific MR or discussion.
Among other, commit history is used in changlogs and to track the project progress, that's why it has to be self-explanatory.
If you have a new need, please contact the main developers to add a type together.
## Update strategy
......@@ -28,18 +28,13 @@ We only use **rebases**, *merges* are strictly fordbidden !
Every time the `master` branch is updated, you must rebase each of your working branch on it. For each of them:
1. Go on your branch
2. Run a rebase on master:
1. Rebase on master with `git rebase master`
1. If you see conflicts, fix them by editing the sources. Once it is done, you must:
1. commit the files that were in conflict
1. continue the rebase with `git rebase --continue`
1. Keep doing until you don't have any more conflict after `git rebase --continue`.
git rebase master
3. If you see conflicts, fix them by editing the sources. Once it is done, you must:
a. commit the files that were in conflict
b. continue the rebase with `git rebase --continue`
c. Do 3. again for each commit that will be in conflict.
4. When you don't have any conflict anymore after `git rebase --continue`, then the rebase succeeded. Then rebase a remaning branch.
To prevent accidental merge commits, we recommend that force the `--ff-only` option on the merge command:
To prevent accidental merge commits, we recommend to force the `--ff-only` option on the merge command:
git config --global merge.ff only
......@@ -58,20 +53,24 @@ any problem with your material.
After complying with the above criteria in your commits, you should check that your branch is up to date with the target branch (`master` in this example). As this branch is moving forward frequently, it is possible that new commits may have occurred while you were working on your branch (named YOUR_BRANCH, here). If this is the case or in case of doubt, to update your branch with respect to `master`, do the following:
```bash
git checkout master # switch to master branch
git pull # updates the remote branch based on remote
git checkout YOU_BRANCH # switch back to your branch
git rebase master # rebase you work on master branch
```
In case of conflict during rebase that you can not solve, contact a lead developer telling them the hash of the commit on which YOUR_BRANCH is currently based so they can reproduce the rebase and see the conflicts. While waiting for their answer, you can cancel the rebase and work on YOUR_BRANCH without updating:
```
git rebase --abort
```
It is better to take your time before integrating a new contribution because the history of the master branch cannot be modified: it is a protected branch. Each commit on this branch remains there *ad vitam aeternam* that is why we make sure to keep a clear and understandable commit history.
## Discussion in a merge request
On Gitlab, a discussion is opened for each merge request. It will allow you to discuss the changes you have made. Feel free to identify someone by writing @pseudo so that they are notified of your request. Don't be impatient, the review of your contribution may take more or less time depending on its content!
On Gitlab, a discussion is opened for each merge request. It will allow you to discuss the changes you have made. Feel free to tag someone by writing @pseudo so that they are notified of your request. Don't be impatient, the review of your contribution may take more or less time depending on its content!
The general discussion is used to comment on the merge request as a whole, for example to tag a developer for a proofreading request. When it comes to discussing a specific change in the code, you should go to the "Changes" tab of the merge request and comment under the code extract involved. This makes it easier to break down the resolution of problems raised by the merge request via the "comment resolution" feature. Each segment can be marked as resolved, but only the reviewer is allowed to do so!
......@@ -81,7 +80,8 @@ When you finished developing, you must compile, run linter and run all tests:
cargo fmt
cargo clippy
cd tests && npm test
cargo tu
cargo cucumber
Then commit everything.
......@@ -93,7 +93,7 @@ If you have a pile of commits, use the useful interactive rebase to clean up you
There you can rename the `wip:` commits, you can "fixup" commits that go together, you can rename and re-order commits,...
After an interactive rebase, your local git history is different that yours in Gitlab, so you need a force push to make it to Gitlab:
After an interactive rebase, your local git history differs from Gitlab's version, so you need a force push to make it to Gitlab:
git push -f
......
docs/dev/img/release-pipeline.png

44.1 KiB

# How to launch a live network
## 1. Choose the currency type
Launching a new live network is more difficult than spawning a local blockchain. Follow this process if you know what you are doing and you already experimented a bit with local blockchains. Part of this process is automated with Rust scripts, including interaction with GitLab's GraphQL API. Do not hesitate to improve and complete it (see TODOs inside `xtask/**/*.rs` files).
Ensure that the currency type you want have the requirements.
## Requirements
In order to build in a standardized environment, you need Docker.
- see docker docs to [install docker](https://docs.docker.com/engine/install/)
- make sure you can run docker as non-root user with `docker info` or so
## Preparation
When launching a new network, you're likely to use a new runtime. See how to [release a new runtime](./release-new-runtime.md).
### Inject runtime in chainspec
ĞDev runtime is automatically embeded in the raw chainspec with the `include_bytes!` macro. An other way to inject the runtime is to use "inject-runtime-code" xtask:
```bash
cargo xtask inject-runtime-code --runtime runtime/gdev/target/srtool/release/wbuild/gdev-runtime/gdev_runtime.compact.compressed.wasm --raw-spec resources/gdev-raw.json
```
## Bootstraping
### Choose the currency type
For now, only `gdev` is supported.
In the commands that will be indicated afterwards, you will have to replace `CURRENCY` by the
currency type you have chosen.
## 2. Choose the docker image
### Choose the docker image
Choose or build the docker image that contains the version of the code thut you want to use.
Choose the docker image that contains the version of the code that you want to use.
In the commands that will be indicated afterwards, you will have to replace `TAG` by the tag of the
docker image that you have chosen.
docker image that you have chosen (example : runtime-400).
## 3. Generate the session keys of genesis authority
### Generate the session keys of genesis authority
Generate a random secret phrase:
```bash
$ docker run --rm -it --entrypoint duniter duniter/duniter-v2s:TAG key generate
$ docker run --rm duniter/duniter-v2s:TAG -- key generate
Secret phrase: noble stay fury mean poverty delay stadium organ evil east vague can
Secret seed: 0xb39c31fb10c5080721738880c2ea45412cb3df33df022bf8d9a51483b3a9b7a6
Public key (hex): 0x90a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d62
......@@ -35,38 +56,64 @@ Keep this secret phrase **carefully**, it will be used **several** times later.
Then, generate the session keys:
```bash
$ docker run --rm -it --entrypoint duniter duniter/duniter-v2s:TAG key generate-session-keys --chain CURRENCY_local --suri "<your secret phrase>"
$ docker run --rm duniter/duniter-v2s:TAG -- key generate-session-keys --chain CURRENCY_local --suri "<your secret phrase>"
Session Keys: 0x87189d723e1b2826c243bc433c718ac26ba60526932216a09102a254d54462b890a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d6290a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d6290a0c2866034db9d05f8193a95fe5af8d5e12ab295a501c17c95cdbeaf226d62
```
## 4. Paste sessions keys in the genesis configuration file
### Paste sessions keys in the genesis configuration file
An example of genesis configuration file: `resources/gdev.json`
An example of genesis configuration file: `resources/gdev.json`. Paste your session keys in your `smith` identity with key `session_keys`.
## 5. Generate raw spec
### Generate raw spec
```docker
docker run -v $HOME/dev/duniter-v2s/resources:/var/lib/duniter/resources -e DUNITER_GENESIS_CONFIG=/var/lib/duniter/resources/gdev.json --rm duniter/duniter-v2s:TAG -- build-spec -lerror --chain=gdev_live --raw > name-raw.json
```
```bash
./scripts/gen-live-network-raw-spec.sh CURRENCY "<path/to/your/genesis/config/file>"
```
## 6. Generate the docker compose and prepare nodes keys
This builds the specs using debug version of Duniter.
### Generate the docker compose and prepare nodes keys
```bash
./scripts/create-live-network.sh "<your secret phrase>" CURRENCY "<path/to/dist/folder>"
```
The distribution folder can then be copied to a server, then the compose must be launched from the
root of the distribution folder:
The new distribution folder can be copied to a server
```bash
scp -r -P SSH_PORT "<path/to/dist/folder>" user@ip:/remote/dist/path
cd "<path/to/dist/folder>"
docker-compose up -d
scp -r "<path/to/dist/folder>" <server>:/remote/dist/path
```
Then, on the server:
then on the server, launch the compose file from the the distribution folder's root:
```bash
ssh <server>
cd "<path/to/dist/folder>"
docker-compose up -d
docker compose up -d
```
This is the first node of the new live network.
## Finalization
The following steps should be completed once you are satisfied with the new live network.
### Rotate session keys
You should rotate session keys for more secured keys produced on the server (the one you used before are still in your develop machine bash history and clipboard).
### Publish image
With these new session keys in the chainspec and the runtime build with srtool, you can release the new runtime again with:
```bash
cargo xtask release-runtime 400
```
### Tell the other smith
Once you completed all these steps, the other smith can pull the docker image with a genesis containing your bootnode with the correct session keys. They can base their `docker-compose.yml` on the `duniter-validator` template.
# Duniter Pallet Conventions
## Call
Custom Duniter pallet calls should adhere to the standard Substrate naming convention:
- `action_` for regular calls (e.g., `create_identity`).
- `force_action_` for calls with a privileged origin (e.g., `force_set_distance_status`).
## Error
In the event of a call failure, it should trigger a pallet error with a self-explanatory name, for instance, `IdtyNotFound`.
## Event
Successful calls should deposit a system event to notify external entities of the change. The event name should be self-explanatory and structured in the form of a Rust struct with named fields, ensuring clarity in autogenerated documentation. An example is:
```rust
IdtyRemoved {
idty_index: T::IdtyIndex,
reason: IdtyRemovalReason<T::IdtyRemovalOtherReason>,
}
```
## Hook
Hooks are inherently infallible, and no errors should be emitted within them. To monitor progression from inside the hook, events can be employed to inform external entities about changes or no-changes.
## Internal Function
Internal functions should adhere to the following naming convention:
- `do_action_` for regular functions executing the base logic of a call (e.g., `do_remove_identity_`). These functions should directly emit events and trigger errors as needed.
- `force_action_` for privileged functions that bypass any checks. This can be useful for specific benchmarking functions.
- `check_` for functions performing checks and triggering errors in case of failure.
# Release a new Runtime
![](img/release-pipeline.png)
> The following instructions have been described in french at: [Créer une release](https://forum.duniter.org/t/industrialiser-le-demarrage-dune-nouvelle-gx/11535/41).
## Process
Example for `runtime-800`.
### New release with new Runtime
* create a `release/runtime-800` branch locally
* update the values:
* update spec version (in `runtime/<currency>/src/lib.rs`)
* eventually update `gdev.yml` (smiths, tech. committee, ...)
* push the `release/runtime-800` branch
* in the CI/CD, wait for `Create release` button to be available and click on it (see above screenshot)
The Runtime is now available on the release page [runtime-800](https://git.duniter.org/nodes/rust/duniter-v2s/-/releases/runtime-800).
### New Client
The Client is published as a Docker image.
You may want to publish a new Client version along with a Runtime update.
#### New raw specs (optional)
For a reboot, you will likely want to update the raw specs:
* in the CI/CD, wait for `release_gdev_specs` button to be available and click on it
* in the CI/CD, wait for `release_gtest_specs` button to be available and click on it
* wait for both jobs to finish
* update the Client raw specs with `cargo xtask update-raw-specs runtime-800`
#### New version (mandatory)
Update Client values:
* update Client version (in `Cargo.toml`)
* update `Cargo.lock` with `cargo build`
#### Publish Docker image
Commit everything and push the branch:
* in the CI/CD, a new pipeline has been launched
* you can stop jobs `create_g1_data`, `gdev_srtool`, `gtest_srtool` (won't be used)
* click on `gdev_docker_deploy` and `gtest_docker_deploy`
The Docker images should now be available at: https://hub.docker.com/r/duniter/duniter-v2s-gdev/tags.
## Runtime tag and spec version
Our runtime tags use `xxyy` version numbers where `x` corresponds to major change and `y` hotfix.
Make sure to move any issue or merge request assigned to the choosen milestone `runtime-xxyy` to the next one. This prevents from forgetting unfinished work.
# How to replay a block
You can use `try-runtime` subcommand to replay a block against a real satte from a live network.
WARN: try-runtime is not properly implemented
You can use `try-runtime` subcommand to replay a block against a real state from a live network.
1. Checkout the git tag of the runtime version at the block you want to replay
2. Build duniter with feature `try-runtime`: `cargo build --features try-runtime`
......@@ -9,13 +11,13 @@ You can use `try-runtime` subcommand to replay a block against a real satte from
5. Replay the block a first time to get the state:
```
duniter try-runtime --exectuion=Native execute-block --block-at 0x2633026e3e428b010cfe08d215b6253843a9fe54db28748ca56de37e6a83c644 live -s tmp/snapshot1 -u ws://localhost:9944
duniter try-runtime --execution=Native execute-block --block-at 0x2633026e3e428b010cfe08d215b6253843a9fe54db28748ca56de37e6a83c644 live -s tmp/snapshot1 -u ws://localhost:9944
```
6. Then, replay the block as many times as you need against your local snapshot:
```
duniter try-runtime --exectuion=Native execute-block --block-at 0x2633026e3e428b010cfe08d215b6253843a9fe54db28748ca56de37e6a83c644 --block-ws-uri ws://localhost:9944 snap -s tmp/snapshot1
duniter try-runtime --execution=Native execute-block --block-at 0x2633026e3e428b010cfe08d215b6253843a9fe54db28748ca56de37e6a83c644 --block-ws-uri ws://localhost:9944 snap -s tmp/snapshot1
```
try-runtime does not allow (for now) to store the block locally, only the storage can be stored.
......@@ -35,7 +35,7 @@ Use a terminal shell to execute the following commands:
```bash
sudo apt update
# May prompt for location information
sudo apt install -y cmake pkg-config libssl-dev git build-essential clang libclang-dev curl
sudo apt install -y cmake pkg-config libssl-dev git build-essential clang libclang-dev curl protobuf-compiler
```
### Arch Linux
......@@ -79,3 +79,16 @@ rustup update nightly
rustup update stable
rustup target add wasm32-unknown-unknown --toolchain nightly
```
### Installing mold linker to decrease build time
Mold (modern linker) (https://github.com/rui314/mold) decreases the build time. Install it through your system package for example then add the following to your `~/.cargo/config`:
```toml
[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=/usr/bin/mold"]
```
(see https://forum.duniter.org/t/decrease-duniter-build-time/10170 on the forum)
\ No newline at end of file
# Polkadot Upgrade Guide
ParityTech frequently releases upgrades of the polkadot-sdk. For each upgrade, Duniter should be upgraded following the instructions below. These instructions are based on upgrading from version 1.8.0 to 1.9.0.
## 1. Upgrade the duniter-polkadot-sdk
* Clone the repository: `git clone git@github.com:duniter/duniter-polkadot-sdk.git`
* Set the upstream repository: `git remote add upstream git@github.com:paritytech/polkadot-sdk.git`
* Fetch the latest released version: `git fetch --tag polkadot-v1.9.0`
* Create a new branch: `git checkout -b duniter-polkadot-v1.9.0`
* Rebase the branch, keeping only specific commits: "fix treasury benchmarks when no SpendOrigin", "allow manual seal to produce non-empty blocks with BABE", "add custom pallet-balance GenesisConfig", and "remove pallet-balances upgrade_account extrinsic", "remove all paritytech sdk dependencies".
* Push the new branch: `git push`
## 2. Upgrade repository
* In the `Cargo.toml` file of Duniter, change the version number from 1.8.0 to 1.9.0 for all polkadot-sdk dependencies. Also, change the version for Subxt. `find . -type f -name "Cargo.toml" -exec sed -i'' -e 's/polkadot-v1.8.0\/polkadot-v1.9.0/g' {} +`.
* Upgrade the version number of all crateio dependencies to ensure compatibility with those used in the polkadot-sdk, see the node template at: [Node Template](https://github.com/paritytech/polkadot-sdk/blob/master/templates/solochain/node/Cargo.toml) (choose the correct branch/tag).
At this point, two cases may arise:
1. If the upgrade only adds some types and minor changes, add the types in the pallet configuration, replace the offending `WeightInfo`, and delete the corresponding weights files until they can be regenerated.
2. If there are many breaking changes, it is recommended to break down the process:
* Start by correcting errors on individual pallets using `cargo check -p my_pallet` to identify and rectify any errors. Then, test using `cargo test -p my_pallet` and benchmark using `cargo test -p my_pallet --feature runtime-benchmark`.
* After correcting all pallets, fix the runtimes using the same approach: check for trait declarations added or removed in each pallet configuration, and use `cargo check -p runtime`, `cargo test -p runtime`, and `cargo test -p runtime --feature runtime-benchmark`.
* Repeat this process with the node part, the distance-oracle, all the tests, xtask, and the client.
* Conclude the process by executing all benchmarks using the command `scripts/run_all_benchmarks.sh`.
## 4. Troubleshooting
As Duniter may sometimes be the only chain implementing advanced features, such as manual sealing, not many references can be found. However, the following projects may be useful:
* Node template for general up-to-date implementation: [Node Template](https://github.com/paritytech/polkadot-sdk/tree/master/templates)
* Acala: [Acala](https://github.com/AcalaNetwork/Acala), which also uses manual sealing add a similar node implementation.
# Compile the runtime with srtool
```docker
When voting for a runtime upgrade, you should check that the proposed hash actually corresponds to the published code you reviewed. Otherwise, a malicious runtime upgrade could be advertised as a legitimate one.
```sh
mkdir runtime/gdev/target
chmod o+w runtime/gdev/target
# Workaround see !239
echo -e "[toolchain]\nchannel = \"1.74.0\"\ncomponents = [ \"rust-std\", \"rust-src\" ]" > runtime/gdev/rust-toolchain.toml
docker run \
-i \
--rm \
-e PACKAGE=gdev-runtime \
-e RUNTIME_DIR=runtime/gdev \
-v $PWD:/build \
paritytech/srtool:1.60.0 build --app --json -cM
paritytech/srtool:1.74.0-0.13.0 build --app --json -cM
```
Then, the runtime wasm bytecode is generated in this location:
......@@ -17,5 +23,6 @@ runtime/gdev/target/srtool/release/wbuild/gdev-runtime/gdev_runtime.compact.comp
```
To compare it to last official :
- download it here : https://git.duniter.org/nodes/rust/duniter-v2s/-/releases
- compare `sha256sum` of it and of the one you've built
# How to benchmarks weights of a Call/Hook/Pallet
# Weights benchmarking
1. Create the benchmarking tests, see commit 31057e37be471e3f27d18c63818d57cc907b4b4f for a
## What is the reference machine?
For now (09/2022), it's a `Raspberry Pi 4 Model B - 4GB` with an SSD connected via USB3.
To cross-compile the benchmarks binary for armv7:
```
./scripts/cross-build-arm.sh --features runtime-benchmarks
```
The cross compiled binary is generated here: `target/armv7-unknown-linux-gnueabihf/release/duniter`
## How to benchmarks weights of a Call/Hook/Pallet
1. Create the benchmarking tests. See commit f5f2ae969ac592ba9957b0e40e18d6e4b0048473 for a
complete real example.
2. Run the benchmark test on your local machine:
`cargo test -p <pallet> --features runtime-benchmarks`
```
cargo test -p <pallet> --features runtime-benchmarks
```
3. If the benchmark tests compiles and pass, compile the binary with benchmarks on your local
machine: `cargo build --release --features runtime-benchmarks`
4. Run the benchmarks on your local machine (to test if it work mith a real runtime). The command
is: `duniter benchmark pallet --chain=CURRENCY-dev --steps=50 --repeat=20 --pallet=pallet_universal_dividend --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=.`
5. If it work, use the generated file content to create or update the `WeightInfo` trait and the `()` dummy implementation. Then use the `WeightInfo` tarit in the real code of the pallet. See 79e0fd4bf3b0579279fc957da5e2fdfc6d8a17fa for a
complete real example.
machine:
```
cargo build --release --features runtime-benchmarks
```
4. Run the benchmarks on your local machine (to test if it work with a real runtime). See 0d1232cd0d8b5809e1586b48376f8952cebc0d27 for a complete real example. The command is:
```
duniter benchmark pallet --chain=CHAINSPEC --steps=50 --repeat=20 --pallet=<pallet> --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/common/src/weights/
```
5. Use the generated file content to create the `WeightInfo` trait and the `()` dummy implementation in `pallets/<pallet>/src/weights.rs`. Then use the `WeightInfo` trait in the real code of the pallet. See 62dcc17f2c0b922e883fbc6337a9e7da97fc3218 for a complete real example.
6. Redo steps `3.` and `4.` on the reference machine.
7. Put the generated file on `runtime/common/src/weights` and use it in the runtimes configuration.
See cee7c3b2763ba402e807f126534d9cd39a8bd025 for a complete real example.
Note 1: You Must replace `CURRENCY` by the currency type, or for ĞDev use directly `--chain=dev`.
7. Use the `runtime/common/src/weights/pallet_<pallet>.rs` generated on the reference machine in the runtimes configuration. See af62a3b9cfc42d6653b3a957836f58540c18e65a for a complete real example.
Note 1: Use relevant chainspec for the benchmarks in place of `CHAINSPEC`, for example `--chain=dev`.
Note2. If the reference machine not support wasmtime, you should replace `--wasm-execution=compiled`
Note 2: If the reference machine does not support wasmtime, you should replace `--wasm-execution=compiled`
by `--wasm-execution=interpreted-i-know-what-i-do`.
## What the reference machine is?
## Generate base block benchmarking
For now (06/2022), it's a `Raspberry Pi 4 Model B - 4GB` with an SSD connected via USB3.
1. Build binary for reference machine and copy it on reference machine.
2. Run base block benchmarks command:
To cross-compile the benchmarks binary for armv7:
```
duniter benchmark overhead --chain=dev --execution=wasm --wasm-execution=compiled --weight-path=./runtime/common/src/weights/ --warmup=10 --repeat=100
```
3. Commit changes and open an MR.
## Generate storage benchmarking
1. Build binary for reference machine and copy it on reference machine.
2. Copy a DB on reference machine (on ssd), example: `scp -r -P 37015 tmp/t1 pi@192.168.1.188:/mnt/ssd1/duniter-v2s/`
3. Run storage benchmarks command, example:
```
./scripts/cross-build-arm.sh --features runtime-benchmarks
duniter benchmark storage -d=/mnt/ssd1/duniter-v2s/t1 --chain=gdev --mul=2 --weight-path=. --state-version=1
```
4. Copy the generated file `paritydb_weights.rs` in the codebase in folder `runtime/common/src/weights/`.
5. Commit changes and open an MR.
The cross compiled binary is generated here: `target/armv7-unknown-linux-gnueabihf/release/duniter`
## How to Write Benchmarks
### Calls
Ensure that any extrinsic call is benchmarked using the most computationally intensive path, i.e., the worst-case scenario.
### Hooks
Benchmark each hook to determine the weight consumed by it; hence, it is essential to benchmark all possible paths.
### Handlers and Internal Functions
When designing handlers and internal functions, it is advisable to avoid having them return weight for the following reasons:
1. **Simplified Benchmarking**: Writing benchmarks for hooks or calls where handlers and internal functions are utilized becomes more straightforward.
2. **Reduced Benchmarking Complexity**: By directly measuring execution and overhead in a single pass, the number of benchmarks is minimized.
3. **Enhanced Readability**: Understanding that weight accounting occurs at the outermost level improves the overall readability of the code.
One notable exception is the internal functions called in hooks like `on_idle` or `on_initialize` that can be easier to benchmark separately when the hook contains numerous branching.
# How to Build Duniter-V2S Debian Package
Compile packages for native integration for Debian-based systems.
## With Docker (on any system)
1. Install Docker and Docker Buildx.
2. Use the `scripts/build-deb.sh` script.
3. The `.deb` packages will be located in the `target/debian` folder.
## Without Docker (on a Debian-based system)
1. Install the necessary dependencies:
```sh
sudo apt-get install -y clang cmake protobuf-compiler libssl-dev
```
2. Compile the project:
```sh
cargo build --release
```
3. Install `cargo-deb`:
```sh
cargo install cargo-deb
```
4. Build the Duniter node `.deb` package:
```sh
cargo deb --no-build -p duniter
```
5. The `.deb` package will be located in the `target/debian` folder.
# How to build duniter-v2s for arm
Cross-compile Duniter to arm (e.g. Raspberry Pi).
## With Docker
1. Create a docker image that contains the build environment
```bash
......@@ -13,3 +17,20 @@ docker build -t duniter-v2s-arm-builder -f docker/cross-arm.Dockerfile .
```
then, get the final binary at `target/armv7-unknown-linux-gnueabihf/release/duniter`.
## Without Docker
**Warning**: armv7 (default for Raspberry Pi) is **not** supported. Linux on RPi can be easily switched to aarch64, please search how to do so on the Internet.
This produces a musl build: the resulting executable is static, hence more portable than a dynamic one. It will be compatible with systems older than the compilation host.
```bash
# Install the tools
rustup target add aarch64-unknown-linux-musl --toolchain nightly-2023-08-23-x86_64-unknown-linux-musl
sudo dpkg --add-architecture arm64
sudo apt update
sudo apt install musl-dev:arm64 musl-tools g++-aarch64-linux-gnu gcc-aarch64-linux-gnu
# Cross-compile
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc cargo build --target=aarch64-unknown-linux-musl --release
```
# How to Build the Duniter RPM Package
1. Install dependencies:
```sh
# Fedora
sudo dnf install clang cmake protobuf-compiler openssl-devel
```
2. Compile the project:
```sh
cargo build --release
```
3. Install `cargo-generate-rpm`:
```sh
cargo install cargo-generate-rpm
```
4. Build the package:
```sh
cargo generate-rpm -p node
```
5. The `.rpm` package will be located in the `target/generate-rpm` folder.
# Autocompletion
One can generate autocompletion for its favorite shell using the following option:
You can generate autocompletion for your favorite shell using the following option:
```sh
cargo run --release -- completion --generator <GENERATOR>
cargo run -- completion --generator <GENERATOR>
```
Where `GENERATOR` can be any of `bash`, `elvish`, `fish`, `powershell` and `zsh`.
......@@ -14,7 +14,7 @@ First, get the completion file in a known place:
```sh
mkdir -p ~/.local/share/duniter
cargo run --release -- completion --generator bash > ~/.local/share/duniter/completion.bash
cargo run -- completion --generator bash > ~/.local/share/duniter/completion.bash
```
You can now manually source the file when needed:
......@@ -30,3 +30,17 @@ Or you can automatically source it at `bash` startup by adding this to your `~/.
```
You can now enjoy semantic completion of the `./target/release/duniter` command using `<Tab>` key.
## Zsh
Zsh equivalent
```sh
# make directory to store completion
mkdir -p ~/.zsh/completion
# write the completion script
cargo run -- completion --generator zsh > ~/.zsh/completion/_duniter.zsh
# add the following lines to your ~/.zshrc
fpath+=(~/.zsh/completion)
compinit # might slow down terminal startup
```
# Distance rule evaluation
The [distance rule](https://duniter.org/blog/duniter-deep-dive-wot/) is computationally too heavy to be handled by the runtime. Therefore it is computed offchain using the distance oracle.
Distance evaluation is operated on a voluntary basis by individual smiths. Since evaluators can lie or make errors, the result considered for applying the distance rule is the median of results published by the different evaluators.
## Running distance evaluation
Any smith member authoring blocks can run a distance evaluation oracle. It is better to have a machine more powerful than the reference machine.
Create a service from this command line, run by the same user as Duniter, on the same system:
/absolute/path/to/duniter distance-oracle --interval <duration>
The duration is the number of seconds between two evaluations. It should be less than the duration of a distance evaluation period. If it is equal, your node may not have the time to evaluate distance.
The oracle communicates with Duniter using its RPC API and using temporary files. Without additional (unsupported) configuration, both must run on the same filesystem. The node also needs to be forging blocks for the evaluations to be published.
### Additional Duniter configuration
Duniter should keep states at least one distance evaluation period old. If this is more than the default 256 and your node is not already an archive (`--state-pruning archive`), use the option `--state-pruning <blocks>`.
## Weights and Fees Calculation
### Introduction
Transaction weights and fees ensure network efficiency, fairness, and security in Substrate-based blockchains. These concepts are designed to manage resource allocation and incentivize proper usage of the blockchain infrastructure.
### Transaction Weights
Transaction weight measures the computational resources required to process a transaction. It is determined by factors such as the complexity of the transaction logic and the amount of data involved. Transactions with higher weights consume more resources and thus contribute to the overall load on the network.
### Transaction Fees
Transaction fees in Substrate-based blockchains are crucial for efficiently managing network resources and sustaining economic viability. They regulate resource allocation by ensuring transactions consuming more computational resources incur higher fees, discouraging spam, and promoting fair use of network capacity.
The fees are computed as follows:
`fee = base_fee + weight2fee * fee_multiplier + length2fee + tip`
## Fees in Duniter
### Fees Implementation Details
Implementing a zero-fee chain in Duniter involves configuring the blockchain to waive transaction fees when the current block weight is below a specified target. This approach aims to promote accessibility and encourage participation by eliminating fees during periods of lower network activity.
However, transaction fees are applied when the block weight or length surpasses the defined targets to ensure network security and stability during increased usage. Additionally, leveraging the fee multiplier mechanism helps deter potential prolonged network attacks by dynamically adjusting fee levels based on previous network conditions.
Duniter members benefit from the quota system, which refunds transaction fees during high network activity periods.
Fees are computed as follows:
* If `current_weight < 0.25 * max_weight` and `current_length < 0.25 * max_length` and `fee_multiplier = 1`, ie. normal load:
`fee = 0`
* If `current_weight > 0.25 * max_weight` or `current_length > 0.25 * max_length` or `fee_multiplier > 1`, ie. heavy usage (approximately more than 135 transactions per second):
`fee = `5cĞ1 + extrinsic_weight * (5cĞ1/base_extrinsic_weight)* fee_multiplier + extrinsic_length/100 + tip`
The multiplier is updated as follows:
* If `current_weight > 0.25 * max_weight` or `current_length > 0.25 * max_length`:
`Min(fee_multiplier += 1, 10)`
* If `current_weight < 0.25 * max_weight` and `current_length < 0.25 * max_length`:
`Max(fee_multiplier -= 1, 1)`
# Debian Setup Instructions
## Mirror Node
1. Download the Duniter .deb file.
2. Install the package: `dpkg -i duniter_vx.y.z.deb`.
3. Change the default configuration (at least the node name) by modifying `/etc/duniter/env_file`.
4. Start the service: `sudo systemctl start duniter-mirror.service`.
5. Enable the service at startup: `sudo systemctl enable duniter-mirror.service`.
## Smith Node
1. Download the Duniter .deb file.
2. Install the package: `dpkg -i duniter_vx.y.z.deb`.
3. Change the default configuration (at least the node name) by modifying `/etc/duniter/env_file`.
4. Create network keys using the same base path as in the config file: `duniter2 key generate-node-key --base-path <YOUR_BASE_PATH> --chain <YOUR_CHAIN>`.
5. Start the service: `sudo systemctl start duniter-smith.service`.
6. Enable the service at startup: `sudo systemctl enable duniter-smith.service`.
## Distance Oracle
A Smith node needs to be installed.
1. Change the default configuration by modifying `/etc/duniter/env_file`.
2. Start the service: `sudo systemctl start distance-oracle.service`.
3. Enable the service at startup: `sudo systemctl enable distance-oracle.service`.
# How to deploy a permanent rpc node on ĞDev network
## Publish a node
### Duniter part
- Add this docker-compose on your server :
[docker/compose/gdev-rpc.docker-compose.yml](https://git.duniter.org/nodes/rust/duniter-v2s/-/blob/master/docker/compose/gdev-rpc.docker-compose.yml)
- Rename the file : `mv gdev-mirror.docker-compose.yml docker-compose.yml`
- In the same folder, create a `.env` file that defime environment variables `SERVER_DOMAIN` and `PEER_ID`:
```
SERVER_DOMAIN=YOUR_DOMAIN
PEER_ID=YOUR_PEER_ID
```
Your `PEER_ID` shoud be generated with this command: `docker run --rm -it --entrypoint duniter -v $PWD:/var/lib/duniter/ duniter/duniter-v2s:v0.1.0 key generate-node-key --file /var/lib/duniter/node.key`
- If you have write access errors run in docker-compose.yml folder : `chmod o+rwX -R .`
- `docker-compose up -d` to start your node
### Reverse-proxy part (with Nginx)
In `/etc/nginx/sites-enabled/gdev.YOUR_DOMAIN` put (you can probably do simpler) :
```
server {
server_name gdev.YOUR_DOMAIN.fr;
listen 443 ssl http2;
listen [::]:443 ssl http2;
ssl_certificate /etc/nginx/ssl/YOUR_DOMAIN.cert;
ssl_certificate_key /etc/nginx/ssl/YOUR_DOMAIN.key;
root /nowhere;
add_header X-Frame-Options SAMEORIGIN;
add_header X-XSS-Protection "1; mode=block";
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_read_timeout 90;
location /http {
proxy_pass http://localhost:9933;
proxy_http_version 1.1;
}
location /ws {
proxy_pass http://localhost:9944;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_read_timeout 1200s;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
}
}
```
and replace `YOUR_DOMAIN` by your domain each time.
- [generate your ssl certificates](https://github.com/acmesh-official/acme.sh) with let's encrypt
if you don't already have a wildcard certificate.
- `service nginx reload`
Your node is now online as a rpc node. It's fully capable for wallet use.
To go further, read [How to become a (black)smith](./smith.md)
## Upgrade your node with minimal interruption
1. Modify docker image tag on your compose file
2. Run `docker compose pull`, this will pull the new image.
3. Run `docker compose up -d --remove-orphans`, this will recreate the container
4. Verify that your node restarted well `docker compose logs duniter-rpc`
5. Remove the old image `docker images rmi duniter/duniter-v2s:OLD_TAG`
# How to become a (black)smith
## Publish a node
### Duniter part
- Add this docker-compose on your server :
[docker/compose/gdev-validator.docker-compose.yml](https://git.duniter.org/nodes/rust/duniter-v2s/-/blob/master/docker/compose/gdev-validator.docker-compose.yml)
- Create a `.env` file that define environment variables `SERVER_DOMAIN`, `PEER_ID` and `VALIDATOR_PEER_ID`:
- `SERVER_DOMAIN`: a domain name that point on your server
- `PEER_ID`: Your rpc node peer id, shoud be generated with this command: `docker run --rm -it --entrypoint -v $PWD:/var/lib/duniter/ duniter duniter/duniter-v2s:v0.1.0 key generate-node-key --file /var/lib/duniter/rpc-node.key`
- `VALIDATOR_PEER_ID`: Your validator node peer id, shoud be generated with this command: `docker run --rm -it --entrypoint -v $PWD:/var/lib/duniter/ duniter duniter/duniter-v2s:v0.1.0 key generate-node-key --file /var/lib/duniter/validator-node.key`
Note: duniter-rpc PEER_ID and duniter-validator PEER_ID isn't the same.
- If you have write access errors run in docker-compose.yml folder : `chmod o+rwX -R .`
- `docker-compose up -d` to start your node
### Reverse-proxy part (with Nginx)
In `/etc/nginx/sites-enabled/gdev.YOUR_DOMAIN` put (you can probably do simpler) :
```
server {
server_name gdev.YOUR_DOMAIN.fr;
listen 443 ssl http2;
listen [::]:443 ssl http2;
ssl_certificate /etc/nginx/ssl/YOUR_DOMAIN.cert;
ssl_certificate_key /etc/nginx/ssl/YOUR_DOMAIN.key;
root /nowhere;
add_header X-Frame-Options SAMEORIGIN;
add_header X-XSS-Protection "1; mode=block";
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_read_timeout 90;
location /http {
proxy_pass http://localhost:9933;
proxy_http_version 1.1;
}
location /ws {
proxy_pass http://localhost:9944;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_read_timeout 1200s;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
}
}
```
and replace `YOUR_DOMAIN` by your domain each time.
- [generate your ssl certificates](https://github.com/acmesh-official/acme.sh) with let's encrypt
if you don't already have a wildcard certificate.
- `service nginx reload`
Your node is now online as a rpc node. It's fully capable for wallet use.
## Join the Smith WoT
- add polkadot webextension to be able to authentificate with your account.
- Go to [any node with polkadotjs ui](https://gdev.1000i100.fr/dev-ui/?rpc=wss://gdev.1000i100.fr/ws)
- Ask to join Smith WoT (you need to already be in the main WoT)
- developer > extrinsics > YOUR_SMITH_ACCOUNT > smithMembership > requestMemberShip(metadata)
- add your p2p endpoint (optional)
- add your session key (follow point 1 to 4 from Validate blocks > Generate and publish your session key)
- Send the query
- Await smith certification : developer > extrinsics > CERTIFIER_SMITH_ACCOUNT > smithCert > addCert(receiver)
When you have at least 3 certifications, your'in !
## Validate blocks (blacksmith work)
- Generate and publish your session keys
1. create an ssh bridge from your desktop/laptop to your server : `ssh -L 9945:localhost:9945 SSH_USER@YOUR_SERVER`
2. In your browser go to [polkadotjs : ws://localhost:9945](https://polkadot.js.org/apps/?rpc=ws%3A%2F%2Flocalhost%3A9945#/explorer)
3. In the UI : developer > appel RPC > author > rotateKey() and run
4. copy the result in clipboard
5. In the UI : developer > extrinsics > YOUR_SMITH_ACCOUNT > authorityMembers > setSessionKeys(keys) then copy your session keys and run the query.
6. **wait 48h to verify you keep sync**
- Join
- In the UI : developer > extrinsics > YOUR_SMITH_ACCOUNT > authorityMembers > goOnline()
If you're not able to monitor, reboot, act on your node, goOffline() to avoid penality to the blockchain and to you.
## Upgrade your node with minimal interruption
1. Modify docker image tag on your compose file
2. Run `docker compose pull`, this will pull the new image.
3. Run `docker compose up -d --remove-orphans`, this will recreate the container
4. Verify that your node restarted well `docker compose logs duniter-validator`
5. Remove the old image `docker images rmi duniter/duniter-v2s:OLD_TAG`