Duniter v2 Substrate support with `polkadot-sdk`: - APIs: RPC, and GraphQL Hasura --- #### Resources - [`py-polkadot-sdk` repository](https://github.com/JAMdotTech/py-polkadot-sdk) - [`Polkadot SDK` documentation](https://jamdottech.github.io/py-polkadot-sdk/) - [`subxtpy` repository](https://github.com/philoniare/subxtpy) - [`subxtpy` documenation](https://philoniare.github.io/subxtpy/) - [Client developer doc](https://duniter.org/wiki/duniter-v2/doc/) - [Constants](https://forum.duniter.org/t/information-sur-les-constantes/12886/4) - https://git.duniter.org/clients/cesium-grp/cesium2s/-/issues - https://forum.duniter.org/t/vocabulaire-de-base-pour-comprendre-duniter-v2s-lecture-fortement-recommandee-pour-tous/9053 - https://forum.duniter.org/t/liste-des-parametres-protocolaires-de-duniter-v2s/9297/1 --- ### Transition plan - Release and maintain v0.12.0 as BMA-only - v0.20.0dev as Substrate-only Allow non-related developments/experimentation to happen in parallel?: Which would not create much git conflicts: - #79+: Seems the best candidate, the class development it-self, not the integration in the commands - #18+, #440+: PyGObject3, Textual TUI console: Just experimentation of the possibilities, interface design, no integration --- ### Consider Hexagonal architecture - duniterpy#117 - https://blog.octo.com/architecture-hexagonale-trois-principes-et-un-exemple-dimplementation/ [en](https://blog.octo.com/en/hexagonal-architecture-three-principles-and-an-implementation-example/) --- ### Set up Duniter v2s and indexer on ĞDev - [x] [Set-up Duniter-v2s](https://duniter.org/wiki/duniter-v2/run-smith/) - [x] [Become smith](https://duniter.org/wiki/duniter-v2/become-smith/) - [x] Set-up Indexer: [Subsquid](https://git.duniter.org/nodes/duniter-squid) ### Outline plan - Set up authentication based on DEWIF kept with `ED25519` encryption algorithm - Set up two network layers: RPC and Indexer GraphQL, make sure the two endpoints are sync and has the same genesis hash - Get currency, parameters, head: `blockchain/tools.py`, `wot/tools.py` identity helpers - port commands one by one to v2 ecosystem (starting by the easy one, then ramping up the difficulty) - v1 and v2 ecosystems support would coexist during the transition - hiding groups, commands and arguments can be done with Click - remove v1 network layer, public key support, tools, functions - remove `duniterpy` dependency (probably later) --- > - Split changes in child items/sub-tickets. Put them in the Kanban board and/or milestone. > - Ideas from [Tikka roadmap](https://forum.duniter.org/t/feuille-de-route-de-tikka/8977) > - [Matograine’s plan](https://git.duniter.org/matograine/silkaj/-/blob/ef232fb934249a15e2bb49580c2a8c8462b47a43/PLAN_TRANSITION_V2S.md) - ~~Delete code of commands which will be rewritten, keep structure, pick code in git history for the reimplementation~~ - Delete step-by-step commands, functions which have been rewritten #### Delete features which don’t make sense anymore and which are not very coupled - !239+ - [x] Delete `blockchain verify` cmd, doesn’t make sense anymore, algorithm can be retrieved in the git history for next `blocks` implementation (#283) - [x] Delete `blockchain diffi`, since the PoW is no longer a thing with BABE/Grandpa consensus algorithm - [x] Delete `checksum` command since the address integrity (checksum) and currency is integrated into the address. - [x] `blockchain blocks`: Does not make sense with https://duniter-portal.axiom-team.fr/ the blocks exploration is more interesting in a web interface. Could be reimplemented later. - [x] `blockchain info` #### Set up network layers - [x] Initializing rpc and gql clients on requests. Eases the testing. - [x] #370+ - [x] Patch/Mock twice same RPC or indexer function call: https://stackoverflow.com/a/75549927 ##### RPC Substrate API, library/SDK - [x] Introduce [`polkadot-sdk`](https://pypi.org/project/substrate-interface/) dependency. [Maintenance concerns](https://forum.duniter.org/t/py-polkadot-sdk-maintenance/12920) - [ ] #489+: Missing Py3.14 wheels for Alpine/`musl` - Consider [`subxtpy`](https://forum.duniter.org/t/12914/2) (async, no Py3.1{3,4} support), no longer maintained - [PolkaBind](https://polkabind.dev/): Project to bind `subxt` official Rust lib to Python - Consider [async-substrate-interface](https://github.com/opentensor/async-substrate-interface) (which can be used without asynchronous property). Dependent on [`from bittensor_wallet.utils import SS58_FORMAT`](https://github.com/opentensor/bittensor/blob/master/bittensor/core/settings.py#L68). Not standalone library. - Check [Tikka substrate own client library](https://git.duniter.org/clients/python/tikka/-/blob/main/tikka/adapters/network/node/substrate_client.py) - [x] Set up RPC API network layer ##### Indexer GraphQL API - [x] Set up GraphQL API network layer with [`gql`](https://github.com/graphql-python/gql), [doc](https://gql.readthedocs.io/en/stable/), [repology](https://repology.org/badge/vertical-allrepos/python:gql.svg) - [x] choose HTTP transporter (`httpx` or `requests`): choosing `requests` because it is the library used by `py-polkadot-sdk` and `pydiscourse` to avoid HTTP lib duplication. - Choosing `websockets` for the subscriptions #### Pre-requisites - [x] #477+ - [x] Ability to pass click `obj` to `silkaj.cli.cli` in order to the RPC client via `CliRunner.invoke(cmd, args, obj=ojb)`. - [x] Store click context obj into a class to ease patching and access - [x] Get rid of `--display`/`--dry-run` options #314+ - [x] Hide commands and groups which have broken tests and will be ported in next release ##### Authentication - Not using SR25519 because this would [reduce our possibilities](https://forum.duniter.org/t/use-ed25519-vs-sr25519-for-v2s-clients/12996/24). With ED25519 (used since Duniter v1), we can still access the seed and do everything. With SR25519 we are locked and can’t get the seed. - Therefore we are not implementing the encrypted JSON authentication format provided by `py-polkadot-sdk` since it only supports ED25519. We therefore stay with (E)WIF authentication format - Do we set mnemonic as default import method, instead of Scrypt? ##### Tools, functions, helpers - [x] Get currency code/name from RPC: `substrate.{chain,currency_symbol}` - [x] Set ss58 format address - [x] Request `uids` from the indexer to set the link between the identity index and the identity username/alias - [x] Implement Click type checking when passing a ss58 format address - [x] Implement mnemonic authentication import - [x] `Amount` class: #193+ - [x] `UDValue` - [x] #211+ #### Port existing commands/features - [x] `wot lookup`: #389, #387 - [x] `wot status`: missing distance, sentry info - [x] `money history`: minor topics to be considered, #339 - [x] #478+ - [x] `wot certify`, #375+ - [x] `money balance`: [Forum Post](https://forum.duniter.org/t//13154), nodes/rust/duniter-v2s!328, nodes/duniter-squid!27 - [x] #350+ Release taking too long, postponing other features. --- #### %"0.21.0" - [ ] `money transfer`: - [x] basic transfer - [x] [transfer reference](https://forum.duniter.org/t/commentaires-de-transaction-dans-tikka/12722/2) - [x] multi-recipients with `batch` - Remove `--outputBackChange` option, rename `--allSources`? #### Important features to be ported Features only provided by Silkaj, so once we switch the Ğ1 to v2s ecosystem, following features will stop functioning: - multi-recipients transfers to Ğ1 technical ecosystem remunerations - DeathReaper: Keep it similar listening to the blockchain events. Listen to exclusions from the indexer. Could be replaced with a feature onchain to report via email the exclusion. The datapod could store the email addresses. Having the emails accessible to everyone would be an issue regarding spam purposes. A private service/datapod should store the emails. #### %"0.22.0" - [ ] `wot exclusions` - [ ] #513+ - [ ] `wot membership` - [ ] `wot revocation`: (support reading v1 revocation format file until a deprecation date, support generating and reading v2 format) - [ ] Drop DuniterPy library usage #### Post tasks after initial porting - [ ] Delete modules: `public_key` (unless needed to display corresponding v1 pubkey) - [ ] Get rid of DuniterPy usage as much as possible - What might remain is authentication support, v1 pubkey support, CS+ usage. Copy code into Silkaj to get rid of DuniterPy dependency. - Remove mention in doc - [ ] Remove `unit` and `integration` tests directories. Doing the opposite from #441. The separation was used for tests requiring network and the others with mocked network. I would like to have integration and unit tests in the same file since they usually share the same patches/mocks otherwise it means maintaining two of them and forgetting about the once in the `integration` directory. End-to-end tests could be kept/added later in a `endtoend` directory for completeness. #### %"0.23.0": Implement new features - [ ] Re-introduce `-gt/--gtest` general option. Needs to set-up official RPC and GraphQL endpoints. - [ ] #48+: parameters from RPC metadata constants, the indexer - [ ] Display fees and refund estimations on all kind of transactions: nodes/rust/duniter-v2s!331 - [ ] Storing rpc and gql clients in Click context/object causes issue in case we would like to use the functions as a library. - [ ] `identity`: `create`, `confirm`, `link-account`, `change-owner-key` #90+ - [ ] `wot status` display expired certifications - [ ] [Claim UD](https://forum.duniter.org/t/implementation-creation-manuelle-du-du-et-guide-pour-dev-clients-wallet/9544) - [ ] Display UDs in `history` command (claimed and unclaimed) - [ ] [Smith sub-WoT](https://forum.duniter.org/t/la-sous-toile-forgerons/9047) handling - [ ] Commands similar to normal wot?: `smith`: `status`: view to check smiths status similar to Duniter Panel, `certify`, `invite`, `accept`, `update-key`/`set-session-key`, [`check`](https://forum.duniter.org/t/noeud-forgeron-avec-probleme-de-cle-de-session/12143/7), `go-online`, `go-offline`, `lookup`? - [ ] [Integrate Smith license](https://git.duniter.org/1000i100/g1_monetary_license/-/blob/master/smith_license/g1_smith_license_fr.adoc) - [Becoming Smith diagram](https://forum.duniter.org/t/becoming-smith-as-a-bpmn-diagram/10980) - [ ] [Check whether the RPC API connected to is in `Unsafe`, `Safe` mode](https://forum.duniter.org/t/session-keys-directement-dans-le-panneau-forgeron/12445/4) - [ ] DeathReaper: report smith exclusions: could be due to main WoT exclusion, session expiration - (`blockchain blocks`) - [ ] Technical Committee: `tech-committee`: `list-members`, `list-proposals`, `vote`, `close` - [ ] Mnemonic derivations (`//0`, `//1`,…) [Ğcli](https://forum.duniter.org/t/request-for-review-adding-db-persistence-for-all-secretformat-of-vault-keys-as-well-as-supporting-derivations/12829/5) - [ ] Endpoint discovery [1](https://forum.duniter.org/t/liste-des-endpoints/12347/60), [2](https://forum.duniter.org/t/nouvelle-version-duniter-0-10-1/13051/3), [3](https://forum.duniter.org/t/nouvelle-version-duniter-0-10-0-attention-utilise-100-du-cpu/13001/3) #### Later - [ ] Display fees per transaction and ask for confirmation (some are reimbursed, so mention the fees are reimbursed). Document it on the documentation. - [ ] [Schedule transfer support](https://forum.duniter.org/t/gdev-virements-reguliers/9459)