Complete migrator CI
https://git.duniter.org/nodes/rust/duniter-v2s/-/blob/56998122e42afd2c2c1642a72a6772a82490ccda/.gitlab-ci.yml#L270-L298
```
create_g1_data:
stage: build
rules:
- if: $CI_PIPELINE_SOURCE != "merge_request_event" && $CI_COMMIT_BRANCH =~ /^(release\/runtime-)[0-9].*/
image: python:3.9.18
variables:
DEBIAN_FRONTEND: noninteractive
LEVELDB_PATH: /dump/duniter_default/data/leveldb
script:
# Duniter 1.8.7 dump
- mkdir /dump
- cd /dump
# Export file constructed using Duniter 1.8.7:
# - bin/duniter sync g1.cgeek.fr --store-txs --nointeractive --mdb 1.8.7
# - mkdir -p /tmp/backup-g1-duniter-1.8.7
# - cp -R $HOME/.config/duniter/1.8.7/data /tmp/backup-g1-duniter-1.8.7
# - cp -R $HOME/.config/duniter/1.8.7/g1 /tmp/backup-g1-duniter-1.8.7
# - cp -R $HOME/.config/duniter/1.8.7/txs.db /tmp/backup-g1-duniter-1.8.7
# - tar -cvzf /tmp/backup-g1-duniter-1.8.7.tgz /tmp/backup-g1-duniter-1.8.7
# Then the file is uploaded to dl.cgeek.fr manually
- curl https://dl.cgeek.fr/public/backup-g1-duniter-1.8.7.tgz -o g1-dump.tgz
- tar xvzf g1-dump.tgz
- rm g1-dump.tgz
- mv tmp/backup-g1-duniter-1.8.7 duniter_default
# py-g1-migrator conversion
- git clone https://git.duniter.org/tools/py-g1-migrator.git -b import_identities_from_leveldb /py-g1-migrator
- cd /py-g1-migrator
- rm -rf inputs/*
- apt-get update
- apt-get install -y sqlite3 libleveldb-dev jq
- pip install -r requirements.txt
# Export identities and wallets
- ./main.py
# Export transaction history
- sqlite3 /dump/duniter_default/txs.db --json "select time,comment,issuers,outputs from txs;" > inputs/transactions_history.json 2>> inputs/txs.err
- ./generate_transactions_history.py
# Merge in one file
- 'jq -s "{ identities: .[0].identities, wallets: .[0].wallets, initial_monetary_mass: .[0].initial_monetary_mass, current_block: .[0].current_block, transactions_history: .[1] }" output/gtest_genesis.json output/history.json > output/g1-data.json'
# Make the exported file available for next jobs
- mkdir -p $CI_PROJECT_DIR/release/
- cp output/g1-data.json $CI_PROJECT_DIR/release/
artifacts:
expire_in: never
paths:
- $CI_PROJECT_DIR/release/
tags:
- kepler
```
> `git clone https://git.duniter.org/tools/py-g1-migrator.git -b import_identities_from_leveldb /py-g1-migrator`
The current branch is `import_identities_from_leveldb`, we should merge this and complete py-g1-migrator with other things detailed below.
> `apt-get install -y sqlite3 libleveldb-dev jq`
Would be nice to have a docker image with that included to gain time in the CI
> `pip install -r requirements.txt`
Same for that, `py-g1-migrator` could have a docker image.
> `sqlite3 /dump/duniter_default/txs.db --json "select time,comment,issuers,outputs from txs;" > inputs/transactions_history.json 2>> inputs/txs.err`
This part has to be replaced by what was developed by @pokapow in https://git.duniter.org/nodes/duniter-squid/-/merge_requests/20. Duniter does not mind the tx history, it is only for indexer purpose. However it makes sense to provide all files necessary for migration as release from Duniter CI.
> `# Merge in one file`
Easier to manage with multiple smaller files.
issue