Complete migrator CI

https://git.duniter.org/nodes/rust/duniter-v2s/-/blob/56998122e42afd2c2c1642a72a6772a82490ccda/.gitlab-ci.yml#L270-L298

create_g1_data:
  stage: build
  rules:
    - if: $CI_PIPELINE_SOURCE != "merge_request_event" && $CI_COMMIT_BRANCH =~ /^(release\/runtime-)[0-9].*/
  image: python:3.9.18
  variables:
    DEBIAN_FRONTEND: noninteractive
    LEVELDB_PATH: /dump/duniter_default/data/leveldb
  script:
    # Duniter 1.8.7 dump
    - mkdir /dump
    - cd /dump
    # Export file constructed using Duniter 1.8.7:
    #  - bin/duniter sync g1.cgeek.fr --store-txs --nointeractive --mdb 1.8.7
    #  - mkdir -p /tmp/backup-g1-duniter-1.8.7
    #  - cp -R $HOME/.config/duniter/1.8.7/data /tmp/backup-g1-duniter-1.8.7
    #  - cp -R $HOME/.config/duniter/1.8.7/g1 /tmp/backup-g1-duniter-1.8.7
    #  - cp -R $HOME/.config/duniter/1.8.7/txs.db /tmp/backup-g1-duniter-1.8.7
    #  - tar -cvzf /tmp/backup-g1-duniter-1.8.7.tgz /tmp/backup-g1-duniter-1.8.7
    # Then the file is uploaded to dl.cgeek.fr manually
    - curl https://dl.cgeek.fr/public/backup-g1-duniter-1.8.7.tgz -o g1-dump.tgz
    - tar xvzf g1-dump.tgz
    - rm g1-dump.tgz
    - mv tmp/backup-g1-duniter-1.8.7 duniter_default
    # py-g1-migrator conversion
    - git clone https://git.duniter.org/tools/py-g1-migrator.git -b import_identities_from_leveldb /py-g1-migrator
    - cd /py-g1-migrator
    - rm -rf inputs/*
    - apt-get update
    - apt-get install -y sqlite3 libleveldb-dev jq
    - pip install -r requirements.txt
    # Export identities and wallets
    - ./main.py
    # Export transaction history
    - sqlite3 /dump/duniter_default/txs.db --json "select time,comment,issuers,outputs from txs;" > inputs/transactions_history.json 2>> inputs/txs.err
    - ./generate_transactions_history.py
    # Merge in one file
    - 'jq -s "{ identities: .[0].identities, wallets: .[0].wallets, initial_monetary_mass: .[0].initial_monetary_mass, current_block: .[0].current_block, transactions_history: .[1] }" output/gtest_genesis.json output/history.json > output/g1-data.json'
    # Make the exported file available for next jobs
    - mkdir -p $CI_PROJECT_DIR/release/
    - cp output/g1-data.json $CI_PROJECT_DIR/release/
  artifacts:
    expire_in: never
    paths:
      - $CI_PROJECT_DIR/release/
  tags:
    - kepler

git clone https://git.duniter.org/tools/py-g1-migrator.git -b import_identities_from_leveldb /py-g1-migrator

The current branch is import_identities_from_leveldb, we should merge this and complete py-g1-migrator with other things detailed below.

apt-get install -y sqlite3 libleveldb-dev jq

Would be nice to have a docker image with that included to gain time in the CI

pip install -r requirements.txt

Same for that, py-g1-migrator could have a docker image.

sqlite3 /dump/duniter_default/txs.db --json "select time,comment,issuers,outputs from txs;" > inputs/transactions_history.json 2>> inputs/txs.err

This part has to be replaced by what was developed by @pokapow in nodes/duniter-squid!20 (closed). Duniter does not mind the tx history, it is only for indexer purpose. However it makes sense to provide all files necessary for migration as release from Duniter CI.

# Merge in one file

Easier to manage with multiple smaller files.