Duniter Datapod

Duniter Datapod is designed for offchain storage of Ğ1 data but does not depend on Duniter and could be used independantly. It contains multiple components:

• a "collector" which listens on pubsub and puts index requests in a timestamped AMT
• an "indexer" which takes index requests of specific kinds and put them in a Postgres database to serve contente with Hasura GraphQL API
• a dev tool in Vue to understand the architecture, explore the data, and debug

sheme of data flow in Duniter Datapod

Use

To start a full indexer in production mode with docker, use the docker-compose.prod.yml and .env.prod.example files:

# edit env file
vim .env
# start services
docker compose up -d

This will pull preconfigured images for postgres/hasura, kubo and datapod. This should:

• connect to existing network
• start collecting from default IPNS
• index to database

You can then do a simple proxy_pass to HASURA_LISTEN_PORT and KUBO_GATEWAY_PORT. Read more on install page.

Dev

Install dev dependencies

# use node version 20
nvm use 20
# install dependencies
pnpm install

Start a kubo node with pubsub and postgres/hasura

# start kubo node (easier to have it on system instead of docker)
ipfs daemon --enable-pubsub-experiment
# start postgres / hasura from dev docker compose file
docker compose up -d

Copy edit and load your .env file

# copy from template
cp .env.example .env
# adapt .env file then export variables
source .env.sh

And start what you want (indexer, dev tool, c+ import...)

# run dev tool app (unmaintained)
pnpm dev
# run datapod indexer
pnpm start
# run given script
pnpm exec tsx ./src/scripts/cesium-plus-import.ts

More detail in the doc below.

Doc

• start the indexer (dev mode)
• using dev tool (for debugging)
• importing cesium plus data (advanced)
• edit the database (dev)

TODO

Bugs

• initialize dd_keys for new node (→ bootstrap)
• fix merging blocked when inode unreachable, timeout seems ignored
• fix pubsub socket closing after 5 minutes
• fix UND_ERR_HEADERS_TIMEOUT that happen often for unknown reasons

Features

• pubkey instead of ss58 address if we want data to be compatible across networks → ss58
• add periodic sync with a list of trusted peers (IPNS entries)
• split indexer vue app from backend indexer and improve node admin app
  • clarify the purpose of the the main TAMT
  • clarify the adressing format in the tables
  • count the number of documents
  • make the app build in prod mode
  • allow connecting the app to a custom RPC endpoint
• manage unpin requests when user/admin wants to delete data (requires ipfs refcount)
• document dev database change with tracking hasura console and squashing migrations
• add transaction comment (onchain + offchain to allow unified search)
• update description of pubkey field to "ss58/address"
• add ability to remove a node (and its parents as well if they become empty)
• [ ]