squid-template
Duniter squid project. It accumulates [ğdev] account balances and serves them via graphql API. For more info consult FAQ.
Prerequisites
- node 16.x
- docker
Quickly running the sample
# 1. Install dependencies
npm ci
# 2. Compile typescript files
npm run build
# 3. Start target Postgres database
docker compose up -d
# 4. Apply database migrations from db/migrations
npx sqd db migrate
# 5. Now start the processor
node -r dotenv/config lib/processor.js
# 6. The above command will block the terminal
# being busy with fetching the chain data,
# transforming and storing it in the target database.
#
# To start the graphql server open the separate terminal
# and run
npx squid-graphql-server
Dev flow
1. Define database schema
Start development by defining the schema of the target database via schema.graphql
.
Schema definition consists of regular graphql type declarations annotated with custom directives.
Full description of schema.graphql
dialect is available here.
2. Generate TypeORM classes
Mapping developers use TypeORM EntityManager
to interact with target database during data processing. All necessary entity classes are
generated by the squid framework from schema.graphql
. This is done by running npx sqd codegen
command.
3. Generate database migration
All database changes are applied through migration files located at db/migrations
.
sqd(1)
tool provides several commands to drive the process.
It is all TypeORM under the hood.
# Connect to database, analyze its state and generate migration to match the target schema.
# The target schema is derived from entity classes generated earlier.
npx sqd db create-migration
# Create template file for custom database changes
npx sqd db new-migration
# Apply database migrations from `db/migrations`
npx sqd db migrate
# Revert the last performed migration
npx sqd db revert
# DROP DATABASE
npx sqd db drop
# CREATE DATABASE
npx sqd db create
4. Generate TypeScript definitions for substrate events and calls
This is an optional part, but it is very advisable.
Event and call data comes to mapping handlers as a raw untyped json. Not only it is unclear what the exact structure of a particular event or call is, but it can also rather frequently change over time.
Squid framework provides tools for generation of type-safe, spec version aware wrappers around events and calls.
The end result looks like this:
/**
* Normalized `balances.Transfer` event data
*/
interface TransferEvent {
from: Uint8Array
to: Uint8Array
amount: bigint
}
function getTransferEvent(ctx: EventHandlerContext): TransferEvent {
// instanciate type-safe facade around event data
let event = new BalancesTransferEvent(ctx)
if (event.isV1020) {
let [from, to, amount, fee] = event.asV1020
return {from, to, amount}
} else if (event.isV1050) {
let [from, to, amount] = event.asV1050
return {from, to, amount}
} else {
// This cast will assert,
// that the type of a given event matches
// the type of generated facade.
return event.asLatest
}
}
Generation of type-safe wrappers for events and calls is currently a two-step process.
First, you need to explore the chain to find blocks which introduce new spec version and fetch corresponding metadata.
npx squid-substrate-metadata-explorer \
--chain ws://127.0.0.1:9944 \
--archive http://localhost:4010/v1/graphql \
--out duniterVersions.json
In the above command --archive
parameter is optional, but it speeds up the process
significantly. From scratch exploration of duniter network without archive takes 20-30 minutes.
You can pass the result of previous exploration to --out
parameter. In that case exploration will
start from the last known block and thus will take much less time.
After chain exploration is complete you can use squid-substrate-typegen(1)
to generate
required wrappers.
npx squid-substrate-typegen typegen.json
Where typegen.json
config file has the following structure:
{
"outDir": "src/types",
"chainVersions": "duniterVersions.json", // the result of chain exploration
"events": [ // list of events to generate
"balances.Transfer"
],
"calls": [ // list of calls to generate
"timestamp.set"
]
}
Project conventions
Squid tools assume a certain project layout.
- All compiled js files must reside in
lib
and all TypeScript sources insrc
. The layout oflib
must reflectsrc
. - All TypeORM classes must be exported by
src/model/index.ts
(lib/model
module). - Database schema must be defined in
schema.graphql
. - Database migrations must reside in
db/migrations
and must be plain js files. -
sqd(1)
andsquid-*(1)
executables consult.env
file for a number of environment variables.
Differences from polkadot.js
Polkadot.js provides lots of specialized classes for various types of data.
Even primitives like u32
are exposed through special classes.
In contrast, squid framework works only with plain js primitives and objects.
This allows to decrease coupling and also simply dictated by the fact, that
there is not enough information in substrate metadata to distinguish between
interesting cases.
Account addresses is one example where such difference shows up. From substrate metadata (and squid framework) point of view account address is simply a fixed length sequence of bytes. On other hand, polkadot.js creates special wrapper for account addresses which aware not only of address value, but also of its ss58 formatting rules. Mapping developers should handle such cases themselves.
Graphql server extensions
It is possible to extend squid-graphql-server(1)
with custom
type-graphql resolvers and to add request validation.
More details will be added later.
Disclaimer
This is alpha-quality software. Expect some bugs and incompatible changes in coming weeks.