{"id":20324151,"url":"https://github.com/opentensor/squid","last_synced_at":"2026-06-22T19:31:40.982Z","repository":{"id":46298659,"uuid":"508370989","full_name":"opentensor/squid","owner":"opentensor","description":null,"archived":false,"fork":false,"pushed_at":"2022-10-31T20:25:52.000Z","size":369,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-19T10:08:44.606Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opentensor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-28T16:14:43.000Z","updated_at":"2022-11-24T14:55:52.000Z","dependencies_parsed_at":"2023-01-20T09:34:01.401Z","dependency_job_id":null,"html_url":"https://github.com/opentensor/squid","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":"subsquid-labs/squid-substrate-template","purl":"pkg:github/opentensor/squid","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentensor%2Fsquid","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentensor%2Fsquid/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentensor%2Fsquid/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentensor%2Fsquid/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opentensor","download_url":"https://codeload.github.com/opentensor/squid/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentensor%2Fsquid/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34663524,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-22T02:00:06.391Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T19:32:37.155Z","updated_at":"2026-06-22T19:31:40.953Z","avatar_url":"https://github.com/opentensor.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Squid template project\n\nA starter [Squid](https://subsquid.io) project to demonstrate its structure and conventions.\nIt accumulates [kusama](https://kusama.network) account balances and serves them via GraphQL API. For a full reference of Subsquid features consult [Docs](https://docs.subsquid.io) and [FAQ](./FAQ.md).\n\n## Summary\n\n- [Quickstart](#quickly-running-the-sample)\n- [Setup for Parachains](#setup-for-parachains)\n- [Setup for Localnets, Devnets and Testnets](#setup-for-devnets-and-testnets)\n- [Development flow](#dev-flow)\n  - [Database Schema](#1-define-database-schema)\n  - [Entity classes](#2-generate-typeorm-classes)\n  - [DB migrations](#3-generate-database-migration)\n  - [Typegen for Events, Extrinsics and Storage Calls](#4-generate-typescript-definitions-for-substrate-events-and-calls)\n- [Deploy the Squid](#deploy-the-squid)\n- [Conventions](#project-conventions)\n- [Type Bundles](#types-bundle)\n\n## Prerequisites\n\n* node 16.x\n* docker\n\n## Quickly running the sample\n\nThe scripts below use [`make`](https://www.gnu.org/software/make/). To get `make` work on windows one can use [WSL](https://docs.microsoft.com/en-us/windows/wsl/install), otherwise just have a look at one-liners in `Makefile`.\n\n```bash\n# 1. Install dependencies\nnpm ci\n\n# 2. Compile typescript files\nmake build\n\n# 3. Start target Postgres database and detach\nmake up\n\n# 4. Now start the processor (blocks the terminal)\nmake process\n\n# 5. The command above will block the terminal\n#    being busy with fetching the chain data, \n#    transforming and storing it in the target database.\n#\n#    To start the graphql server open the separate terminal\n#    and run\nmake serve\n```\n\n## Setup for parachains\n\nSubsquid provides Squid Archive data sources for most parachains. Use `lookupArchive(\u003cnetwork name\u003e)` to lookup the archive endpoint by the network name, e.g.\n\n```typescript\nprocessor.setDataSource({\n  archive: lookupArchive(\"kusama\", { release: \"FireSquid\" })\n  //...\n});\n```\n\nTo make sure you're indexing the right chain one can additionally filter by genesis hash and other options provided by `LookupOptions`:\n\n```typescript\nprocessor.setDataSource({\n  archive: lookupArchive(\"kusama\", { \n    release: \"FireSquid\", \n    genesis: \"0xb0a8d493285c2df73290dfb7e61f870f17b41801197a149ca93654499ea3dafe\" \n  }),\n  //...\n});\n```\n\nIf the chain is not yet supported, please fill out the [form](https://forms.gle/Vhr3exPs4HrF4Zt36) to submit a request.\n\n## Setup for devnets and testnets\n\nNon-production chains, e.g. Devnets and Testnets are not supported by `lookupArchive` and one has to provide a local Squid Archive as a data source.\n\nInspect `archive/docker-compose.yml` and provide the WebSocket endpoint for your node. \n\nThen run (in a separate terminal window)\n\n```bash\ndocker compose -f archive/docker-compose.yml up\n```\n\nThe `docker-compose` will start the archive gateway at port `8888` and it can immediately be used with the processor (even if it's not in sync it will eventually catch up):\n\n```typescript\nprocessor.setDataSource({\n  archive: `http://localhost:8888/graphql`,\n});\n```\n\nAdditionally, an explorer GraphQL API and a playground is started at `http://localhost:4350/graphql`. While it's optional to run it, it proved to be a very useful tool for debugging, developing and exploring on-chain data due to the rich filtering interfaces it provides.\n\n\nTo drop the archive, run\n```bash\ndocker compose -f archive/docker-compose.yml down -v\n```\n\n\n## Dev flow\n\n### 1. Define database schema\n\nStart development by defining the schema of the target database via `schema.graphql`.\nSchema definition consists of regular graphql type declarations annotated with custom directives.\nFull description of `schema.graphql` dialect is available [here](https://docs.subsquid.io/schema-spec).\n\n### 2. Generate TypeORM classes\n\nMapping developers use TypeORM [EntityManager](https://typeorm.io/#/working-with-entity-manager)\nto interact with the target database during data processing. All necessary entity classes are\ngenerated by the squid framework from `schema.graphql`. This is done by running `npx squid-typeorm-codegen`\ncommand.\n\n### 3. Generate database migration\n\nAll database changes are applied through migration files located at `db/migrations`.\n`sqd(1)` tool provides several commands to drive the process.\nIt is all [TypeORM](https://typeorm.io/#/migrations) under the hood.\n\n```bash\n# Connect to database, analyze its state and generate migration to match the target schema.\n# The target schema is derived from entity classes generated earlier.\nnpx sqd db create-migration\n\n# Create template file for custom database changes\nnpx sqd db new-migration\n\n# Apply database migrations from `db/migrations`\nnpx sqd db migrate\n\n# Revert the last performed migration\nnpx sqd db revert\n\n# DROP DATABASE\nnpx sqd db drop\n\n# CREATE DATABASE\nnpx sqd db create            \n```\n\n### 4. Generate TypeScript definitions for substrate events and calls\n\nThis is an optional part, but it is very advisable. \n\nEvent, call and runtime storage data come to mapping handlers as raw untyped json. \nWhile it is possible to work with raw untyped json data, it's extremely error-prone and the json structure may change over time due to runtime upgrades.\n\nSquid framework provides tools for the generation of type-safe, spec version aware wrappers around events, calls and runtime storage items. Typegen generates type-safe classes in `types/events.ts`, `types/calls.ts` and `types/storage.ts` respectively. All historical runtime upgrades are accounted out of the box. Typical usage is as follows (see `src/processor.ts`):\n\n```typescript\nfunction getTransferEvent(ctx: EventHandlerContext): TransferEvent {\n  // instantiate the autogenerated type-safe class for Balances.Transfer event\n  const event = new BalancesTransferEvent(ctx);\n  // for each runtime version, reduce the data to a common interface\n  if (event.isV1020) {\n    const [from, to, amount] = event.asV1020;\n    return { from, to, amount };\n  } else if (event.isV1050) {\n    const [from, to, amount] = event.asV1050;\n    return { from, to, amount };\n  } else {\n    const { from, to, amount } = event.asV9130;\n    return { from, to, amount };\n  }\n}\n``` \n\nGeneration of type-safe wrappers for events, calls and storage items can be now done with a single \ncommand as specVersions are provided by FireSquid Archives (see `typegen.json`)\n\n```bash\nnpx squid-substrate-typegen typegen.json\n```\n\nIf for some reason a FireSquid Archive is not available, it is still possible to \ngenerate the spec versions file using `squid-substrate-metadata-explorer` (it may take some time):\n\n```bash\nnpx squid-substrate-metadata-explorer \\ \n  --chain wss://kusama-rpc.polkadot.io \\\n  --out kusamaVersions.jsonl\n```\n\nand then source the generated file in `typegen.json`:\n\n```json5\n{\n  \"outDir\": \"src/types\",\n  \"specVersions\": \"kusamaVersions.jsonl\", // the result of chain exploration\n  \"events\": [ // list of events to generate\n    \"Balances.Transfer\"\n  ],\n  \"calls\": [ // list of calls to generate\n    \"Timestamp.set\"\n  ],\n  \"storage\": [\n    \"System.Account\" // list of storage items. To generate wrappers for all storage items, set \"storage\": true\n  ]\n}\n```\n\n## Deploy the Squid\n\nSubsquid offers a free hosted service for deploying your Squid. First, build and run the docker image locally and fix any errors or missing files in Dockerfile:\n\n```sh\nbash scripts/docker-run.sh # optionally specify DB port as an argument\n```\n\nAfter the local run, follow the [instructions](https://docs.subsquid.io/recipes/deploying-a-squid) for obtaining a deployment key and submitting the Squid to [Aquarium](https://app.subsquid.io).\n\n\n## Project conventions\n\nSquid tools assume a certain project layout.\n\n* All compiled js files must reside in `lib` and all TypeScript sources in `src`. \nThe layout of `lib` must reflect `src`.\n* All TypeORM classes must be exported by `src/model/index.ts` (`lib/model` module).\n* Database schema must be defined in `schema.graphql`.\n* Database migrations must reside in `db/migrations` and must be plain js files.\n* `sqd(1)` and `squid-*(1)` executables consult `.env` file for a number of environment variables.\n\n## Types bundle\n\nSubstrate chains that have blocks with metadata versions below 14 don't provide enough \ninformation to decode their data. For those chains, external [type](https://polkadot.js.org/docs/api/start/types.extend) definitions](https://polkadot.js.org/docs/api/start/types.extend) are required.\n\nType definitions (`typesBundle`) can be given to squid tools in two forms:\n\n1. as a name of a known chain (currently only `kusama`)\n2. as a json file of a structure described below.\n\n```json5\n{\n  \"types\": {\n    \"AccountId\": \"[u8; 32]\"\n  },\n  \"typesAlias\": {\n    \"assets\": {\n      \"Balance\": \"u64\"\n    }\n  },\n  \"versions\": [\n    {\n      \"minmax\": [0, 1000], // block range with inclusive boundaries\n      \"types\": {\n        \"AccountId\": \"[u8; 16]\"\n      },\n      \"typesAlias\": {\n        \"assets\": {\n          \"Balance\": \"u32\"\n        }\n      }\n    }\n  ]\n}\n```\n\n* `.types` - scale type definitions similar to [polkadot.js types](https://polkadot.js.org/docs/api/start/types.extend#extension)\n* `.typesAlias` - similar to [polkadot.js type aliases](https://polkadot.js.org/docs/api/start/types.extend#type-clashes)\n* `.versions` - per-block range overrides/patches for above fields.\n\nAll fields in the types bundle are optional and applied on top of a fixed set of well-known frame types.\n\n## Differences from polkadot.js\n\nPolkadot.js provides lots of [specialized classes](https://polkadot.js.org/docs/api/start/types.basics) for various types of data. \nEven primitives like `u32` are exposed through special classes.\nIn contrast, the squid framework works only with plain js primitives and objects.\nFor instance, account data is passed to the handler context as a plain byte array.  To convert it into a standard human-readable format one should explicitly use a utility lib `@subsquid/ss58`:\n\n```typescript \n    // ...\n    from: ss58.codec('kusama').encode(rec.from),\n    to: ss58.codec('kusama').encode(rec.to),\n```\n\n\n\n## Graphql server extensions\n\nIt is possible to extend `squid-graphql-server(1)` with custom\n[type-graphql](https://typegraphql.com) resolvers and to add request validation.\nMore details will be added later.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopentensor%2Fsquid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopentensor%2Fsquid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopentensor%2Fsquid/lists"}