{"id":51630173,"url":"https://github.com/subsquid/pipes-sdk","last_synced_at":"2026-07-13T06:32:51.680Z","repository":{"id":317351714,"uuid":"1063847893","full_name":"subsquid/pipes-sdk","owner":"subsquid","description":"TypeScript SDK for streaming blockchain data: tap SQD Portal sources, decode EVM/Solana/Bitcoin/Tron/Hyperliquid onchain data, handle reorgs, write to Postgres, ClickHouse, BigQuery, or Parquet.","archived":false,"fork":false,"pushed_at":"2026-07-11T07:22:23.000Z","size":2499,"stargazers_count":19,"open_issues_count":11,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-07-11T08:12:13.760Z","etag":null,"topics":["bigquery","bitcoin","blockchain","blockchain-indexer","clickhouse","data-pipeline","ethereum","etl","evm","hyperliquid","indexer","parquet","postgresql","real-time","sdk","solana","streaming","tron","typescript","web3"],"latest_commit_sha":null,"homepage":"https://docs.sqd.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/subsquid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-25T07:43:30.000Z","updated_at":"2026-07-11T06:31:08.000Z","dependencies_parsed_at":"2026-01-29T01:05:38.316Z","dependency_job_id":null,"html_url":"https://github.com/subsquid/pipes-sdk","commit_stats":null,"previous_names":["subsquid-labs/pipes-sdk","subsquid/pipes-sdk"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/subsquid/pipes-sdk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subsquid%2Fpipes-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subsquid%2Fpipes-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subsquid%2Fpipes-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subsquid%2Fpipes-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/subsquid","download_url":"https://codeload.github.com/subsquid/pipes-sdk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subsquid%2Fpipes-sdk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35413538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-13T02:00:06.543Z","response_time":119,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","bitcoin","blockchain","blockchain-indexer","clickhouse","data-pipeline","ethereum","etl","evm","hyperliquid","indexer","parquet","postgresql","real-time","sdk","solana","streaming","tron","typescript","web3"],"created_at":"2026-07-13T06:32:50.824Z","updated_at":"2026-07-13T06:32:51.676Z","avatar_url":"https://github.com/subsquid.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SQD Pipes SDK\n\n[![npm](https://img.shields.io/npm/v/@subsquid/pipes.svg)](https://www.npmjs.com/package/@subsquid/pipes)\n[![license](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](./LICENSE)\n[![docs](https://img.shields.io/badge/docs-docs.sqd.dev-3b82f6)](https://docs.sqd.dev)\n\nDocumentation: https://docs.sqd.dev  ·  Website: https://sqd.dev\n\nSQD Pipes is a TypeScript-first toolkit for streaming blockchain data, transforming it in-flight, and delivering the results to your own systems. It glues together:\n\n- **Sources** that tap into managed SQD Portal datasets for chains like Ethereum and Solana.\n- **Transforms/decoders** that turn raw blocks and logs into strongly-typed objects.\n- **Targets** that persist or forward processed data (ClickHouse today, with community hooks for more sinks).\n- **Observability** utilities such as profiling, structured logging, and Prometheus metrics.\n\nEvery pipeline is described as a composition of these pieces via the `pipe()` helper.\nYou can run the same code in CLIs, backend services, or long-running workers.\n\n\n---\n\n## 1. Install the SDK\n\nAdd the Pipes package to any TypeScript/Node project.\n\n```bash\npnpm add @subsquid/pipes\n# or\nnpm install @subsquid/pipes\n```\n\n---\n\n## 2. Create your first pipeline\n\nThe snippet below streams ERC-20 transfers from Ethereum Mainnet via the SQD Portal and prints them to the console.\n\nCreate `src/erc20-transfers.ts`:\n\n```ts\nimport { commonAbis, evmDecoder, evmPortalStream } from '@subsquid/pipes/evm'\n\nasync function main() {\n  const stream = evmPortalStream({\n    portal: 'https://portal.sqd.dev/datasets/ethereum-mainnet',\n  }).pipe(\n    evmDecoder({\n      profiler: { name: 'erc20-transfers' },\n      range: { from: '12,000,000' },\n      events: {\n        transfers: commonAbis.erc20.events.Transfer,\n      },\n    }),\n  )\n\n  for await (const { data } of stream) {\n    console.log(`parsed ${data.transfers.length} transfers`)\n  }\n}\n\nvoid main()\n```\n\nRun it with [`tsx`](https://github.com/privatenumber/tsx) (fast TypeScript executor):\n\n```bash\npnpm dlx tsx src/erc20-transfers.ts\n```\n\nYou should see logs as transfers are decoded.\n\n---\n\n## 3. Persist data (optional)\n\n### ClickHouse target\n\nIf you have ClickHouse and want automatic offset management, read the [ClickHouse example](https://github.com/subsquid-labs/pipes-sdk/blob/main/docs/examples/evm/04.clickhouse.example.ts).\nIt uses the `createClickhouseTarget` from the core package to batch writes and handle forks gracefully.\n\n#### Rollbacks and materialized views\n\nOn a blockchain fork the target removes rolled-back rows by inserting CollapsingMergeTree\ncancel rows (`sign = -1`) — the only delete mechanism in ClickHouse that propagates through\nmaterialized views. To get this behavior, a table must use a `CollapsingMergeTree`\n(or `VersionedCollapsingMergeTree`) engine with a `sign` column, and materialized views\nbuilt on top of it should be written rollback-aware: aggregate with the sign, e.g.\n`sum(value * sign)` for sums and `sum(sign)` for counts, so cancel rows revert the\naggregate automatically.\n\nTables on any other engine still roll back — the target falls back to a lightweight\n`DELETE` and logs a warning — but materialized views built on such tables keep the\nrolled-back data, because ClickHouse fires MVs on `INSERT` only. Picking the mechanism\nrequires read access to `system.tables` / `system.columns`; without it the target logs a\nwarning and uses the legacy `FINAL`-based cancel-row rollback, which assumes a\n`CollapsingMergeTree` table with a `sign` column.\n\nIrreversible aggregates — `min`, `max`, `uniq`, `argMax` and similar — cannot \"subtract\" a\nvalue, so no write mechanism can roll them back. If you need such an MV, the only correct\nrecovery after a fork is recomputing its affected tail (drop the tail partition and\n`INSERT ... SELECT` the range from the base table).\n\nRollback reads are pruned by a small `minmax` skip index on `block_number` that\n`store.removeAllRows` creates automatically on first use; call\n`store.ensureRollbackIndex({ table })` in `onStart` to set it up eagerly.\n\n### PostgreSQL with Drizzle\n\nIf you prefer PostgreSQL, check out the [Drizzle example](https://github.com/subsquid-labs/pipes-sdk/blob/main/docs/examples/evm/08.drizzle.example.ts),\nwhich demonstrates how to use Drizzle ORM to define your schema and persist decoded data.\n\n---\n\n## 4. Explore more examples\n\n- [`docs/examples/evm`](https://github.com/subsquid-labs/pipes-sdk/tree/main/docs/examples/evm): combining sources, decoders, and targets for EVM chains.\n- [`docs/examples/solana`](https://github.com/subsquid-labs/pipes-sdk/tree/main/docs/examples/solana): Solana Portal pipelines, including token balance over-fetch and parallel processing demos.\n\nFrom the repository root you can run any example with `pnpm tsx \u003cpath/to/example.ts\u003e`.\n\n---\n\n## 5. Next steps\n\n1. Wire your own sinks by implementing `createTarget` (see `packages/subsquid-pipes/src/targets` for references).\n2. Add instrumentation with the built-in profiler and Prometheus metrics (`packages/subsquid-pipes/src/core`).\n3. Try the UI tooling in `@sqd-pipes/pipe-ui`.\n\nNeed help or found a bug? Open an issue or discussion on the repository. Happy hacking!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubsquid%2Fpipes-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsubsquid%2Fpipes-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubsquid%2Fpipes-sdk/lists"}