{"id":18542759,"url":"https://github.com/coinbase/chainstorage","last_synced_at":"2025-10-26T02:50:22.838Z","repository":{"id":66163482,"uuid":"567071568","full_name":"coinbase/chainstorage","owner":"coinbase","description":"The File System For a Multi-Blockchain World","archived":false,"fork":false,"pushed_at":"2024-09-24T00:17:01.000Z","size":15861,"stargazers_count":86,"open_issues_count":13,"forks_count":30,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-03-22T14:11:16.853Z","etag":null,"topics":["bitcoin","blockchain","ethereum","golang"],"latest_commit_sha":null,"homepage":"https://time-wilderness-a70.notion.site/ChainStorage-5c173d1cafd842ceb9e38c26bfbd6265?pvs=4","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coinbase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-17T02:14:18.000Z","updated_at":"2025-03-17T03:04:24.000Z","dependencies_parsed_at":"2024-01-07T06:40:33.352Z","dependency_job_id":"7367f8d3-c812-4959-b628-4f506d2262be","html_url":"https://github.com/coinbase/chainstorage","commit_stats":{"total_commits":73,"total_committers":11,"mean_commits":6.636363636363637,"dds":0.5753424657534247,"last_synced_commit":"ae0dd7773f17e36175ed26719c57ac14bee677e4"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fchainstorage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fchainstorage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fchainstorage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fchainstorage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coinbase","download_url":"https://codeload.github.com/coinbase/chainstorage/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246200315,"owners_count":20739566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcoin","blockchain","ethereum","golang"],"created_at":"2024-11-06T20:10:25.378Z","updated_at":"2025-10-26T02:50:17.803Z","avatar_url":"https://github.com/coinbase.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*\n\n- [Overview](#overview)\n- [Quick Start](#quick-start)\n- [Command Line](#command-line)\n  - [Block Command](#block-command)\n  - [Backfill Command (development)](#backfill-command-development)\n  - [Stream Command](#stream-command)\n- [Testing](#testing)\n  - [Unit Test](#unit-test)\n  - [Integration Test](#integration-test)\n  - [Functional Test](#functional-test)\n- [Configuration](#configuration)\n  - [Dependency overview](#dependency-overview)\n  - [Template location and generated config](#template-location-and-generated-config)\n  - [Environment Variables](#environment-variables)\n  - [Creating New Configurations](#creating-new-configurations)\n  - [Template Format and Inheritance](#template-format-and-inheritance)\n  - [Endpoint Group](#endpoint-group)\n  - [Overriding the Configuration](#overriding-the-configuration)\n- [Development](#development)\n  - [Running Server](#running-server)\n  - [AWS localstack](#aws-localstack)\n  - [Temporal Workflow](#temporal-workflow)\n- [Failover](#failover)\n  - [Nodes Failover](#nodes-failover)\n  - [Failover in Workflows](#failover-in-workflows)\n  - [Automatic Failover in Poller Workflow](#automatic-failover-in-poller-workflow)\n  - [Checking Workflow Statuses](#checking-workflow-statuses)\n  - [APIs](#apis)\n  - [APIs List](#apis-list)\n- [SDK](#sdk)\n  - [Data Processing Pattern](#data-processing-pattern)\n- [Examples](#examples)\n  - [Batch](#batch)\n  - [Stream](#stream)\n  - [Unified](#unified)\n- [Contact Us](#contact-us)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n## Overview\n\nChainStorage is the foundational component of [ChainNode](https://github.com/coinbase/chainnode) and [Chainsformer](https://github.com/coinbase/chainsformer).\nThese projects represent the crypto data processing suites (code name ChainStack) widely adopted within Coinbase.\n\nChainStorage is inspired by the Change Data Capture paradigm, commonly used in the big data world. It continuously\nreplicates the changes (i.e. new blocks) on the blockchain, and acts like a distributed file system for the blockchain.\n\nIt aims to provide an efficient and flexible way to access the on-chain data:\n\n- Efficiency is optimized by storing data in horizontally-scalable storage with a key-value schema. At Coinbase's production environment, ChainStorage can serve up to 1,500 blocks per second, enabling teams to build various indexers cost-effectively.\n- Flexibility is improved by decoupling data interpretation from data ingestion. ChainStorage stores the raw data, and the parsing is deferred until the data is consumed. The parsers are shipped as part of the SDK and run on the consumer side. Thanks to the ELT (Extract, Load, Transform) architecture, we can quickly iterate on the parser without ingesting the data from the blockchain again.\n\n## Quick Start\n\nMake sure your local go version is 1.22 by running the following commands:\n```shell\nbrew install go@1.22\nbrew unlink go\nbrew link go@1.22\n\nbrew install protobuf@25.2\nbrew unlink protobuf\nbrew link protobuf\n\n```\n\nTo set up for the first time (only done once):\n```shell\nmake bootstrap\n```\n\nRebuild everything:\n```shell\nmake build\n```\n\nGenerate Protos:\n```shell\nmake proto\n```\n\n## Command Line\n\nthe `cmd/admin` tool consists of multiple sub command.\n\n```\nadmin is a utility for managing chainstorage\n\nUsage:\n  admin [command]\n\nAvailable Commands:\n  backfill    Backfill a block\n  block       Fetch a block\n  completion  Generate the autocompletion script for the specified shell\n  event       tool for managing events storage\n  help        Help about any command\n  sdk\n  validator\n  workflow    tool for managing chainstorage workflows\n\nFlags:\n      --blockchain string   blockchain full name (e.g. ethereum)\n      --env string          one of [local, development, production]\n  -h, --help                help for admin\n      --meta                output metadata only\n      --network string      network name (e.g. mainnet)\n      --out string          output filepath: default format is json; use a .pb extension for protobuf format\n      --parser string       parser type: one of native, mesh, or raw (default \"native\")\n\nUse \"admin [command] --help\" for more information about a command.\n```\n\nAll sub-commands require the `blockchain`, `env`, `network` flags.\n\n### Block Command\n\nFetch a block from ethereum mainnet:\n```shell\ngo run ./cmd/admin block --blockchain ethereum --network mainnet --env local --height 46147\n```\n\nFetch a block from ethereum goerli:\n```shell\ngo run ./cmd/admin block --blockchain ethereum --network goerli --env local --height 46147\n```\n\n### Backfill Command (development)\n\nBackfill a block from BSC mainnet:\n```\ngo run ./cmd/admin backfill --blockchain bsc --network mainnet --env development --start-height 10408613 --end-height 10408614\n```\n\n### Stream Command\n\nStream block events from a specific event sequence id:\n```shell\ngo run ./cmd/admin sdk stream --blockchain ethereum --network mainnet --env development --sequence 2228575 --event-tag 1\n```\n\n\n## Testing\n### Unit Test\n\n```shell\n# Run everything\nmake test\n\n# Run the blockchain package only\nmake test TARGET=internal/blockchain/...\n```\n\n### Integration Test\n```shell\n# Run everything\nmake integration\n\n# Run the storage package only\nmake integration TARGET=internal/storage/...\n\n# If test class implemented with test suite, add suite name before the test name\nmake integration TARGET=internal/blockchain/... TEST_FILTER=TestIntegrationPolygonTestSuite/TestPolygonGetBlock\n```\n\n### Functional Test\n\nBefore running the functional test, you need to provide the endpoint group config by creating `secrets.yml`.\nSee [here](#endpoint-group) for more details.\n\n```shell\n# Run everything\nmake functional\n\n# Run the workflow package only\nmake functional TARGET=internal/workflow/...\n\n# Run TestIntegrationEthereumGetBlock only\nmake functional TARGET=internal/blockchain/... TEST_FILTER='TestIntegrationEthereumGetBlock$$'\n\n# If test class implemented with test suite, add suite name before the test name\nmake functional TARGET=internal/blockchain/... TEST_FILTER=TestIntegrationPolygonTestSuite/TestPolygonGetBlock\n```\n\n## Configuration\n\n### Dependency overview\n\nTo understand the structure and elements of ChainStorage's config, it's important to comprehend its dependencies.\n\n- **Temporal**: Temporal is a workflow engine that orchestrates the data ingestion workflow. It calls ChainStorage service endpoint to complete various of tasks.\n- **Blob storage** - current implementation is on AWS S3, and the local service is provied by localstack\n- **Key value storage** - current implemnentation is based on dynamodb and the local service is provied by localstack\n- **Dead Letter queue** - current implementation is on SQS and the local service is provied by localstack\n\n### Template location and generated config\n\nThe config template directory is in `config_templates/config`, which `make config` reads this directory and generates the config into the `config/chainstorage` directory.\n\n### Environment Variables\n\nChainStorage depends on the following environment variables to resolve the path of the configuration. \nThe directory structure is as follows: `config/{namespace}/{blockchain}/{network}/{environment}.yml`.\n\n- `CHAINSTORAGE_NAMESPACE`:\n  A `{namespace}` is logical grouping of several services, each of which manages its own blockchain and network. The\n  default namespace is [chainstorage](/config/chainstorage).\n  To deploy a different namespace, set the env var to the name of a subdirectory of [./config](/config).\n- `CHAINSTORAGE_CONFIG`:\n  This env var, in the format of `{blockchain}-{network}`, determines the blockchain and network managed by the service.\n  The naming is defined\n  in [c3/common](/protos/coinbase/c3/common/common.proto).\n- `CHAINSTORAGE_ENVIRONMENT`:\n  This env var controls the `{environment}` in which the service is deployed. Possible values include `production`\n  , `development`, and `local` (which is also the default value).\n\n### Creating New Configurations\n\nEvery new asset in ChainStorage consists of ChainStorage configuration files.\nThese configuration files are generated from `.template.yml` template files using:\n\n```shell\nmake config\n```\n\nthese templates will be under a directory dedicated to storing the config templates\nin a structure that mirrors the final config structure of the `config`\ndirectories. All configurations from this directory will be generated within the final\nrespective config directories\n\n### Template Format and Inheritance\n\nConfiguration templates are composable and inherit configuration properties from\n\"parent templates\", which can be defined in `base.template.yml`, `local.template.yml`, `development.template.yml`,\nand `production.template.yml`.\nThese parent templates are merged into the final blockchain and network specific `base.template.yml`,\n`local.template.yml`, `development.template.yml`, `production.template.yml` configurations respectively.\n\nIn the following example, `config/chainstorage/ethereum/mainnet/base.yml`\ninherits from `config_templates/base.template.yml` and `config_templates/chainstorage/ethereum/mainnet/base.template.yml`,\nwith the latter taking precedence over the former.\n\n```\nconfig\n  chainstorage\n    ethereum\n      mainnet\n        base.yml\n        development.yml\n        local.yml\n        production.yml\nconfig_templates\n  chainstorage\n    ethereum\n      mainnet\n        base.template.yml\n        development.template.yml\n        local.template.yml\n        production.template.yml\n    base.template.yml\n    development.template.yml\n    local.template.yml\n    production.template.yml\n\n```\n\nThe template language supports string substitution for the Config-Name and Environment using the `{{`, `}}` tags.\n\nExample:\n\n```yaml\nfoo: {{blockchain}}-{{network}}-{{environment}}\n```\n\nThe blockchain, `{{blockchain}}`, network, `{{network}}`, and environment, `{{environment}}` template variables\nare derived from the directory and file naming schemes associated with cloud and ChainStorage configurations.\n\n### Endpoint Group\n\nEndpoint group is an abstraction for one or more JSON-RPC endpoints.\n[EndpointProvider](./internal/blockchain/endpoints/endpoint_provider.go) uses the `endpoint_group` config to implement\nclient-side routing to the node provider.\n\nChainStorage utilizes two endpoint groups to speed up data ingestion:\n* master: This endpoint group is used to resolve the canonical chain and determine what blocks to ingest next.\n  Typically, sticky session is turned on for this group to ensure stronger data consistency between the requests.\n* slave: This endpoint group is used to ingest the data from the blockchain. During data ingestion, the new blocks are\n  ingested in parallel and out of order. Typically, the endpoints are selected in a round-robin fashion, but you may\n  increase the weights to send more traffic to certain endpoints.\n\nIf your node provider, e.g. QuickNode, already has built-in load balancing, your endpoint group may contain only one \nendpoint, as illustrated by the following configuration:\n```yaml\nchain:\n  client:\n    master:\n      endpoint_group: |\n        {\n          \"endpoints\": [\n            {\n              \"name\": \"quicknode-foo-bar-sticky\",\n              \"url\": \"https://foo-bar.matic.quiknode.pro/****\",\n              \"weight\": 1\n            }\n          ],\n          \"sticky_session\": {\n            \"header_hash\": \"x-session-hash\"\n          }\n        }\n    slave:\n      endpoint_group: |\n        {\n          \"endpoints\": [\n            {\n              \"name\": \"quicknode-foo-bar-round-robin\",\n              \"url\": \"https://foo-bar.matic.quiknode.pro/****\",\n              \"weight\": 1\n            }\n          ]\n        }\n```\n\n### Overriding the Configuration\n\nYou may override any configuration using an environment variable. The environment variable should be prefixed with\n\"CHAINSTORAGE_\". For nested dictionary, use underscore to separate the keys.\n\nFor example, you may override the endpoint group config at runtime by injecting the following environment variables:\n* master: CHAINSTORAGE_CHAIN_CLIENT_MASTER_ENDPOINT_GROUP\n* slave: CHAINSTORAGE_CHAIN_CLIENT_SLAVE_ENDPOINT_GROUP\n\nAlternatively, you may override the configuration by creating `secrets.yml` within the same directory. Its attributes\nwill be merged into the runtime configuration and take the highest precedence. Note that this file may contain\ncredentials and is excluded from check-in by `.gitignore`.\n\n## Development\n\n### Running Server\nStart the dockers by the docker-compose file from project root folder:\n```shell\nmake localstack\n```\n\u003e If you have developed ChainStorage before locally with previous docker compose file and got below error message\n\u003e ```shell\n\u003e nc: bad address 'postgresql'\n\u003e ```\n\u003e Please remove existing ChainStorage container and reran `make localstack`\n\nThe next step is to start the server locally:\n\n```shell\n# Ethereum Mainnet\n# Use aws local stack\nmake server\n\n# If want to start testnet (goerli) server\n# Use aws local stack\nmake server CHAINSTORAGE_CONFIG=ethereum_goerli\n```\n\n### AWS localstack\n\nCheck S3 files:\nYou can checkout the config from `config/chainstorage/{{Blockchain}}/{{network}}/{{evironment}}`  for the value of S3 bucket name, dynamoDB tables, and SQS name.\n```shell\naws s3 --no-sign-request --region local --endpoint-url http://localhost:4566 ls --recursive example-chainstorage-ethereum-mainnet-dev/\n```\n\nCheck DynamoDB rows:\n```shell\naws dynamodb --no-sign-request --region local --endpoint-url http://localhost:4566 scan --table-name example_chainstorage_blocks_ethereum_mainnet\n```\n\nCheck DLQ:\n```shell\naws sqs --no-sign-request --region local --endpoint-url http://localhost:4566/000000000000/example_chainstorage_blocks_ethereum_mainnet_dlq receive-message --queue-url \"http://localhost:4566/000000000000/example_chainstorage_blocks_ethereum_mainnet_dlq\" --max-number-of-messages 10 --visibility-timeout 2\n```\n\n### Temporal Workflow\n\nOpen Temporal UI in a browser by entering the\nURL: http://localhost:8080/namespaces/chainstorage-ethereum-mainnet/workflows\n\nStart the backfill workflow:\n```shell\ngo run ./cmd/admin workflow start --workflow backfiller --input '{\"StartHeight\": 11000000, \"EndHeight\": 11000100, \"NumConcurrentExtractors\": 24}' --blockchain ethereum --network mainnet --env local\n```\n\nStart the benchmarker workflow:\n```shell\ngo run ./cmd/admin workflow start --workflow benchmarker --input '{\"StartHeight\": 1, \"EndHeight\": 12000000, \"NumConcurrentExtractors\": 24, \"StepSize\":1000000, \"SamplesToTest\":500}' --blockchain ethereum --network mainnet --env local\n```\n\nStart the monitor workflow:\n```shell\ngo run ./cmd/admin workflow start --workflow monitor --blockchain ethereum --network mainnet --env local --input '{\"StartHeight\": 11000000}'\n```\n\nStart the poller workflow:\n```shell\ngo run ./cmd/admin workflow start --workflow poller --input '{\"Tag\": 0, \"MaxBlocksToSync\": 100, \"Parallelism\":4}' --blockchain ethereum --network mainnet --env local\n```\nNOTE: the recommended value for \"parallelism\" depend on the capacity of your node provider. If you are not sure what\nvalue should be used, just drop it from the command.\n\nStart the streamer workflow:\n```shell\ngo run ./cmd/admin workflow start --workflow streamer --input '{}' --blockchain ethereum --network goerli --env local\n```\n\nStop the monitor workflow:\n```shell\ngo run ./cmd/admin workflow stop --workflow monitor --blockchain ethereum --network mainnet --env local\n```\n\nStop a versioned streamer workflow:\n```shell\ngo run ./cmd/admin workflow stop --workflow streamer --blockchain ethereum --network mainnet --env local --workflowID {workflowID}\n```\n\nUsing Temporal CLI to check the status of the workflow: \n```shell\nbrew install tctl\n\ntctl --address localhost:7233 --namespace chainstorage-ethereum-mainnet workflow show --workflow_id workflow.backfiller\n````\n\n## Failover\n\n### Nodes Failover\nChainStorage supports nodes failover feature to mitigate the issue from nodes which may impact our SLA.\nWhen primary clusters are down, we can choose to switch over to failover clusters to mitigate incidents, instead of waiting for nodes get fully recovered.\n\nCheck [this comment](https://github.com/coinbase/chainstorage/blob/master/internal/blockchain/endpoints/endpoint_provider.go#L21-26) to get more details of the primary/secondary cluster definition.\n\nTo check if failover clusters are provided, please go to config service and check the endpoints configurations.\n\n\n### Failover in Workflows\nBoth `Backfiller` and `Poller` workflows provide failover feature without updating configs in the config service which requires approvals as well as redeployment.\n\nTo use failover clusters for those two workflows, you simply need to set the `Failover` workflow param as `true` when you trigger it.\n\nNote: By default, the `Failover` workflow param should be set as `false`, which means the primary clusters should always be first choice.\nIf you intend to use failover clusters, please update endpoints configs in config service instead.\n\n\n### Automatic Failover in Poller Workflow\n`Poller` workflow provides an automatic failover mechanism so that we don't need to manually restart workflows with `Failover` param.\n\nThis feature is guarded by the `failover_enabled` configuration.\nOnce this feature is enabled, when the workflow execution using primary clusters fails, it will automatically trigger a new workflow run with `Failover=true`.\n[Implementation Code](https://github.com/coinbase/chainstorage/blob/master/internal/workflow/poller.go#L186).\n\nRef: [Temporal Continue-As-New](https://docs.temporal.io/concepts/what-is-continue-as-new/#:~:text=Continue%2DAs%2DNew%20is%20a,warn%20you%20every%2010%2C000%20Events.)\n\n\n### Checking Workflow Statuses\n\nInstall tctl, it is a command-line tool that you can use to interact with a Temporal cluster. More info can be found\nhere: https://docs.temporal.io/tctl/\n\n```shell\nbrew install tctl\n```\n\n### APIs\n\n### APIs List\n[Supported APIs List](https://github.com/coinbase/chainstorage/blob/master/protos/coinbase/chainstorage/api.proto)\n\n```shell\n# local\ngrpcurl --plaintext localhost:9090 coinbase.chainstorage.ChainStorage/GetLatestBlock\ngrpcurl --plaintext -d '{\"start_height\": 0, \"end_height\": 10}' localhost:9090 coinbase.chainstorage.ChainStorage/GetBlockFilesByRange\ngrpcurl --plaintext -d '{\"sequence_num\": 2223387}' localhost:9090 coinbase.chainstorage.ChainStorage/StreamChainEvents\ngrpcurl --plaintext -d '{\"initial_position_in_stream\": \"EARLIEST\"}' localhost:9090 coinbase.chainstorage.ChainStorage/StreamChainEvents\ngrpcurl --plaintext -d '{\"initial_position_in_stream\": \"LATEST\"}' localhost:9090 coinbase.chainstorage.ChainStorage/StreamChainEvents\ngrpcurl --plaintext -d '{\"initial_position_in_stream\": \"13222054\"}' localhost:9090 coinbase.chainstorage.ChainStorage/StreamChainEvents\n```\n\n## SDK\nChainstorage also provides SDK, and you can find supported\nmethods [here](https://github.com/coinbase/chainstorage/blob/master/sdk/client.go)\n\nNote:\n- `GetBlocksByRangeWithTag` is not equivalent to the batch version of `GetBlockWithTag` since you don't have a way to\n  specify the block hash.\n  So when you use `GetBlocksByRangeWithTag` and if it goes beyond the current tip of chain due to reorg, you'll get back\n  the `FailedPrecondition` error because it exceeds the latest watermark.\n\n  In conclusion, it's safe to use `GetBlocksByRangeWithTag` for backfilling since the reorg will not happen for past\n  blocks, however, you'd be suggested to use `GetBlockWithTag` for recent blocks (e.g. streaming case).\n\n### Data Processing Pattern\nBelow are several patterns you can choose for data processing.\n\n1. If you want the most up-to-date blocks, you need to use the streaming APIs to handle the chain reorg events:\n  1. Unified batch and streaming:\n    - Download, let's say 10k events, using `GetChainEvents`.\n    - Break down 10k events into small batches, e.g. 20 events/batch.\n    - Process those batches in parallel.\n    - For events in each batch, it can be processed either sequentially or in parallel using `GetBlockWithTag`.\n    - Update watermark once all small batches have been processed.\n    - Repeat above steps.\n\n     With the above pattern, you can unify batch and streaming use cases. When your data pipeline is close to the tip,\n     `GetChainEvents` will simply return all available blocks.\n  2. Separate workflows for backfilling and live streaming:\n     Use `GetBlocksByRangeWithTag` for backfilling and then switch over to `StreamChainEvents` for live streaming.\n2. If you don't want to deal with chain reorg, you may use the batch APIs as follows:\n  - Maintain a distance (`irreversibleDistance`) to the tip, the irreversible distance can be queried using `GetChainMetadata`.\n  - Get the latest block height (`latest`) using `GetLatestBlock`.\n  - Poll for new data from current watermark block to the block (`latest - irreversibleDistance`) using `GetBlocksByRangeWithTag`.\n  - Repeat above steps periodically.\n\n## Examples\n\nSee below for a few examples for implementing a simple indexer using the SDK.\nNote that the examples are provided in increasing complexity.\n\n### Batch\n\nIn [this example](/examples/batch/main.go), we use the blocks API to fetch the confirmed blocks as follows:\n1. Fetch the maximum reorg distance (`irreversibleDistance`).\n2. Fetch the latest block height (`latest`).\n3. Poll for new blocks from the checkpoint up to the latest confirmed block  (`latest - irreversibleDistance`).\nusing `GetBlocksByRange`.\n4. Update the checkpoint.\n5. Repeat above steps periodically.\n\n### Stream\n\n[This example](/examples/stream/main.go) demonstrates how to stream the latest blocks and handle chain reorgs.\nThe worker processes the events sequentially and relies on [BlockchainEvent_Type](/protos/coinbase/chainstorage/api.proto)\nto construct the canonical chain.\nFor example, given `+1, +2, +3, -3, -2, +2', +3'` as the events, the canonical chain would be `+1, +2', +3'`.\n\n### Unified\n\nThe [last example](/examples/unified/main.go) showcases how to turn the data processing into an embarrassingly parallel\nproblem by leveraging the mono-increasing sequence number. In this example, though the events are processed in parallel\nand out of order, the logical ordering guarantee is preserved.\n1. Download, say 10k events, using `GetChainEvents`. Note that this API is non-blocking, and it returns all the\n   available events if the requested amount is not available. This enables us to unify batch and stream processing.\n2. Break down 10k events into small batches, e.g. 20 events/batch.\n3. Distribute those batches to a number of workers for parallel processing.\n   Note that this step is not part of the example.\n4. For events in each batch, it can be processed either sequentially or in parallel using `GetBlockWithTag`.\n5. Implement versioning using the mono-increasing sequence numbers provided by the events.\n   See [here](https://aws.amazon.com/blogs/database/implementing-version-control-using-amazon-dynamodb/) for more details.\n6. Update watermark once all the batches have been processed.\n7. Repeat above steps.\n\n## Contact Us\n\nWe have set up a Discord server soon. Here is the link to join (limited 10) https://discord.com/channels/1079683467018764328/1079683467786334220.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoinbase%2Fchainstorage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoinbase%2Fchainstorage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoinbase%2Fchainstorage/lists"}