{"id":18542778,"url":"https://github.com/coinbase/chainsformer","last_synced_at":"2025-04-09T18:32:16.414Z","repository":{"id":167029395,"uuid":"632106262","full_name":"coinbase/chainsformer","owner":"coinbase","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-08T01:02:41.000Z","size":357,"stargazers_count":14,"open_issues_count":4,"forks_count":9,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-24T10:38:48.023Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coinbase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-24T18:12:08.000Z","updated_at":"2024-11-10T18:34:27.000Z","dependencies_parsed_at":"2024-11-06T20:10:42.612Z","dependency_job_id":"3733c888-f67b-4dd4-a89d-480927acc704","html_url":"https://github.com/coinbase/chainsformer","commit_stats":null,"previous_names":["coinbase/chainsformer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fchainsformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fchainsformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fchainsformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fchainsformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coinbase","download_url":"https://codeload.github.com/coinbase/chainsformer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248087871,"owners_count":21045604,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T20:10:38.609Z","updated_at":"2025-04-09T18:32:11.405Z","avatar_url":"https://github.com/coinbase.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*\n\n- [Overview](#overview)\n- [Quick Start](#quick-start)\n- [Configuration](#configuration)\n  - [Environment Variables](#environment-variables)\n  - [Service Configurations](#service-configurations)\n  - [New Blockchain Configurations](#new-blockchain-configurations)\n- [Development](#development)\n  - [Running Chainsformer Server](#running-chainsformer-server)\n  - [Run test client](#run-test-client)\n  - [Use grpcurl](#use-grpcurl)\n    - [Query Chainsformer for a range of blocks](#query-chainsformer-for-a-range-of-blocks)\n    - [Query Chainsformer for a range of blocks events](#query-chainsformer-for-a-range-of-blocks-events)\n- [Testing](#testing)\n  - [Unit Test](#unit-test)\n  - [Integration Test](#integration-test)\n  - [Functional Test](#functional-test)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n## Overview\n\nChainsformer is an [Apache Arrow Flight](https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/) service built on top of [ChainStorage](https://github.com/coinbase/chainstorage) as a stateless adaptor service. It currently supports batch data processing and micro batch data streaming from ChainStorage service to the Spark data processing platform.\n\nIt aims to provide a set of easy to use interfaces to support spark consumers to read and process ChainStorage Data on the Spark platform:\n* It defines a set of standardized block and transaction data schema for each asset class (i.e EVM assets or bitcoin).\n* It provides data transformation capability from protobuf to Arrow format.\n* It can be easily scaled up to support higher data throughput.\n* It can be easily integrated via the Chainsformer Spark Connector (https://github.com/coinbase/chainsformer-spark-source) for structured data streaming.\n\n## Quick Start\n\nMake sure your local go version is 1.18 by running the following commands:\n\n```shell\nbrew install go@1.18\nbrew unlink go\nbrew link go@1.18\n\nbrew install protobuf@3.21.12\nbrew unlink protobuf\nbrew link protobuf\n```\n\nTo set up for the first time (only done once):\n\n```shell\nmake bootstrap\n```\n\nRebuild everything:\n\n```shell\nmake build\n```\n\n## Configuration\n\n### Environment Variables\n\nChainsformer depends on the following environment variables to resolve the path of the configuration.\nThe directory structure is as follows: `config/chainsformer/{blockchain}/{network}/{environment}.yml`.\n\n- `CHAINSFORMER_CONFIG`:\n  This env var, in the format of `{blockchain}-{network}`, determines the blockchain and network managed by the service.\n  The naming is defined in [chainstorage/protos/coinbase/c3/common/common.protp](https://github.com/coinbase/chainstorage/blob/master/protos/coinbase/c3/common/common.proto)\n- `CHAINSFORMER_ENVIRONMENT`:\n  This env var controls the `{environment}` in which the service is deployed. Possible values include `production`\n  , `development`, and `local` (which is also the default value).\n\n### Service Configurations\n\nAsset specific configurations are stored in the `config` directory under the Chainsformer service repo. The config folder structure follows the following form `./config/chainsformer/{blockchain}/{network}/base.yml`\n\n### New Blockchain Configurations\n* Simply follow the config folder structure to add new configurations for any new blockchains or new networks of existing blockchains.\n* Add new tests in the [config_test.go](/internal/config/config_test.go)\n* Add new test configs in teh [testapp.go](/internal/utils/testapp/testapp.go)\n\n## Development\n  \n### Running Chainsformer Server\n\nClone the Chainsformer service repo:\n```shell\ngit clone https://github.com/coinbase/chainsformer.git\n```\n\nChange directory to the Chainsformer service repo:\n```shell\ncd chainsformer\n```\n\nSetup Chainstorage SDK credentials\n```shell\nexport CHAINSTORAGE_SDK_AUTH_HEADER=cb-nft-api-token\nexport CHAINSTORAGE_SDK_AUTH_TOKEN=****\n```\n\nTo set up Chainsformer for the first time (only done once):\n```shell\nmake bootstrap\n```\n\nRebuild Chainsformer:\n\n```shell\nmake build\n```\n\nStart the Chainsformer service with default `CHAINSFORMER_CONFIG=ethereum-mainnet`:\n```shell\nmake server\n```\n\n### Run test client\n\nQuery Chainsformer for a range of blocks\n```shell\ngo run ./cmd/client --env local --blockchain ethereum --network mainnet --start 0 --end 10 --table blocks\n```\n\nQuery Chainsformer for a range of block events\n```shell\ngo run ./cmd/client --env local --blockchain ethereum --network mainnet --start 0 --end 10 --table streamed_blocks\n```\n\n### Use grpcurl\n\n#### Query Chainsformer for a range of blocks\nCalling the `GetSchema` API\n```shell\ncmd=$(echo -n '{\"table\": \"blocks\"}' | base64)\ngrpcurl --plaintext -d '{\"cmd\":'\"\\\"$cmd\\\"\"',\"type\":2}' localhost:9090 arrow.flight.protocol.FlightService.GetSchema\n```\n\nCalling the `GetFlightInfo` API to partition the data\n```shell\ncmd=$(echo -n '{\"batch_query\": {\"start_height\": 0, \"end_height\": 10, \"table\": \"blocks\"}}' | base64)\ngrpcurl --plaintext -d '{\"cmd\":'\"\\\"$cmd\\\"\"',\"type\":2}' localhost:9090 arrow.flight.protocol.FlightService.GetFlightInfo\n```\n\nTake one of the `ticket` returned by the above command\n```\n...\n\"endpoint\": [\n    {\n      \"ticket\": {\n        \"ticket\": \"eyJiYXRjaF9xdWVyeSI6eyJlbmRfaGVpZ2h0IjoiMTAiLCJ0YWJsZSI6ImJsb2NrcyJ9fQ==\"\n      }\n    }\n  ]\n...\n```\n\nCalling the `DoGet` API to get data for one of the partition\n```shell\ngrpcurl --plaintext -d '{\"ticket\": \"eyJiYXRjaF9xdWVyeSI6eyJlbmRfaGVpZ2h0IjoiMTAiLCJ0YWJsZSI6ImJsb2NrcyJ9fQ==\"}' localhost:9090 arrow.flight.protocol.FlightService.DoGet\n```\n\nCalling the `DoGet` API to get data of a specific partition\n```shell\ncmd=$(echo -n '{\"batch_query\":{\"start_height\":\"1\", \"end_height\":\"2\", \"table\":\"blocks\"}}' | base64)\ngrpcurl --plaintext -d '{\"ticket\": '\"\\\"$cmd\\\"\"'}' localhost:9090 arrow.flight.protocol.FlightService.DoGet\n```\n\nCalling the `DoAction` API to get the tip in ChainStorage via Chainsformer\n```shell\ngrpcurl --plaintext -d '{\"type\": \"TIP\"}' localhost:9090 arrow.flight.protocol.FlightService.DoAction | jq '.body | @base64d'\n```\n\n#### Query Chainsformer for a range of blocks events\nCalling the `GetSchema` API\n```shell\ncmd=$(echo -n '{\"table\": \"streamed_blocks\"}' | base64)\ngrpcurl --plaintext -d '{\"cmd\":'\"\\\"$cmd\\\"\"',\"type\":2}' localhost:9090 arrow.flight.protocol.FlightService.GetSchema\n```\n\nCalling the `GetFlightInfo` API to partition the data\n```shell\ncmd=$(echo -n '{\"stream_query\": {\"start_sequence\": 0, \"end_sequence\": 10, \"table\": \"streamed_blocks\"}}' | base64)\ngrpcurl --plaintext -d '{\"cmd\":'\"\\\"$cmd\\\"\"',\"type\":2}' localhost:9090 arrow.flight.protocol.FlightService.GetFlightInfo\n```\n\nTake one of the `ticket` returned by the above command\n```\n...\n\"endpoint\": [\n    {\n      \"ticket\": {\n        \"ticket\": \"eyJzdHJlYW1fcXVlcnkiOnsic3RhcnRfc2VxdWVuY2UiOiIxIiwiZW5kX3NlcXVlbmNlIjoiMTAiLCJ0YWJsZSI6InN0cmVhbWVkX2Jsb2NrcyJ9fQ==\"\n      }\n    }\n  ]\n...\n```\n\nCalling the `DoGet` API to get data for one of the partition\n```shell\ngrpcurl --plaintext -d '{\"ticket\": \"eyJzdHJlYW1fcXVlcnkiOnsic3RhcnRfc2VxdWVuY2UiOiIxIiwiZW5kX3NlcXVlbmNlIjoiMTAiLCJ0YWJsZSI6InN0cmVhbWVkX2Jsb2NrcyJ9fQ==\"}' localhost:9090 arrow.flight.protocol.FlightService.DoGet\n```\n\nCalling the `DoGet` API to get data of a specific partition\n```shell\ncmd=$(echo -n '{\"stream_query\":{\"start_sequence\":\"1\", \"end_sequence\":\"2\", \"table\":\"streamed_blocks\"}}' | base64)\ngrpcurl --plaintext -d '{\"ticket\": '\"\\\"$cmd\\\"\"'}' localhost:9090 arrow.flight.protocol.FlightService.DoGet\n```\n\nCalling the `DoAction` API to get the tip in ChainStorage via Chainsformer\n```shell\ngrpcurl --plaintext -d '{\"type\": \"STREAM_TIP\"}' localhost:9090 arrow.flight.protocol.FlightService.DoAction | jq '.body | @base64d'\n```\n\n## Testing\n### Unit Test\n\n```shell\n# Run everything\nmake test\n```\n\n### Integration Test\nUnder development\n\n### Functional Test\nUnder development","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoinbase%2Fchainsformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoinbase%2Fchainsformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoinbase%2Fchainsformer/lists"}