{"id":28414371,"url":"https://github.com/blockchain-etl/blockchain-etl-streaming","last_synced_at":"2025-06-25T06:32:07.169Z","repository":{"id":43959462,"uuid":"149081094","full_name":"blockchain-etl/blockchain-etl-streaming","owner":"blockchain-etl","description":"Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes ","archived":false,"fork":false,"pushed_at":"2021-12-09T12:37:10.000Z","size":66,"stargazers_count":80,"open_issues_count":4,"forks_count":22,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-06-24T05:42:40.372Z","etag":null,"topics":["apache-beam","bitcoin","blockchain","blockchain-analytics","crypto","cryptocurrency","data-analytics","data-engineering","ethereum","etl","gcp","google-bigquery","google-cloud-platform","google-dataflow","google-pubsub","on-chain-analysis","real-time","real-time-analytics","stream-processing","web3"],"latest_commit_sha":null,"homepage":"https://medium.com/google-cloud/live-ethereum-and-bitcoin-data-in-google-bigquery-and-pub-sub-765b71cd57b5","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blockchain-etl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-17T06:48:27.000Z","updated_at":"2025-03-07T11:17:07.000Z","dependencies_parsed_at":"2022-09-03T00:42:01.151Z","dependency_job_id":null,"html_url":"https://github.com/blockchain-etl/blockchain-etl-streaming","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/blockchain-etl/blockchain-etl-streaming","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fblockchain-etl-streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fblockchain-etl-streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fblockchain-etl-streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fblockchain-etl-streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blockchain-etl","download_url":"https://codeload.github.com/blockchain-etl/blockchain-etl-streaming/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fblockchain-etl-streaming/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261818979,"owners_count":23214406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-beam","bitcoin","blockchain","blockchain-analytics","crypto","cryptocurrency","data-analytics","data-engineering","ethereum","etl","gcp","google-bigquery","google-cloud-platform","google-dataflow","google-pubsub","on-chain-analysis","real-time","real-time-analytics","stream-processing","web3"],"created_at":"2025-06-03T08:14:57.428Z","updated_at":"2025-06-25T06:32:07.161Z","avatar_url":"https://github.com/blockchain-etl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Blockchain ETL Streaming\n\nStreams the following Ethereum entities to Pub/Sub or Postgres using \n[ethereum-etl stream](https://github.com/blockchain-etl/ethereum-etl/tree/develop/docs/commands.md#stream):\n\n- blocks\n- transactions\n- logs\n- token_transfers \n- traces\n- contracts\n- tokens\n\nStreams blocks and transactions to Pub/Sub using \n[bitcoin-etl stream](https://github.com/blockchain-etl/bitcoin-etl#stream). Supported chains:\n\n- bitcoin\n- bitcoin_cash\n- dogecoin\n- litecoin\n- dash\n- zcash\n\n## Deployment Instructions\n\n1. Create a cluster:\n\n```bash\ngcloud container clusters create ethereum-etl-streaming \\\n--zone us-central1-a \\\n--num-nodes 1 \\\n--disk-size 10GB \\\n--machine-type custom-2-4096 \\\n--network default \\\n--subnetwork default \\\n--scopes pubsub,storage-rw,logging-write,monitoring-write,service-management,service-control,trace\n```\n\n2. Get `kubectl` credentials:\n\n```bash\ngcloud container clusters get-credentials ethereum-etl-streaming \\\n--zone us-central1-a\n```\n\n3. Create Pub/Sub topics (use `create_pubsub_topics_ethereum.sh`). Skip this step if you need to stream to Postgres.\n  - \"crypto_ethereum.blocks\" \n  - \"crypto_ethereum.transactions\" \n  - \"crypto_ethereum.token_transfers\" \n  - \"crypto_ethereum.logs\" \n  - \"crypto_ethereum.traces\" \n  - \"crypto_ethereum.contracts\"\n  - \"crypto_ethereum.tokens\" \n\n4. Create GCS bucket. Upload a text file with block number you want to start streaming from to \n`gs://\u003cYOUR_BUCKET_HERE\u003e/ethereum-etl/streaming/last_synced_block.txt`.\n\n5. Create \"ethereum-etl-app\" service account with roles:\n    - Pub/Sub Editor\n    - Storage Object Admin\n    - Cloud SQL Client\n\nDownload the key. Create a Kubernetes secret:\n\n```bash\nkubectl create secret generic streaming-app-key --from-file=key.json=$HOME/Downloads/key.json -n eth\n```\n\n6. Install [helm] (https://github.com/helm/helm#install) \n\n```bash\nbrew install helm\nhelm init  \nbash patch-tiller.sh\n```\n7. Copy [example values](example_values) directory to `values` dir and adjust all the files at least with your bucket and project ID.\n8. Install ETL apps via helm using chart from this repo and values we adjust on previous step, for example:\n```bash\nhelm install --name btc --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/bitcoin/values.yaml\nhelm install --name bch --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/bitcoin_cash/values.yaml\nhelm install --name dash --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/dash/values.yaml\nhelm install --name dogecoin --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/dogecoin/values.yaml\nhelm install --name litecoin --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/litecoin/values.yaml\nhelm install --name zcash --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/zcash/values.yaml\n\nhelm install --name eth-blocks --namespace eth charts/blockchain-etl-streaming \\ \n--values values/ethereum/values.yaml --values values/ethereum/block_data/values.yaml\nhelm install --name eth-traces --namespace eth charts/blockchain-etl-streaming \\ \n--values values/ethereum/values.yaml --values values/ethereum/trace_data/values.yaml \n\n``` \nEthereum block and trace data streaming are decoupled for higher reliability.\n\nTo stream to Postgres:\n\n```bash\nhelm install --name eth-postgres --namespace eth charts/blockchain-etl-streaming \\ \n--values values/ethereum/values-postgres.yaml\n``` \n\nRefer to https://github.com/blockchain-etl/ethereum-etl-postgres for table schema and initial data load.\n\n9. Use `describe` command to troubleshoot, f.e.:\n\n```bash\nkubectl describe pods -n btc\nkubectl describe node [NODE_NAME]\n```\n\nRefer to [blockchain-etl-dataflow](https://github.com/blockchain-etl/blockchain-etl-dataflow)\nfor connecting Pub/Sub to BigQuery.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockchain-etl%2Fblockchain-etl-streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblockchain-etl%2Fblockchain-etl-streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockchain-etl%2Fblockchain-etl-streaming/lists"}