{"id":24491262,"url":"https://github.com/blockchain-etl/iotex-etl","last_synced_at":"2025-04-14T00:21:14.426Z","repository":{"id":62571462,"uuid":"275166252","full_name":"blockchain-etl/iotex-etl","owner":"blockchain-etl","description":"ETL (extract, transform and load) tools for ingesting IoTeX blockchain data to Google BigQuery and Pub/Sub","archived":false,"fork":false,"pushed_at":"2023-07-05T21:04:55.000Z","size":456,"stargazers_count":5,"open_issues_count":5,"forks_count":7,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-27T14:21:43.134Z","etag":null,"topics":["bigquery","blockchain-data","iotex","sql"],"latest_commit_sha":null,"homepage":"https://console.cloud.google.com/bigquery?page=dataset\u0026d=crypto_iotex\u0026p=public-data-finance","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blockchain-etl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-26T13:46:13.000Z","updated_at":"2023-08-28T21:41:48.000Z","dependencies_parsed_at":"2022-11-04T00:33:40.746Z","dependency_job_id":null,"html_url":"https://github.com/blockchain-etl/iotex-etl","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fiotex-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fiotex-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fiotex-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fiotex-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blockchain-etl","download_url":"https://codeload.github.com/blockchain-etl/iotex-etl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248519553,"owners_count":21117761,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","blockchain-data","iotex","sql"],"created_at":"2025-01-21T18:17:25.109Z","updated_at":"2025-04-14T00:21:14.402Z","avatar_url":"https://github.com/blockchain-etl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IoTeX ETL\n\n[![Build Status](https://travis-ci.org/blockchain-etl/iotex-etl.svg?branch=master)](https://travis-ci.org/blockchain-etl/iotex-etl)\n[![Telegram](https://img.shields.io/badge/telegram-join%20chat-blue.svg)](https://t.me/joinchat/GsMpbA3mv1OJ6YMp3T5ORQ)\n\n## Overview\n\nIoTeX ETL allows you to setup an ETL pipeline in Google Cloud Platform for ingesting IoTeX blockchain data \ninto BigQuery and Pub/Sub. It comes with [CLI tools](/cli) for exporting IoTeX data into JSON newline-delimited files\npartitioned by day. \n\nData is available for you to query right away in \n[Google BigQuery](https://console.cloud.google.com/bigquery?page=dataset\u0026d=crypto_iotex\u0026p=public-data-finance).\n\n## Architecture\n\n![iotex_etl_architecture.svg](iotex_etl_architecture.svg)\n\n[Google Slides version](https://docs.google.com/presentation/d/1VFMR4f8lghnpGZWZTevRTv6Zn9n9IUWHRnNrQsNE-8Y/edit#slide=id.p89)\n\n1. The nodes are run in a Kubernetes cluster. \n    Refer to [IoTeX Node in Kubernetes](https://github.com/blockchain-etl/iotex-kubernetes) for deployment instructions.\n\n2. [Airflow DAGs](https://airflow.apache.org/) export and load IoTeX data to BigQuery daily. \n    Refer to [IoTeX ETL Airflow](/airflow) for deployment instructions.\n  \n3. IoTeX data is polled periodically from the nodes and pushed to Google Pub/Sub. \n    Refer to [IoTeX ETL Streaming](/streaming) for deployment instructions.  \n  \n4. IoTeX data is pulled from Pub/Sub, transformed and streamed to BigQuery. \n    Refer to [IoTeX ETL Dataflow](/dataflow) for deployment instructions.  \n \n## Setting Up\n\n1. Follow the instructions in [IoTeX Node in Kubernetes](https://github.com/blockchain-etl/iotex-kubernetes) to deploy\n    an IoTeX node in GKE. Wait until it's fully synced. Make note of the Load Balancer IP from the node deployment, it\n    will be used in Airflow and Streamer components below.\n\n2. Follow the instructions in [IoTeX ETL Airflow](/airflow) to deploy a Cloud Composer cluster for \n    exporting and loading historical IoTeX data. It may take several hours for the export DAG to catch up. During this\n    time \"load\" and \"verify_streaming\" DAGs will fail. \n\n3. Follow the instructions in [IoTeX ETL Streaming](/streaming) to deploy the Streamer component. For the value in \n    `last_synced_block.txt` specify the last block number of the previous day. You can query it in BigQuery:\n    `SELECT height FROM crypto_iotex.blocks ORDER BY height DESC LIMIT 1`.\n\n4. Follow the instructions in [IoTeX ETL Dataflow](/dataflow) to deploy the Dataflow component. Monitor \n    \"verify_streaming\" DAG in Airflow console, once the Dataflow job catches up the latest block, the DAG will succeed.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockchain-etl%2Fiotex-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblockchain-etl%2Fiotex-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockchain-etl%2Fiotex-etl/lists"}