{"id":28414402,"url":"https://github.com/blockchain-etl/ethereum-etl-airflow","last_synced_at":"2025-06-24T23:30:27.152Z","repository":{"id":37492943,"uuid":"145594246","full_name":"blockchain-etl/ethereum-etl-airflow","owner":"blockchain-etl","description":"Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethereum-smart-contract-into-bigquery-in-8-mins-bab5db1fdeee","archived":false,"fork":false,"pushed_at":"2025-03-19T22:36:00.000Z","size":3305,"stargazers_count":416,"open_issues_count":28,"forks_count":199,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-06-03T18:28:07.713Z","etag":null,"topics":["apache-airflow","blockchain-analytics","crypto","cryptocurrency","data-analytics","data-engineering","ethereum","etl","gcp","google-cloud","google-cloud-platform","on-chain-analysis","web3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blockchain-etl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"custom":"https://gitcoin.co/grants/233/ethereumetl"}},"created_at":"2018-08-21T17:01:21.000Z","updated_at":"2025-04-03T18:20:57.000Z","dependencies_parsed_at":"2024-04-30T14:32:50.656Z","dependency_job_id":"91ae2c22-6345-406b-91b5-28680b653083","html_url":"https://github.com/blockchain-etl/ethereum-etl-airflow","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/blockchain-etl/ethereum-etl-airflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fethereum-etl-airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fethereum-etl-airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fethereum-etl-airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fethereum-etl-airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blockchain-etl","download_url":"https://codeload.github.com/blockchain-etl/ethereum-etl-airflow/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Fethereum-etl-airflow/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261774410,"owners_count":23207732,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-airflow","blockchain-analytics","crypto","cryptocurrency","data-analytics","data-engineering","ethereum","etl","gcp","google-cloud","google-cloud-platform","on-chain-analysis","web3"],"created_at":"2025-06-03T08:15:15.994Z","updated_at":"2025-06-24T23:30:27.146Z","avatar_url":"https://github.com/blockchain-etl.png","language":"Python","funding_links":["https://gitcoin.co/grants/233/ethereumetl"],"categories":[],"sub_categories":[],"readme":"# Ethereum ETL Airflow\n\nRead this article: https://cloud.google.com/blog/products/data-analytics/ethereum-bigquery-how-we-built-dataset\n\n## Local Development Prerequisites\n\n- direnv\n- pyenv\n\n\nWe are using direnv to automatically set up and load the correct python version. We also create a venv in the root folder, \nthat is automatically activated when entering the project folder.\n\n## Setting up Airflow DAGs using Google Cloud Composer\n\n### Create BigQuery Datasets\n\n- Sign in to BigQuery https://bigquery.cloud.google.com/\n- Create new datasets called `crypto_ethereum`, `crypto_ethereum_raw`, `crypto_ethereum_temp`\n\n### Create Google Cloud Storage bucket\n\n- Create a new Google Storage bucket to store exported files https://console.cloud.google.com/storage/browser\n\n### Create Google Cloud Composer (version 2) environment\n\nCreate a new Cloud Composer environment:\n\n```bash\nexport ENVIRONMENT_NAME=ethereum-etl-0\n\nAIRFLOW_CONFIGS_ARR=(\n    \"celery-worker_concurrency=8\"\n    \"scheduler-dag_dir_list_interval=300\"\n    \"scheduler-min_file_process_interval=120\"\n)\nexport AIRFLOW_CONFIGS=$(IFS=, ; echo \"${AIRFLOW_CONFIGS_ARR[*]}\")\n\ngcloud composer environments create \\\n    $ENVIRONMENT_NAME \\\n    --location=us-central1 \\\n    --image-version=composer-2.1.14-airflow-2.5.1 \\\n    --environment-size=medium \\\n    --scheduler-cpu=2 \\\n    --scheduler-memory=13 \\\n    --scheduler-storage=1 \\\n    --scheduler-count=1 \\\n    --web-server-cpu=1 \\\n    --web-server-memory=2 \\\n    --web-server-storage=512MB \\\n    --worker-cpu=2 \\\n    --worker-memory=13 \\\n    --worker-storage=10 \\\n    --min-workers=1 \\\n    --max-workers=8 \\\n    --airflow-configs=$AIRFLOW_CONFIGS\n\ngcloud composer environments update \\\n    $ENVIRONMENT_NAME \\\n    --location=us-central1 \\\n    --update-pypi-packages-from-file=requirements_airflow.txt\n```\n\nCreate variables in Airflow (**Admin \u003e Variables** in the UI):\n\n| Variable                                | Description                             |\n|-----------------------------------------|-----------------------------------------|\n| ethereum_output_bucket                  | GCS bucket to store exported files      |\n| ethereum_provider_uris                  | Comma separated URIs of Ethereum nodes  |\n| ethereum_destination_dataset_project_id | Project ID of BigQuery datasets         |\n| notification_emails                     | email for notifications                 |\n\nCheck other variables in `dags/ethereumetl_airflow/variables.py`.\n\n### Updating package requirements\nSuggested package requirements for Composer are stored in `requirements_airflow.txt`.\n\nYou can update the Composer environment using the following script:\n```bash\nENVIRONMENT_NAME=\"ethereum-etl-0\"\nLOCAL_REQUIREMENTS_PATH=\"$(mktemp)\"\n\n# grep pattern removes comments and whitespace:\ncat \"./requirements_airflow.txt\" | grep -o '^[^#| ]*' \u003e \"$LOCAL_REQUIREMENTS_PATH\"\n\ngcloud composer environments update \\\n  \"$ENVIRONMENT_NAME\" \\\n  --location=\"us-central1\" \\\n  --update-pypi-packages-from-file=\"$LOCAL_REQUIREMENTS_PATH\"\n```\n\n**Note:** Composer can be _very_ pedantic about conflicts in additional packages. You may have to fix dependency conflicts where you had no issues testing locally (when updating dependencies, Composer does something \"cleverer\" than just `pip install -r requirements.txt`). This is why `eth-hash` is currently pinned in `requirements_airflow.txt`. Typically we have found that pinning `eth-hash` and/or `eth-rlp` may make things work, though Your Mileage May Vary.\n\nSee [this issue](https://github.com/blockchain-etl/ethereum-etl-airflow/issues/481#issuecomment-1332878533) for further ideas on how to unblock problems you may encounter.\n\n### Upload DAGs\n\n```bash\n\u003e ./upload_dags.sh \u003cairflow_bucket\u003e\n```\n\n### Running Tests\n\n```bash\npip install \\\n    -r requirements_test.txt \\\n    -r requirements_local.txt \\\n    -r requirements_airflow.txt\npytest -vv -s\n```\n\n### Running locally\nA docker compose definition has been provided to easily spin up a local Airflow instance.\n\nTo build the required image:\n```bash\ndocker compose build\n```\nTo start Airflow:\n```bash\ndocker compose up airflow\n```\n\nThe instance requires the `CLOUDSDK_CORE_PROJECT` environment variable to be set in most cases. Airflow Variables can be defined in [variables.json](./docker/variables.json).\n\n### Creating Table Definition Files for Parsing Events and Function Calls\n\nRead this article: https://medium.com/@medvedev1088/query-ens-and-0x-events-with-sql-in-google-bigquery-4d197206e644\n\n### Debugging Table Defenition Files\n\nA utility script for debugging and verifying contract parsing in Ethereum data processing pipelines is available. You can simply run \n\n```\npython3 generate_parse_sql.py \u003cpath_to_table_definition_file\u003e \u003cdate\u003e\n```\n\nThis will output some example SQL that can be used to debug if the generated json files from the contract parser are correct. \n\nNOTE: certain files may not have the `contract_address` field specified as a valid address [ERC20Pool_event_TransferLP](dags/resources/stages/parse/table_definitions/ajna_v2/ERC20Pool_event_TransferLP.json) but use a select statement on another table instead. For these you can simply pass the contract address yourself like below:\n\n```\npython3 generate_parse_sql.py \u003cpath_to_table_definition_file\u003e \u003cdate\u003e --contract_address \u003ccontract_address\u003e\n```\n\n\n\n### More Information\n\nYou can follow the instructions here for Polygon DAGs https://github.com/blockchain-etl/polygon-etl. The architecture\nthere is very similar to Ethereum so in most case substituting `polygon` for `ethereum` will work. Contributions \nto this README file for porting documentation from Polygon to Ethereum are welcome.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockchain-etl%2Fethereum-etl-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblockchain-etl%2Fethereum-etl-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockchain-etl%2Fethereum-etl-airflow/lists"}