{"id":13651943,"url":"https://github.com/blockchain-etl/eos-etl","last_synced_at":"2025-07-14T01:15:43.747Z","repository":{"id":33304511,"uuid":"156980663","full_name":"blockchain-etl/eos-etl","owner":"blockchain-etl","description":"ETL scripts for EOS.","archived":false,"fork":false,"pushed_at":"2023-05-22T21:37:21.000Z","size":231,"stargazers_count":9,"open_issues_count":2,"forks_count":7,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-03-25T10:44:31.112Z","etag":null,"topics":["apache-beam","blockchain-analytics","crypto","cryptocurrency","data-analytics","data-engineering","eos","eosio","etl","gcp","google-bigquery","google-cloud","google-cloud-platform","google-dataflow","google-pubsub","on-chain-analysis","web3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blockchain-etl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-11-10T12:49:12.000Z","updated_at":"2022-10-18T12:12:26.000Z","dependencies_parsed_at":"2024-01-03T05:13:53.293Z","dependency_job_id":"998ea1a9-c9d9-4c20-9861-2cdb5075be05","html_url":"https://github.com/blockchain-etl/eos-etl","commit_stats":{"total_commits":39,"total_committers":4,"mean_commits":9.75,"dds":0.3589743589743589,"last_synced_commit":"04854203284c4c0945278ac18972208369d18f85"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Feos-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Feos-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Feos-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockchain-etl%2Feos-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blockchain-etl","download_url":"https://codeload.github.com/blockchain-etl/eos-etl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223909848,"owners_count":17223575,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-beam","blockchain-analytics","crypto","cryptocurrency","data-analytics","data-engineering","eos","eosio","etl","gcp","google-bigquery","google-cloud","google-cloud-platform","google-dataflow","google-pubsub","on-chain-analysis","web3"],"created_at":"2024-08-02T02:00:53.638Z","updated_at":"2024-11-10T03:30:20.332Z","avatar_url":"https://github.com/blockchain-etl.png","language":"Python","funding_links":[],"categories":["Developers"],"sub_categories":["Developer Tools"],"readme":"# EOS ETL\n\n[![Join the chat at https://gitter.im/ethereum-eth](https://badges.gitter.im/ethereum-etl.svg)](https://gitter.im/ethereum-etl/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n[![Build Status](https://travis-ci.org/blockchain-etl/eos-etl.png)](https://travis-ci.org/blockchain-etl/eos-etl)\n[Join Telegram Group](https://t.me/joinchat/GsMpbA3mv1OJ6YMp3T5ORQ)\n\nInstall EOS ETL:\n\n```bash\npip install eos-etl\n```\n\nExport blocks, transactions and actions ([Schema](#schema), [Reference](#export_blocks)):\n\n```bash\n\u003e eosetl export_blocks --start-block 1 --end-block 500000 \\\n--provider-uri http://api.main.alohaeos.com \\\n--blocks-output blocks.json --transactions-output transactions.json --actions-output actions.json\n```\n\nStream blockchain data continually to console:\n\n```bash\n\u003e pip install eos-etl[streaming]\n\u003e eosetl stream -p http://api.main.alohaeos.com --start-block 500000\n```\n\nFor the latest version, check out the repo and call \n```bash\n\u003e pip install -e .[streaming] \n\u003e python eosetl.py\n```\n\n## Table of Contents\n\n- [Schema](#schema)\n- [Exporting the Blockchain](#exporting-the-blockchain)\n  - [Running in Docker](#running-in-docker)\n  - [Command Reference](#command-reference)\n- [Public Datasets in BigQuery](#public-datasets-in-bigquery)\n\n## Schema\n\n- [blocks](https://github.com/blockchain-etl/eos-etl-airflow/tree/master/dags/resources/stages/raw/schemas/blocks.json)\n- [transactions](https://github.com/blockchain-etl/eos-etl-airflow/tree/master/dags/resources/stages/raw/schemas/transactions.json)\n- [actions](https://github.com/blockchain-etl/eos-etl-airflow/tree/master/dags/resources/stages/enrich/raw/actions.json)\n\n## Exporting the Blockchain\n\n1. Install python 3.6.0+ https://www.python.org/downloads/\n\n1. Install EOS node or get access to EOS node maintained by someone else (because running your own node is not so easy).\nSome docs:\n- [https://developers.eos.io/eosio-nodeos/docs/](https://developers.eos.io/eosio-nodeos/docs/)\n- [https://eosnode.tools/](https://eosnode.tools/)\n- [https://github.com/CryptoLions/EOS-MainNet](https://github.com/CryptoLions/EOS-MainNet)\n\n1. Make sure it downloaded the blocks that you need by executing in the terminal:\n```bash\ncurl --request POST \\\n  --url https://localhost:8080/v1/chain/get_info \\\n  --header 'accept: application/json'\n```\nYou can export blocks below `last_irreversible_block_num`, there is no need to wait until the full sync\n\n1. Install EOS ETL:\n\n    ```bash\n    \u003e pip install eos-etl\n    ```\n\n1. Export blocks, transactions and actions:\n\n    ```bash\n    \u003e eosetl export_all --start 1 --end 499999  \\\n    --provider-uri http://api.main.alohaeos.com\n    ```\n    \n    In case `eosetl` command is not available in PATH, use `python -m eosetl` instead.\n  \n    The result will be in the `output` subdirectory, partitioned in Hive style:\n\n    ```bash\n    output/blocks/start_block=00000000/end_block=00000099/blocks_00000000_00000099.csv\n    output/blocks/start_block=00000100/end_block=00000199/blocks_00000100_=00000199.csv\n    ...\n    output/transactions/start_block=00000000/end_block=00000099/transactions_00000000_00000099.csv\n    ...\n    output/actions/start_block=00000000/end_block=00000099/actions_00000000_00000099.csv\n    ...\n    ```\n    \n### Running in Docker\n\n1. Install Docker https://docs.docker.com/install/\n\n1. Build a docker image\n    ```bash\n    \u003e docker build -t eos-etl:latest .\n    \u003e docker image ls\n    ```\n\n1. Run a container out of the image\n    ```bash\n    \u003e MSYS_NO_PATHCONV=1 docker run -v $HOME/output:/eos-etl/output eos-etl:latest \\\n        export_blocks --max-workers 50 --start-block 30000000 \\\n        --end-block 30000100 --provider-uri http://your_eos_node:node_port \\\n        --blocks-output ./output/blocks.csv --transactions-output ./output/transactions.csv \\\n        --actions-output ./output/actions.csv\n    ```\n    \n1. Run streaming to console or Pub/Sub\n    ```bash\n    \u003e MSYS_NO_PATHCONV=1 docker build -t eos-etl:latest-streaming -f Dockerfile_with_streaming .\n    \u003e echo \"Stream to console\"\n    \u003e MSYS_NO_PATHCONV=1 docker run eos-etl:latest-streaming stream -p http://api.main.alohaeos.com --start-block 500000\n    \u003e echo \"Stream to Pub/Sub\"\n    \u003e MSYS_NO_PATHCONV=1 docker run -v /path_to_credentials_file/:/eos-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/eos-etl/credentials_file.json eos-etl:latest-streaming stream -p http://api.main.alohaeos.com --start-block 500000 --output projects/your-project/topics/crypto_eos\n    ```\n\n1. Refer to https://github.com/blockchain-etl/blockchain-etl-streaming for deploying the streaming app to \nGoogle Kubernetes Engine.\n\n### Command Reference\n\n- [export_blocks](#export_blocks)\n- [get_block_range_for_date](#get_block_range_for_date)\n- [export_all](#export_all)\n- [stream](#stream)\n\nAll the commands accept `-h` parameter for help, e.g.:\n\n```bash\n\u003e python eosetl.py export_blocks --help\nUsage: eosetl.py export_blocks [OPTIONS]\n\n  Export blocks, transactions and actions.\n\nOptions:\n  -s, --start-block INTEGER   Start block\n  -e, --end-block INTEGER     End block  [required]\n  -p, --provider-uri TEXT     The URI of the remote EOS node\n  -w, --max-workers INTEGER   The maximum number of workers.\n  --blocks-output TEXT        The output file for blocks. If not provided\n                              blocks will not be exported. Use \"-\" for stdout\n  --transactions-output TEXT  The output file for transactions. If not\n                              provided transactions will not be exported. Use\n                              \"-\" for stdout\n  --actions-output TEXT       The output file for actions. If not provided\n                              transactions will not be exported. Use \"-\"\n                              for stdout\n  --help                      Show this message and exit.\n```\n\nFor the `--output` parameters the supported type is json. The format type is inferred from the output file name.\n\n#### export_blocks\n\n```bash\n\u003e python eosetl.py export_blocks --start-block 1 --end-block 500000 \\\n  --provider-uri http://api.main.alohaeos.com \\\n  --blocks-output blocks.json --transactions-output transactions.json --actions-output actions.json\n```\n\nOmit `--blocks-output` or `--transactions-output` or `--actions-output` options if you want to export only transactions/blocks/actions.\n\nYou can tune `--max-workers` for performance.\n\n#### get_block_range_for_date\n\n```bash\n\u003e python eosetl.py get_block_range_for_date --provider-uri http://api.main.alohaeos.com --date=2018-06-09\n```\n\n#### export_all\n\n```bash\n\u003e python eosetl.py export_all --provider-uri http://api.main.alohaeos.com --start 2018-06-08 --end 2018-06-09\n```\n\nYou can tune `--export-batch-size`, `--max-workers` for performance.\n\n#### stream\n\n```bash\n\u003e python eosetl.py stream --provider-uri http://api.main.alohaeos.com --start-block 500000\n```\n\n- This command outputs blocks and transactions to the console by default.\n- Use `--output` option to specify the Google Pub/Sub topic where to publish blockchain data, \ne.g. `projects/your-project/topics/eos_blockchain`.\n- The command saves its state to `last_synced_block.txt` file where the last synced block number is saved periodically.\n- Specify either `--start-block` or `--last-synced-block-file` option. `--last-synced-block-file` should point to the \nfile where the block number, from which to start streaming the blockchain data, is saved.\n- Use the `--lag` option to specify how many blocks to lag behind the head of the blockchain. It's the simplest way to \nhandle chain reorganizations - they are less likely the further a block from the head.\n- You can tune `--period-seconds`, `--batch-size`, `--max-workers` for performance.\n \n\n### Running Tests\n\n```bash\n\u003e pip install -e .[dev]\n\u003e echo \"The below variables are optional\"\n\u003e export EOSETL_PROVIDER_URI=http://api.main.alohaeos.com\n\u003e pytest -vv\n```\n\n### Running Tox Tests\n\n```bash\n\u003e pip install tox\n\u003e tox\n```\n\n### Public Datasets in BigQuery\n\nTODO: https://cloud.google.com/blog/products/data-analytics/introducing-six-new-cryptocurrencies-in-bigquery-public-datasets-and-how-to-analyze-them\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockchain-etl%2Feos-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblockchain-etl%2Feos-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockchain-etl%2Feos-etl/lists"}