{"id":20336727,"url":"https://github.com/blockchain-data-analytics/cardano_mainchain_parquet","last_synced_at":"2026-03-09T10:01:51.627Z","repository":{"id":180456712,"uuid":"627008931","full_name":"Blockchain-Data-Analytics/Cardano_Mainchain_Parquet","owner":"Blockchain-Data-Analytics","description":"Prepare Parquet tables from Db-sync snapshots and blazingly fast query them using \"duckdb\" or \"spark\"","archived":false,"fork":false,"pushed_at":"2023-04-12T16:01:02.000Z","size":7,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-04T13:46:56.814Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Blockchain-Data-Analytics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-12T15:44:34.000Z","updated_at":"2023-04-12T15:44:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"c4aadbe9-e6dc-499a-aca7-f9f4f766a466","html_url":"https://github.com/Blockchain-Data-Analytics/Cardano_Mainchain_Parquet","commit_stats":null,"previous_names":["blockchain-data-analytics/cardano_mainchain_parquet"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Blockchain-Data-Analytics/Cardano_Mainchain_Parquet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockchain-Data-Analytics%2FCardano_Mainchain_Parquet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockchain-Data-Analytics%2FCardano_Mainchain_Parquet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockchain-Data-Analytics%2FCardano_Mainchain_Parquet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockchain-Data-Analytics%2FCardano_Mainchain_Parquet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Blockchain-Data-Analytics","download_url":"https://codeload.github.com/Blockchain-Data-Analytics/Cardano_Mainchain_Parquet/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockchain-Data-Analytics%2FCardano_Mainchain_Parquet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30290912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T02:57:19.223Z","status":"ssl_error","status_checked_at":"2026-03-09T02:56:26.373Z","response_time":61,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T21:06:40.263Z","updated_at":"2026-03-09T10:01:51.611Z","avatar_url":"https://github.com/Blockchain-Data-Analytics.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Extract Cardano on-chain data to Parquet files\n\n[Db-sync](https://github.com/input-output-hk/cardano-db-sync) maps on-chain data for the Cardano blockchain to PostgreSQL.\n\nFor extended use cases, instead of querying the database directly we would like to use a computer cluster to run such queries in parallel at MaxSpeed\u0026trade;.\n\nWe already provided an export to [BigQuery](https://github.com/input-output-hk/data-analytics-bigquery) on Google's cloud. But this comes with a cost.\n\nInstead, this project let's you export Db-sync's tables to Parquet files which can be queried locally on your machine or in your local network using many computers.\n\nWe use [duckdb](https://duckdb.org/) to output Parquet files which can be queried using _duckdb_ or [Spark SQL](https://spark.apache.org/sql/).\n\n## snapshot\n\nget a db-sync snapshot from: https://update-cardano-mainnet.iohk.io/cardano-db-sync/index.html\n\ninstall it in a PostgreSQL db\n\n\n\n## PostgreSQL setup\n\ncreate a database with name e.g. \"cardanomainnet13\" and a user with the same name.\n\nalso add the views from ext/data-analytics-bigquery.git/schema/views to schema analytics.\n\nfirst, change the target users in the SQL files in that directory: \n\n`for F in *.sql; sed -i -e 's/db_sync_master/cardanomainnet13/g;s/db_sync_reader/cardanomainnet13/g'; done`\n\n\n\n## copy to Parquet files\n\nthen run the continuous export to Parquet files\n(create a .pgpass file or set the password in an env. var. PGPASSWORD)\n\n```sh\nexport PGHOST=host\nexport PGPORT=5432\nexport PGDATABASE=cardanomainnet13\nexport PGUSER=cardanomainnet13\nexport PGPASSFILE=.pgpass\n```\n\nthe target path:\n```sh\nPARQUET_PATH=/usr/local/spark/data\n```\n\n```sh\nTBL=block  # for example\nDBFILE=${TBL}.ddb\nINCREMENT=20\n./duckdb ${DBFILE} \u003c schema/${TBL}.sql\nfor E in `seq 0 ${INCREMENT} 500`; do\n  echo \"exporting ${TBL} from ${E} to $((E+${INCREMENT})) as CSV\"\n  psql -c \"\\\\copy ( SELECT * FROM analytics.vw_bq_${TBL} WHERE epoch_no \u003e= ${E} AND epoch_no \u003c ${E} + ${INCREMENT} ) to '${TBL}_${E}.csv' csv;\"\n  ./duckdb -c \"COPY ${TBL} FROM '${TBL}_${E}.csv';\" ${DBFILE}\ndone\n./duckdb -c \"COPY ${TBL} TO '${PARQUET_PATH}/cardano_mainchain/${TBL}.parquet' (FORMAT PARQUET, COMPRESSION SNAPPY);\" ${DBFILE}\n```\n\ntable tx_in_out\n---------------\n\nthis table is very large and thus we will split it into pieces.\nfirst, generate the CSV files with the above loop.\nthen, load groups of CSV files into Parquet files.\n\n```sh\nTBL=tx_in_out\nDBFILE=${TBL}.ddb\nINCREMENT=20\nfor E in `seq 0 ${INCREMENT} 500`; do\n    if [ -e $DBFILE ]; then rm -v $DBFILE; fi\n    ./duckdb ${DBFILE} \u003c schema/${TBL}.sql\n    for F in `seq $E 1 $((E+INCREMENT-1))`; do\n        if [ -e ${TBL}_${F}.csv ]; then\n            ./duckdb -c \"COPY ${TBL} FROM '${TBL}_${F}.csv';\" ${DBFILE}\n        fi\n    done\n    ./duckdb -c \"COPY ${TBL} TO '${PARQUET_PATH}/cardano_mainchain/${TBL}/${TBL}_${E}.parquet' (FORMAT PARQUET, COMPRESSION SNAPPY);\" ${DBFILE}\ndone\n\nquerying those files:\n\n`duckdb -c \"SELECT epoch_no, COUNT(*) FROM read_parquet('${PARQUET_PATH}/cardano_mainchain/tx_in_out/tx_in_out*.parquet') GROUP BY epoch_no ORDER BY epoch_no;\"`\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockchain-data-analytics%2Fcardano_mainchain_parquet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblockchain-data-analytics%2Fcardano_mainchain_parquet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockchain-data-analytics%2Fcardano_mainchain_parquet/lists"}