{"id":27259790,"url":"https://github.com/neurone/hedera-raw-data","last_synced_at":"2025-04-11T04:07:01.260Z","repository":{"id":226636902,"uuid":"769237839","full_name":"Neurone/hedera-raw-data","owner":"Neurone","description":"Tools to download, manage and share Hedera raw data","archived":false,"fork":false,"pushed_at":"2025-01-10T03:04:59.000Z","size":82,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-11T04:06:58.462Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Neurone.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-08T16:17:30.000Z","updated_at":"2025-01-10T03:05:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"98b5f5e0-f8e2-4f3b-87a5-77887000f250","html_url":"https://github.com/Neurone/hedera-raw-data","commit_stats":null,"previous_names":["neurone/hedera-raw-data"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neurone%2Fhedera-raw-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neurone%2Fhedera-raw-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neurone%2Fhedera-raw-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neurone%2Fhedera-raw-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Neurone","download_url":"https://codeload.github.com/Neurone/hedera-raw-data/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248339266,"owners_count":21087215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-11T04:07:00.793Z","updated_at":"2025-04-11T04:07:01.249Z","avatar_url":"https://github.com/Neurone.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hedera raw data\n\nTools to download, manage and share Hedera raw data\n\n## Script list\n\n|Name | Description |\n|-|-|\n|check-node-file-lists | Check if all the record file list and corresponding metadata are created. If not, it downloads the list. This checks from the start 2019-09-13 up to yesterday (current date -1 day). Use `fix` parameter in case you want to try to download the list again. |\n|create-metadata-from-file-list-by-day | Create metadata from AWS file list for a specific day. |\n|create-multiple-file-lists-by-day | Create the file list for a specific day for all consensus nodes. |\n|create-single-file-list-by-day | Create a single file list from AWS. |\n|download-and-check-records-by-md5sum-list | Download and check record files for a node. |\n|download-and-check-signatures-by-md5sum-list | Download and check signature files for a node. |\n|download-and-check-sidecars-by-md5sum-list | Download and check sidecar files for a node. |\n|download-file | Download a single file from AWS S3 |\n|with-nodes-range | Helper script to loop through node ranges. |\n|loop-date-range | Helper script to loop through dates. |\n\nYou can run multiple scripts that download files (signature only, soon record and sidecar) in parallel on the same list. The script randomize by default the file order, so it reduces the odds to overlap with other script. In case the file is already there, the script skip that file.\n\nIn any case, the script check the hash of the file at the end of the process and, in case it's wrong, it downloads the file again.\n\nTo be 100% sure, you can run the download again at the end. In case some file is missing or the hash does not correspond, it will be downloaded again.\n\nFor example, here I modified an existing file `2019-09-13T23_57_50.186388Z.rcd` and, running the script, I was able to detect the error and download it again:\n\n```zsh\n❯ ./download-and-check-records-by-md5sum-list.sh 0.0.3 2019-09-13\n2024-09-20T13:33:44.419Z-251049 ⚑ Started ./download-and-check-records-by-md5sum-list.sh (PID 251049) with the following configuration\n2024-09-20T13:33:44.420Z-251049 ⛶ Day (UTC) .........................: 2019-09-13\n2024-09-20T13:33:44.421Z-251049 ⛶ Node ID ......................: 0.0.3\n2024-09-20T13:33:44.422Z-251049 ⛶ Node's file lists folder .....: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T13:33:44.423Z-251049 ⛶ Node's records destination ...: /media/sf_Hedera/hedera-streams/records/0.0.3/2019-09-13\n2024-09-20T13:33:44.426Z-251049 ☕ Downloading the record files listed in the MD5 list but missing from the node's folder\nbytes\t1672\tapplication/octet-stream\t\"ac8b4d9696a8669b1ff09b1e4007826a\"\t2019-09-13T23:57:57+00:00\trequester\thNsdT8hs.Fno0s7pMpUGao8V65Fw.WRN\n2024-09-20T13:33:45.746Z-251049 ✔ Download operations ended\n2024-09-20T13:33:45.748Z-251049 ⚙ Checking MD5 checksum for files in /media/sf_Hedera/hedera-streams/lists/0.0.3/2019-09-13.records.md5sum over /media/sf_Hedera/hedera-streams/records/0.0.3/2019-09-13\n2024-09-20T13:33:46.120Z-251049 ✔ All the listed files match with the files in the folder.\n2024-09-20T13:33:46.121Z-251049 🏁 Script ./download-and-check-records-by-md5sum-list.sh (PID 251049) ended\n```\n\n## Usage examples\n\nCreate file lists and metadata files (used to download the actual data) for all current consensus nodes for one day with:\n\n```zsh\n./create-multiple-file-lists-by-day.sh 2024-09-18\n```\n\nExample output\n\n```zsh\n2024-09-20T12:37:40.313Z-225082 ⚑ Started ./create-multiple-file-lists-by-day.sh (PID 225082) with the following configuration\n2024-09-20T12:37:40.313Z-225082 ⛶ Day (UTC) ...............: 2024-09-18\n2024-09-20T12:37:40.314Z-225082 ⛶ First node ID ......: 0.0.3\n2024-09-20T12:37:40.315Z-225082 ⛶ Last node ID .......: 0.0.34\n2024-09-20T12:37:40.315Z-225082 ⛶ Logs folder .............: /media/sf_Hedera/hedera-streams/logs/2024-09-20T12_37_40.312Z-225082-create-single-file-list-by-day-2024-09-18\n2024-09-20T12:37:40.316Z-225082 ⛶ Logs files format .......: 0.0.\u003cNODE_ID\u003e.log\n2024-09-20T12:37:40.317Z-225082 ⛶ Monitor all logs with ...: tail -f /media/sf_Hedera/hedera-streams/logs/2024-09-20T12_37_40.312Z-225082-create-single-file-list-by-day-2024-09-18/*\n2024-09-20T12:37:40.320Z-225082 ✔ Created folder /media/sf_Hedera/hedera-streams/logs/2024-09-20T12_37_40.312Z-225082-create-single-file-list-by-day-2024-09-18\n2024-09-20T12:37:40.322Z-225082 ✔ Starting a single processes for each nodes from 0.0.3 to 0.0.34\n2024-09-20T12:37:40.325Z-225082 ☕ Waiting for all the processes to finish. Closing this script does not stop the sub processes.\n2024-09-20T12:38:41.924Z-225082 🏁 Script ./create-multiple-file-lists-by-day.sh (PID 225082) ended\n```\n\nDownload actual files with dedicated script.\n\nFor example, the following script downloads all the record files from node 0.0.3 for the day 2024-09-19:\n\n```zsh\n./download-and-check-records-by-md5sum-list.sh 0.0.3 2024-09-18\n```\n\nExample output\n\n```zsh\n2024-09-20T12:39:49.815Z-228749 ⚑ Started ./download-and-check-records-by-md5sum-list.sh (PID 228749) with the following configuration\n2024-09-20T12:39:49.816Z-228749 ⛶ Day (UTC) .........................: 2024-09-18\n2024-09-20T12:39:49.817Z-228749 ⛶ Node ID ......................: 0.0.3\n2024-09-20T12:39:49.818Z-228749 ⛶ Node's file lists folder .....: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:39:49.818Z-228749 ⛶ Node's records destination ...: /media/sf_Hedera/hedera-streams/records/0.0.3/2024-09-18\n2024-09-20T12:39:49.826Z-228749 ✔ Created folder /media/sf_Hedera/hedera-streams/records/0.0.3/2024-09-18\n2024-09-20T12:39:49.827Z-228749 ☕ Downloading the record files listed in the MD5 list but missing from the node's folder\nbytes\t1097620\tbinary/octet-stream\t\"52645018244e75b8982d41de2b7c2288\"\t2024-09-18T00:00:06+00:00\tCOMPLETED\trequester\tAES256\tAoExpkNZAeqZeOKgPK0KYIClgsUjieTN\nbytes\t874688\tbinary/octet-stream\t\"bbe2b15f78689010ee7a3e5102b16950\"\t2024-09-18T00:00:08+00:00\tCOMPLETED\trequester\tAES256\t1dUMsQ62gIh80y.x8OKSZvAqtf2bUPwT\nbytes\t1227287\tbinary/octet-stream\t\"a94c2bc713b94643a4d64efdc3dff501\"\t2024-09-18T00:00:09+00:00\tCOMPLETED\trequester\tAES256\tl32AegROxF2ltsrZo5gg14bS23UaJ4Am\nbytes\t1152128\tbinary/octet-stream\t\"be21511fc29927db226da67babeba8d1\"\t2024-09-18T00:00:12+00:00\tCOMPLETED\trequester\tAES256\ttCqKfS7k_8HmZbbfImhrwYdGe9.CcF4Q\nbytes\t1170972\tbinary/octet-stream\t\"f41411f72d30860537761a5c0beef60e\"\t2024-09-18T00:00:14+00:00\tCOMPLETED\trequester\tAES256\tdnzNP.PfnEAB9pkA_JpejYVg2fCReM57\nbytes\t1259463\tbinary/octet-stream\t\"1a66b849d68fef2e4bfb15cd5c3b83d4\"\t2024-09-18T00:00:16+00:00\tCOMPLETED\trequester\tAES256\tyvbteE71Z2EaQqWgMItjNtOsjiUC7Vvc\nbytes\t1214243\tbinary/octet-stream\t\"09d8889da6ee349ad76bdd7b03f2f179\"\t2024-09-18T00:00:18+00:00\tCOMPLETED\trequester\tAES256\tA3fpR4Uv2M8En60ewNfBrJ2hWIaidh92\nbytes\t1230140\tbinary/octet-stream\t\"e86780ec73eec20274fa4b84baf557e1\"\t2024-09-18T00:00:20+00:00\tCOMPLETED\trequester\tAES256\tRhbPzl9uuwvWv4gmQqDIPhXP4_TNmr3B\nbytes\t1260969\tbinary/octet-stream\t\"967ff03fe805e97f2e8045f2d628ff44\"\t2024-09-18T00:00:23+00:00\tCOMPLETED\trequester\tAES256\t6QXIkd0lggrDjjRgz47EY7PeU9QUBjAm\nbytes\t1146352\tbinary/octet-stream\t\"b5c4584dc3bc9645e8fc713cb046b8c2\"\t2024-09-18T00:00:24+00:00\tCOMPLETED\trequester\tAES256\tg7.FeNGp.AXCvzE35Kq9IO2IdXXOhj07\nbytes\t1315552\tbinary/octet-stream\t\"7fe2be8cb6ce2426be263c8dc39d0433\"\t2024-09-18T00:00:27+00:00\tCOMPLETED\trequester\tAES256\tsXIf_p2nCzo1wZw.0iP5bdIPOgYIQ1Mq\n...\n```\n\nApply operations to multiple consensus nodes with the `with-nodes-range` script:\n\n```zsh\n./with-nodes-range.sh 3 11 ./create-single-file-list-by-day.sh 2024-09-18\n```\n\nExample output\n\n```zsh\n2024-09-20T12:41:34.648Z-229386 ⚑ Started ./with-nodes-range.sh (PID 229386) with the following configuration\n2024-09-20T12:41:34.649Z-229386 ⛶ Logs folder .............: /media/sf_Hedera/hedera-streams/logs/2024-09-20T12_41_34.647Z-229386-with-nodes-range-3-11\n2024-09-20T12:41:34.650Z-229386 ⛶ Logs files format .......: 0.0.\u003cNODE_ID\u003e.log\n2024-09-20T12:41:34.650Z-229386 ⛶ Monitor all logs with ...: tail -f /media/sf_Hedera/hedera-streams/logs/2024-09-20T12_41_34.647Z-229386-with-nodes-range-3-11/*\n2024-09-20T12:41:34.651Z-229386 ⛶ Command .................: ./create-single-file-list-by-day.sh \u003c0.0.NODE_ID\u003e 3 11 ./create-single-file-list-by-day.sh 2024-09-18 \u003e /media/sf_Hedera/hedera-streams/logs/2024-09-20T12_41_34.647Z-229386-with-nodes-range-3-11/\u003c0.0.NODE_ID\u003e.log \u0026\n2024-09-20T12:41:34.656Z-229386 ✔ Created folder /media/sf_Hedera/hedera-streams/logs/2024-09-20T12_41_34.647Z-229386-with-nodes-range-3-11\n2024-09-20T12:41:34.656Z-229386 ⚙ Executing command for each nodes from 0.0.3 to 0.0.11\n2024-09-20T12:41:34.659Z-229386 ☕ Waiting for all the processes to finish. Closing this script does not stop the sub processes.\n2024-09-20T12:42:07.638Z-229386 🏁 Script ./with-nodes-range.sh (PID 229386) ended\n```\n\nApply operations to multiple days with the `loop-date-range` script:\n\n```zsh\n./utils/loop-date-range.sh 2019-09-13 2019-09-27 | xargs -I {} ./download-and-check-records-by-md5sum-list.sh 0.0.5 {}\n```\n\nExample output\n\n```zsh\n❯ ./utils/loop-date-range.sh 2019-09-13 2019-09-27 | xargs -I {} ./download-and-check-records-by-md5sum-list.sh 0.0.5 {}\n2024-09-20T12:45:19.797Z-230724 ⚑ Started ./download-and-check-records-by-md5sum-list.sh (PID 230724) with the following configuration\n2024-09-20T12:45:19.798Z-230724 ⛶ Day (UTC) .........................: 2019-09-13\n2024-09-20T12:45:19.799Z-230724 ⛶ Node ID ......................: 0.0.5\n2024-09-20T12:45:19.800Z-230724 ⛶ Node's file lists folder .....: /media/sf_Hedera/hedera-streams/lists/0.0.5\n2024-09-20T12:45:19.800Z-230724 ⛶ Node's records destination ...: /media/sf_Hedera/hedera-streams/records/0.0.5/2019-09-13\n2024-09-20T12:45:19.804Z-230724 ✔ Created folder /media/sf_Hedera/hedera-streams/records/0.0.5/2019-09-13\n2024-09-20T12:45:19.805Z-230724 ☕ Downloading the record files listed in the MD5 list but missing from the node's folder\nbytes\t455\tapplication/octet-stream\t\"874b534374528255a18f0459f0f4a79e\"\t2019-09-13T22:46:06+00:00\trequester\tipZYloKsa8HNnKwDRADHxALgcJgionp7\nbytes\t453\tapplication/octet-stream\t\"0ad15899ec4576886fe10a61354a099c\"\t2019-09-13T22:46:06+00:00\trequester\tu1QojP9YlVwUbcvdPl1JXRiyzURNkxgT\nbytes\t455\tapplication/octet-stream\t\"9de088e826ca33095ee2e387c8924860\"\t2019-09-13T22:46:06+00:00\trequester\teksBHvM5MRx4QL7s3i9krneRb03_0b2O\n...\n```\n\nCheck if all the record file list and corresponding metadata are created. If not, it downloads the list. This checks from the network start date (i.e. 2019-09-13 for Mainnet, but you configure it with HD_HEDERA_MAINNET_START_DATE) up to yesterday (current date -1 day).\n\n```zsh\n./check-node-file-lists.sh 0.0.3\n```\n\nExample output\n\n```zsh\n2024-09-20T12:49:06.710Z-233689 ⚑ Started ./check-node-file-lists.sh (PID 233689) with the following configuration\n2024-09-20T12:49:06.711Z-233689 ⛶ Fix list if missing or incomplete ...: false\n2024-09-20T12:49:06.711Z-233689 ⛶ Checking from .......................: 2019-09-13\n2024-09-20T12:49:06.712Z-233689 ⛶ Current time (System time zone)......: Fri Sep 20 02:49:06 PM CEST 2024\n2024-09-20T12:49:06.713Z-233689 ⛶ Current time (UTC)...................: Fri Sep 20 12:49:06 PM UTC 2024\n2024-09-20T12:49:06.714Z-233689 ⛶ Toady (UTC)..........................: 2024-09-20\n2024-09-20T12:49:06.715Z-233689 ⛶ Yesterday (UTC)......................: 2024-09-19\n2024-09-20T12:49:06.716Z-233689 ⛶ Hedera Mainnet Start Date ...........: 2019-09-13\n2024-09-20T12:49:06.717Z-233689 ⛶ Node ID ........................: 0.0.3\n2024-09-20T12:49:06.718Z-233689 ⛶ Node Join Date .................: 2020-05-26\n2024-09-20T12:49:06.719Z-233689 ⛶ Node Start Date ................: 2019-09-13\n2024-09-20T12:49:06.719Z-233689 ⛶ Node's file lists folder .......: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:49:07.390Z-233689 ⚠ 2021-05-07 is missing or incomplete\n2024-09-20T12:49:07.391Z-233689 ⚠ 2021-05-08 is missing or incomplete\n2024-09-20T12:49:07.393Z-233689 ⚠ 2021-05-09 is missing or incomplete\n2024-09-20T12:49:07.394Z-233689 ⚠ 2021-05-10 is missing or incomplete\n2024-09-20T12:49:08.207Z-233689 ⚠ 2023-04-21 is missing or incomplete\n2024-09-20T12:49:08.209Z-233689 ⚠ 2023-04-22 is missing or incomplete\n2024-09-20T12:49:08.210Z-233689 ⚠ 2023-04-23 is missing or incomplete\n2024-09-20T12:49:08.212Z-233689 ⚠ 2023-04-24 is missing or incomplete\n2024-09-20T12:49:08.823Z-233689 🏁 Script ./check-node-file-lists.sh (PID 233689) ended\n```\n\nUse `fix` parameter in case you want to try to download the list again.\n\n```zsh\n./check-node-file-lists.sh 0.0.3 fix\n```\n\nExample output\n\n```zsh\n2024-09-20T12:52:24.890Z-238196 ⚑ Started ./check-node-file-lists.sh (PID 238196) with the following configuration\n2024-09-20T12:52:24.891Z-238196 ⛶ Fix list if missing or incomplete ...: true\n2024-09-20T12:52:24.892Z-238196 ⛶ Checking from .......................: 2019-09-13\n2024-09-20T12:52:24.892Z-238196 ⛶ Current time (System time zone)......: Fri Sep 20 02:52:24 PM CEST 2024\n2024-09-20T12:52:24.893Z-238196 ⛶ Current time (UTC)...................: Fri Sep 20 12:52:24 PM UTC 2024\n2024-09-20T12:52:24.895Z-238196 ⛶ Toady (UTC)..........................: 2024-09-20\n2024-09-20T12:52:24.895Z-238196 ⛶ Yesterday (UTC)......................: 2024-09-19\n2024-09-20T12:52:24.897Z-238196 ⛶ Hedera Mainnet Start Date ...........: 2019-09-13\n2024-09-20T12:52:24.898Z-238196 ⛶ Node ID ........................: 0.0.3\n2024-09-20T12:52:24.899Z-238196 ⛶ Node Join Date .................: 2020-05-26\n2024-09-20T12:52:24.899Z-238196 ⛶ Node Start Date ................: 2019-09-13\n2024-09-20T12:52:24.900Z-238196 ⛶ Node's file lists folder .......: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:52:25.596Z-238196 ⚠ 2021-05-07 is missing or incomplete\n2024-09-20T12:52:25.599Z-238831 ⚑ Started ./create-single-file-list-by-day.sh (PID 238831) with the following configuration\n2024-09-20T12:52:25.599Z-238831 ⛶ Day (UTC) .......................: 2021-05-07\n2024-09-20T12:52:25.600Z-238831 ⛶ Node ID ....................: 0.0.3\n2024-09-20T12:52:25.601Z-238831 ⛶ Node's file lists folder ...: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:52:25.603Z-238831 ☕ Filtering files for 2021-05-07 and consensus node 0.0.3 from AWS S3\n2024-09-20T12:52:27.452Z-238831 ⛔ Requested file recordstreams/record0.0.3/sidecar/2021-05-07 missing from AWS S3. Exiting.\n2024-09-20T12:52:27.454Z-238196 ⚠ 2021-05-08 is missing or incomplete\n2024-09-20T12:52:27.456Z-238887 ⚑ Started ./create-single-file-list-by-day.sh (PID 238887) with the following configuration\n2024-09-20T12:52:27.457Z-238887 ⛶ Day (UTC) .......................: 2021-05-08\n2024-09-20T12:52:27.458Z-238887 ⛶ Node ID ....................: 0.0.3\n2024-09-20T12:52:27.458Z-238887 ⛶ Node's file lists folder ...: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:52:27.460Z-238887 ☕ Filtering files for 2021-05-08 and consensus node 0.0.3 from AWS S3\n2024-09-20T12:52:29.043Z-238887 ⛔ Requested file recordstreams/record0.0.3/sidecar/2021-05-08 missing from AWS S3. Exiting.\n2024-09-20T12:52:29.045Z-238196 ⚠ 2021-05-09 is missing or incomplete\n2024-09-20T12:52:29.047Z-238943 ⚑ Started ./create-single-file-list-by-day.sh (PID 238943) with the following configuration\n2024-09-20T12:52:29.048Z-238943 ⛶ Day (UTC) .......................: 2021-05-09\n2024-09-20T12:52:29.049Z-238943 ⛶ Node ID ....................: 0.0.3\n2024-09-20T12:52:29.050Z-238943 ⛶ Node's file lists folder ...: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:52:29.051Z-238943 ☕ Filtering files for 2021-05-09 and consensus node 0.0.3 from AWS S3\n2024-09-20T12:52:30.955Z-238943 ⛔ Requested file recordstreams/record0.0.3/sidecar/2021-05-09 missing from AWS S3. Exiting.\n2024-09-20T12:52:30.957Z-238196 ⚠ 2021-05-10 is missing or incomplete\n2024-09-20T12:52:30.960Z-238999 ⚑ Started ./create-single-file-list-by-day.sh (PID 238999) with the following configuration\n2024-09-20T12:52:30.961Z-238999 ⛶ Day (UTC) .......................: 2021-05-10\n2024-09-20T12:52:30.962Z-238999 ⛶ Node ID ....................: 0.0.3\n2024-09-20T12:52:30.962Z-238999 ⛶ Node's file lists folder ...: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:52:30.964Z-238999 ☕ Filtering files for 2021-05-10 and consensus node 0.0.3 from AWS S3\n2024-09-20T12:52:32.778Z-238999 ⛔ Requested file recordstreams/record0.0.3/sidecar/2021-05-10 missing from AWS S3. Exiting.\n2024-09-20T12:52:33.559Z-238196 ⚠ 2023-04-21 is missing or incomplete\n2024-09-20T12:52:33.561Z-239765 ⚑ Started ./create-single-file-list-by-day.sh (PID 239765) with the following configuration\n2024-09-20T12:52:33.562Z-239765 ⛶ Day (UTC) .......................: 2023-04-21\n2024-09-20T12:52:33.563Z-239765 ⛶ Node ID ....................: 0.0.3\n2024-09-20T12:52:33.563Z-239765 ⛶ Node's file lists folder ...: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:52:33.566Z-239765 ☕ Filtering files for 2023-04-21 and consensus node 0.0.3 from AWS S3\n2024-09-20T12:52:35.124Z-239765 ⛔ Requested file recordstreams/record0.0.3/sidecar/2023-04-21 missing from AWS S3. Exiting.\n2024-09-20T12:52:35.126Z-238196 ⚠ 2023-04-22 is missing or incomplete\n2024-09-20T12:52:35.129Z-239821 ⚑ Started ./create-single-file-list-by-day.sh (PID 239821) with the following configuration\n2024-09-20T12:52:35.129Z-239821 ⛶ Day (UTC) .......................: 2023-04-22\n2024-09-20T12:52:35.130Z-239821 ⛶ Node ID ....................: 0.0.3\n2024-09-20T12:52:35.131Z-239821 ⛶ Node's file lists folder ...: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:52:35.133Z-239821 ☕ Filtering files for 2023-04-22 and consensus node 0.0.3 from AWS S3\n2024-09-20T12:52:37.120Z-239821 ⛔ Requested file recordstreams/record0.0.3/sidecar/2023-04-22 missing from AWS S3. Exiting.\n2024-09-20T12:52:37.122Z-238196 ⚠ 2023-04-23 is missing or incomplete\n2024-09-20T12:52:37.125Z-239877 ⚑ Started ./create-single-file-list-by-day.sh (PID 239877) with the following configuration\n2024-09-20T12:52:37.126Z-239877 ⛶ Day (UTC) .......................: 2023-04-23\n2024-09-20T12:52:37.126Z-239877 ⛶ Node ID ....................: 0.0.3\n2024-09-20T12:52:37.127Z-239877 ⛶ Node's file lists folder ...: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:52:37.129Z-239877 ☕ Filtering files for 2023-04-23 and consensus node 0.0.3 from AWS S3\n2024-09-20T12:52:38.891Z-239877 ⛔ Requested file recordstreams/record0.0.3/sidecar/2023-04-23 missing from AWS S3. Exiting.\n2024-09-20T12:52:38.893Z-238196 ⚠ 2023-04-24 is missing or incomplete\n2024-09-20T12:52:38.895Z-239933 ⚑ Started ./create-single-file-list-by-day.sh (PID 239933) with the following configuration\n2024-09-20T12:52:38.896Z-239933 ⛶ Day (UTC) .......................: 2023-04-24\n2024-09-20T12:52:38.897Z-239933 ⛶ Node ID ....................: 0.0.3\n2024-09-20T12:52:38.898Z-239933 ⛶ Node's file lists folder ...: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:52:38.900Z-239933 ☕ Filtering files for 2023-04-24 and consensus node 0.0.3 from AWS S3\n2024-09-20T12:52:40.355Z-239933 ⛔ Requested file recordstreams/record0.0.3/sidecar/2023-04-24 missing from AWS S3. Exiting.\n2024-09-20T12:52:40.884Z-238196 ⚠ 2024-08-19 is missing or incomplete\n2024-09-20T12:52:40.887Z-240471 ⚑ Started ./create-single-file-list-by-day.sh (PID 240471) with the following configuration\n2024-09-20T12:52:40.887Z-240471 ⛶ Day (UTC) .......................: 2024-08-19\n2024-09-20T12:52:40.888Z-240471 ⛶ Node ID ....................: 0.0.3\n2024-09-20T12:52:40.889Z-240471 ⛶ Node's file lists folder ...: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:52:40.891Z-240471 ☕ Filtering files for 2024-08-19 and consensus node 0.0.3 from AWS S3\n2024-09-20T12:53:12.666Z-240471 ⚙ Creating metadata: /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-19.metadata\n2024-09-20T12:53:13.026Z-240471 ⚙ Extracting MD5 checksums for records to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-19.records.md5sum\n2024-09-20T12:53:13.398Z-240471 ⚙ Extracting MD5 checksums for signatures to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-19.signatures.md5sum\n2024-09-20T12:53:13.775Z-240471 ⚙ Extracting MD5 checksums for sidecars to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-19.sidecars.md5sum\n2024-09-20T12:53:14.168Z-240471 ⚙ Computing records total size to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-19.records.size\n2024-09-20T12:53:14.528Z-240471 ⚙ Computing signatures total size to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-19.signatures.size\n2024-09-20T12:53:14.868Z-240471 ⚙ Computing sidecars total size to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-19.sidecars.size\n2024-09-20T12:53:15.203Z-240471 🏁 Script ./create-single-file-list-by-day.sh (PID 240471) ended\n2024-09-20T12:53:15.204Z-238196 ⚠ 2024-08-20 is missing or incomplete\n2024-09-20T12:53:15.207Z-240609 ⚑ Started ./create-single-file-list-by-day.sh (PID 240609) with the following configuration\n2024-09-20T12:53:15.207Z-240609 ⛶ Day (UTC) .......................: 2024-08-20\n2024-09-20T12:53:15.208Z-240609 ⛶ Node ID ....................: 0.0.3\n2024-09-20T12:53:15.209Z-240609 ⛶ Node's file lists folder ...: /media/sf_Hedera/hedera-streams/lists/0.0.3\n2024-09-20T12:53:15.210Z-240609 ☕ Filtering files for 2024-08-20 and consensus node 0.0.3 from AWS S3\n2024-09-20T12:53:44.276Z-240609 ⚙ Creating metadata: /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-20.metadata\n2024-09-20T12:53:44.642Z-240609 ⚙ Extracting MD5 checksums for records to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-20.records.md5sum\n2024-09-20T12:53:45.023Z-240609 ⚙ Extracting MD5 checksums for signatures to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-20.signatures.md5sum\n2024-09-20T12:53:45.405Z-240609 ⚙ Extracting MD5 checksums for sidecars to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-20.sidecars.md5sum\n2024-09-20T12:53:45.841Z-240609 ⚙ Computing records total size to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-20.records.size\n2024-09-20T12:53:46.190Z-240609 ⚙ Computing signatures total size to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-20.signatures.size\n2024-09-20T12:53:46.534Z-240609 ⚙ Computing sidecars total size to /media/sf_Hedera/hedera-streams/lists/0.0.3/2024-08-20.sidecars.size\n2024-09-20T12:53:46.870Z-240609 🏁 Script ./create-single-file-list-by-day.sh (PID 240609) ended\n2024-09-20T12:53:46.907Z-238196 🏁 Script ./check-node-file-lists.sh (PID 238196) ended\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneurone%2Fhedera-raw-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneurone%2Fhedera-raw-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneurone%2Fhedera-raw-data/lists"}