{"id":19294567,"url":"https://github.com/web3-storage/backup-progress","last_synced_at":"2025-04-22T08:30:33.299Z","repository":{"id":150298437,"uuid":"621281015","full_name":"web3-storage/backup-progress","owner":"web3-storage","description":"Show the progress of the backup job by tailing the cloudwatch logs","archived":true,"fork":false,"pushed_at":"2023-03-30T10:59:37.000Z","size":28,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-02-24T00:28:04.774Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/web3-storage.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-30T10:55:21.000Z","updated_at":"2024-04-18T16:17:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"d4f4e2b9-3ce2-4af5-8325-a0f1aec09f94","html_url":"https://github.com/web3-storage/backup-progress","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fbackup-progress","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fbackup-progress/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fbackup-progress/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fbackup-progress/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/web3-storage","download_url":"https://codeload.github.com/web3-storage/backup-progress/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250205971,"owners_count":21392157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T22:38:44.214Z","updated_at":"2025-04-22T08:30:33.293Z","avatar_url":"https://github.com/web3-storage.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# backup progress\n\nShow the progress of the [backup] job by tailing the cloudwatch logs\n\n```\n❯ tail -n +1 -f backup-log-full.log | node progress.js\n ██████████████████░░░░░░░░░░░░░░░░░░░░░░ nft-0.json | 46% | 537,994 of 1,000,000 left\n ████████████████████████░░░░░░░░░░░░░░░░ nft-1.json | 61% | 389,359 of 1,000,000 left\n ████████████████████████████████████████ nft-2.json | 99% | 5,035 of 1,000,000 left\n ████████████████████████████████░░░░░░░░ nft-3.json | 79% | 203,383 of 1,000,000 left\n ████████████████████████████░░░░░░░░░░░░ nft-4.json | 69% | 300,458 of 1,000,000 left\n ████████████████████████████████████████ nft-5.json | 99% | 3,425 of 1,000,000 left\n ███████████████████████░░░░░░░░░░░░░░░░░ nft-6.json | 58% | 413,764 of 1,000,000 left\n ████████████████████████████████████████ nft-7.json | 99% | 4,462 of 1,000,000 left\n ██████████████████████████████░░░░░░░░░░ nft-8.json | 75% | 246,987 of 1,000,000 left\n ████████████████████████████████████████ nft-9.json | 99% | 4,529 of 1,000,000 left\n ████████████████████████████████████████ nft-10.json | 99% | 8,039 of 1,000,000 left\n ████████████████████████████████████████ nft-11.json | 99% | 6,205 of 1,000,000 left\n ████████████████████████████████████████ nft-12.json | 99% | 1,460 of 1,000,000 left\n ████████████████████████████████████████ nft-13.json | 99% | 2,841 of 1,000,000 left\n ████████████████████████████████████████ nft-14.json | 99% | 4,457 of 1,000,000 left\n ████████████████████████████████████████ nft-15.json | 99% | 4,746 of 1,000,000 left\n ████████████████████████████████████████ nft-16.json | 99% | 5,340 of 1,000,000 left\n ████████████████████████████████████████ nft-17.json | 99% | 8,138 of 1,000,000 left\n ████████████████████████░░░░░░░░░░░░░░░░ nft-18.json | 59% | 401,216 of 1,000,000 left\n ███████████████████████████████████████░ nft-19.json | 97% | 22,078 of 1,000,000 left\n ████████████████████████████████████████ nft-20.json | 99% | 4,593 of 1,000,000 left\n ████████████████████████████████████████ nft-21.json | 99% | 3,960 of 1,000,000 left\n ███████████████████████████░░░░░░░░░░░░░ nft-22.json | 66% | 336,298 of 1,000,000 left\n ████████████████████████████████████████ nft-23.json | 99% | 5,865 of 1,000,000 left\n ████████████████████████████████████████ nft-24.json | 99% | 5,492 of 1,000,000 left\n ██████████████████████████░░░░░░░░░░░░░░ nft-25.json | 64% | 358,388 of 1,000,000 left\n ████████████████████████████████████████ nft-26.json | 99% | 3,114 of 1,000,000 left\n ██████████████████████████████░░░░░░░░░░ nft-27.json | 75% | 245,303 of 1,000,000 left\n ████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░ nft-28.json | 30% | 694,551 of 1,000,000 left\n ████████████████████████████████████████ nft-29.json | 99% | 5,593 of 1,000,000 left\n ██████████████████████░░░░░░░░░░░░░░░░░░ nft-30.json | 55% | 87,707 of 196,294 left\n ██░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ web3-0.json | 3% | 960,808 of 1,000,000 left\n ███████████████████░░░░░░░░░░░░░░░░░░░░░ web3-1.json | 46% | 533,830 of 1,000,000 left\n █████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░ web3-2.json | 32% | 675,264 of 1,000,000 left\n ██████████████████████████░░░░░░░░░░░░░░ web3-3.json | 64% | 354,639 of 1,000,000 left\n ███████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ web3-4.json | 17% | 474,317 of 576,770 left\n ████████████████████████████████░░░░░░░░ total       | 78% | 7,323,638 of 34,773,064 left\n```\n\n## Getting started\n\nYou need:\n\n- macos or linux (or a shell that can run `tail`)\n- aws cli installed and configured with access to the aws account that backup is runnning in\n- node v18\n- `pnpm`\n\nRun `pnpm i` to install the deps\n\n### Tailing the logs\n\nRun `node tail.js` to print out the command you can copy and paste to run that will tail the cloudwatch logs for the backup process. e.g\n\n```bash\n$ node tail.js\n## Run this (AWS_PROFILE in env as needed)\naws logs tail prod-backup-ipfs-cluster --since 2023-02-01T12:00:00.000Z --region us-west-2 --format short --filter-pattern '{ $.status = \"ok\" }' --follow \u003e\u003e backup-log-full.log\n```\n\nThis will create `backup-log-full.log` in the root of the project, and may take a few hours to fetch ~3GB of logs on the first run.\n\nThe command is provided for you to run as you see fit, as you are the boss here, we're just trying to make it easier. It's pretty handy to keep the log tailing serperate from the progress indicating.\n\n### Resuming the tail\n\nYou will at some point want to close your laptop or your network will fail, or aws will just give up. No bother! You can pick up where you left off.\n\nRun `node tail.js` again, and it will read `backup-log-full.log`, pickout the last timestamp and provide you with an updated version of that command you can run with the `--since` value set so you start streaming from where you left off.\n\nNote that the `\u003e\u003e` in the command is important as you want to append the subsequent logs on to the end of the file rather than overwriting it.\n\n### Showing progress\n\nVisualise the `backup-log-full.log` by piping it in to `progress.js`\n\n```bash\ntail -n +1 -f backup-log-full.log | node progress.js\n```\n\nHere we tell `tail -n +1` asks tail to start from the first line of the file, and `-f` has it follow along, outputting new lines as they are added. This means we can start showing progress immediately, and continue as our local copy of the log catches up to the current time.\n\n## Notes\n\nAt time of writing (2023-03-29) the following batches have completed. These show up as 99% in the progress indicator when i run it, so it's likely that my local copy of the log has dropped some log lines along the way.\n\n```csv\n@timestamp,@message,file\n2023-03-03 15:57:26.849,2023-03-03T15:57:26.848Z backup:nft-2.json backup complete 🎉,nft-02.json\n2023-02-17 06:53:33.598,2023-02-17T06:53:33.598Z backup:nft-5.json backup complete 🎉,nft-05.json\n2023-02-18 02:58:31.295,2023-02-18T02:58:31.295Z backup:nft-7.json backup complete 🎉,nft-07.json\n2023-02-16 05:48:44.567,2023-02-16T05:48:44.567Z backup:nft-9.json backup complete 🎉,nft-09.json\n2023-03-05 16:34:02.234,2023-03-05T16:34:02.234Z backup:nft-11.json backup complete 🎉,nft-11.json\n2023-02-20 19:43:27.253,2023-02-20T19:43:27.253Z backup:nft-12.json backup complete 🎉,nft-12.json\n2023-02-26 09:31:34.912,2023-02-26T09:31:34.912Z backup:nft-13.json backup complete 🎉,nft-13.json\n2023-03-02 20:51:45.711,2023-03-02T20:51:45.711Z backup:nft-13.json backup complete 🎉,nft-13.json\n2023-03-12 01:38:20.245,2023-03-12T01:38:20.245Z backup:nft-14.json backup complete 🎉,nft-14.json\n2023-03-05 07:18:52.838,2023-03-05T07:18:52.838Z backup:nft-15.json backup complete 🎉,nft-15.json\n2023-03-04 10:33:40.773,2023-03-04T10:33:40.773Z backup:nft-16.json backup complete 🎉,nft-16.json\n2023-03-03 17:15:20.788,2023-03-03T17:15:20.787Z backup:nft-17.json backup complete 🎉,nft-17.json\n2023-03-03 06:28:01.071,2023-03-03T06:28:01.071Z backup:nft-20.json backup complete 🎉,nft-20.json\n2023-03-11 10:04:48.225,2023-03-11T10:04:48.222Z backup:nft-21.json backup complete 🎉,nft-21.json\n2023-03-04 23:52:48.021,2023-03-04T23:52:48.021Z backup:nft-23.json backup complete 🎉,nft-23.json\n2023-03-06 01:34:56.563,2023-03-06T01:34:56.562Z backup:nft-24.json backup complete 🎉,nft-24.json\n2023-02-19 07:14:27.854,2023-02-19T07:14:27.854Z backup:nft-26.json backup complete 🎉,nft-26.json\n2023-03-04 00:30:03.750,2023-03-04T00:30:03.750Z backup:nft-29.json backup complete 🎉,nft-29.json\n```\n\nPer the following query across the last 2 months\n\n```\nfields @timestamp, @message\n| filter @message like \"backup complete\"\n| sort @timestamp desc\n```\n\nhttps://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:logs-insights$3FqueryDetail$3D~(end~'2023-03-30T08*3a20*3a41.000Z~start~'2023-02-01T09*3a08*3a42.000Z~timeType~'ABSOLUTE~tz~'Local~editorString~'fields*20*40timestamp*2c*20*40message*0a*7c*20filter*20*40message*20like*20*22backup*20complete*22*0a*7c*20sort*20*40timestamp*20desc~queryId~'78b3f82c-5b6c-4914-9105-8131076af8a5~source~(~'prod-backup-ipfs-cluster))$26tab$3Dlogs\n\n\n\n[backup]: https://github.com/web3-storage/backup\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweb3-storage%2Fbackup-progress","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweb3-storage%2Fbackup-progress","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweb3-storage%2Fbackup-progress/lists"}