{"id":19294574,"url":"https://github.com/web3-storage/backup-infra","last_synced_at":"2025-04-22T08:30:33.380Z","repository":{"id":66220436,"uuid":"593313729","full_name":"web3-storage/backup-infra","owner":"web3-storage","description":"AWS Infrastructure managed by Terraform for running Backup Tool","archived":true,"fork":false,"pushed_at":"2023-05-15T15:56:16.000Z","size":45,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-02-24T00:28:04.122Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/web3-storage.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-25T18:13:29.000Z","updated_at":"2024-04-18T16:17:06.000Z","dependencies_parsed_at":"2024-11-09T22:38:52.588Z","dependency_job_id":"18fc0107-21d6-48b1-a682-8fe957dbb1df","html_url":"https://github.com/web3-storage/backup-infra","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fbackup-infra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fbackup-infra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fbackup-infra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fbackup-infra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/web3-storage","download_url":"https://codeload.github.com/web3-storage/backup-infra/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250205971,"owners_count":21392157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T22:38:44.676Z","updated_at":"2025-04-22T08:30:33.375Z","avatar_url":"https://github.com/web3-storage.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Backup Infra\n\nThis project contains code to provision Cloud infrastructure for [Backup tool](https://github.com/web3-storage/backup). \n\n## Pre Requisites\n\n- [Configure AWS Access](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html)\n- Install [Terraspace](https://terraspace.cloud/docs/install/) and [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)\n\n## Stacks\n\n- **artifactory**: ECR repository for docker image\n- **ecs-cluster**: \n    - Cluster (`main.tf`)\n    - One Task Definition, and one service, for each source data file URL (`main.tf`)\n    - Networking (`network.tf`)\n    - Access permissionning (`auth.tf`)\n- **logs**: Log Group. Increase `retention_in_days` if you want to keep logs for a longer period of time. `prevent_destroy=true` makes sure this log group can't be accidentaly deleted before retention period\n\n## Deploy\n\n``` sh\nAWS_REGION=us-west-2 TS_ENV=prod terraspace up artifactory\nAWS_REGION=us-west-2 TS_ENV=prod terraspace up log\n```\n\nMake a first **image push to registry**. Also, remember to feed source CIDs bucket list (See next topic). `terraspace up ecs-cluster` won't work before doing so\n\nMerge a PR to main in https://github.com/web3-storage/backup and an updated docker image is built and pushed to a private ECR repository with the tag format `${iso-date}-${short-commit-sha}`. \n\nGrab that new tag name from the workflow run (or check the ECR listing in aws console) and update `backup_image_version` in `terraspace/app/stacks/ecs-cluster/tfvars/base.tfvars`, then run `terraspace up ecs-cluster`\n\n``` sh\nAWS_REGION=us-west-2 TS_ENV=prod terraspace up ecs-cluster\n```\n\n## Add/Remove source data files URLs\n\nUpdate `data_urls_map` (`terraspace/app/stacks/ecs-cluster/tfvars/base.tfvars`) by adding a new file, or removing an existing, to/from set. Then, run:\n\n``` sh\nAWS_REGION=us-west-2 TS_ENV=prod terraspace up ecs-cluster\n```\n\n## Stop\n\n``` sh\nAWS_REGION=us-west-2 TS_ENV=prod terraspace down ecs-cluster\n```\n\nCloudWatch logs are preserved for further analysis\n\n\n## Workflows\n\nCurrently, there isn't workflows configured for this project. Commands need to be executed from developer machine\n\n## Troubleshooting\n\n### Error acquiring the state lock\n\nDon't `ctrl+c` a terraspace command. You'll have claimed the lock, and it doesn't gracefully handle a kill signal, and now you've snapped the key off line lock and subsequent commands will fail with a:\n\n```\nAWS_REGION=us-west-2 TS_ENV=prod terraspace all plan\n...\nterraspace plan artifactory:  │ Error: Error acquiring the state lock\nterraspace plan log:  │ Error: Error acquiring the state lock\n```\n\nTo fix it, go to the dynamo db explorer, find the ID of the stack that are failing \n\n![screenshot showing lock id in dynamo db](https://bafybeigx5w6kctovp6n4jemzgdht7iwsbdqceunz7qkwmfne6rt5nk4kim.ipfs.w3s.link/Screenshot%202023-03-30%20at%2014.16.10.png)\n\n\nand run `terraspace force_unlock \u003cstack\u003e \u003cid\u003e` to free the lock. You may have to do this for multple stack'n'lock combos.\n\n```\n# e.g.\nAWS_REGION=us-west-2 TS_ENV=prod terraspace force_unlock artifactory 689fa081-b637...\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweb3-storage%2Fbackup-infra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweb3-storage%2Fbackup-infra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweb3-storage%2Fbackup-infra/lists"}