{"id":22312222,"url":"https://github.com/ajejoseph22/redesigned-guide","last_synced_at":"2026-04-10T17:07:48.105Z","repository":{"id":60234952,"uuid":"496815749","full_name":"ajejoseph22/redesigned-guide","owner":"ajejoseph22","description":"POC - AWS-based solution for non-disruptive bucket migration [WIP]","archived":false,"fork":false,"pushed_at":"2022-10-13T20:38:30.000Z","size":81,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-30T23:41:32.901Z","etag":null,"topics":["aws","docker","golang","postgresql","s3","sqs"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ajejoseph22.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-27T00:45:51.000Z","updated_at":"2023-02-11T15:15:17.000Z","dependencies_parsed_at":"2023-01-19T22:45:43.554Z","dependency_job_id":null,"html_url":"https://github.com/ajejoseph22/redesigned-guide","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajejoseph22%2Fredesigned-guide","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajejoseph22%2Fredesigned-guide/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajejoseph22%2Fredesigned-guide/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajejoseph22%2Fredesigned-guide/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ajejoseph22","download_url":"https://codeload.github.com/ajejoseph22/redesigned-guide/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245576522,"owners_count":20638123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","docker","golang","postgresql","s3","sqs"],"created_at":"2024-12-03T21:35:46.397Z","updated_at":"2025-12-30T23:24:52.206Z","avatar_url":"https://github.com/ajejoseph22.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Installation\nThis assumes that you have: \n- a source and destination bucket created on s3\n- a remote postgres DB\n- executed [the SQL script](https://drive.google.com/file/d/1DC46CHhtYLZ5lm3V174aGz9zf5_4mq5g/view?usp=sharing) on the remote db and run the \n[seeding script](https://drive.google.com/file/d/1VLlhFh2xDQJJ0pPH-usOaBlj5Vkxw_hv/view?usp=sharing) to insert data into the source bucket \n- created AWS user account credentials with s3 (read, write, copy) and SQS (read, write, delete) permissions\n- created an SQS standard queue\n\n\n## Steps\n* clone the repository.\n* `cd` into the `feeder` directory. create a `.env` file using the `env.sample` file as a reference.\n* `cd` into the `consumer` directory. create a `.env` file using the `env.sample` file as a reference.\n* cd to the root directory and run `docker-compose up`\n\n[PS: docker and docker-compose must be installed on machine]\n\n\n## Comments and considerations\nFor an ideal solution in a production environment, the feeder service would be running on a serverless \ncontainerization service like AWS fargate (Docker compose is used here for easy demonstration/setup locally). \nThis enables it to run continuously and auto-scale without us having to\nthink of servers. \nThe feeder service would feed the keys to the SQS (standard) queue which stores them until they are consumed by a worker\n(consumer service). \nThe consumer service would **ideally** be deployed to a container orchestration service like AWS ECS, with an \nautoscaling policy setup that allows several instances (as many as needed) of the container to run and process the\nmessages from the queue, while scaling up and down as needed. The idea here is to have several workers processing the\nmessages so the overall execution is much faster. Each worker updates the path of the image in the DB on every successful\ncopy to s3 - production flow is not disrupted.\nIn the event of a failure to copy an object, the key (message) doesn't get deleted from the SQS queue, and it would\nautomatically get processed/consumed again (either by the same worker or another worker) after the visibility timeout \nhas expired.   \n\n\n## Architecture\n![](./architecture.jpg)\n\n\n## Stack\n* Go\n* Docker\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fajejoseph22%2Fredesigned-guide","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fajejoseph22%2Fredesigned-guide","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fajejoseph22%2Fredesigned-guide/lists"}