{"id":14969830,"url":"https://github.com/loomly/s3_asset_deploy","last_synced_at":"2025-04-08T04:16:02.683Z","repository":{"id":43684851,"uuid":"335763776","full_name":"Loomly/s3_asset_deploy","owner":"Loomly","description":"Deploy \u0026 manage static assets on S3 with rolling deploys \u0026 rollbacks in mind.","archived":false,"fork":false,"pushed_at":"2024-09-23T19:48:36.000Z","size":55,"stargazers_count":65,"open_issues_count":3,"forks_count":3,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-10-30T04:28:39.805Z","etag":null,"topics":["aws","devops","rails","ruby","ruby-on-rails","s3"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Loomly.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-03T21:45:53.000Z","updated_at":"2024-09-23T19:47:46.000Z","dependencies_parsed_at":"2024-11-13T00:03:49.214Z","dependency_job_id":"55d7ef60-c3ff-4889-9541-cd08f5d6efa9","html_url":"https://github.com/Loomly/s3_asset_deploy","commit_stats":{"total_commits":46,"total_committers":3,"mean_commits":"15.333333333333334","dds":0.04347826086956519,"last_synced_commit":"40ca09407651562018622d49ed2a36995783f527"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Loomly%2Fs3_asset_deploy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Loomly%2Fs3_asset_deploy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Loomly%2Fs3_asset_deploy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Loomly%2Fs3_asset_deploy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Loomly","download_url":"https://codeload.github.com/Loomly/s3_asset_deploy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247773720,"owners_count":20993639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","devops","rails","ruby","ruby-on-rails","s3"],"created_at":"2024-09-24T13:42:27.787Z","updated_at":"2025-04-08T04:16:02.620Z","avatar_url":"https://github.com/Loomly.png","language":"Ruby","readme":"# S3AssetDeploy\n\n[![CircleCI](https://circleci.com/gh/Loomly/s3_asset_deploy.svg?style=shield)](https://circleci.com/gh/Loomly/s3_asset_deploy)\n[![Gem Version](https://badge.fury.io/rb/s3_asset_deploy.svg)](https://badge.fury.io/rb/s3_asset_deploy)\n\nDuring rolling deploys to our web instances, this is what we use at\n[Loomly](https://www.loomly.com) to safely deploy our web assets to S3 to be served via Cloudfront.\nThis gem is designed to upload and clean unneeded assets from S3 in a safe manner such that older\nversions or recently removed assets are kept on S3 during the rolling deploy process.\nIt also maintains a version limit and TTL (time-to-live) on assets to avoid deleting\nrecent and outdated versions (up to a limit) or those that have been recently removed.\n\n## Background\n\nAt the very beginning, we were serving our assets from our webservers. This isn't ideal for many reasons, but one big one is that it's problematic during rolling deploys where you temporarily have some web servers with new assets and some with old assets during the deploy. When round-robbining requests to instances behind a load balancer, this can result in requests for assets hitting web servers that don't have the asset being requested (either the new or the old depending on what web server and what's being requested). One way to fix this problem is to serve your assets from a CDN and keep both old and new versions of assets available on the CDN during the deploy process. So we decided to serve our assets from Cloudfront, backed by S3. In order to upload our assets to S3 during our deploy process, we started using [`asset_sync`](https://github.com/AssetSync/asset_sync). `asset_sync` served us well for quite some time, but our needs started to diverge a bit. Namely, `asset_sync`:\n\n- Depends on the [`fog`](https://github.com/fog/fog) gem which was an extra dependency we didn't need since we already had the [`aws-sdk-s3`](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3.html) gem as a dependency.\n- Uses a global configuration, which made it difficult to deploy to different S3 buckets depending on the environment (development, staging, production, etc.).\n- Didn't have a way to remove or retire outdated or old assets from storage (in this case S3).\n\nWe took inspiration from `asset_sync` and ended up writing our own library inside our Rails app. We figured this could be useful to others, so we then moved it to an open source gem. While Rails is a \"first-class citizen\", this gem can be used with other frameworks by writing your own `S3AssetDeploy::LocalAssetCollector`. See the `Usage` section below for more details.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem \"s3_asset_deploy\"\n```\n\nAnd then execute:\n\n    $ bundle install\n\nOr install it yourself as:\n\n    $ gem install s3_asset_deploy\n\n## Usage\n\nBefore using `S3AssetDeploy` you want to make sure to compile your assets. Assets must also be compiled using [fingerprinting](https://guides.rubyonrails.org/asset_pipeline.html#what-is-fingerprinting-and-why-should-i-care-questionmark) for things to work correctly. By default, `S3AssetDeploy` works with Rails and will find your locally compiled assets after running `rake assets:precompile`. Once you've compiled your assets, you can deploy them with:\n\n\n```ruby\nmanager = S3AssetDeploy::Manager.new(\"my-s3-bucket\")\nmanager.deploy do\n  # Perform deploy to web instances in this block\nend\n```\n\n`S3AssetDeploy::Manager#deploy` will perform the following steps:\n- Upload your assets to the S3 bucket you specify\n- Yield to the block\n- Clean old versions assets or removed assets\n\nSince it's yielding to the block after uploading, but before cleaning, the block is an ideal place to perform a deploy, especially if it's a rolling deploy across multiple servers. If you want to perform an upload or a clean without using `#deploy`, you can call `#upload` or `#clean` directly. For more configuration options, see below.\n\n### Initializing [`S3AssetDeploy::Manager`](https://github.com/Loomly/s3_asset_deploy/blob/main/lib/s3_asset_deploy/manager.rb)\nYou'll need to initialize `S3AssetDeploy::Manager` with an S3 bucket name and **optionally**:\n\n- **s3_client_options** (Hash) -\u003e A hash that is passed directly to [`Aws::S3::Client#initialize`](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#initialize-instance_method) to configure the S3 client. By default the region is set to `us-east-1`.\n- **logger** (Logger) -\u003e A custom logger. By default things are logged to `STDOUT`.\n- **local_asset_collector** (S3AssetDeploy::LocalAssetCollector) -\u003e A custom instance of `S3AssetDeploy::LocalAssetCollector`. This allows you to customize how locally compiled assets are collected.\n- **upload_options** (Hash) -\u003e A hash consisting of options that are passed directly to [`Aws::S3::Client#put_object`](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_object-instance_method) when each asset is uploaded. By default `cache_control` is set to `public, max-age=31536000`.\n- **remove_fingerprint** (Lambda) -\u003e Lambda for overriding how fingerprints are removed from asset paths. Fingerprints need to be removed during the cleaning process in order to group versions of the same file. If no Lambda is provided, [`S3AssetDeploy::AssetHelper.remove_fingerprint`](https://github.com/Loomly/s3_asset_deploy/blob/main/lib/s3_asset_deploy/asset_helper.rb#L8) is used by default.\n\nHere's an example:\n\n```ruby\nmanager = S3AssetDeploy::Manager.new(\n  \"mybucket\",\n  s3_client_options: { region: \"us-west-1\", profile: \"my-aws-profile\" },\n  logger: Logger.new(STDOUT),\n  remove_fingerprint: -\u003e(path) { path.gsub(\"-myfingerprint\", \"\") }\n)\n```\n\n### Deploying Assets\nOnce you have an instance of `S3AssetDeploy::Manager`, you can deploy your precompiled assets with `S3AssetDeploy::Manager#deploy`:\n\n```ruby\nmanager.deploy(version_limit: 2, version_ttl: 3600, removed_ttl: 172800) do\n  # Perform deploy to web instances in this block\nend\n```\n\nThis will upload new assets and perform a clean, which deletes removed assets and old versions from your bucket after the block is executed.\nWith the arguments used above, the clean process will keep the latest version on S3, two of the most recent older versions (`version_limit`), and any versions created within the last hour (`version_ttl`).\nIf you there are assets that are in your S3 bucket but no longer included in your locally compiled bundle, they will be deleted from S3 using the `removed_ttl` (after two days in the case above). This process uses [S3 object tagging](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_object_tagging-instance_method) to track `removed_at` timestamps. Here are a list of all the options you can pass to `#deploy`:\n\n- **version_limit** (Integer) -\u003e Max number of older versions of an asset to keep around. Note that this limit does **not** include the current version. Therefore, setting this to 0 will keep the current version and delete any older versions. Default is `2`.\n- **version_ttl** (Integer) -\u003e Number of seconds to keep newly uploaded versions before deleting according to `version_limit`. If an older version is still within the `version_ttl`, it will be kept on S3 even if the total number of older versions is beyond `version_limit`. Default is `3600`.\n- **removed_ttl** (Integer) -\u003e Number of seconds to keep assets on S3 that have been removed from your compiled set of assets. If the age of a removed asset expires according to `removed_ttl`, it will be deleted on the next deploy. Default is `172800`.\n- **clean** (Boolean) -\u003e Skip the clean process during a deploy. Default is `true`.\n- **dry_run** (Boolean) -\u003e Run deploy in read-only mode. This is helpful for debugging purposes and seeing plan of what would happen without performing any writes or deletes. Default is `false`.\n\n`S3AssetDeploy::Manager#deploy` performs its work by delegating to `S3AssetDeploy#upload` and `S3AssetDeploy#clean`, which you can call yourself if you need some more control.\n\n\n```ruby\n# Upload new assets\nmanager.upload\n\n# Delete old versions and removed assets from S3\nmanager.clean\n```\n\n`S3AssetDeploy::Manager#deploy` and `S3AssetDeploy::Manager#clean` both accept `dry_run` as a keyword argument.\n`S3AssetDeploy::Manager#clean` also accepts `version_limit`, `version_ttl`, and `removed_ttl` just like `S3AssetDeploy::Manager#deploy`.\n\n### Practical Example of Usage\nThere are many ways to use and invoke `S3AssetDeploy`. How you use it will depend on your deploy process and pipeline. At Loomly, we have some rake tasks that are invoked from our CI/CD pipeline to perform deploys.\nHere's a basic example of how we use `S3AssetDeploy`:\n\n\n```ruby\n# lib/tasks/deploy.rake\n\nrequire \"s3_asset_deploy\"\n\nnamespace :deploy do\n  task precompile: :environment do\n    puts \"Precompiling assets...\"\n    sh(\"RAILS_ENV=production SECRET_KEY_BASE='secret key' bundle exec rake assets:precompile\")\n  end\n\n  task clobber_assets: :environment do\n    puts \"Clobbering assets...\"\n    sh(\"RAILS_ENV=production SECRET_KEY_BASE='secret key' bundle exec rake assets:clobber\")\n  end\n\n  task :production do\n    Rake::Task[\"deploy:precompile\"].invoke\n\n    manager = S3AssetDeploy::Manager.new(\"my-s3-bucket\")\n    manager.deploy do\n      # Perform deploy to web instances in this block.\n      # How you do this will depend on where you are hosting your application and what tools you use to deploy.\n    end\n\n    Rake::Task[\"deploy:clobber_assets\"].invoke # \u003c-- If you are running on CI where the precompiled assets directory is ephemeral, this may be unnecessary\n  end\nend\n```\n\nGiven the example above, we can perform a deploy by running `bundle exec rake deploy:production`. This task will:\n1. Precompile assets\n2. Upload any new assets to S3 using `S3AssetDeploy`\n3. Deploy a new version of our application\n4. Clean any outdated or unused assets from S3 using `S3AssetDeploy`\n\n## Customizing local asset collection\nBy default, `S3AssetDeploy::Manager` will use [`S3AssetDeploy::RailsLocalAssetCollector`](https://github.com/Loomly/s3_asset_deploy/blob/main/lib/s3_asset_deploy/rails_local_asset_collector.rb) to collect locally compiled assets. This will use the `Sprockets::Manifest` and `Webpacker`/`Shakapacker` config (if either are installed) to locate the compiled assets. `S3AssetDeploy::RailsLocalAssetCollector` inherits from the [`S3AssetDeploy::LocalAssetCollector`](https://github.com/Loomly/s3_asset_deploy/blob/main/lib/s3_asset_deploy/local_asset_collector.rb) base class. You can completely customize how your local assets are collected for deploys by creating your own class that inherits from `S3AssetDeploy::LocalAssetCollector` and passing it into the manager. You'll want override `S3AssetDeploy::LocalAssetCollector#assets` in your custom collector such that it returns an array of `S3AssetDeploy::LocalAsset` instances. Here's a basic example:\n\n\n```ruby\nclass MyCustomLocalAssetCollector \u003c S3AssetDeploy::LocalAssetCollector\n  def assets\n    # Override this method to return an array of your locally compiled assets\n    # as instances of S3AssetDeploy::LocalAsset\n    [S3AssetDeploy::LocalAsset.new(\"path-to-my-asset.jpg\")]\n  end\nend\n\nmanager = S3AssetDeploy::Manager.new(\n  \"mybucket\",\n  local_asset_collector: MyCustomLocalAssetCollector.new\n)\n```\n\n## Dry run\nAs mentioned above, you can run operations in \"dry\" mode by passing `dry_run: true`.\nThis will skip any write or delete operations and only perform read opeartions with log output.\nThis is helpful for debugging or planning purposes.\n\n```ruby\n\u003e manager = S3AssetDeploy::Manager.new(\"my-s3-bucket\")\n\u003e manager.deploy(dry_run: true)\n\nI, [2021-02-17T16:12:23.703677 #65335]  INFO -- : S3AssetDeploy::Manager: Cleaning assets from test-bucket S3 bucket. Dry run: true\nI, [2021-02-17T16:12:23.703677 #65335]  INFO -- : S3AssetDeploy::Manager: Determining how long ago assets/file-2-34567.jpg was removed - removed on 2021-02-15 23:12:22 UTC (172801.703677 seconds ago)\nI, [2021-02-17T16:12:23.703677 #65335]  INFO -- : S3AssetDeploy::Manager: Determining how long ago assets/file-3-9876666.jpg was removed - removed on 2021-02-15 23:12:24 UTC (172799.703677 seconds ago)\n```\n\n## AWS IAM Permissions\n`S3AsetDeploy` requires the following AWS IAM permissions to list, put, and delete objects in your S3 Bucket:\n\n```json\n\"Statement\": [\n  {\n    \"Action\": [\n      \"s3:ListBucket\",\n      \"s3:PutObject*\",\n      \"s3:DeleteObject\"\n    ],\n    \"Effect\": \"Allow\",\n    \"Resource\": [\n      \"arn:aws:s3:::#{YOUR_BUCKET}\",\n      \"arn:aws:s3:::#{YOUR_BUCKET}/*\"\n    ]\n  }\n]\n```\n\n## Configuration with Cloudfront\n\n### Restricting Access with Origin Access Identity\nIf you want to setup Cloudfront to serve your assets, you can [restrict access to the bucket by using an Origin Access Identity](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html#private-content-granting-permissions-to-oai) so that only Cloudfront can access the objects in your bucket.\n\nIf you do this, your bucket policy will look something like this:\n\n```json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"AllowGetObject\",\n            \"Effect\": \"Allow\",\n            \"Principal\": {\n                \"AWS\": [\n                  \"arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity #{YOUR_OAI_ID}\"\n                ]\n            },\n            \"Action\": \"s3:GetObject\",\n            \"Resource\": \"arn:aws:s3:::#{YOUR_BUCKET}/*\"\n        },\n        {\n            \"Sid\": \"DenyGetObject\",\n            \"Effect\": \"Deny\",\n            \"Principal\": {\n                \"AWS\": \"arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity #{YOUR_OAI_ID}\"\n            },\n            \"Action\": \"s3:GetObject\",\n            \"Resource\": \"arn:aws:s3:::#{YOUR_BUCKET}/s3-asset-deploy-removal-manifest.json\"\n        }\n    ]\n}\n```\n\nThis policy allows Cloudfront to access everything **except** the removal manifest uploaded and maintained by this gem since this manifest does not need to be served to clients.\n\n### CORS\nYour CORS configuration on the bucket might look something like this:\n\n```json\n[\n    {\n        \"AllowedHeaders\": [\n            \"Authorization\"\n        ],\n        \"AllowedMethods\": [\n            \"GET\",\n            \"HEAD\"\n        ],\n        \"AllowedOrigins\": [\n            \"https://*.#{YOUR_SITE}.com\"\n        ],\n        \"ExposeHeaders\": [],\n        \"MaxAgeSeconds\": 3000\n    }\n]\n```\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `bundle exec rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/Loomly/s3_asset_deploy. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/Loomly/s3_asset_deploy/blob/main/CODE_OF_CONDUCT.md).\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n\n## Code of Conduct\n\nEveryone interacting in the S3AssetDeploy project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/Loomly/s3_asset_deploy/blob/main/CODE_OF_CONDUCT.md).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floomly%2Fs3_asset_deploy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Floomly%2Fs3_asset_deploy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floomly%2Fs3_asset_deploy/lists"}