{"id":22608802,"url":"https://github.com/atombrenner/aws-image-optimizer","last_synced_at":"2025-04-11T06:13:36.373Z","repository":{"id":91519443,"uuid":"571978031","full_name":"atombrenner/aws-image-optimizer","owner":"atombrenner","description":"Serverless image optimization (avif, webp and jpeg) with focus point cropping.","archived":false,"fork":false,"pushed_at":"2023-12-28T10:18:00.000Z","size":1031,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-11T06:13:29.481Z","etag":null,"topics":["aws","cropping","image","image-processing","serverless","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/atombrenner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-11-29T09:50:40.000Z","updated_at":"2024-11-07T23:46:05.000Z","dependencies_parsed_at":"2023-12-08T16:28:33.000Z","dependency_job_id":"7907b906-48f7-4562-994f-1bee114d780b","html_url":"https://github.com/atombrenner/aws-image-optimizer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atombrenner%2Faws-image-optimizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atombrenner%2Faws-image-optimizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atombrenner%2Faws-image-optimizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atombrenner%2Faws-image-optimizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/atombrenner","download_url":"https://codeload.github.com/atombrenner/aws-image-optimizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248351393,"owners_count":21089270,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cropping","image","image-processing","serverless","typescript"],"created_at":"2024-12-08T15:09:26.350Z","updated_at":"2025-04-11T06:13:36.350Z","avatar_url":"https://github.com/atombrenner.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS Image Optimizer\n\n- can resize and crop images to arbitrary aspect ratios on the fly\n- converts images to modern image formats `webp` and `avif` to reduce image size\n- works well with [focus-point based cropping](https://github.com/atombrenner/focus-crop-react)\n- secure, fast and cheap implementation with AWS Cloudfront, S3 and Lambda\n- completely serverless and scalable\n- easy to integrate into existing projects that already store images in S3 buckets\n\n## Architecture\n\n![System Overview](/doc/overview.drawio.svg)\n\nThe heart of this solution is leveraging the ability of a CloudFront Origin Group\nto first look up an image on an S3 bucket, and if not found retry the request\nwith a lambda function. The lambda function will read the original image,\ntransform it and then save and return the optimized image. That ensures that\nevery image is transformed only once, and all successive requests will be\neither served from the CloudFront cache or the S3 Bucket (like a secondary cache).\nLet's go through the details:\n\n1. An image is requested.\n2. Optional: the URL signature is verified by a CloudFront Function.\n   If invalid the request is rejected with a 400 status code. This protects\n   against Denial-of-Wallet attacks.\n3. CloudFront will forward the verified request to an Origin Group.\n   An Origin Group acts like a normal origin but is composed of two origins.\n   One is the primary origin. If this returns an error status code the\n   request is retried with the secondary (failover) origin.\n4. The request is forwarded to the S3 Bucket that stores optimized images.\n   If found it will be returned and the request is complete.\n5. If the S3 Bucket returns a 403 or 404 CloudFront will forward the request\n   to our Image Optimizer Lambda which has a Lambda Function Url, no\n   extra API Gateway or ALB is necessary. Cloudfront will add an\n   X-Security-Token header to each request.\n6. The Lambda Function first verifies the X-Security-Token header to prevent\n   direct access (from bots or attackers). Then it reads the original image\n   from an S3 bucket and optimizes it by using sharp and libvips.\n7. The optimized image is stored in the Optimized Images S3 Bucket for future\n   requests. It is also returned directly to Cloudfront.\n   The request is now complete.\n\nTechnical constraints: As we are limited by how an S3 bucket can act as a\nCloudFront origin, we can use URL parameters but have to encode all parameters into\nthe path. Which is easy, as each parameter just maps to a path segment.\nExample: `https://example.com/some/IMAGEID/webp/100x200/fp=200,80/paramA=1/paramB=2`\n\nThis solution is very similar to what is written in this\n[AWS blog post](https://aws.amazon.com/blogs/networking-and-content-delivery/image-optimization-using-amazon-cloudfront-and-aws-lambda/)\nbut was implemented without knowledge of it.\n\n## URL path structure\n\nExample: `/image/ab5234-2346-ab34f/webp/400x300/fp=2000,1000/crop=0,0,4000,3000`\n\nThe start of the path contains the image id. Often it is prefixed or postfixed\nby arbitrary strings. To be flexible, we use a regex, `IMAGE_PATH_ID_PATTERN`\nto extract the id. The regex must match the beginning of the path and the first\ngroup must capture the id. In the above example, we would use the pattern\n`^/image/(?\u003cID\u003e[^/]+)/` to capture the first segment after `/image/`. With more\ncomplex patterns the id can even span multiple segments.\n\nOnce we have the ID of the original image, we need to construct the key of\nthe original image. The `ORIGINAL_IMAGE_KEY` environment variable defines a\nkey template, e.g. `foo/${ID}/bar`, where the `${ID}` is replaced with the\nextracted id.\n\nAll segments after the `IMAGE_PATH_ID_PATTERN` are encoded parameters.\n(Path segments are delimited by `/`). As the path of the request equals\nthe S3 key for the optimized image, we can't use query parameters but need\npath parameters. Conceptually they are the same, but path parameters are\neasier to use and build (no CloudFront whitelisting for example).\nIf you don't like this approach you can use a CloudFront function to\nconvert query parameters to path parameters.\n\n| Parameter   | Explanation                     | Example                | Default                                             |\n| ----------- | ------------------------------- | ---------------------- | --------------------------------------------------- |\n| format      | `jpeg` or `webp` or `avif`      | `/avif`                | pick smallest `jpeg` or `webp` impage               |\n| dimensions  | `\u003cwidth\u003ex\u003cheight`\u003e              | `/800x600`             | `320x200`                                           |\n| width only  | `\u003cwidth\u003e` or `\u003cwidth\u003ex`         | `/800`                 | height calculated to keep source aspect ratio       |\n| height only | `x\u003cheight\u003e`                     | `/x600`                | width calculated to keep source aspect ratio        |\n| focus point | `fp=\u003cx\u003e,\u003cy\u003e`                    | `/fp=2000,1200`        | (original_width / 2), (original_height / 3)         |\n| cropping    | `crop=\u003cx\u003e,\u003cy\u003e,\u003cwidth\u003e,\u003cheight\u003e` | `/crop=96,0,3904,2850` | original image size                                 |\n| quality     | `q=\u003c0..100\u003e`                    | `/q=80`                | automatic, depending on imagesize and format        |\n| background  | `bg=\u003chex-rgb-color\u003e`            | `/bg=ff0000`           | blend transparent pixels with this background color |\n\nJpeg encoding uses `mozjpg` settings, so it has a similar compression ratio as `webp` for photos.\nIf you don't specify a format, the image optimizer will internally try\n`webp` and `jpeg`. The format that produces the smallest image will be chosen and returned.\n\n- [Is WebP better than JPEG?](https://siipo.la/blog/is-webp-really-better-than-jpeg)\n- [Modern Data Compression in 2021](https://chipsandcheese.com/2021/02/28/modern-data-compression-in-2021-part-2-the-battle-to-dethrone-jpeg-with-jpeg-xl-avif-and-webp/)\n\n## Supported Image Formats\n\nOnly general purpose image formats are supported:\n\n- `jpeg` (with `mozjpg` settings)\n- `webp` (similar to `jpeg` and always better than `png` or `gif`)\n- `avif` (better compression than above, but slow and in some edge cases I noticed problems with visual quality)\n- coming soon: `jxl` excellent compression, quality and speed\n\nFor `jpeg` format, the `mozjpg` settings are used which gives us a comparable or better compression than `webp`.\nBecause `jpeg` and `webp` are widespread I recommend not specifying a format and letting the image\noptimizer pick the one that produces the smallest image.\n\n## Cost effective\n\nEach image will be generated only once and stored on S3.\nAn S3 lifetime policy will remove optimized images after a while (300 days by default).\nIf the image is still in use it will be optimized at max every 300 days.\nThis could even improve quality because in the meantime encoders probably improved.\n\n## Security\n\n- original images can be never accessed from the outside, they live in a separate private bucket\n- metadata (which can contain PII data) is always removed in optimized images\n- security token to protect against direct calls of the Lambda Function URL\n- URL signing to prevent malicious tampering\n\n## Prerequisites\n\n- run `npm ci`\n\n## Commands\n\n- `npm test` executes tests with jest\n- `npm run build` creates ./dist/lambda.js bundle\n- `npm run zip` creates the ./dist/lambda.zip from ./dist/lambda.js and ./dist/lambda.js.map\n- `npm run dist` runs all of the above steps\n- `npm run stack` creates or updates the CloudFormation stack\n- `npm run deploy` used to deploy ./dist/lambda.zip to the created lambda function\n- `npm start` will start the lambda function locally\n- `npm sign \u003cpath\u003e [secret]`\n\n## Configuration\n\nThe following environment variables must be specified. For `npm start` it is recommended\nto create a `.env` file and configure AWS credentials\n\n| Environment Variable      | Explanation                                                                 |\n| ------------------------- | --------------------------------------------------------------------------- |\n| `ORIGINAL_IMAGES_BUCKET`  | S3 bucket with orginal images (read only access)                            |\n| `OPTIMIZED_IMAGES_BUCKET` | S3 bucket with optimized images (read write access)                         |\n| `ORIGINAL_IMAGE_KEY`      | S3 key pattern for the original image, e.g. `image/${ID}/original`          |\n| `IMAGE_PATH_ID_PATTERN`   | regex to extract path prefix and image id, e.g. `^/path/to/image/([^/]+)/`  |\n| `CACHE_CONTROL`           | the Cache-Control header to set for optimized images                        |\n| `SECURITY_TOKEN`          | configured in Cloudfront to prevent direct lambda access without Cloudfront |\n| `AWS\\_\\*`                 | configure AWS SDK, e.g. credentials or region, for local development only   |\n\n## URL Signing\n\nURL Signing is done by a CloudFront Function. It needs a secret that is shared between\nthe client who signs a URL and CloudFront who verifies the signature.\nBecause CloudFront Functions don't have environment variables,\nwe embed the secret directly in the source code of the CloudFront Function.\nSee [stack.ts](infrastructure/stack.ts) and [viewerRequest.js](infrastructure/cloudFrontFunctions/viewerRequest.js)\nfor an implementation that reads it from [Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html).\nYou should configure a parameter with the name `image-optimizer-url-signing-secret` to store the secret.\n\nWhen you use URL signing each image URL must be signed. If you don't use it, you must remove\nthe CloudFront Function. If you use it, it is your responsibility to give the URL builder secure access\nto the shared secret. If for example some code (think React) runs in the browser and creates URLs (e.g.\nsetting image srcset), signing is no longer useful as the secret is visible in the browser and can be easily stolen.\nOnly if you can guarantee that all URLs are signed in a secure environment (server rendered) signing makes sense.\n\n## Caveats\n\n- **Building on a non-Linux environment:** Adjust the build process to include\n  the correct native sharp and libvips library in the lambda artifact.\n  See `infrastructure/zip.ts` and package.json prezip script.\n\n- **Lambda Return Size Limit:** If an optimized image is very large (roughly 6MB),\n  the Lambda function can't return a response. In this case, the result is still\n  written to the optimized images bucket, and a 503 response with a\n  [retry-after header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After)\n  is returned. On the next request, the image will be present on S3 and returned\n  normally via CloudFront.\n\n## Tools\n\n- [sharp](https://github.com/lovell/sharp) high-performance image processing\n- [tsx](https://github.com/privatenumber/tsx) execute typescript scripts (and typescript lambdas)\n- [esbuild](https://esbuild.github.io/) fast Typescript transpiler and bundler\n- [Jest](https://jestjs.io/) for testing\n- [Babel](https://babeljs.io/) as a Jest transformer\n- [Prettier](https://prettier.io/) for code formatting\n- [Husky](https://github.com/typicode/husky) for managing git hooks, e.g. run tests before committing\n- [@atombrenner/cfn-stack](https://github.com/atombrenner/cfn-stack) execute Cloudformation stacks with Typescript\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatombrenner%2Faws-image-optimizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fatombrenner%2Faws-image-optimizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatombrenner%2Faws-image-optimizer/lists"}