{"id":16293556,"url":"https://github.com/badsyntax/s3-etag","last_synced_at":"2025-03-20T03:31:09.314Z","repository":{"id":37911414,"uuid":"439007522","full_name":"badsyntax/s3-etag","owner":"badsyntax","description":"Generate an accurate S3 ETAG in Node.js for any file (including multipart)","archived":false,"fork":false,"pushed_at":"2024-05-22T10:15:42.000Z","size":199,"stargazers_count":4,"open_issues_count":2,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-05-22T11:30:01.270Z","etag":null,"topics":["aws","etag","hash","md5","s3"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/s3-etag","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/badsyntax.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-16T13:54:43.000Z","updated_at":"2024-05-22T11:30:09.637Z","dependencies_parsed_at":"2024-05-22T11:30:07.161Z","dependency_job_id":"b0744631-96ca-4485-9a03-38ad595fcb26","html_url":"https://github.com/badsyntax/s3-etag","commit_stats":{"total_commits":30,"total_committers":2,"mean_commits":15.0,"dds":"0.33333333333333337","last_synced_commit":"e4a3d6da7ae22f73ec96fc725a290a3268705485"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/badsyntax%2Fs3-etag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/badsyntax%2Fs3-etag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/badsyntax%2Fs3-etag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/badsyntax%2Fs3-etag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/badsyntax","download_url":"https://codeload.github.com/badsyntax/s3-etag/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244543787,"owners_count":20469562,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","etag","hash","md5","s3"],"created_at":"2024-10-10T20:11:43.509Z","updated_at":"2025-03-20T03:31:08.961Z","avatar_url":"https://github.com/badsyntax.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# S3 ETAG\n\n[![Build, Test \u0026 Publish](https://github.com/badsyntax/s3-etag/actions/workflows/build-test-publish.yml/badge.svg)](https://github.com/badsyntax/s3-etag/actions/workflows/build-test-publish.yml)\n[![CodeQL](https://github.com/badsyntax/s3-etag/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/badsyntax/s3-etag/actions/workflows/codeql-analysis.yml)\n\nGenerate an accurate S3 ETAG in Node.js for any file (including multipart). \n\nPlease Note, this only works for unencrypted buckets.\n\n## Installation\n\n```console\nnpm install s3-etag\n```\n\n## Usage\n\n```ts\nimport { generateETag } from 's3-etag';\n\n// Simple MD5 hash of contents for non-multipart files\nconst etag = generateETag(absoluteFilePath);\n\n// MD5 hash of combined contents \u0026 part number (see below) for multipart files\nconst partSizeInBytes = 10 * 1024 * 1024; // 10mb\nconst etag = generateETag(absoluteFilePath, partSizeInBytes);\n```\n\n## How It Works\n\nThis is a Node.js implementation of [this algorithm](https://stackoverflow.com/a/19896823/492325).\n\nAt a high level:\n\n- If no `partSizeInBytes` is specified, return MD5 hash of file contents\n- If `partSizeInBytes` is specified:\n  - Generate parts by comparing `partSizeInBytes` to the file size\n  - Read each part from the file, MD5 hash the part, and append it to a global combined hash\n  - Once all parts are processed, generate a new MD5 from the global combined hash, and suffix with the amount of parts\n\nIf the partSizeInBytes is unknown, you can find it by using AWS CLI:\n   \n\n 1. Use `head-object` to retrieve the object's metadata using the following command, where `raw-files` is the bucket name, and `IMG2345.CR2` is the key\n\t `aws s3api head-object --bucket raw-files --key IMG2345.CR2`\n    this will return\n      ```\n        {\n        \"AcceptRanges\": \"bytes\",\n        \"ContentType\": \"text/html\",\n        \"LastModified\": \"2024-05-22T08:45:10+00:00\"\n        \"ContentLength\": 28333464,\n        \"ETag\": \"\\\"85ae33db28930d3afe594da14cd190bb-2\\\"\",\n        \"VersionId\": \"28P4UkX5sCO.8vbyMojvecHndkHDwf\",\n        \"ContentType\": \"binary/octet-stream\",\n        \"ServerSideEncryption\": \"AES256\",\n        \"Metadata\": {}\n        }\n    ```\n 2. Look for a `-n` at the end of the eTag, where `n` is a number \u003e= 2, representing the number of parts/chunks. For single part objects, the eTag will simply be the MD5 of the object.\n 3. Use  `aws s3api head-object --bucket raw-files --key IMG2345.CR2 --part-number 1`to get the metadata of the first part/chunk. This will return \n\t ```\n        {\n        \"AcceptRanges\": \"bytes\",\n        \"ContentType\": \"text/html\",\n        \"LastModified\": \"2024-05-22T08:45:10+00:00\"\n        \"ContentLength\": 17179870,\n        \"ETag\": \"\\\"85ae33db28930d3afe594da14cd190bb-2\\\"\",\n        \"VersionId\": \"28P4UkX5sCO.8vbyMojvecHndkHDwf\",\n        \"ContentType\": \"binary/octet-stream\",\n        \"ServerSideEncryption\": \"AES256\",\n        \"Metadata\": {},\n        \"PartsCount\": 2\n        }\n    ```\n 4. Use the `ContentLength` value as the `partSizeInBytes`\n\n## License\n\nSee [LICENSE.md](./LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbadsyntax%2Fs3-etag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbadsyntax%2Fs3-etag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbadsyntax%2Fs3-etag/lists"}