{"id":15673501,"url":"https://github.com/lmammino/s3-object-content-stream","last_synced_at":"2026-03-15T10:41:11.399Z","repository":{"id":138311211,"uuid":"171695767","full_name":"lmammino/s3-object-content-stream","owner":"lmammino","description":"A Node.js transform stream that gets S3 object keys and emits their actual content","archived":false,"fork":false,"pushed_at":"2025-05-23T21:45:11.000Z","size":214,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-29T15:58:43.943Z","etag":null,"topics":["library","module","node","node-js","s3","s3-bucket","s3-storage","streams","streams-api"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lmammino.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-02-20T15:13:39.000Z","updated_at":"2019-03-06T21:40:44.000Z","dependencies_parsed_at":"2025-07-29T02:45:01.205Z","dependency_job_id":null,"html_url":"https://github.com/lmammino/s3-object-content-stream","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/lmammino/s3-object-content-stream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fs3-object-content-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fs3-object-content-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fs3-object-content-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fs3-object-content-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lmammino","download_url":"https://codeload.github.com/lmammino/s3-object-content-stream/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fs3-object-content-stream/sbom","scorecard":{"id":595989,"data":{"date":"2025-08-11","repo":{"name":"github.com/lmammino/s3-object-content-stream","commit":"ccf751c8e5322a93618e3c4ec9014422abead666"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.7,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 0/19 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"70 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-968p-4wvh-cqc8","Warn: Project is vulnerable to: GHSA-67hx-6x53-jw92","Warn: Project is vulnerable to: GHSA-6chw-6frg-f759","Warn: Project is vulnerable to: GHSA-v88g-cgmw-v5xw","Warn: Project is vulnerable to: GHSA-93q8-gq69-wqmw","Warn: Project is vulnerable to: GHSA-fwr7-v2mv-hh25","Warn: Project is vulnerable to: GHSA-v6h2-p8h4-qcjw","Warn: Project is vulnerable to: GHSA-grv7-fg5c-xmjg","Warn: Project is vulnerable to: GHSA-w8qv-6jwh-64r5","Warn: Project is vulnerable to: GHSA-mh2h-6j8q-x246","Warn: Project is vulnerable to: GHSA-5q88-cjfq-g2mh","Warn: Project is vulnerable to: GHSA-xp63-6vf5-xf3v","Warn: Project is vulnerable to: GHSA-3xgq-45jj-v275","Warn: Project is vulnerable to: GHSA-gxpj-cx7g-858c","Warn: Project is vulnerable to: GHSA-w573-4hg7-7wgq","Warn: Project is vulnerable to: GHSA-3gx7-xhv7-5mx3","Warn: Project is vulnerable to: GHSA-fjxv-7rqg-78g4","Warn: Project is vulnerable to: GHSA-8r6j-v8pm-fqw3","Warn: Project is vulnerable to: MAL-2023-462","Warn: Project is vulnerable to: GHSA-q42p-pg8m-cqh6","Warn: Project is vulnerable to: GHSA-w457-6q6x-cgp9","Warn: Project is vulnerable to: GHSA-62gr-4qp9-h98f","Warn: Project is vulnerable to: GHSA-f52g-6jhx-586p","Warn: Project is vulnerable to: GHSA-2cf5-4w76-r9qv","Warn: Project is vulnerable to: GHSA-3cqr-58rm-57f8","Warn: Project is vulnerable to: GHSA-g9r4-xpmj-mj65","Warn: Project is vulnerable to: GHSA-q2c6-c6pm-g3gh","Warn: Project is vulnerable to: GHSA-765h-qjxv-5f44","Warn: Project is vulnerable to: GHSA-f2jv-r9rf-7988","Warn: Project is vulnerable to: GHSA-43f8-2h32-f4cj","Warn: Project is vulnerable to: GHSA-pc5p-h8pf-mvwp","Warn: Project is vulnerable to: GHSA-qqgx-2p2h-9c37","Warn: Project is vulnerable to: GHSA-2pr6-76vf-7546","Warn: Project is vulnerable to: GHSA-8j8c-7jfh-h6hx","Warn: Project is vulnerable to: GHSA-896r-f27r-55mw","Warn: Project is vulnerable to: GHSA-9c47-m6qq-7p4h","Warn: Project is vulnerable to: GHSA-6c8f-qphg-qjgp","Warn: Project is vulnerable to: GHSA-jf85-cpcp-j695","Warn: Project is vulnerable to: GHSA-p6mc-m468-83gw","Warn: Project is vulnerable to: GHSA-29mw-wpgm-hmr9","Warn: Project is vulnerable to: GHSA-35jh-r3h4-6jhm","Warn: Project is vulnerable to: GHSA-7wpw-2hjm-89gp","Warn: Project is vulnerable to: GHSA-952p-6rrq-rcjv","Warn: Project is vulnerable to: GHSA-f8q6-p94x-37v3","Warn: Project is vulnerable to: GHSA-vh95-rmgr-6w4m","Warn: Project is vulnerable to: GHSA-xvch-5gv4-984h","Warn: Project is vulnerable to: GHSA-fhjf-83wg-r2j9","Warn: Project is vulnerable to: GHSA-w7rc-rwvf-8q5r","Warn: Project is vulnerable to: GHSA-r683-j2x4-v87g","Warn: Project is vulnerable to: GHSA-5fw9-fq32-wv5p","Warn: Project is vulnerable to: GHSA-hj48-42vr-x3v9","Warn: Project is vulnerable to: GHSA-hrpp-h998-j3pp","Warn: Project is vulnerable to: GHSA-p8p7-x288-28g6","Warn: Project is vulnerable to: GHSA-gcx4-mw62-g8wm","Warn: Project is vulnerable to: GHSA-c2qf-rxjj-qqgw","Warn: Project is vulnerable to: GHSA-4g88-fppr-53pp","Warn: Project is vulnerable to: GHSA-4jqc-8m5r-9rpr","Warn: Project is vulnerable to: GHSA-3jfq-g458-7qm9","Warn: Project is vulnerable to: GHSA-r628-mhmh-qjhw","Warn: Project is vulnerable to: GHSA-9r2w-394v-53qc","Warn: Project is vulnerable to: GHSA-5955-9wpr-37jh","Warn: Project is vulnerable to: GHSA-qq89-hq3f-393p","Warn: Project is vulnerable to: GHSA-f5x3-32g6-xq36","Warn: Project is vulnerable to: GHSA-52f5-9888-hmc6","Warn: Project is vulnerable to: GHSA-jgrx-mgxx-jf9v","Warn: Project is vulnerable to: GHSA-72xf-g2v4-qvf3","Warn: Project is vulnerable to: GHSA-6fc8-4gx4-v693","Warn: Project is vulnerable to: GHSA-3h5v-q93c-6h6q","Warn: Project is vulnerable to: GHSA-c4w7-xm78-47vh","Warn: Project is vulnerable to: GHSA-p9pc-299p-vxgp"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-20T23:09:42.027Z","repository_id":138311211,"created_at":"2025-08-20T23:09:42.028Z","updated_at":"2025-08-20T23:09:42.028Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274670709,"owners_count":25328293,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["library","module","node","node-js","s3","s3-bucket","s3-storage","streams","streams-api"],"created_at":"2024-10-03T15:40:58.591Z","updated_at":"2025-10-05T10:39:46.870Z","avatar_url":"https://github.com/lmammino.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3-object-content-stream\n\nA Node.js transform stream that gets S3 object keys and emits their actual content\n\n[![npm version](https://badge.fury.io/js/s3-object-content-stream.svg)](https://badge.fury.io/js/s3-object-content-stream)\n[![CircleCI](https://circleci.com/gh/lmammino/s3-object-content-stream.svg?style=shield)](https://circleci.com/gh/lmammino/s3-object-content-stream)\n[![Codecov coverage](https://codecov.io/gh/lmammino/s3-object-content-stream/branch/master/graph/badge.svg)](https://codecov.io/gh/lmammino/s3-object-content-stream)\n[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)\n\n## Install\n\nUsing npm:\n\n```bash\nnpm i --save s3-object-content-stream\n```\n\nor yarn\n\n```bash\nyarn add s3-object-content-stream\n```\n\n**Note:** In order to use this package you need to have the [`aws-sdk`](https://www.npmjs.com/package/aws-sdk) module installed\n(or any other library that allows you to instantiate an S3 client with the `listBucketV2` method).\n\n## Usage\n\nThis library works very well in conjunction with [`s3-list-bucket-stream`](https://www.npmjs.com/package/s3-list-bucket-stream), a library that allows you to create a readable stream to list files from an S3 bucket.\n\nHere's a simple example that allows to output the content of objects\nin an S3 bucket:\n\n```javascript\nconst S3ListBucketStream = require('s3-list-bucket-stream')\nconst S3ObjectContentStream = require('s3-object-content-stream')\n\n// create the S3 client\nconst AWS = require('aws-sdk')\nconst s3 = new AWS.S3()\n\n// create the instance for the list bucket stream\nconst listBucketStream = new S3ListBucketStream(\n  s3,\n  'some-bucket',\n  'path/to/files'\n)\n\n// create the instance for the object content stream\nconst objectContentStream = new S3ObjectContentStream(s3, 'some-bucket')\n\n// pipe the two streams together and outputs on the stdout\nlistBucketStream.pipe(objectContentStream).pipe(process.stdout)\n```\n\nThis will print the content of all the files in `some-bucket` with the prefix `path/to/files`.\n\nNote that there will be no separator between different files.\n\n## Content transformation\n\nMost often your data in S3 will be in a compressed and/or encrypted form.\n\nTo deal with these cases, you can pass a factory function while instantiating a new `S3ObjectContentStream`.\n\nThis factory function has the goal of creating a new Transform stream that will be used to convert the data while it is emitted to the next phase of the pipeline.\n\nThe factory function receives the current chunk, so you can use that to do smart transformation based, for instance on the Object key (file name).\n\nIn this example we use this feature to automatically decompress gzipped files before emitting their content:\n\n```javascript\nconst { createGunzip } = require('zlib')\nconst { extname } = require('path')\nconst { PassThrough } = require('stream')\nconst S3ListBucketStream = require('s3-list-bucket-stream')\nconst S3ObjectContentStream = require('s3-object-content-stream')\n\n// definition of our factory function\nconst ungzipIfNeeded = key =\u003e {\n  const extension = extname(key.toString())\n  if (['.gz', '.gzip'].includes(extension)) {\n    return createGunzip() // if the file is gzip return a transform stream\n  }\n\n  // otherwise returns a passthrough stream (do not modify the content)\n  return new PassThrough()\n}\n\n// create the S3 client\nconst AWS = require('aws-sdk')\nconst s3 = new AWS.S3()\n\n// create the instance for the list bucket stream\nconst listBucketStream = new S3ListBucketStream(\n  s3,\n  'some-bucket',\n  'path/to/files'\n)\n\n// create the instance for the object content stream\nconst objectContentStream = new S3ObjectContentStream(\n  s3,\n  'some-bucket',\n  ungzipIfNeeded // pass our transform stream factory function\n)\n\n// pipe the two streams together and outputs on the stdout\nlistBucketStream.pipe(objectContentStream).pipe(process.stdout)\n```\n\n## Full metadata mode\n\nIf your readable source of S3 Objects emits objects (where for every element you have a property called `Key` that identifies the object name), you need to enable the `fullMetadata` flag for the stream to switch to object mode and read the object keys correctly.\n\nThis mode is useful when you want to operate to a level that is closer to what you get from the AWS SDK `ListObjectsV2` API where objects are identified as follows:\n\n```plain\n{ Key: 'path/to/files/file1',\n  LastModified: 2019-02-08T11:11:19.000Z,\n  ETag: '\"7e97db1005fe07801a3e3737103ceab8\"',\n  Size: 49152,\n  StorageClass: 'STANDARD' }\n{ Key: 'path/to/files/file2',\n  LastModified: 2019-02-07T11:11:19.000Z,\n  ETag: '\"6a97db1005fe07801a3e3737103ceab8\"',\n  Size: 39152,\n  StorageClass: 'STANDARD' }\n{ Key: 'path/to/files/file3',\n  LastModified: 2019-02-05T11:11:19.000Z,\n  ETag: '\"b097db1005fe07801a3e3737103ceab8\"',\n  Size: 29152,\n  StorageClass: 'STANDARD' }\n...\n```\n\nIn the following example we will use `S3ListBucketStream` in full metadata mode, hence we will need to enable full metadata even in our instance of `S3ObjectContentStream`:\n\n```javascript\nconst S3ListBucketStream = require('s3-list-bucket-stream')\nconst S3ObjectContentStream = require('s3-object-content-stream')\n\n// create the S3 client\nconst AWS = require('aws-sdk')\nconst s3 = new AWS.S3()\n\n// create the instance for the list bucket stream\nconst listBucketStream = new S3ListBucketStream(\n  s3,\n  'some-bucket',\n  'path/to/files',\n  { fullMetadata: true } // full metadata enabled\n)\n\n// create the instance for the object content stream\nconst objectContentStream = new S3ObjectContentStream(\n  s3,\n  'some-bucket',\n  undefined, // no transformation needed\n  { fullMetadata: true } // full metadata enabled\n)\n\n// pipe the two streams together and outputs on the stdout\nlistBucketStream.pipe(objectContentStream).pipe(process.stdout)\n```\n\n## Contributing\n\nEveryone is very welcome to contribute to this project. You can contribute just by submitting bugs or\nsuggesting improvements by [opening an issue on GitHub](https://github.com/lmammino/s3-object-content-stream/issues).\n\nYou can also submit PRs as long as you adhere with the code standards and write tests for the proposed changes.\n\n## License\n\nLicensed under [MIT License](LICENSE). © Luciano Mammino.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmammino%2Fs3-object-content-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flmammino%2Fs3-object-content-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmammino%2Fs3-object-content-stream/lists"}