{"id":15673841,"url":"https://github.com/lmammino/s3-list-bucket-stream","last_synced_at":"2025-05-12T19:26:34.885Z","repository":{"id":57355940,"uuid":"171479683","full_name":"lmammino/s3-list-bucket-stream","owner":"lmammino","description":"Node.js stream library that allows you to stream a list of objects from an S3 bucket","archived":false,"fork":false,"pushed_at":"2019-02-20T14:42:29.000Z","size":466,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-01T02:41:19.575Z","etag":null,"topics":["aws","aws-s3","library","node","node-js","node-module","nodejs","s3","s3-bucket","stream","streams"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lmammino.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-02-19T13:35:31.000Z","updated_at":"2022-07-21T21:08:06.000Z","dependencies_parsed_at":"2022-09-05T22:13:22.180Z","dependency_job_id":null,"html_url":"https://github.com/lmammino/s3-list-bucket-stream","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fs3-list-bucket-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fs3-list-bucket-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fs3-list-bucket-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fs3-list-bucket-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lmammino","download_url":"https://codeload.github.com/lmammino/s3-list-bucket-stream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253807267,"owners_count":21967331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-s3","library","node","node-js","node-module","nodejs","s3","s3-bucket","stream","streams"],"created_at":"2024-10-03T15:42:25.447Z","updated_at":"2025-05-12T19:26:34.862Z","avatar_url":"https://github.com/lmammino.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3-list-bucket-stream\n\nNode.js stream library that allows you to stream a list of objects from an S3 bucket\n\n[![npm version](https://badge.fury.io/js/s3-list-bucket-stream.svg)](https://badge.fury.io/js/s3-list-bucket-stream)\n[![CircleCI](https://circleci.com/gh/lmammino/s3-list-bucket-stream.svg?style=shield)](https://circleci.com/gh/lmammino/s3-list-bucket-stream)\n[![Codecov coverage](https://codecov.io/gh/lmammino/s3-list-bucket-stream/branch/master/graph/badge.svg)](https://codecov.io/gh/lmammino/s3-list-bucket-stream)\n[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)\n\n## Install\n\nUsing npm:\n\n```bash\nnpm i --save s3-list-bucket-stream\n```\n\nor yarn\n\n```bash\nyarn add s3-list-bucket-stream\n```\n\n**Note:** In order to use this package you need to have the [`aws-sdk`](https://www.npmjs.com/package/aws-sdk) module installed\n(or any other library that allows you to instantiate an S3 client with the `listBucketV2` method).\n\n\n## Usage\n\nHere's a simple example that allows to list all the files in a bucket\n\n```javascript\nconst S3ListBucketStream = require('s3-list-bucket-stream')\n\n// create the S3 client\nconst AWS = require('aws-sdk')\nconst s3 = new AWS.S3()\n\n// create the instance for the stream\nconst listBucketStream = new S3ListBucketStream(s3, 'some-bucket', 'path/to/files')\n\n// attach an 'on data' event which will start the stream flow\nlistBucketStream\n  .on('data', (key) =\u003e console.log(key.toString()))\n```\n\nThis will output:\n\n```plain\npath/to/files/file1\npath/to/files/file2\npath/to/files/file3\n...\n```\n\nIf you want to emit objects containing the entire S3 object metadata, you can do so\nby enabling the `fullMetadata` option while creating the stream instance:\n\n```javascript\nconst S3ListBucketStream = require('s3-list-bucket-stream')\n\n// create the S3 client\nconst AWS = require('aws-sdk')\nconst s3 = new AWS.S3()\n\n// create the instance for the stream\nconst listBucketStream = new S3ListBucketStream(\n  s3,\n  'some-bucket',\n  'path/to/files',\n  { fullMetadata: true }\n)\n\n// attach an 'on data' event which will start the stream flow\nlistBucketStream\n  .on('data', console.log)\n```\n\nThis will produce this output:\n\n```plain\n{ Key: 'path/to/files/file1',\n  LastModified: 2019-02-08T11:11:19.000Z,\n  ETag: '\"7e97db1005fe07801a3e3737103ceab8\"',\n  Size: 49152,\n  StorageClass: 'STANDARD' }\n{ Key: 'path/to/files/file2',\n  LastModified: 2019-02-07T11:11:19.000Z,\n  ETag: '\"6a97db1005fe07801a3e3737103ceab8\"',\n  Size: 39152,\n  StorageClass: 'STANDARD' }\n{ Key: 'path/to/files/file3',\n  LastModified: 2019-02-05T11:11:19.000Z,\n  ETag: '\"b097db1005fe07801a3e3737103ceab8\"',\n  Size: 29152,\n  StorageClass: 'STANDARD' }\n...\n```\n\n**Note**: with this option enabled, the stream will automatically enable `objectMode`.\n\n\n## Programmatic API\n\nHere you can find more details on how to configure the stream instances and what kind\nof events are available.\n\n### Constructor arguments\n\nWhen creating a new instance of the stream these are the arguments you can pass as\nconstructor arguments:\n\n - `s3` (`Object`): An S3 client from the AWS SDK (or any object that implements a compatible `listObjectsV2` method)\n - `bucket` (`string`): The name of the bucket to list\n - `[bucketPrefix]` (`string`): A prefix to list only files with the given prefix (optional)\n - `[options]` (`S3ListBucketStreamOptions`): Stream options (optional)\n - `[listObjectsV2args]` (`ListObjectsV2args`): Extra arguments to be passed to the listObjectsV2 call in the S3 client (optional)\n\n#### S3ListBucketStreamOptions\n\n`S3ListBucketStreamOptions` is an object that can contain arbitrary `Readable` stream options.\nYou can also specify the following extra options:\n\n - `fullMetadata` (`boolean`): switches the stream to `objectMode: true` and emits objects containing the full metadata for a given bucket object. See the example above for more details (default value is `false`).\n\n#### ListObjectsV2args\n\n`ListObjectsV2args` is an object that can contain arbitrary [`s3.listObjectsV2` parameters](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property) like `MaxKeys` (set to `1000` by default), `FetchOwner`, `RequestPayer` or `StartAfter`.\n\nThese parameters will be propagated to every internal `listObjectsV2` call to the S3 client\nprovided at construction time.\n\n**Note**: be careful not to specify values for `Bucket`, `Prefix` and `ContinuationToken`\nas these values will be managed by the internals according to the internal state and configuration\nof the given stream instance.\n\n\n### Events\n\nThe stream is a `Readable` stream instance, so it can fire all the common\n[`Readable` stream events](https://nodejs.org/api/stream.html#stream_class_stream_readable).\n\nIn addition to these events there some new events that will be fired by a given instance:\n\n - `page`: fired when a new page is fetched through the S3 client. If listening to this event,\n   your callback will receive an object containing the properties `params` (original parameters for the `listObjectsV2` call)\n   and `data` (the raw response to the `listObjectsV2` call).\n - `stopped`: fired when the readable buffer is full and the fetch from S3 is stopped\n - `restarted`: fired when the fetch from S3 is restarted after a pause\n\n## Contributing\n\nEveryone is very welcome to contribute to this project. You can contribute just by submitting bugs or\nsuggesting improvements by [opening an issue on GitHub](https://github.com/lmammino/s3-list-bucket-stream/issues).\n\nYou can also submit PRs as long as you adhere with the code standards and write tests for the proposed changes.\n\n## License\n\nLicensed under [MIT License](LICENSE). © Luciano Mammino.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmammino%2Fs3-list-bucket-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flmammino%2Fs3-list-bucket-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmammino%2Fs3-list-bucket-stream/lists"}