{"id":19878712,"url":"https://github.com/ironsource/aws-api-read-stream","last_synced_at":"2025-05-02T13:30:49.420Z","repository":{"id":42902291,"uuid":"251734566","full_name":"ironSource/aws-api-read-stream","owner":"ironSource","description":"Turn an AWS api call into a readable stream","archived":false,"fork":false,"pushed_at":"2023-05-23T17:51:39.000Z","size":602,"stargazers_count":24,"open_issues_count":4,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-04-27T06:03:49.856Z","etag":null,"topics":["api","aws","nodejs","npm-package","stream-api"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ironSource.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-31T21:20:58.000Z","updated_at":"2022-02-27T17:10:50.000Z","dependencies_parsed_at":"2022-09-07T14:21:02.569Z","dependency_job_id":null,"html_url":"https://github.com/ironSource/aws-api-read-stream","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Faws-api-read-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Faws-api-read-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Faws-api-read-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Faws-api-read-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ironSource","download_url":"https://codeload.github.com/ironSource/aws-api-read-stream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224315370,"owners_count":17290992,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","aws","nodejs","npm-package","stream-api"],"created_at":"2024-11-12T17:06:28.056Z","updated_at":"2024-11-12T17:06:28.756Z","avatar_url":"https://github.com/ironSource.png","language":"JavaScript","readme":"# aws-api-read-stream\nTurn an aws api call into a readable stream.\n\n## Install\n```\nnpm i aws-api-read-stream\n```\n\n## example\n\nPiping the result of `s3.listObjectsV2()` \n\nTake care to use `NextToken` or `ContinuationToken` accordingly.\n\n```js\nconst aws = require('aws-sdk')\nconst APIStream = require('aws-api-read-stream')\nconst { promisify } = require('util')\nconst pipeline = promisify(require('stream').pipeline)\n\nasync function main() {\n    const s3 = new aws.S3()\n\n    const s = APIStream.from((nextToken) =\u003e {\n        return s3.listObjectsV2({\n            Bucket: 'your-bucket-here',\n            ContinuationToken: nextToken\n        }).promise()\n    })\n\n    // convert the object stream to strings using async generator\n    // (node 13.* and above)\n    const transform = async function*(source) {\n        for await (const chunk of source) {\n            yield JSON.stringify(chunk)\n        }\n    }\n\n    await pipeline(s, transform, process.stdout)\n}\n\nmain()\n```\n\nKeep reading until the stream finishes. This will buffer the results in an internal array, _be wary though, because this might crash the process if it runs out of memory_\n\n```js\nconst aws = require('aws-sdk')\nconst APIStream = require('aws-api-read-stream')\nconst { promisify } = require('util')\nconst pipeline = promisify(require('stream').pipeline)\n\nasync function main() {\n    const s3 = new aws.S3()\n\n    const s = APIStream.from((nextToken) =\u003e {\n        return s3.listObjectsV2({\n            Bucket: 'your-bucket-here',\n            ContinuationToken: nextToken\n        }).promise()\n    })\n\n    const results = await s.readAll()\n}\n\nmain()\n```\n\nProvide `Readable` stream options during initialization.\n`objectMode` will always be set to `true`\n\n```js\nconst s = APIStream.from((nextToken) =\u003e {\n    return s3.listObjectsV2({\n        Bucket: 'your-bucket-here',\n        ContinuationToken: nextToken\n    }).promise()\n}, { options: { ... your options here } })\n```\n\nStart with an existing `nextToken`\n\n```js\nconst s = APIStream.from((nextToken) =\u003e {\n    return s3.listObjectsV2({\n        Bucket: 'your-bucket-here',\n        ContinuationToken: nextToken\n    }).promise()\n}, { nextToken: '123123' })\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fironsource%2Faws-api-read-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fironsource%2Faws-api-read-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fironsource%2Faws-api-read-stream/lists"}