https://github.com/ironsource/aws-api-read-stream
Turn an AWS api call into a readable stream
https://github.com/ironsource/aws-api-read-stream
api aws nodejs npm-package stream-api
Last synced: 10 months ago
JSON representation
Turn an AWS api call into a readable stream
- Host: GitHub
- URL: https://github.com/ironsource/aws-api-read-stream
- Owner: ironSource
- Created: 2020-03-31T21:20:58.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2023-05-23T17:51:39.000Z (almost 3 years ago)
- Last Synced: 2024-04-27T06:03:49.856Z (almost 2 years ago)
- Topics: api, aws, nodejs, npm-package, stream-api
- Language: JavaScript
- Size: 588 KB
- Stars: 24
- Watchers: 5
- Forks: 1
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# aws-api-read-stream
Turn an aws api call into a readable stream.
## Install
```
npm i aws-api-read-stream
```
## example
Piping the result of `s3.listObjectsV2()`
Take care to use `NextToken` or `ContinuationToken` accordingly.
```js
const aws = require('aws-sdk')
const APIStream = require('aws-api-read-stream')
const { promisify } = require('util')
const pipeline = promisify(require('stream').pipeline)
async function main() {
const s3 = new aws.S3()
const s = APIStream.from((nextToken) => {
return s3.listObjectsV2({
Bucket: 'your-bucket-here',
ContinuationToken: nextToken
}).promise()
})
// convert the object stream to strings using async generator
// (node 13.* and above)
const transform = async function*(source) {
for await (const chunk of source) {
yield JSON.stringify(chunk)
}
}
await pipeline(s, transform, process.stdout)
}
main()
```
Keep reading until the stream finishes. This will buffer the results in an internal array, _be wary though, because this might crash the process if it runs out of memory_
```js
const aws = require('aws-sdk')
const APIStream = require('aws-api-read-stream')
const { promisify } = require('util')
const pipeline = promisify(require('stream').pipeline)
async function main() {
const s3 = new aws.S3()
const s = APIStream.from((nextToken) => {
return s3.listObjectsV2({
Bucket: 'your-bucket-here',
ContinuationToken: nextToken
}).promise()
})
const results = await s.readAll()
}
main()
```
Provide `Readable` stream options during initialization.
`objectMode` will always be set to `true`
```js
const s = APIStream.from((nextToken) => {
return s3.listObjectsV2({
Bucket: 'your-bucket-here',
ContinuationToken: nextToken
}).promise()
}, { options: { ... your options here } })
```
Start with an existing `nextToken`
```js
const s = APIStream.from((nextToken) => {
return s3.listObjectsV2({
Bucket: 'your-bucket-here',
ContinuationToken: nextToken
}).promise()
}, { nextToken: '123123' })
```