{"id":15375546,"url":"https://github.com/crccheck/kinesis-streams","last_synced_at":"2025-04-14T12:45:25.813Z","repository":{"id":57164103,"uuid":"88582182","full_name":"crccheck/kinesis-streams","owner":"crccheck","description":"AWS Kinesis readable and writable streams","archived":false,"fork":false,"pushed_at":"2024-05-23T16:51:47.000Z","size":209,"stargazers_count":8,"open_issues_count":5,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-07T03:08:03.825Z","etag":null,"topics":["aws","kinesis","nodejs","readable","stream","writable"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/crccheck.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-18T04:38:36.000Z","updated_at":"2024-05-23T16:51:51.000Z","dependencies_parsed_at":"2024-10-01T14:04:00.315Z","dependency_job_id":"dc5dd01a-a0c0-43d6-a88e-9ce0f81018b9","html_url":"https://github.com/crccheck/kinesis-streams","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crccheck%2Fkinesis-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crccheck%2Fkinesis-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crccheck%2Fkinesis-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crccheck%2Fkinesis-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/crccheck","download_url":"https://codeload.github.com/crccheck/kinesis-streams/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248883505,"owners_count":21177220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","kinesis","nodejs","readable","stream","writable"],"created_at":"2024-10-01T14:03:58.451Z","updated_at":"2025-04-14T12:45:25.779Z","avatar_url":"https://github.com/crccheck.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"Kinesis Streams\n===============\n\n[![Build Status](https://travis-ci.org/crccheck/kinesis-streams.svg?branch=master)](https://travis-ci.org/crccheck/kinesis-streams)\n[![npm version](https://badge.fury.io/js/kinesis-streams.svg)](https://badge.fury.io/js/kinesis-streams)\n[![Test Coverage](https://codeclimate.com/github/crccheck/kinesis-streams/badges/coverage.svg)](https://codeclimate.com/github/crccheck/kinesis-streams/coverage)\n\nThere once was a [Kinesis readable stream][kinesis-console-consumer] without a\nhome, and a Kinesis writable stream without a home, so now they're roommates.\n\nNOTE: Kinesis was a bad idea, and we're switching to Kafka. So I won't be using\nmy own library for much longer.\n\n\nInstalling\n----------\n\n    npm install kinesis-streams\n\n\nWriteable stream\n----------------\n\n    const AWS = require('aws-sdk')\n    const { KinesisWritable } = require('kinesis-streams')\n    const client = new AWS.Kinesis()\n    client.config.update({ maxRetries: 10 })\n    const writable = new KinesisWritable(client, 'streamName', options)\n    inputStream.pipe(writable)\n\n### Options\n\n* `options.logger` ([optional](#loggers)) [bunyan], [winston], or logger with `debug`, `error` and `info`\n* `options.highWaterMark` (default: 16) Buffer this many records before writing to Kinesis. Equivalent to `CollectionMaxCount`\n* `options.wait` (default: 500) How many milliseconds it should periodically flush. Equivalent to `RecordMaxBufferedTime`\n\nSome of these options have equivalents in the official [KPL].\n\n### Custom events\n\nThese events are emitted:\n\n* `kinesis.putRecords` Fires after records are put and the response is processed. You'll get the original response from AWS. See [`demo.js`](./demo.js) for an example of how to interpret it\n\n        reader.on('kinesis.putRecords', (response: {FailedRecordCount: number, Records: Record[]}) =\u003e {})\n\n### Setting the partition key\n\nBy default, the [partition key] is to a dummy value, `'0'`. If you have\nmultiple shards, you need to set a [partition key] in a way that makes sense\nfor your data. Here are two ways to do this:\n\n1. Set the `getPartitionKey` method of the writable stream instance:\n\n  ```\n  const AWS = require('aws-sdk')\n  const { KinesisWritable } = require('kinesis-streams')\n  const client = new AWS.Kinesis()\n  const writable = new KinesisWritable(client, 'streamName', options)\n  writable.getPartitionKey = (data) =\u003e data.foo.substr(5)\n  inputStream.pipe(writable)\n  ```\n\n2. Subclass `KinesisWritable` and provide your own `getPartitionKey`. See the [source](https://github.com/crccheck/kinesis-streams/blob/master/lib/writable.js) for reference.\n\n\nReadable stream\n---------------\n\n    const AWS = require('aws-sdk')\n    const { KinesisReadable } = require('kinesis-streams')\n    const client = new AWS.Kinesis()\n    const reader = new KinesisReadable(client, streamName, options)\n    reader.pipe(yourDestinationHere)\n\n### Options\n\n* `options.logger` ([optional](#loggers)) [bunyan], [winston], or logger with `debug`, `error` and `info`\n* `options.interval: number` (default: `2000`) Milliseconds between each Kinesis read. The AWS limit is 5 reads / second / shard\n* `options.parser: Function` If this is set, this function is applied to the data. Example:\n\n        const reader = new KinesisReadable(client, streamName, {parser: JSON.parse})\n        reader.on('data', console.log(data.id))\n\n* `options.restartOnClose: boolean` (default: `false`) Rediscover new shards once all current shards have been closed\n* And any [getShardIterator] parameter\n\n### Custom events\n\nThese events are emitted:\n\n* `checkpoint` This fires when data is received so you can keep track of the last successful sequence read:\n\n        reader.on('checkpoint', (sequenceNumber: string) =\u003e {})\n\n\nLoggers\n-------\n\n`KinesisWritable` and `KinesisReadable` both take an optional `logger` option.\nIf this is omitted, the [debug] logger will be used instead. To see output, set\n`DEBUG=kinesis-streams:*` in your environment.\n\n\nPrior art\n---------\n\nThe writable stream is based on the interface of [kinesis-write-stream]. The\n`checkpoint` event in readable stream is based on [kinesis-readable]. The\nreadable stream was originally written as a proof of concept in\n[kinesis-console-consumer].\n\n[kinesis-write-stream] was forked because at the time, it didn't support\nperiodic flushes. Since then the configuration of the readable and writable\nstreams have been rewritten to be consistent, and both emit lots of events now\nthat consumers can use for instrumentation.\n\n\nLicense\n-------\n\nThis package is licensed under Apache License 2.0, but the\n`tests/writable.spec.js` and `test/fixture/*` are originally from\n[kinesis-write-stream] MIT licensed from Espen Volden.\n\n\n[bunyan]: https://github.com/trentm/node-bunyan\n[debug]: https://github.com/visionmedia/debug\n[getShardIterator]: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Kinesis.html#getShardIterator-property\n[Kafka quickstart]: http://kafka.apache.org/documentation.html#quickstart_consume\n[kinesis-console-consumer]: https://github.com/crccheck/kinesis-console-consumer\n[kinesis-readable]: https://github.com/rclark/kinesis-readable\n[kinesis-write-stream]: https://github.com/voldern/kinesis-write-stream\n[KPL]: https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-kpl.html\n[partition key]: https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#partition-key\n[winston]: https://github.com/winstonjs/winston\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrccheck%2Fkinesis-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcrccheck%2Fkinesis-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrccheck%2Fkinesis-streams/lists"}