https://github.com/sidravic/kinesis_stream_fetcher
Is a simple wrapper over the AWS SDK for Kinesis to fetch data from kinesis streams using a simple polling strategy. The stream fetcher launches processes for each partition within a stream and polls for updates. The stream fetcher is not highly configurable but is built to eliminate the use of the java based multilang daemon provided by AWS
https://github.com/sidravic/kinesis_stream_fetcher
kinesis kinesis-stream-fetcher partition polls
Last synced: 4 months ago
JSON representation
Is a simple wrapper over the AWS SDK for Kinesis to fetch data from kinesis streams using a simple polling strategy. The stream fetcher launches processes for each partition within a stream and polls for updates. The stream fetcher is not highly configurable but is built to eliminate the use of the java based multilang daemon provided by AWS
- Host: GitHub
- URL: https://github.com/sidravic/kinesis_stream_fetcher
- Owner: sidravic
- Created: 2015-12-12T19:35:21.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2016-01-04T05:45:59.000Z (over 10 years ago)
- Last Synced: 2025-09-28T05:37:07.037Z (8 months ago)
- Topics: kinesis, kinesis-stream-fetcher, partition, polls
- Language: JavaScript
- Homepage:
- Size: 1.18 MB
- Stars: 2
- Watchers: 0
- Forks: 2
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
### Kinesis Stream Fetcher
Kinesis Stream Fetcher is a basic library to fetch messages from a Kinesis
stream without using the multilang daemon. It works as a simple poller by
launching child processes for each stream that needs to be monitored.
Version 1.0.0 supports multiple partitions and launches a process for each paritition.
The `partitions` config is no longer relevant as the information is not directly fetched
from Amazon API.
### Install
```
npm install kinesis_stream_fetcher --save
```
### Usage
Before starting development ensure that your AWS configuration parameters
are present in your environment under the following variable.
```
export AWS_ACCESS_KEY_ID="ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="SECRET_ACCESS_KEY"
export AWS_REGION="ap-southeast-1"
export KINESIS_PARTITION_KEY="sweetcoffee"
```
**Update**
Version 1.2.0 includes a backward compatible support for instanceIds. Each instanceId
identifies a deployment uniquely, thus allowing you to run multiple instances of Kinesis
Stream Fetcher.
This was a bug in the earlier versions where it assumed that the library would be used for a
single instance
```
let streamConfig = {
instanceId: 'instance-1',
redisUrl: 'redis://localhost:6379',
streams: [{
name: 'development-transaction_events-1',
partitions: 1
},{
name: 'test-transaction_events',
partitions: 1
}]
};
```
```
const KinesisStreamFetcher = require('kinesis_stream_fetcher');
```
```
let fetcher = new KinesisStreamFetcher(streamConfig);
```
```
fetcher.on('message', (data) => {
console.log(data);
})
```
### Description
The consumer expects a redis connection to store the last read state of a stream.
The first attempt is always to begin from the start of the stream.