https://github.com/amekkawi/cwlogs-writable
Writable stream for AWS CloudWatch Logs
https://github.com/amekkawi/cwlogs-writable
aws cloudwatch-logs nodejs streams writable-streams
Last synced: about 1 month ago
JSON representation
Writable stream for AWS CloudWatch Logs
- Host: GitHub
- URL: https://github.com/amekkawi/cwlogs-writable
- Owner: amekkawi
- Created: 2017-02-11T00:43:08.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2017-10-09T16:07:57.000Z (over 7 years ago)
- Last Synced: 2025-04-14T00:12:33.144Z (about 1 month ago)
- Topics: aws, cloudwatch-logs, nodejs, streams, writable-streams
- Language: JavaScript
- Homepage:
- Size: 94.7 KB
- Stars: 18
- Watchers: 3
- Forks: 2
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
# cwlogs-writable #
Writable stream for AWS CloudWatch Logs, inspired by [bunyan-cloudwatch](https://github.com/mirkokiefer/bunyan-cloudwatch).
[](https://travis-ci.org/amekkawi/cwlogs-writable)
[](https://coveralls.io/github/amekkawi/cwlogs-writable?branch=v1.0.0)
[](https://david-dm.org/amekkawi/cwlogs-writable/v1.0.0)
[](https://david-dm.org/amekkawi/cwlogs-writable/v1.0.0?type=optional)
* [Features](#features)
* [API Docs](#api-docs)
* [Quick Start](#quick-start)
* [Bunyan Example](#bunyan-example)
* [Capturing Log Record Stringification Errors](#capturing-log-record-stringification-errors)
* [Picking LogStream Names](#picking-logstream-names)
* [Recovering from Errors](#recovering-from-errors)
* [Custom Handling of InvalidSequenceTokenException AWS Errors](#custom-handling-of-invalidsequencetokenexception-aws-errors)
* [CWLogsWritable Options](#cwlogswritable-options)
* [Change Log](CHANGELOG.md)
* [License](#license)
---
## Features ##
* Uses [aws-sdk](https://www.npmjs.com/package/aws-sdk).
* Can be used anywhere Writable streams are allowed.
* Allows for recovery from AWS errors.
* Creates log groups and streams if they do not exist.
* Filtering of log events by the stream itself.
* Safe stringification of log events.
## API Docs ##
There are two forms of the API docs:
* [Normal API docs](docs/api.md) - Use this if you using cwlogs-writable
as-is and not customizing/extending it's functionality.
* [Extended API docs](docs/api-protected.md) - Use this to also view
`protected` methods that you can use to customize/extend cwlogs-writable.
## Quick Start ##
Install the library using NPM into your existing node project:
```
npm install cwlogs-writable
```
Create and write to the stream.
```javascript
var CWLogsWritable = require('cwlogs-writable');
// Make stream name as unique as possible (see "Picking LogStream Names").
var streamName = 'my-log-stream/' + Date.now()
+ '/' + Math.round(Math.random() * 4026531839 + 268435456).toString(16);
var stream = new CWLogsWritable({
logGroupName: 'my-aws-log-group',
logStreamName: streamName,
// Options passed to the AWS.CloudWatchLogs service.
cloudWatchLogsOptions: {
// Change the AWS region as needed.
region: 'us-east-1',
// Example authenticating using access key.
accessKeyId: '{AWS-IAM-USER-ACCESS-KEY-ID}',
secretAccessKey: '{AWS-SECRET-ACCESS-KEY}'
}
});
stream.write('example-log-message');
```
Also consider this checklist:
* Are my log stream names [unique enough](#picking-logstream-names)?
* What if a log record [stringification fails](#capturing-log-record-stringification-errors)?
* Should my logging [recover from errors](#recovering-from-errors) or fail fast?
## Bunyan Example ##
```javascript
var bunyan = require('bunyan');
var CWLogsWritable = require('cwlogs-writable');
// Make stream name as unique as possible (see "Picking LogStream Names").
var streamName = 'my-log-stream/' + Date.now()
+ '/' + Math.round(Math.random() * 4026531839 + 268435456).toString(16);
var logger = bunyan.createLogger({
name: 'foo',
streams: [
{
level: 'debug',
// If 'raw' the CloudWatch log event timestamp will
// be taken from the bunyan JSON (i.e. rec.time).
type: 'raw',
stream: new CWLogsWritable({
logGroupName: 'my-aws-log-group',
logStreamName: streamName,
cloudWatchLogsOptions: { /* ... */ }
})
}
]
});
```
## Picking LogStream Names ##
In AWS CloudWatch Logs a LogStream represents "a sequence of log events from a
single emitter of logs".
The important part is "single emitter", as this implies that log events should
not be put into a LogStream concurrently by multiple emitters.
This is enforced by the [PutLogEvents API action](http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html)
which requires each call to include a "sequenceToken". That token is changed
each time a call is successful, and the new token is used for the next call.
If an emitter provides an incorrect token, the API will respond with
an __InvalidSequenceTokenException__.
To avoid this error, you must pick LogStream names that are unique to the
emitter or at least include enough randomness.
```javascript
// Example generation of LogStream name
var logStreamName = [
// Environment identifier (e.g. "production")
process.env.NODE_ENV || 'development',
// Current UTC date
new Date().toISOString().substr(0, 10),
// EC2 instance ID, optionally provided as an env variable
process.env.EC2_INSTANCE_ID || null,
// Process ID
'p' + process.pid,
// Random hex string (from "10000000" to "ffffffff")
Math.round(Math.random() * 4026531839 + 268435456).toString(16),
].filter(Boolean).join('/').replace(/[:*]/g, '');
```
## Capturing Log Record Stringification Errors ##
Before log records are sent to AWS they must be stringified.
cwlogs-writable uses safe stringification techniques to handle
circular references that would normally cause JSON.stringify to fail.
Other errors thrown during stringification (e.g. one thrown by a
[property getter](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/get))
will also be handled if the optional dependency
[safe-json-stringify](https://npmjs.com/package/safe-json-stringify)
is installed.
If it is not installed, cwlogs-writable will catch the error and emit a
`stringifyError` event.
```javascript
var stream = new CWLogsWritable({ ... });
// Catch errors if safe-json-stringify is not installed
stream.on('stringifyError', function(err, record) {
console.log('Failed to stringify log entry!', err);
// You can attempt to manually process the record argument,
// but be careful as you will may hit the same error.
//customBadRecordProcessing(record);
});
```
## Recovering from Errors ##
By default cwlogs-writable will handle the two most common AWS errors,
`InvalidSequenceTokenException` and `DataAlreadyAcceptedException`, to
give your application as much resiliency as possible.
For all other errors, the default behavior of a CWLogsWritable stream is to
emit an 'error' event, clear any queued logs, and ignore all further writes to
the stream to prevent memory leaks.
To override this behavior you can provide a `onError` callback that will
allow you to recover from these errors.
```javascript
var CWLogsWritable = require('cwlogs-writable');
function onError(err, logEvents, next) {
// Use built-in behavior if the error is not
// from a PutLogEvents action (logEvents will be null).
if (!logEvents) {
next(err);
return;
}
// Requeue the log events after a delay,
// if the queue is small enough.
if (this.getQueueSize() < 100) {
setTimeout(function() {
// Pass the logEvents to the "next" callback
// so they are added back to the head of the queue.
next(logEvents);
}, 2000);
}
// Otherwise, log the events to the console
// and resume streaming.
else {
console.error(
'Failed to send logEvents: ' +
JSON.stringify(logEvents)
);
next();
}
}
// Make stream name as unique as possible (see "Picking LogStream Names").
var streamName = 'my-log-stream/' + Date.now()
+ '/' + Math.round(Math.random() * 4026531839 + 268435456).toString(16);
var stream = new CWLogsWritable({
logGroupName: 'my-aws-log-group',
logStreamName: streamName,
cloudWatchLogsOptions: { /* ... */ },
// Pass the onError callback to CWLogsWritable
onError: onError
});
```
## Custom Handling of InvalidSequenceTokenException AWS Errors ##
Frequent `InvalidSequenceTokenException` AWS errors may indicate a
problem with the uniqueness of your LogStream name
(see [Picking LogStream Names](#picking-logstream-names)).
If you are experiencing throttling on
[PutLogEvents](http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html)
or [DescribeLogStreams](http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_DescribeLogStreams.html)
actions, you may want to add custom handling of `InvalidSequenceTokenException` errors.
```javascript
// Example of changing the logStreamName on
// InvalidSequenceTokenException errors
// to attempt to avoid further collisions.
function getLogStreamName() {
return 'my-log-stream/' + Date.now()
+ '/' + Math.round(Math.random() * 4026531839 + 268435456).toString(16);
}
function onError(err, logEvents, next) {
// Change the LogStream name to get a new
// randomized value, and requeue the log events.
if (err.code === 'InvalidSequenceTokenException') {
this.logStreamName = getLogStreamName();
next(logEvents);
}
// Default to built-in behavior.
else {
next(err);
}
}
var stream = new CWLogsWritable({
logGroupName: 'my-aws-log-group',
logStreamName: getLogStreamName(),
cloudWatchLogsOptions: { /* ... */ },
// Disable the default handling of
// InvalidSequenceTokenException errors
// so onError will get them instead.
retryOnInvalidSequenceToken: false,
// Pass the onError callback to CWLogsWritable
onError: onError
});
```
## CWLogsWritable Options ##
- **logGroupName**
Required
Type: string
AWS CloudWatch [LogGroup](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatchLogs.html#putLogEvents-property) name. It will be created if it doesn't exist.
- **logStreamName**
Required
Type: string
AWS CloudWatch [LogStream](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatchLogs.html#putLogEvents-property) name. It will be created if it doesn't exist.
- **cloudWatchLogsOptions**
Optional
Type: object
Default: {}
Options passed to [AWS.CloudWatchLogs](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatchLogs.html#constructor-property) service.
- **writeInterval**
Optional
Type: string | number
Default: "nextTick"
Amount of wait time after a Writable#_write call to allow batching of log events. Must be a positive number or "nextTick". If "nextTick", `process.nextTick` is used. If a number, `setTimeout` is used.
- **retryableDelay**
Optional
Type: string | number
Default: 150
- **retryableMax**
Optional
Type: number
Default: 100
Maximum number of times an AWS error marked as "retryable" will be retried before the error is instead passed to [CWLogsWritable#onError](docs/api.md#CWLogsWritable+onError).
- **maxBatchCount**
Optional
Type: number
Default: 10000
Maximum number of log events allowed in a single PutLogEvents API call.
- **maxBatchSize**
Optional
Type: number
Default: 1048576
Maximum number of bytes allowed in a single PutLogEvents API call.
- **ignoreDataAlreadyAcceptedException**
Optional
Type: boolean
Default: true
Ignore `DataAlreadyAcceptedException` errors. This will bypass [CWLogsWritable#onError](docs/api.md#CWLogsWritable+onError). See [cwlogs-writable/issues/10](https://github.com/amekkawi/cwlogs-writable/issues/10).
- **retryOnInvalidSequenceToken**
Optional
Type: boolean
Default: true
Retry on `InvalidSequenceTokenException` errors. This will bypass [CWLogsWritable#onError](docs/api.md#CWLogsWritable+onError). See [cwlogs-writable/issues/12](https://github.com/amekkawi/cwlogs-writable/issues/12).
- **onError**
Optional
Type: function
Called when an AWS error is encountered. Overwrites [CWLogsWritable#onError](docs/api.md#CWLogsWritable+onError) method.
- **filterWrite**
Optional
Type: function
Filter writes to CWLogsWritable. Overwrites [CWLogsWritable#filterWrite](docs/api.md#CWLogsWritable+filterWrite) method.
- **objectMode**
Optional
Type: boolean
Default: true
Passed to the Writable constructor. See https://nodejs.org/api/stream.html#stream_object_mode.
## Change Log ##
See [CHANGELOG.md](CHANGELOG.md)
## License ##
The MIT License (MIT)
Copyright (c) 2017 Andre Mekkawi <[email protected]>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.