{"id":23976020,"url":"https://github.com/amekkawi/cwlogs-writable","last_synced_at":"2025-04-14T00:12:56.351Z","repository":{"id":145334468,"uuid":"81619959","full_name":"amekkawi/cwlogs-writable","owner":"amekkawi","description":"Writable stream for AWS CloudWatch Logs","archived":false,"fork":false,"pushed_at":"2017-10-09T16:07:57.000Z","size":97,"stargazers_count":18,"open_issues_count":3,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-14T00:12:33.144Z","etag":null,"topics":["aws","cloudwatch-logs","nodejs","streams","writable-streams"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amekkawi.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-11T00:43:08.000Z","updated_at":"2021-01-28T02:59:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"f928a766-dfa1-4099-89a8-33fb7718b5f3","html_url":"https://github.com/amekkawi/cwlogs-writable","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amekkawi%2Fcwlogs-writable","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amekkawi%2Fcwlogs-writable/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amekkawi%2Fcwlogs-writable/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amekkawi%2Fcwlogs-writable/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amekkawi","download_url":"https://codeload.github.com/amekkawi/cwlogs-writable/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248799956,"owners_count":21163404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloudwatch-logs","nodejs","streams","writable-streams"],"created_at":"2025-01-07T06:55:51.872Z","updated_at":"2025-04-14T00:12:56.316Z","avatar_url":"https://github.com/amekkawi.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cwlogs-writable #\n\nWritable stream for AWS CloudWatch Logs, inspired by [bunyan-cloudwatch](https://github.com/mirkokiefer/bunyan-cloudwatch).\n\n[![Build Status](https://travis-ci.org/amekkawi/cwlogs-writable.svg?branch=v1.0.0)](https://travis-ci.org/amekkawi/cwlogs-writable)\n[![Coverage Status](https://coveralls.io/repos/github/amekkawi/cwlogs-writable/badge.svg?branch=v1.0.0)](https://coveralls.io/github/amekkawi/cwlogs-writable?branch=v1.0.0)\n[![Dependencies Status](https://david-dm.org/amekkawi/cwlogs-writable/v1.0.0/status.svg)](https://david-dm.org/amekkawi/cwlogs-writable/v1.0.0)\n[![Optional Dependencies Status](https://david-dm.org/amekkawi/cwlogs-writable/v1.0.0/optional-status.svg)](https://david-dm.org/amekkawi/cwlogs-writable/v1.0.0?type=optional)\n\n* [Features](#features)\n* [API Docs](#api-docs)\n* [Quick Start](#quick-start)\n* [Bunyan Example](#bunyan-example)\n* [Capturing Log Record Stringification Errors](#capturing-log-record-stringification-errors)\n* [Picking LogStream Names](#picking-logstream-names)\n* [Recovering from Errors](#recovering-from-errors)\n* [Custom Handling of InvalidSequenceTokenException AWS Errors](#custom-handling-of-invalidsequencetokenexception-aws-errors)\n* [CWLogsWritable Options](#cwlogswritable-options)\n* [Change Log](CHANGELOG.md)\n* [License](#license)\n\n---\n\n## Features ##\n\n* Uses [aws-sdk](https://www.npmjs.com/package/aws-sdk).\n* Can be used anywhere Writable streams are allowed.\n* Allows for recovery from AWS errors.\n* Creates log groups and streams if they do not exist.\n* Filtering of log events by the stream itself.\n* Safe stringification of log events.\n\n## API Docs ##\n\nThere are two forms of the API docs:\n\n* [Normal API docs](docs/api.md) - Use this if you using cwlogs-writable\n  as-is and not customizing/extending it's functionality.\n* [Extended API docs](docs/api-protected.md) - Use this to also view\n  `protected` methods that you can use to customize/extend cwlogs-writable.\n\n## Quick Start ##\n\nInstall the library using NPM into your existing node project:\n\n```\nnpm install cwlogs-writable\n```\n\nCreate and write to the stream.\n\n```javascript\nvar CWLogsWritable = require('cwlogs-writable');\n\n// Make stream name as unique as possible (see \"Picking LogStream Names\").\nvar streamName = 'my-log-stream/' + Date.now()\n  + '/' + Math.round(Math.random() * 4026531839 + 268435456).toString(16);\n\nvar stream = new CWLogsWritable({\n  logGroupName: 'my-aws-log-group',\n  logStreamName: streamName,\n\n  // Options passed to the AWS.CloudWatchLogs service.\n  cloudWatchLogsOptions: {\n    // Change the AWS region as needed.\n    region: 'us-east-1',\n\n    // Example authenticating using access key.\n    accessKeyId: '{AWS-IAM-USER-ACCESS-KEY-ID}',\n    secretAccessKey: '{AWS-SECRET-ACCESS-KEY}'\n  }\n});\n\nstream.write('example-log-message');\n```\n\nAlso consider this checklist:\n\n* Are my log stream names [unique enough](#picking-logstream-names)?\n* What if a log record [stringification fails](#capturing-log-record-stringification-errors)?\n* Should my logging [recover from errors](#recovering-from-errors) or fail fast?\n\n## Bunyan Example ##\n\n```javascript\nvar bunyan = require('bunyan');\nvar CWLogsWritable = require('cwlogs-writable');\n\n// Make stream name as unique as possible (see \"Picking LogStream Names\").\nvar streamName = 'my-log-stream/' + Date.now()\n  + '/' + Math.round(Math.random() * 4026531839 + 268435456).toString(16);\n\nvar logger = bunyan.createLogger({\n  name: 'foo',\n  streams: [\n    {\n      level: 'debug',\n\n      // If 'raw' the CloudWatch log event timestamp will\n      // be taken from the bunyan JSON (i.e. rec.time).\n      type: 'raw',\n\n      stream: new CWLogsWritable({\n        logGroupName: 'my-aws-log-group',\n        logStreamName: streamName,\n        cloudWatchLogsOptions: { /* ... */ }\n      })\n    }\n  ]\n});\n```\n\n## Picking LogStream Names ##\n\nIn AWS CloudWatch Logs a LogStream represents \"a sequence of log events from a\nsingle emitter of logs\".\n\nThe important part is \"single emitter\", as this implies that log events should\nnot be put into a LogStream concurrently by multiple emitters.\n\nThis is enforced by the [PutLogEvents API action](http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html)\nwhich requires each call to include a \"sequenceToken\". That token is changed\neach time a call is successful, and the new token is used for the next call.\n\nIf an emitter provides an incorrect token, the API will respond with\nan __InvalidSequenceTokenException__.\n\nTo avoid this error, you must pick LogStream names that are unique to the\nemitter or at least include enough randomness.\n\n```javascript\n// Example generation of LogStream name\nvar logStreamName = [\n\n  // Environment identifier (e.g. \"production\")\n  process.env.NODE_ENV || 'development',\n\n  // Current UTC date\n  new Date().toISOString().substr(0, 10),\n\n  // EC2 instance ID, optionally provided as an env variable\n  process.env.EC2_INSTANCE_ID || null,\n\n  // Process ID\n  'p' + process.pid,\n\n  // Random hex string (from \"10000000\" to \"ffffffff\")\n  Math.round(Math.random() * 4026531839 + 268435456).toString(16),\n\n].filter(Boolean).join('/').replace(/[:*]/g, '');\n```\n\n## Capturing Log Record Stringification Errors ##\n\nBefore log records are sent to AWS they must be stringified.\ncwlogs-writable uses safe stringification techniques to handle\ncircular references that would normally cause JSON.stringify to fail.\n\nOther errors thrown during stringification (e.g. one thrown by a\n[property getter](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/get))\nwill also be handled if the optional dependency\n[safe-json-stringify](https://npmjs.com/package/safe-json-stringify)\nis installed.\n\nIf it is not installed, cwlogs-writable will catch the error and emit a\n`stringifyError` event.\n\n```javascript\nvar stream = new CWLogsWritable({ ... });\n\n// Catch errors if safe-json-stringify is not installed\nstream.on('stringifyError', function(err, record) {\n  console.log('Failed to stringify log entry!', err);\n\n  // You can attempt to manually process the record argument,\n  // but be careful as you will may hit the same error.\n  //customBadRecordProcessing(record);\n});\n```\n\n## Recovering from Errors ##\n\nBy default cwlogs-writable will handle the two most common AWS errors,\n`InvalidSequenceTokenException` and `DataAlreadyAcceptedException`, to\ngive your application as much resiliency as possible.\n\nFor all other errors, the default behavior of a CWLogsWritable stream is to\nemit an 'error' event, clear any queued logs, and ignore all further writes to\nthe stream to prevent memory leaks.\n\nTo override this behavior you can provide a `onError` callback that will\nallow you to recover from these errors.\n\n```javascript\nvar CWLogsWritable = require('cwlogs-writable');\n\nfunction onError(err, logEvents, next) {\n  // Use built-in behavior if the error is not\n  // from a PutLogEvents action (logEvents will be null).\n  if (!logEvents) {\n    next(err);\n    return;\n  }\n\n  // Requeue the log events after a delay,\n  // if the queue is small enough.\n  if (this.getQueueSize() \u003c 100) {\n    setTimeout(function() {\n      // Pass the logEvents to the \"next\" callback\n      // so they are added back to the head of the queue.\n      next(logEvents);\n    }, 2000);\n  }\n\n  // Otherwise, log the events to the console\n  // and resume streaming.\n  else {\n    console.error(\n      'Failed to send logEvents: ' +\n      JSON.stringify(logEvents)\n    );\n\n    next();\n  }\n}\n\n// Make stream name as unique as possible (see \"Picking LogStream Names\").\nvar streamName = 'my-log-stream/' + Date.now()\n  + '/' + Math.round(Math.random() * 4026531839 + 268435456).toString(16);\n\nvar stream = new CWLogsWritable({\n  logGroupName: 'my-aws-log-group',\n  logStreamName: streamName,\n  cloudWatchLogsOptions: { /* ... */ },\n\n  // Pass the onError callback to CWLogsWritable\n  onError: onError\n});\n```\n\n## Custom Handling of InvalidSequenceTokenException AWS Errors ##\n\nFrequent `InvalidSequenceTokenException` AWS errors may indicate a\nproblem with the uniqueness of your LogStream name\n(see [Picking LogStream Names](#picking-logstream-names)).\n\nIf you are experiencing throttling on\n[PutLogEvents](http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html)\nor [DescribeLogStreams](http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_DescribeLogStreams.html)\nactions, you may want to add custom handling of `InvalidSequenceTokenException` errors.\n\n```javascript\n// Example of changing the logStreamName on\n// InvalidSequenceTokenException errors\n// to attempt to avoid further collisions.\n\nfunction getLogStreamName() {\n  return 'my-log-stream/' + Date.now()\n    + '/' + Math.round(Math.random() * 4026531839 + 268435456).toString(16);\n}\n\nfunction onError(err, logEvents, next) {\n  // Change the LogStream name to get a new\n  // randomized value, and requeue the log events.\n  if (err.code === 'InvalidSequenceTokenException') {\n    this.logStreamName = getLogStreamName();\n    next(logEvents);\n  }\n\n  // Default to built-in behavior.\n  else {\n    next(err);\n  }\n}\n\nvar stream = new CWLogsWritable({\n  logGroupName: 'my-aws-log-group',\n  logStreamName: getLogStreamName(),\n  cloudWatchLogsOptions: { /* ... */ },\n\n  // Disable the default handling of\n  // InvalidSequenceTokenException errors\n  // so onError will get them instead.\n  retryOnInvalidSequenceToken: false,\n\n  // Pass the onError callback to CWLogsWritable\n  onError: onError\n});\n```\n\n## CWLogsWritable Options ##\n\n- **logGroupName**\n\n   Required  \n   Type: \u003ccode\u003estring\u003c/code\u003e\n\n   AWS CloudWatch [LogGroup](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatchLogs.html#putLogEvents-property) name. It will be created if it doesn't exist.\n\n- **logStreamName**\n\n   Required  \n   Type: \u003ccode\u003estring\u003c/code\u003e\n\n   AWS CloudWatch [LogStream](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatchLogs.html#putLogEvents-property) name. It will be created if it doesn't exist.\n\n- **cloudWatchLogsOptions**\n\n   Optional  \n   Type: \u003ccode\u003eobject\u003c/code\u003e  \n   Default: \u003ccode\u003e{}\u003c/code\u003e\n\n   Options passed to [AWS.CloudWatchLogs](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatchLogs.html#constructor-property) service.\n\n- **writeInterval**\n\n   Optional  \n   Type: \u003ccode\u003estring\u003c/code\u003e | \u003ccode\u003enumber\u003c/code\u003e  \n   Default: \u003ccode\u003e\u0026quot;nextTick\u0026quot;\u003c/code\u003e\n\n   Amount of wait time after a Writable#_write call to allow batching of log events. Must be a positive number or \"nextTick\". If \"nextTick\", `process.nextTick` is used. If a number, `setTimeout` is used.\n\n- **retryableDelay**\n\n   Optional  \n   Type: \u003ccode\u003estring\u003c/code\u003e | \u003ccode\u003enumber\u003c/code\u003e  \n   Default: \u003ccode\u003e150\u003c/code\u003e\n\n- **retryableMax**\n\n   Optional  \n   Type: \u003ccode\u003enumber\u003c/code\u003e  \n   Default: \u003ccode\u003e100\u003c/code\u003e\n\n   Maximum number of times an AWS error marked as \"retryable\" will be retried before the error is instead passed to [CWLogsWritable#onError](docs/api.md#CWLogsWritable+onError).\n\n- **maxBatchCount**\n\n   Optional  \n   Type: \u003ccode\u003enumber\u003c/code\u003e  \n   Default: \u003ccode\u003e10000\u003c/code\u003e\n\n   Maximum number of log events allowed in a single PutLogEvents API call.\n\n- **maxBatchSize**\n\n   Optional  \n   Type: \u003ccode\u003enumber\u003c/code\u003e  \n   Default: \u003ccode\u003e1048576\u003c/code\u003e\n\n   Maximum number of bytes allowed in a single PutLogEvents API call.\n\n- **ignoreDataAlreadyAcceptedException**\n\n   Optional  \n   Type: \u003ccode\u003eboolean\u003c/code\u003e  \n   Default: \u003ccode\u003etrue\u003c/code\u003e\n\n   Ignore `DataAlreadyAcceptedException` errors. This will bypass [CWLogsWritable#onError](docs/api.md#CWLogsWritable+onError). See [cwlogs-writable/issues/10](https://github.com/amekkawi/cwlogs-writable/issues/10).\n\n- **retryOnInvalidSequenceToken**\n\n   Optional  \n   Type: \u003ccode\u003eboolean\u003c/code\u003e  \n   Default: \u003ccode\u003etrue\u003c/code\u003e\n\n   Retry on `InvalidSequenceTokenException` errors. This will bypass [CWLogsWritable#onError](docs/api.md#CWLogsWritable+onError). See [cwlogs-writable/issues/12](https://github.com/amekkawi/cwlogs-writable/issues/12).\n\n- **onError**\n\n   Optional  \n   Type: \u003ccode\u003efunction\u003c/code\u003e\n\n   Called when an AWS error is encountered. Overwrites [CWLogsWritable#onError](docs/api.md#CWLogsWritable+onError) method.\n\n- **filterWrite**\n\n   Optional  \n   Type: \u003ccode\u003efunction\u003c/code\u003e\n\n   Filter writes to CWLogsWritable. Overwrites [CWLogsWritable#filterWrite](docs/api.md#CWLogsWritable+filterWrite) method.\n\n- **objectMode**\n\n   Optional  \n   Type: \u003ccode\u003eboolean\u003c/code\u003e  \n   Default: \u003ccode\u003etrue\u003c/code\u003e\n\n   Passed to the Writable constructor. See https://nodejs.org/api/stream.html#stream_object_mode.\n\n## Change Log ##\n\nSee [CHANGELOG.md](CHANGELOG.md)\n\n## License ##\n\nThe MIT License (MIT)\n\nCopyright (c) 2017 Andre Mekkawi \u0026lt;github@andremekkawi.com\u0026gt;\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famekkawi%2Fcwlogs-writable","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famekkawi%2Fcwlogs-writable","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famekkawi%2Fcwlogs-writable/lists"}