{"id":24659115,"url":"https://github.com/coggle/s3-streamlogger","last_synced_at":"2025-11-03T23:04:17.338Z","repository":{"id":20709825,"uuid":"23993741","full_name":"Coggle/s3-streamlogger","owner":"Coggle","description":"Node.js stream for logging to s3 with rotated object names. Usable as a winston-file stream.","archived":false,"fork":false,"pushed_at":"2025-03-19T11:24:58.000Z","size":320,"stargazers_count":82,"open_issues_count":2,"forks_count":42,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-09T17:55:19.491Z","etag":null,"topics":["aws","javascript","logging","nodejs","s3"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Coggle.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-09-13T12:28:07.000Z","updated_at":"2025-03-19T11:25:01.000Z","dependencies_parsed_at":"2024-05-28T17:27:31.762Z","dependency_job_id":"12dffaa2-bb8b-46f0-b023-c2bb58c802d9","html_url":"https://github.com/Coggle/s3-streamlogger","commit_stats":{"total_commits":92,"total_committers":19,"mean_commits":4.842105263157895,"dds":0.3804347826086957,"last_synced_commit":"485b146fc4ae63d8871caf30b02fff7c7a2ba455"},"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Coggle%2Fs3-streamlogger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Coggle%2Fs3-streamlogger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Coggle%2Fs3-streamlogger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Coggle%2Fs3-streamlogger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Coggle","download_url":"https://codeload.github.com/Coggle/s3-streamlogger/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254459088,"owners_count":22074605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","javascript","logging","nodejs","s3"],"created_at":"2025-01-26T02:31:56.569Z","updated_at":"2025-11-03T23:04:17.292Z","avatar_url":"https://github.com/Coggle.png","language":"JavaScript","readme":"## s3-streamlogger\n[![NPM version](https://badge.fury.io/js/s3-streamlogger.svg)](http://badge.fury.io/js/s3-streamlogger)\n\n\nA Writable Stream object that uploads to s3 objects, periodically rotating to a\nnew object name.\n\nSee also [tails3](http://github.com/coggle/tails3) for a script to tail the log\nfiles produced by s3-streamlogger.\n\n### Installation\n```bash\nnpm install --save s3-streamlogger\n```\n\n### Basic Usage\n```js\nconst { S3StreamLogger } = require('s3-streamlogger');\n\nconst s3stream = new S3StreamLogger({\n     bucket: \"mys3bucket\",\n});\n\ns3stream.write(\"hello S3\");\n```\n\n### Use with Winston: Log to S3\n```sh\nnpm install --save winston\nnpm install --save s3-streamlogger\n```\n\n```js\nconst winston            = require('winston');\nconst { S3StreamLogger } = require('s3-streamlogger');\n\nconst s3_stream = new S3StreamLogger({\n     bucket: \"mys3bucket\",\n});\n\nconst transport = new (winston.transports.Stream)({\n  stream: s3_stream\n});\n// see error handling section below\ntransport.on('error', function(err){/* ... */});\n\nconst logger = winston.createLogger({\n  transports: [transport]\n});\n\nlogger.info('Hello Winston!');\n```\n\n### Define subfolder\n```js\nconst { S3StreamLogger } = require('s3-streamlogger');\n\nconst s3stream = new S3StreamLogger({\n     bucket: \"mys3bucket\",\n     folder: \"my/nested/subfolder\"\n});\n\ns3stream.write(\"hello S3\");\n```\n\n### Assign tags\n```js\nconst { S3StreamLogger } = require('s3-streamlogger');\n\nconst s3stream = new S3StreamLogger({\n     bucket: \"mys3bucket\",\n     folder: \"my/nested/subfolder\",\n       tags: {type: 'myType', project: 'myProject'}\n});\n\ns3stream.write(\"hello S3\");\n```\n\n### Add hostname information for tails3\ntails3 expects messages to be logged as json (the default for the file\ntransport), with hostname and (for critical errors), stack properties to each\nlog object, in addition to the standard timestamp, level and message\nproperties. You can provide these using the third \"metadata\" option to\nwinston's log method:\n\n```js\nlogger.log(level, message, {hostname: ... , stack: ...});\n```\n\n### Handling logging errors\nWhen there is an error writing to s3, the stream emits an 'error' event with\ndetails. You should take care **not** to log these errors back to the same\nstream (as that is likely to cause infinite recursion). Instead log them to the\nconsole, to a file, or to SNS using [winston-sns](https://github.com/jesseditson/winston-sns).\n\nNote that these errors will result in uncaught exceptions unless you have an\n`error` event handler registered, for example:\n\n```js\ns3_stream.on('error', function(err){\n    // there was an error!\n    some_other_logging_transport.log('error', 'logging transport error', err)\n});\n```\n\nWhen using s3-streamlogger with the Winston Stream transport, the Stream transport\nattaches its own error handler to the stream, so you do not need your own,\nhowever it will re-emit the errors on itself which must be handled instead:\n\n```js\nconst transport = new (winston.transports.Stream)({\n  stream: s3_stream\n});\ntransport.on('error', function(err){\n  /* handle s3 stream errors (e.g. invalid credentials, EHOSTDOWN) here */\n});\n\nconst logger = winston.createLogger({\n  transports: [transport]\n});\n```\n\n### Options\n\n#### bucket *(required)*\nName of the S3 bucket to upload data to. Must exist.\nCan also be provided as the environment variable `BUCKET_NAME`.\n\n#### folder\nAn optional folder to stream log files to. Takes a path string,\neg: \"my/subfolder\" or \"nested\".\n\n#### tags\nAn optional set of tags to assign to the log files. Takes an object,\neg: `{type: \"myType\"}` or `{type: \"myType\", project: \"myProject\"}`.\n\n#### access_key_id *deprecated*\nAWS access key ID, must have putObject permission on the specified bucket.  Provide\ncredentials through the environment variable `AWS_ACCESS_KEY_ID`, or as any\nof the other [authentication\nmethods](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-credentials-node.html)\nsupported by the AWS SDK instead.\n\n\n#### secret_access_key *deprecated*\nAWS secret key for the `access_key_id` specified.  Provide\ncredentials through the environment variable `AWS_SECRET_ACCESS_KEY`, or as any\nof the other [authentication\nmethods](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-credentials-node.html)\nsupported by the AWS SDK instead.\n\n#### config\n\nConfiguration object for the AWS SDK. The full list of options is available on the [AWS SDK Configuration page](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/global-config-object.html). This is an alternative to using access_key_id and secret_access_key and is overwritten by them if both are used.\n\n#### name_format\nFormat of file names to create, accepts [strftime specifiers](https://github.com/samsonjs/strftime). Defaults to `\"%Y-%m-%d-%H-%M-%S-%L-unknown-\u003chostname\u003e.log\"`. The Date() used to fill the format specifiers is created with the current UTC time, but still *has the current timezone*, so any specifiers that perform timezone conversion will return incorrect dates.\n\nIf you use a format of the form `%Y-%m-%d-%H-%M-\u003cstage\u003e-\u003chostname\u003e.log`, then\nyou can use [tails3](http://github.com/coggle/tails3) to tail the log files\nbeing generated by `S3StreamLogger`.\n\nIf `compress` is set to true, then the default extension is `.log.gz` instead of\n`.log`.\n\n#### rotate_every\nFiles will be rotated every `rotate_every` milliseconds. Defaults to 3600000 (60\nminutes).\n\n#### max_file_size\nFiles will be rotated when they reach `max_file_size` bytes. Defaults to 200000 (i.e. 200 KB).\n\n#### upload_every\nFiles will be uploaded every `upload_every` milliseconds. Defaults to 20\nseconds.\n\n#### buffer_size\nFiles will be uploaded if the un-uploaded data exceeds `buffer_size` bytes.\nDefaults to 10 KB.\n\n#### server_side_encryption\nThe server side encryption `AES256` algorithm used when storing objects in S3.\nDefaults to false.\n\n#### storage_class\nThe S3 StorageClass (STANDARD, REDUCED_REDUNDANCY, etc.). If omitted, no value\nis used and aws-sdk will fill in its default.\n\n#### acl\nThe canned ACL (access control list) to apply to uploaded objects.\nDefaults to no ACL.\n\n#### compress\nIf true, the files will be gzipped before uploading (may reduce s3 storage costs).\nDefaults to false.\n\n## Changelog\n * **1.11.1**: Include typescript definitions in the files published to npm.\n * **1.11.0**: Adds typescript definitions.\n * **1.10.1**: Corrects readme.\n * **1.10.0**: Removes use of deprecated git-branch module. The default filename format no longer includes the git branch name.\n\n## License\n[ISC](http://opensource.org/licenses/ISC): equivalent to 2-clause BSD.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoggle%2Fs3-streamlogger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoggle%2Fs3-streamlogger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoggle%2Fs3-streamlogger/lists"}