{"id":17109669,"url":"https://github.com/msimerson/safe-log-reader","last_synced_at":"2026-04-30T03:02:25.348Z","repository":{"id":35121134,"uuid":"39315390","full_name":"msimerson/safe-log-reader","owner":"msimerson","description":"Safe Log Reader","archived":false,"fork":false,"pushed_at":"2020-01-28T20:59:45.000Z","size":120,"stargazers_count":7,"open_issues_count":3,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-26T20:21:23.057Z","etag":null,"topics":["filewatcher","logfile","reader"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/safe-log-reader","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"SheetJS/js-xlsx","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/msimerson.png","metadata":{"files":{"readme":"README.md","changelog":"Changes.md","contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"msimerson"}},"created_at":"2015-07-18T23:00:26.000Z","updated_at":"2021-04-21T01:25:33.000Z","dependencies_parsed_at":"2022-09-16T14:11:14.878Z","dependency_job_id":null,"html_url":"https://github.com/msimerson/safe-log-reader","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msimerson%2Fsafe-log-reader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msimerson%2Fsafe-log-reader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msimerson%2Fsafe-log-reader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msimerson%2Fsafe-log-reader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/msimerson","download_url":"https://codeload.github.com/msimerson/safe-log-reader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248658375,"owners_count":21140931,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["filewatcher","logfile","reader"],"created_at":"2024-10-14T16:24:00.513Z","updated_at":"2026-04-30T03:02:20.272Z","avatar_url":"https://github.com/msimerson.png","language":"JavaScript","funding_links":["https://github.com/sponsors/msimerson"],"categories":[],"sub_categories":[],"readme":"[![Build Status][ci-img]][ci-url]\n[![Code Coverage][cov-img]][cov-url]\n[![Code Climate][clim-img]][clim-url]\n[![NPM][npm-img]][npm-url]\n\n# Safe Log Reader\n\nRead plain or compressed log files from disk, deliver as [batches of] lines to a log consumer. Wait for the log consumer to report success. Advance bookmark. Repeat ad infinitum.\n\n# Install\n\n    npm i safe-log-reader\n\n# Usage\n\n```js\nconst read = require('safe-line-reader');\nread.createReader(filePath, {\n    batchLimit: 1024,\n    bookmark: {\n        dir: path.resolve('someDir', '.bookmark'),\n    }\n})\n.on('readable', function () { this.readLine(); })\n.on('read', function (line, count) {\n    // do something with this line of text\n})\n.on('end', function (done) {\n    // close up shop and go home\n});\n```\n\n## Features\n\n- [x] Read plain text files\n- [x] Handles common log file events\n    - [x] reads growing files (aka: tail -F)\n    - [x] reads rotated logs\n        - [x] reads the new file when it appears\n            - [x] fs.watch tested on:\n                - [x] Mac OS X\n                - [x] Linux\n                - [x] FreeBSD\n        - [ ] continues reading old log file until quiet\n    - [ ] file truncation (echo '' \u003e foo.log)\n    - [x] watches for non-existent log to appear\n- [x] Read compressed log files\n    - [x] gzip (zlib)\n    - [ ] bzip2\n- [x] Emits data as lines, upon request (paused mode streaming)\n    - [x] Uses a [Transform Stream](https://nodejs.org/api/stream.html#stream_class_stream_transform_1) to efficiently convert buffer streams to lines\n    - [x] waits for confirmation, then advances bookmarks\n- [x] handles utf-8 multibyte characters properly\n- [x] Remembers previously read files (bookmarks)\n    - [x] Perists across program restarts\n        - [x] identifies files by inode\n        - [x] saves file data: name, size, line count, inode\n    - [x] When safe, uses byte position to efficiently resume reading\n- [ ] cronolog naming syntax (/var/log/http/YYYY/MM/DD/access.log)\n    - [ ] watches existing directory ancestor\n- [ ] winston naming syntax (app.log1, app.log2, etc.)\n- [x] zero dependencies\n\n# Shippers\n\n- [x] [log-ship-elastic-postfix](https://github.com/msimerson/log-ship-elastic-postfix)\n    - reads batches of log entries\n    - parses with [postfix-parser](https://github.com/msimerson/postfix-parser)\n    - fetches matching docs from ES\n    - updates/creates normalized postfix docs\n    - saves docs to elasticsearch\n- [x] [log-ship-elastic-qpsmtpd](https://github.com/msimerson/log-ship-elastic-qpsmtpd)\n    - receives JSON parsed log lines\n    - saves to elasticsearch\n\n# Similar Projects\n\n* [tail-stream](https://github.com/Juul/tail-stream) has good options for\n  reading a file and handling rotation, truncation, and resuming. It had no\n  tests so I wrote them and most have been merged. Bugs remain\n  (demonstrated with Travis-CI test integration) unresolved. The author\n  offered a license in exchange for the tests but the GPL is problematic.\n* [tail-forever](https://github.com/mingqi/tail-forever) has character\n  encoding detection and very basic file watching.\n* [always-tail](https://github.com/jandre/always-tail)\n\nThe key \"missing\" feature of the node \"tail\" libraries is the ability to\nresume correctly after the app has stopped reading (think: kill -9)\nin the middle of a file.\n\nBecause files are read as [chunks of] bytes and log entries are lines,\nresuming at the files last byte position is likely to be in the middle of a\nline, or even splitting a multi-byte character. Extra buffered bytes not yet\nemitted as lines are lost, unless at restart, one rewinds and replays the\nlast full $bufferSize. Then the consuming app needs to have duplicate line\ndetection and suppression.\n\nThe key to resuming reading a log file _safely_ is tracking line numbers and\nthe byte steam offset the consuming app has committed. When saving bookmarks,\nthe file position advances to the byte offset coinciding with the byte\nposition of the last line your application has safely commited.\n\nSafe-log-reader uses a Transform Stream to convert the byte stream into\nlines. This makes it dead simple to read compressed files by adding\na `.pipe(ungzip())` into the stream.\n\n\n\u003csub\u003eCopyright 2015 by eFolder, Inc.\u003c/sub\u003e\n\n\n[ci-img]: https://travis-ci.org/msimerson/safe-log-reader.svg\n[ci-url]: https://travis-ci.org/msimerson/safe-log-reader\n[cov-img]: https://codecov.io/github/msimerson/safe-log-reader/badge.svg\n[cov-url]: https://codecov.io/github/msimerson/safe-log-reader\n[clim-img]: https://codeclimate.com/github/msimerson/safe-log-reader/badges/gpa.svg\n[clim-url]: https://codeclimate.com/github/msimerson/safe-log-reader\n[npm-img]: https://nodei.co/npm/safe-log-reader.png\n[npm-url]: https://www.npmjs.com/package/safe-log-reader\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsimerson%2Fsafe-log-reader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmsimerson%2Fsafe-log-reader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsimerson%2Fsafe-log-reader/lists"}