{"id":16759184,"url":"https://github.com/alcalzone/jsonl-db","last_synced_at":"2025-07-22T15:07:56.050Z","repository":{"id":39479056,"uuid":"258458065","full_name":"AlCalzone/jsonl-db","owner":"AlCalzone","description":"Simple JSONL-based key-value store","archived":false,"fork":false,"pushed_at":"2024-12-01T04:17:23.000Z","size":4000,"stargazers_count":15,"open_issues_count":12,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-06T19:27:59.353Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AlCalzone.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"AlCalzone","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":["https://paypal.me/dgriesel"]}},"created_at":"2020-04-24T08:52:19.000Z","updated_at":"2025-05-29T17:32:59.000Z","dependencies_parsed_at":"2023-02-17T01:46:06.942Z","dependency_job_id":"8692c0a8-baf1-42a0-9358-d742e7385476","html_url":"https://github.com/AlCalzone/jsonl-db","commit_stats":{"total_commits":480,"total_committers":4,"mean_commits":120.0,"dds":"0.32499999999999996","last_synced_commit":"1f38725028295776af226c3ad14b2fd6af076278"},"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"purl":"pkg:github/AlCalzone/jsonl-db","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlCalzone%2Fjsonl-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlCalzone%2Fjsonl-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlCalzone%2Fjsonl-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlCalzone%2Fjsonl-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AlCalzone","download_url":"https://codeload.github.com/AlCalzone/jsonl-db/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlCalzone%2Fjsonl-db/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266516232,"owners_count":23941374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T04:07:31.870Z","updated_at":"2025-07-22T15:07:55.826Z","avatar_url":"https://github.com/AlCalzone.png","language":"TypeScript","funding_links":["https://github.com/sponsors/AlCalzone","https://paypal.me/dgriesel"],"categories":[],"sub_categories":[],"readme":"# jsonl-db\n\nSimple JSONL-based key-value store. Uses an append-only file to store the data. With support for database dumps and compressing the db file.\n\n![Build Status](https://action-badges.now.sh/AlCalzone/jsonl-db)\n\u003c!-- [![Coverage Status](https://img.shields.io/coveralls/github/AlCalzone/jsonl-db.svg)](https://coveralls.io/github/AlCalzone/jsonl-db) --\u003e\n[![node](https://img.shields.io/node/v/@alcalzone/jsonl-db.svg) ![npm](https://img.shields.io/npm/v/@alcalzone/jsonl-db.svg)](https://www.npmjs.com/package/@alcalzone/jsonl-db)\n\n\n## Usage\n\nLoad the module:\n\n```ts\nimport { DB } from \"@alcalzone/jsonl-db\";\n```\n\nOpen or create a database file and use it like a  [`Map`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map)\n\n```ts\n// Open\nconst db = new DB(\"/path/to/file\");\nawait db.open();\n// db.isOpen is now true\n\n// and use it\ndb.set(\"key\", value);\ndb.delete(\"key\");\ndb.clear();\nif (db.has(\"key\")) {\n\tresult = db.get(\"key\");\n}\n// ...forEach, keys(), entries(), values(), ...\n```\n\n### Handling invalid data\n\nIf corrupt data is encountered while opening the DB, the call to `open()` will be rejected. If this is to be expected, use the options parameter on the constructor to turn on forgiving behavior:\n```ts\nconst db = new DB(\"/path/to/file\", { ignoreReadErrors: true });\nawait db.open();\n```\n**Warning:** This may result in inconsistent data since invalid lines are silently ignored.\n\n### Support custom objects/values\n\nYou can optionally transform the parsed values by passing a reviver function. This allows storing non-primitive objects in the database if those can be transformed to JSON (e.g. by overwriting the `toJSON` method). To control the transformation values before they are saved to the database, use the serializer function. This is necessary for `Map`s, `Set`s, `WeakMap`s and `WeakSet`s.\n```ts\nfunction reviver(key: string, serializedValue: any) {\n\t// MUST return a value. If you don't want to transform `serializedValue`, return it.\n}\n\nfunction serializer(key: string, value: any) {\n\t// MUST return a value. If you don't want to transform `value`, return it.\n}\n\nconst db = new DB(\"/path/to/file\", { reviver, serializer });\nawait db.open();\n```\n\n### Closing the database\n\nData written to the DB is persisted asynchronously. Be sure to call `close()` when you no longer need the database in order to flush all pending writes and close all files:\n\n```ts\nawait db.close();\n```\nNow, `db.isOpen` is `false`. While the db is not open, any calls that access the data will throw an error.\n\n### Controlling file system access\n\nBy default, the database immediately writes to the database file. You can throttle the write accesses using the `throttleFS` constructor option. Be aware that buffered data will be lost in case the process crashes.\n```ts\nconst db = new DB(\"/path/to/file\", { throttleFS: { /* throttle options */ } });\n```\nThe following options exist:\n| Option | Default | Description |\n|-----------------|---------|-------------|\n| `intervalMs` | `0` | Write to the database file no more than every `intervalMs` milliseconds. |\n| `maxBufferedCommands` | `+Infinity` | Force a write after `maxBufferedCommands` have been buffered. This reduces memory consumption and data loss in case of a crash. |\n\nTo create a compressed copy of the database in `/path/to/file.dump`, use the `dump()` method. If any data is written to the db during the dump, it is appended to the dump but most likely compressed.\n\n### Lockfile-related options\n\nA lockfile is used to avoid concurrent access to the DB file. Multiple options exist to control where this lockfile is created and how it is accessed:\n```ts\nconst db = new DB(\"/path/to/file\", { lockfile: { /* lockfile options */ } });\n```\n\n| Option | Default | Description |\n|-----------------|---------|-------------|\n| `directory` | - | Change where the lockfile is created, e.g. to put the lockfile into a `tmpfs`. By default the lockfile is created in the same directory as the DB file. If the directory does not exist, it will be created when opening the DB. |\n| `staleMs` | `10000` | Duration after which the lock is considered stale. Minimum: `2000` |\n| `updateMs` | `staleMs/2` | The interval in which the lockfile's `mtime` will be updated. Range: `1000 ... staleMs/2` |\n| `retries` | `0` | How often to retry acquiring a lock before giving up. The retries progressively wait longer with an exponential backoff strategy. |\n| `retryMinTimeoutMs` | `updateMs/2` or `1000` | The start interval used for retries. Minimum: `100` |\n\n\n### Copying and compressing the database\n\n```ts\nawait db.dump(); // To use the default dump filename `/path/to/file.dump`\nawait db.dump(\"/somewhere/else.jsonl\"); // To use a different filename\n```\n\nAfter a while, the main db file may contain unnecessary entries. The raw number of entries can be read using the `uncompressedSize` property. To remove unnecessary entries, use the `compress()` method.\n\n```ts\nawait db.compress();\n```\n\n**Note:** During this call, `/path/to/file.dump` is overwritten and then renamed, `/path/to/file.bak` is overwritten and then deleted. So make sure you don't have any important data in these files.\n\nThe database can automatically compress the database file under some conditions. To do so, use the `autoCompress` parameter of the constructor options:\n```ts\nconst db = new DB(\"/path/to/file\", { autoCompress: { /* auto compress options */ }});\n```\nThe following options exist (all optional) and can be combined:\n| Option | Default | Description |\n|-----------------|---------|-------------|\n| `sizeFactor` | `+Infinity` | Compress when `uncompressedSize \u003e= size * sizeFactor` |\n| `sizeFactorMinimumSize` | 0 | Configure the minimum size necessary for auto-compression based on size |\n| `intervalMs` | `+Infinity` | Compress after a certain time has passed |\n| `intervalMinChanges` | 1 | Configure the minimum count of changes for auto-compression based on time |\n| `onClose` | `false` | Compress when closing the DB |\n| `onOpen` | `false` | Compress after opening the DB |\n\n### Keeping track of timestamps\n\nThe DB can automatically keep track of the last time a key was written. To do so, set the `enableTimestamps` option to `true`. Every `set` call will then update the timestamp of a value, which can be retrieved using `getTimestamp(key)`. To opt out of this behavior for individual writes, pass `false` as the 3rd argument to `set`.\n\n```ts\ndb.set(\"key\", \"value\", false); // Don't update the timestamp\n```\n\n\u003e **Note**\n\u003e Recording timestamps decreases the DB throughput by ~10...30% depending on how it is used.\n\n### Import / Export\n\nImporting JSON files can be done this way:\n```ts\n// pass a filename, the import will be asynchronous\nawait db.importJson(filename);\n// pass the object directly, the import will be synchronous\ndb.importJson({key: \"value\"});\n```\nIn both cases, existing entries in the DB will not be deleted but will be overwritten if they exist.\n\nExporting JSON files is also possible:\n```ts\nawait db.exportJson(filename[, options]);\n```\nThe file will be overwritten if it exists. The 2nd options argument can be used to control the file formatting. Since `fs-extra`'s `writeJson` is used under the hood, take a look at that [method documentation](https://github.com/jprichardson/node-fs-extra/blob/master/docs/writeJson.md) for details on the options object.\n\n## Changelog\n\n\u003c!--\n\tPlaceholder for next release:\n\t### __WORK IN PROGRESS__\n--\u003e\n### 3.1.1 (2024-01-25)\n* Reduced CPU load while idle (#475)\n\n### 3.1.0 (2023-03-15)\n* Support opt-out of updating timestamps for individual set calls (#415)\n\n### 3.0.0 (2023-03-09)\n* Support recording timestamps for each value (#413)\n* Drop support for Node.js 12 (#414)\n\n### 2.5.3 (2022-09-06)\n* Consider lockfiles from the future to be stale (#371)\n\n### 2.5.2 (2022-05-02)\n* Dependency updates\n\n### 2.5.1 (2022-02-13)\n* Allow setting minimum retry timeout\n* Correct minimum value for the lockfile's stale timeout\n\n### 2.5.0 (2022-02-13)\n* Add the ability to control the lockfile's stale/update timeouts and retrying acquiring a lock\n\n### 2.4.3 (2022-02-09)\n* Errors while automatically compressing the DB are now caught\n\n### 2.4.2 (2022-02-09)\n* Errors while automatically restoring the DB from a backup or dump are now caught\n* Simplified and decoupled the persistence code. Individual commands like `dump` and `compress` are now properly sequenced and should no longer conflict with each other.\n* Increased throughput for primitive entries by ~2x\n\n### 2.4.1 (2021-12-30)\n* Individual writes are now collected in a string and written at once, increasing throughput for larger entries by ~10x.\n\n### 2.4.0 (2021-12-27)\n* Stringifying the individual lines now happens lazily and only when actually necessary, increasing the throughput by 30...50x.\n\n### 2.3.0 (2021-12-19)\n* Add the ability to dump the database to a different location\n\n### 2.2.0 (2021-10-15)\n* Add the ability to specify where the lockfile is created\n\n### 2.1.0 (2021-06-22)\n* When opening the DB, recover from crashes that happened while compressing the DB\n* Ensure that the DB files are flushed to disk when closing or renaming files\n* Tests now work with the real filesystem instead of `mock-fs`, which breaks in Node.js `16.3+`\n* Modernized TypeScript build output\n\n### 2.0.0 (2021-06-19)\nUpdate dependencies and drop support for Node.js 10\n\n### 1.2.5 (2021-05-29)\nPrevent opening one DB file in multiple instances of the DB using lockfiles\n\n### 1.2.4 (2021-02-15)\nReduced the work done while opening a DB\n\n### 1.2.3 (2021-01-02)\nFixed a crash that happens while compressing the DB when the `.bak` file exists\n\n### 1.2.2 (2020-10-16)\nWhen consuming this library without `skipLibCheck`, `@types/fs-extra` is no longer required\n\n### 1.2.1 (2020-08-20)\nUpdate dependencies\n\n### 1.2.0 (2020-05-25)\nAdded an optional serializer function to transform non-primitive objects before writing to the DB file\n\n### 1.1.2 (2020-05-11)\nFixed a timeout leak that would prevent Node.js from exiting\n\n### 1.1.1 (2020-05-07)\nLeading directories are now created if they don't exist\n\n### 1.1.0 (2020-05-02)\nAdded functionality to throttle write accesses\n\n### 1.0.1 (2020-04-29)\nExport `JsonlDBOptions` from the main entry point\n\n### 1.0.0 (2020-04-29)\nAdded auto-compress functionality\n\n### 0.5.1 (2020-04-28)\nFix: The main export no longer exports `JsonlDB` as `DB`.\n\n### 0.5.0 (2020-04-27)\nAdded an optional reviver function to transform non-primitive objects while loading the DB\n\n### 0.4.0 (2020-04-27)\n* Renamed the `DB` class to `JsonlDB`\n* `open()` now skips empty lines\n* `open()` throws an error with the line number when it encounters an invalid line. These errors can be ignored using the new constructor options argument.\n\n### 0.3.0 (2020-04-26)\n* Added `importJson` and `exportJson` methods\n\n### 0.2.0 (2020-04-25)\n* Added `isOpen` property\n\n### 0.1.3 (2020-04-25)\n* Writes that happen while `compress()` replaces files are now persisted\n\n### 0.1.2 (2020-04-25)\n* `compress()` no longer overwrites the main file while the DB is being closed\n\n### 0.1.1 (2020-04-25)\n* Fixed some race conditions\n\n### 0.1.0 (2020-04-25)\nFirst official release\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falcalzone%2Fjsonl-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falcalzone%2Fjsonl-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falcalzone%2Fjsonl-db/lists"}