{"id":18087893,"url":"https://github.com/jamesbontempo/njodb","last_synced_at":"2025-05-06T20:43:36.518Z","repository":{"id":42648400,"uuid":"338457610","full_name":"jamesbontempo/njodb","owner":"jamesbontempo","description":"A persistent, partitioned, concurrency-controlled, Node.js JSON object database","archived":false,"fork":false,"pushed_at":"2025-02-26T05:52:22.000Z","size":556,"stargazers_count":16,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-01T05:17:56.287Z","etag":null,"topics":["database","json","node","object"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jamesbontempo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-12T23:29:21.000Z","updated_at":"2025-04-09T10:17:49.000Z","dependencies_parsed_at":"2024-04-09T05:32:35.465Z","dependency_job_id":"342af9b5-5693-4437-b3c0-c0fe01563481","html_url":"https://github.com/jamesbontempo/njodb","commit_stats":{"total_commits":90,"total_committers":6,"mean_commits":15.0,"dds":0.0888888888888889,"last_synced_commit":"95ee05dda0faed6ff58f5fee80fc110ed0984ffb"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesbontempo%2Fnjodb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesbontempo%2Fnjodb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesbontempo%2Fnjodb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesbontempo%2Fnjodb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jamesbontempo","download_url":"https://codeload.github.com/jamesbontempo/njodb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252768988,"owners_count":21801373,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","json","node","object"],"created_at":"2024-10-31T17:09:21.113Z","updated_at":"2025-05-06T20:43:36.453Z","avatar_url":"https://github.com/jamesbontempo.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# njodb\n\n![Version](https://img.shields.io/github/package-json/v/jamesbontempo/njodb?color=blue) ![Coverage](https://img.shields.io/codecov/c/github/jamesbontempo/njodb/\u0026lt;master\u003e.svg?style=flat) ![Dependencies](https://img.shields.io/librariesio/release/npm/njodb) ![License](https://img.shields.io/github/license/jamesbontempo/njodb?color=red)\n\n`njodb` is a persistent, partitioned, concurrency-controlled, Node JSON object database. Data is written to the file system and distributed across multiple files that are protected by read and write locks. By default, all methods are asynchronous and use read/write streams to improve performance and reduce memory requirements (this should be particularly useful for large databases).\n\n## What makes `njodb` different than other Node JSON object databases?\n\n**Persistence** - Data is saved to the file system so that it remains after the application that created it is no longer running, unlike the many existing in-memory solutions. This persistence also allows data to be made available to other applications.\n\n**Asynchronous and streaming** - By default, all methods are asynchronous and non-blocking, and also use read and write streams, to make data access and manipulation efficient. Synchronous methods are also provided for those cases where they are desired or appropriate.\n\n**JSON records, not JSON files** - Records are stored as individual lines of JSON objects in a file, so a read stream can be used to retrieve data rapidly, parse it in small chunks, and dispense with it when done. This removes the time and memory overhead required by solutions that store data as a single, monolithic JSON object. They must read all of that data into memory, and then parse all of it, before allowing you to use any of it.\n\n**Completely schemaless** - While the JSON data itself is schemaless, it is also the case that data is not siloed into tables, or forced into collections, so the entire database, from top to bottom, is schemaless, not just the data. For a user or application, this means that there is no need to know anything about the database structure, only what data is being sought.\n\n**Balanced partitions** - When inserting data, records are randomly distributed across partitions so that partition sizes are kept roughly equal, making data access times consistent. Manually resizing the database performs a similar distribution, so, as the database grows or shrinks, the partitions remain well-balanced.\n\n**Concurrency-controlled** - File locks are used during read and write operations, ensuring data integrity can be maintained in a multi-user/multi-application environment. There are few, if any, existing solutions that are designed for such scenarios and that include this sort of data protection.\n\n`njodb` even has its own command-line interface: check out [njodb-cli](https://www.npmjs.com/package/njodb-cli).\n\n## Table of contents\n- [Install](#install)\n- [Test](#test)\n- [Introduction](#introduction)\n- [Constructor](#constructor)\n  - [Database properties](#database-properties)\n- [Database management methods](#database-management-methods)\n  - [stats](#stats) / [statsSync](#statsSync)\n  - [grow](#grow) / [growSync](#growSync)\n  - [shrink](#shrink) / [shrinkSync](#shrinkSync)\n  - [resize](#resize) / [resizeSync](#resizeSync)\n  - [drop](#drop) / [dropSync](#dropSync)\n  - [getProperties](#getproperties) / [setProperties](#setproperties)\n- [Data manipulation methods](#data-manipulation-methods)\n  - [insert](#insert) / [insertSync](#insertSync)\n  - [insertFile](#insertFile) / [insertFileSync](#insertFileSync)\n  - [select](#select) / [selectSync](#selectSync)\n  - [update](#update) / [updateSync](#updateSync)\n  - [delete](#delete) / [deleteSync](#deleteSync)\n  - [aggregate](#aggregate) / [aggregateSync](#aggregateSync)\n- [Finding and fixing problematic data](#finding-and-fixing-problematic-data)\n\n## Install\n```js\nnpm install njodb\n```\n\n## Test\n```js\nnpm test\n```\n\n## Introduction\nLoad the module:\n```js\nconst njodb = require(\"njodb\");\n```\n\nCreate an instance of an NJODB:\n```js\nconst db = new njodb.Database();\n```\n\nCreate some JSON data objects:\n```js\nconst data = [\n    {\n        id: 1,\n        name: \"James\",\n        nickname: \"Good Times\",\n        modified: Date.now()\n    },\n    {\n        id: 2,\n        name: \"Steve\",\n        nickname: \"Esteban\",\n        modified: Date.now()\n    }\n];\n```\n\nInsert them into the database:\n```js\ndb.insert(data).then(results =\u003e /* do something */ );\n```\n\nSelect some records from the database by supplying a function to find matches:\n```js\ndb.select(\n    record =\u003e record.id === 1 || record.name === \"Steve\"\n).then(results =\u003e /* do something */ );\n```\n\nUpdate some records in the database by supplying a function to find matches and another function to update them:\n```js\ndb.update(\n    record =\u003e record.name === \"James\",\n    record =\u003e { record.nickname = \"Bulldog\"; return record; }\n).then(results =\u003e /* do something */ );\n```\n\nDelete some records from the database by supplying a function to find matches:\n```js\ndb.delete(\n    record =\u003e record.modified \u003c Date.now()\n).then(results =\u003e /* do something */ );\n```\n\nDelete the database:\n```js\ndb.drop().then(results =\u003e /* do something */ );\n```\n\n## Constructor\nCreates a new instance of an NJODB `Database`.\n\nParameters:\n\nName|Type|Description|Default\n----|----|-----------|-------\n`root`|string|Path to the root directory of the `Database`|`process.cwd()`\n`properties`|object|User-specific properties to set for the `Database`|`{}` (see [Database properties](#database-properties))\n\nIf an `njodb.properties` file already exists in the `root` directory, a connection to the existing `Database` will be created. If the `root` directory does not exist it will be created. If no user-specific properties are supplied, an `njodb.properties` file will be created using default values; otherwise, the user-supplied properties will be merged with the default values (see [Database properties](#database-properties) below). If the data and temp directories do not exist, they will be created.\n\nExample:\n\n```js\nconst db = new njodb.Database() // created in or connected to the current directory\n\nconst db = new njodb.Database(\"/path/to/some/other/place\", {datadir: \"mydata\", datastores: 2}) // created or connected to elsewhere with user-supplied properties\n```\n\n### Database properties\nAn NJODB `Database` has several properties that control its functioning. These properties can be set explicitly in the `njodb.properties` file in the `root` directory; otherwise, default properties will be used. For a newly created `Database`, an `njodb.properties` file will be created using default values.\n\nProperties:\n\nName|Type|Description|Default\n----|----|-----------|-------\n`datadir`|string|The name of the subdirectory of `root` where data files will be stored|`data`\n`dataname`|string|The file name that will be used when creating or accessing data files|`data`\n`datastores`|number|The number of data partitions that will be used|`5`\n`tempdir`|string|The name of the subdirectory of `root` where temporary data files will be stored|`tmp`\n`lockoptions`|object|The options that will be used by [proper-lockfile](https://www.npmjs.com/package/proper-lockfile) to lock data files|`{\"stale\": 5000, \"update\": 1000, \"retries\": { \"retries\": 5000, \"minTimeout\": 250, \"maxTimeout\": 5000, \"factor\": 0.15, \"randomize\": false } }`\n\n\n## Database management methods\n\n### stats\n\n`stats`\n\nReturns statistics about the `Database`. Resolves with the following information:\n\nName|Description\n----|-----------\n`root`|The path of the root directory of the `Database`\n`data`|The path of the data subdirectory of the `Database`\n`temp`|The path of the temp subdirectory of the `Database`\n`records`|The number of records in the `Database` (the sum of the number of records in each `datastore`)\n`errors`|The number of problematic records in the `Database`\n`size`|The total size of the `Database` in \"human-readable\" format (the sum of the sizes of the individual `datastores`)\n`stores`|The total number of `datastores` in the `Database`\n`min`|The minimum number of records in a `datastore`\n`max`|The maximum number of records in a `datastore`\n`mean`|The mean (i.e., average) number of records in each `datastore`\n`var`|The variance of the number of records across `datastores`\n`std`|The standard deviation of the number of records across `datastores`\n`start`|The timestamp of when the `stats` call started\n`end`|The timestamp of when the `stats` call finished\n`elapsed`|The amount of time in milliseconds required to run the `stats` call\n`details`|An array of detailed stats for each `datastore`\n\n### statsSync\n\nA synchronous version of `stats`.\n\n### grow\n\n`grow()`\n\nIncreases the number of `datastores` by one and redistributes the data across them.\n\n### growSync\n\n`growSync()`\n\nA synchronous version of `grow`.\n\n### shrink\n\n`shrink()`\n\nDecreases the number of `datastores` by one and redistributes the data across them. If the current number of `datastores` is one, calling `shrink()` will throw an error.\n\n### shrinkSync\n\n`shrinkSync()`\n\nA synchronous version of `shrink`.\n\n### resize\n\n`resize(size)`\n\nChanges the number of `datastores` and redistributes the data across them.\n\nParameters:\n\nName|Type|Description\n----|----|-----------\n`size`|number|The number of `datastores` (must be greater than zero)\n\n### resizeSync\n\n`resizeSync(size)`\n\nA synchronous version of `resize`.\n\n### drop\n\n`drop()`\n\nDeletes the database, including the data and temp directories, and the properties file.\n\n### dropSync\n\n`dropSync()`\n\nA synchronous version of `drop`.\n\n### getProperties\n\n`getProperties()`\n\nReturns the properties set for the `Database`. Will likely be deprecated in a future version.\n\n### setProperties\n\n`setProperties(properties)`\n\nSets the properties for the the `Database`. Will likely be deprecated in a future version.\n\nParameters:\n\nName|Type|Description|Default\n----|----|-----------|-------\n`properties`|object|The properties to set for the `Database`|See [Database properties](#database-properties)\n\n\n## Data manipulation methods\n\n### insert\n\n`insert(data)`\n\nInserts data into the `Database`.\n\nParameters:\n\nName|Type|Description\n----|----|-----------\n`data`|array|An array of JSON objects to insert into the `Database`\n\nResolves with an object containing results from the `insert`:\n\nName|Type|Description\n----|----|-----------\n`inserted`|number|The number of objects inserted into the `Database`\n`start`|date|The timestamp of when the insertions began\n`end`|date|The timestamp of when the insertions finished\n`elapsed`|number|The amount of time in milliseconds required to execute the `insert`\n`details`|array|An array of insertion results for each individual `datastore`\n\n### insertSync\n\n`insertSync(data)`\n\nA synchronous version of `insert`.\n\n### insertFile\n\n`insertFile(file)`\n\nInserts data into the `database` from a file containing JSON data. The file itself does not need to be a valid JSON object, rather it should contain a single stringified JSON object per line. Blank lines are ignored and problematic data is collected in an `errors` array.\n\nResolves with an object containing results from the `insertFile`:\n\nName|Type|Description\n----|----|-----------\n`inspected`|number|The number of lines of the file inspected\n`inserted`|number|The number of objects inserted into the `Database`\n`blanks`|number|The number of blank lines in the file\n`errors`|array|An array of problematic records in the file\n`start`|date|The timestamp of when the insertions began\n`end`|date|The timestamp of when the insertions finished\n`elapsed`|number|The amount of time in milliseconds required to execute the `insert`\n`details`|array|An array of insertion results for each individual `datastore`\n\nAn example data file, `data.json`, is included in the `test` subdirectory. Among many valid records, it also includes blank lines and a malformed JSON object. To insert its data into the `database`:\n\n```js\ndb.insertFile(\"./test/data.json\").then(results =\u003e /* do something */ );\n```\n\n### insertFileSync\n\n`insertFileSync(file)`\n\nA synchronous version of `insertFile`.\n\n### select\n\n`select(selecter [, projector])`\n\nSelects data from the `Database`.\n\nParameters:\n\nName|Type|Description\n----|----|-----------\n`selecter`|function|A function that returns a boolean that will be used to identify the records that should be returned\n`projecter`|function| A function that returns an object that identifies the fields that should be returned\n\nResolves with an object containing results from the `select`:\n\nName|Type|Description\n----|----|-----------\n`data`|array|An array of objects selected from the `Database`\n`selected`|number|The number of objects selected from the `Database`\n`ignored`|number|The number of objects that were not selected from the `Database`\n`errors`|array|An array of problematic (i.e., un-parseable) records in the `Database`\n`start`|date|The timestamp of when the selections began\n`end`|date|The timestamp of when the selections finished\n`elapsed`|number|The amount of time in milliseconds required to execute the `select`\n`details`|array|An array of selection results, including error details, for each individual `datastore`\n\nExample with projection that selects all records, returns only the `id` and `modified` fields, but also creates a new one called `newID`:\n\n```js\ndb.select(\n    () =\u003e true,\n    record =\u003e { return {id: record.id, newID: record.id + 1, modified: record.modified }; }\n);\n```\n\n### selectSync\n\n`selectSync(selecter [, projector])`\n\nA synchronous version of `select`.\n\n### update\n\n`update(selecter, updater)`\n\nUpdates data in the `Database`.\n\nParameters:\n\nName|Type|Description\n----|----|-----------\n`selecter`|function|A function that returns a boolean that will be used to identify the records that should be updated\n`updater`|function|A function that applies an update to a selected record and returns it\n\nResolves with an object containing results from the `update`:\n\nName|Type|Description\n----|----|-----------\n`selected`|number|The number of objects selected from the `Database` for updating\n`updated`|number|The number of objects updated in the `Database`\n`unchanged`|number|The number of objects that were not updated in the `Database`\n`errors`|array|An array of problematic (i.e., un-parseable) records in the `Database` or records that were unable to be updated\n`start`|date|The timestamp of when the updates began\n`end`|date|The timestamp of when the updates finished\n`elapsed`|number|The amount of time in milliseconds required to execute the `update`\n`details`|array|An array of update results, including error details, for each individual `datastore`\n\n### updateSync\n\n`updateSync(selecter, updater)`\n\nA synchronous version of `update`\n\n### delete\n\n`delete(selecter)`\n\nDeletes data from the `Database`.\n\nParameters:\n\nName|Type|Description\n----|----|-----------\n`selecter`|function|A function that returns a boolean that will be used to identify the records that should be deleted\n\nResolves with an object containing results from the `delete`:\n\nName|Type|Description\n----|----|-----------\n`deleted`|number|The number of objects deleted from the `Database`\n`retained`|number|The number of objects that were not deleted from the `Database`\n`errors`|array|An array of problematic (i.e., un-parseable) records in the `Database` or records that were unable to be deleted\n`start`|date|The timestamp of when the deletions began\n`end`|date|The timestamp of when the deletions finished\n`elapsed`|number|The amount of time in milliseconds required to execute the `delete`\n`details`|array|An array of deletion results, including error details, for each individual `datastore`\n\n### deleteSync\n\n`deleteSync(selecter)`\n\nA synchronous version of `delete`.\n\n### aggregate\n\n`aggregate(selecter, indexer [, projecter])`\n\nAggregates data in the database.\n\nParameters:\n\nName|Type|Description\n----|----|-----------\n`selecter`|function|A function that returns a boolean that will be used to identify the records that should be aggregated\n`indexer`|function| A function that returns an object that creates the index by which data will be grouped\n`projecter`|function| A function that returns an object that identifies the fields that should be returned\n\nResolves with an object containing results from the `aggregate`:\n\nName|Type|Description\n----|----|-----------\n`data`|array|An array of index objects selected from the `Database`\n`indexed`|number|The number of records that were indexable (i.e., processable by the indexer function)\n`unindexed`|number|The number of records that were un-indexable\n`errors`|number|The number of problematic (i.e., un-parseable) records in the `Database`\n`start`|date|The timestamp of when the aggregations began\n`end`|date|The timestamp of when the aggregations finished\n`elapsed`|number|The amount of time in milliseconds required to execute the `aggregate`\n`details`|array|An array of selection results, including error details, for each individual `datastore`\n\nEach index object contains the following:\n\nName|Type|Description\n----|----|-----------\n`index`|any valid type|The value of the index created by the indexer function\n`count`|number|The count of records that contained the index\n`data`|array|An array of aggregation objects for each field of the records returned\n\nEach aggregation object contains one or more of the following (non-numeric fields do not contain numeric aggregate data):\n\nName|Type|Description\n----|----|-----------\n`min`|any valid type|Minimum value of the field\n`max`|any valid type|Maximum value of the field\n`sum`|number|The sum of the values of the field (undefined if not a number)\n`mean`|number|The mean (i.e., average) of the values of the field (undefined if not a number)\n`varp`|number|The population variance of the values of the field (undefined if not a number)\n`vars`|number|The sample variance of the values of the field (undefined if not a number)\n`stdp`|number|The population standard deviation of the values of the field (undefined if not a number)\n`stds`|number|The sample standard deviation of the values of the field (undefined if not a number)\n\nAn example that generates aggregates for all records and fields, grouped by state and lastName:\n```js\ndb.aggregate(\n    () =\u003e true,\n    record =\u003e [record.state, record.lastName]\n);\n```\n\nAnother example that generates aggregates for records with an ID less than 1000, grouped by state, but for only two fields (note the non-numeric fields do not include numeric aggregate data):\n```js\ndb.aggregate(\n    record =\u003e record.id \u003c 1000,\n    record =\u003e record.state,\n    record =\u003e { return {favoriteNumber: record.favoriteNumber, firstName: record.firstName}; }\n);\n```\n\nExample aggregate data array:\n```js\n[\n    {\n        index: \"Maryland\",\n        count: 50,\n        aggregates: [\n            {\n                field: \"favoriteNumber\",\n                data: {\n                      min: 0,\n                      max: 98,\n                      sum: 2450,\n                      mean: 49,\n                      varp: 833,\n                      vars: 850,\n                      stdp: 28.861739379323623,\n                      stds: 29.154759474226502\n                  }\n            },\n            {\n                field: \"firstName\",\n                data: {\n                    min: \"Elizabeth\",\n                    max: \"William\"\n                }\n            }\n        ]\n    },\n    {\n        index: \"Virginia\",\n        count: 50,\n        aggregates: [\n            {\n                field: \"favoriteNumber\",\n                data: {\n                    min: 0,\n                    max: 49,\n                    sum: 1225,\n                    mean: 24.5,\n                    varp: 208.25000000000003,\n                    vars: 212.50000000000003,\n                    stdp: 14.430869689661813,\n                    stds: 14.577379737113253\n                }\n            },\n            {\n                field: \"firstName\",\n                data: {\n                    min: \"James\",\n                    max: \"Robert\"\n                }\n            }\n        ]\n    }\n]\n```\n\n### aggregateSync\n\n`aggregate(selecter, indexer [, projecter])`\n\nA synchronous version of `aggregate`.\n\n## Finding and fixing problematic data\n\nMany methods return information about problematic records encountered (e.g., records that are not parseable using `JSON.parse()`, or ones that couldn't be updated or deleted); both a count of them, as well as details about them in the `details` array. The objects in the `details` array - one for each `datastore` - contain an `errors` array that is a collection of objects about problematic records.\n\nFor un-parseable records, each error object includes the line of the `datastore` file where the problematic record was found as well as a copy of the record itself. With this information, if one wants to address these problematic data they can simply load the `datastore` file in a text editor and either correct the record or remove it. For records that couldn't be deleted or updated, each error object includes a copy of the record itself. With this information, one could make another attempt to update or delete the record(s), or otherwise handle the failure.\n\nHere is an example of the `details` for a `datastore` that contains an un-parseable record. As you can see, the record is on the tenth line of the file, and the problem is that the `lastname` key name is missing an enclosing quote. Simply adding the quote fixes the record.\n\n```js\n{\n  store: '/Users/jamesbontempo/github/njodb/data/data.0.json',\n  size: 1512464,\n  lines: 8711,\n  records: 8709,\n  errors: [\n    {\n      error: 'Unexpected token D in JSON at position 42',\n      line: 10,\n      data: '{\"id\":232,\"firstName\":\"Robert\",\"lastName:\"Davis\",\"state\":\"Illinois\",\"birthdate\":\"1990-10-22\",\"favoriteNumbers\":[5,34,1],\"favoriteNumber\":183,\"modified\":1616806973645}'\n    }\n  ],\n  blanks: 1,\n  created: 2021-03-27T01:20:21.562Z,\n  modified: 2021-03-27T01:28:32.686Z,\n  start: 1616808517081,\n  end: 1616808517124,\n  elapsed: 43\n}\n ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesbontempo%2Fnjodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjamesbontempo%2Fnjodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesbontempo%2Fnjodb/lists"}