{"id":24118384,"url":"https://github.com/fantasyui-com/multiprocess-store","last_synced_at":"2025-10-26T09:10:36.482Z","repository":{"id":19302942,"uuid":"86636181","full_name":"fantasyui-com/multiprocess-store","owner":"fantasyui-com","description":"ES6-and-beyond multiprocess-safe file-system-based object-store. [Object Database, Multiprocess, Multiprocess Database]","archived":false,"fork":false,"pushed_at":"2024-07-09T19:16:16.000Z","size":108,"stargazers_count":2,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-19T22:29:51.557Z","etag":null,"topics":["db","filesystem","library","multi-process","multiprocessing","store"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fantasyui-com.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-03-29T22:47:26.000Z","updated_at":"2024-07-09T19:16:20.000Z","dependencies_parsed_at":"2024-04-10T20:28:13.618Z","dependency_job_id":"507a6002-7552-4752-846e-7217a46cc03c","html_url":"https://github.com/fantasyui-com/multiprocess-store","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fantasyui-com%2Fmultiprocess-store","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fantasyui-com%2Fmultiprocess-store/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fantasyui-com%2Fmultiprocess-store/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fantasyui-com%2Fmultiprocess-store/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fantasyui-com","download_url":"https://codeload.github.com/fantasyui-com/multiprocess-store/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233456637,"owners_count":18679049,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["db","filesystem","library","multi-process","multiprocessing","store"],"created_at":"2025-01-11T08:19:23.945Z","updated_at":"2025-09-18T07:30:52.905Z","avatar_url":"https://github.com/fantasyui-com.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"multiprocess-store\n==================\n\nES6-and-beyond multiprocess-safe file-system-based JSON-formatted object-store.\nSee https://github.com/fantasyui-com/und for an independent sister project.\n\n## Node Version\n\n  Use the 7.x branch as it supports async/await via --harmony flag.\n  Note that node 7.6.0 and above does not require the --harmony flag.\n  Overall, use latest version of node.\n\n## Install\n\n```shell\n\nnpm i multiprocess-store\n\n```\n\n### Require and Initialize\n\n```JavaScript\n\nconst objectStore = require('multiprocess-store');\n\nconst store = await objectStore.createStore('~/Tests/my-objects');\n\nawait store.upsertObject({\n    _id: 'helloWorldObject',\n    text: 'Cats are little people!'\n});\n\nconsole.log(await store.getObject('helloObject'));\n```\n\n### Async/Await and ES6+\n\nUse the --harmony flag to enable async/await in older 7.x.\nNode 7.6 and up does not require the --harmony flag.\n\nnode \u003c7.6.0 example\n\n```shell\n\nnode --harmony my-app.js\n\n```\n\nnode \u003e=7.6.0 example\n\n```shell\n\nnode my-app.js\n\n```\n\n## Examples\n\n### Insert or update an object (upsert)\n\n```JavaScript\n\nconst objectStore = require('multiprocess-store');\nconst myApp = async function() {\n\n    const store = await objectStore.createStore('~/Tests/test-store-1');\n\n    await store.upsertObject({\n        _id: 'helloObject',\n        text: 'Hello Object Store!'\n    });\n\n    console.log(await store.getObject('helloObject'));\n\n};\n\nmyApp();\n\n```\n\n### Insert or Update, and then modify the object.\n\n```JavaScript\n\nconst objectStore = require('multiprocess-store');\nconst myApp = async function() {\n    const store = await objectStore.createStore('~/Tests/test-store-2');\n\n    await store.upsertObject({\n        _id: 'helloObject',\n        text: 'Hello Object Store!'\n    })\n\n    // Make an update\n    let storeObject = await store.getObject('helloObject');\n    storeObject.secret = 'Cats are little people!';\n    await store.updateObject(storeObject);\n\n    console.log(await store.getObject('helloObject'));\n};\n\nmyApp();\n\n```\n\n## API\n\nCreate the disk based store:\n\n```JavaScript\n\nconst store = await objectStore.createStore('/tmp/a-user-manager-store');\n=\u003e store\n\n// tilde will resolve to user home directory\nconst store = await objectStore.createStore('~/Tests/hello-store');\n=\u003e store // in user's home: /Users/captain-fantasy/Tests/hello-store\n\n```\n\nCrete an object with id hello.\n\n```JavaScript\n\nawait store.createObject({_id:'hello', email:'alice@example.com'});\n=\u003e true;\n\n```\n\nRetrieve an object from the store by its id.\n\n```JavaScript\n\nlet _id = 'hello';\nawait store.getObject(_id);\n=\u003e obj // the requested object, it will contain _id and _rev fields\n\n```\n\nUpdate a recently retrieved and edited object.\n\n```JavaScript\n\nawait store.updateObject(obj);\n=\u003e true;\n\n```\n\nDelete an object using its id.\n\n```JavaScript\n\nlet _id = 'hello';\nawait store.deleteObject(_id)\n=\u003e true;\n\n```\n\nGet revision conflicts.\n\n```JavaScript\n\nlet _id = 'hello';\nawait store.getConflicts(_id);\n=\u003e [\n    [\n      \"17-2351d3253f494398acb7b6c1f3ae293d\",\n      \"17-3636fcd8b0dd4de9b851acb734204baa\"\n    ],\n    [\n      \"31-16f79c01537740efab08ccac7417c22b\",\n      \"31-71b27cfb062d4105a733fc69e728abd9\"\n    ]\n  ];\n\n```\n\nGet all objects from the store. (Use .filter(), .map() and/or lodash on returned data)\n\n```JavaScript\n\nawait store.getAllObjects();\n=\u003e [\n    {\n      \"_id\": \"alice\",\n      \"password\": \"42b3fdf4-12f7-4915-b218-599d08b001c4\",\n      \"_rev\": \"56-34a2bb6f366143b999241f099e3e29c3\"\n    },\n    {\n      \"_id\": \"bob\",\n      \"password\": \"123\",\n      \"_rev\": \"0-94bdde39692f48018e097cf4d1698931\"\n    },\n    {\n      \"_id\": \"carol\",\n      \"password\": \"hunter2\",\n      \"_rev\": \"0-fe93b45cb46d4a9bbe4a44f11a62ee27\"\n    }\n  ];\n\n```\n\n\n## A quick note on revision conflicts in a distributed, unreliable, multicore, networked world.\n\nRevision conflicts occur when two actors write at the same time to the same revision number. That is to say, two separate programs requested a copy of an object at the same time/object-state (ex. on a Monday Evening), modified the data and performed an ```updateObject``` slipping through the cracks between atomic disk IO and node multiprocessing ```race conditions``` at the same time, on High Noon Tuesday.\n\nRevision conflicts occur due to race conditions that emerge between multiple users, multiprocessing, and non-atomic disk IO. Revision conflicts occur in rare conditions, and solving them depends on your particular application.\n\n- If it is a user manager for administrators, flag account as needing attention.\n- If it is a wiki, proudly parade the conflict as needing human attention.\n- If it is a fabulous RGB color generator for your new website, just ignore it.\n\nA new revision of a document means that a conflict is ```probably``` not important anymore becasue a new document with a higher revision has been created. However, a conflict inside a previous revision becomes a signal that something may potentially need attention, maybe a bit of information can be moved into the latest revision all the way back from few weeks ago when some administrator made a note on an unreliable network connection somewhere.\n\nRevision conflicts can only be solved by a human in context of a program that is using the store, however there is great potential for auto-solving depending on your specific use case. Consider an event where both a very old conflict and the latest master have matching data for example email:alice@example.com this means that in this particular scenario the old revision conflict can be removed automatically as its data has been captured by the latest revision, be it by chance or because someone looked at the revision and moved that bit of data to the latest revision.\n\nEvery revision is a file with a random name, saved in a directory named after the object id: ```alice-profile/3f700747-033f-486a-afbe-57e4f6662153-3```\nWhenever a new revision is made, a whole new random filename is generated: ```alice-profile/ef15f947-fe03-4de9-9926-0745c69373f5-4```\n\nTo track revisions, the revision number is added to the filename: alice-profile/f94bc318-0ffd-4f13-b258-84420e97601c```-5``` when document is updated the new information is saved to a new random filename with a higher revision postfix alice-profile/```5b4c17f0-546f-47f6-95bb-1a10a107ebeb```-```6``` now.\n\nWhat you must understand is that when a race condition occurs, it does not endanger the data, information cannot be lost due to random filenames (UUIDs) both operations will succed in saving two separate files, with the same revision number. THe same revision number is then used as a red-flag for discovering that a conflict occured.\n\nThusly,\n\nUser #1 saves: alice-profile/ee30f92d-a785-4603-b0b8-b681b8707e39-3\nUser #2 saves: alice-profile/56b1d51a-e2b8-4d6c-837b-61af06c6b272-3\n\nNo information can be lost, but both will be unaware that they updated the same piece of data unless you quickly check for conflicts and ask either to merge them - depending on use case.\n\nWhen ```alice-profile``` is read, and a conflict is present, the winner is elected by sort() thus in a situation where two revisions are saved at the same time, and we have a tie, we choose the winner by just sorting the randomly generated filename, in effect we toss the dice; and the winner is chosen at random (by means of a random UUID).\n\n### One Thousand Hosts, One Thousand Processes, One Million Revision Conflicts, No Headache.\n\nIf you had a 1,000 servers, with 1,000 processes all creating revision 2 of object named important-passwords, thus 1,000 conflicts on 1,000 machines.\n\nIf every machine contacted all other 999 machines, and copied their revision data about 1,000 revision files from each of those 999 machines, a total of 1,000,000 files. At the end of the day, they would each arrive at the same state of revision 2, independently agreeing on a single lucky revision file; thus reaching ```eventual consistency```.\n\nIf one of those machines made another change later in the night, and saved revision 3 of important-passwords. All those 999 other machines, hoping to synchronize, would only copy that single revision 3 from that host. The other 999 machines would request revision 3.\n\nAgain conflict resolution is not a theoretical problem, nor is it a general problem for generic databases, it depends on your particular application, needs, network, customers, administrators, foresight, and technology. \n\n### LICENSE\n\nGPLv3\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffantasyui-com%2Fmultiprocess-store","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffantasyui-com%2Fmultiprocess-store","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffantasyui-com%2Fmultiprocess-store/lists"}