{"id":17054578,"url":"https://github.com/grantcarthew/node-scalable-blob-store","last_synced_at":"2025-04-12T17:07:16.687Z","repository":{"id":140994761,"uuid":"43920224","full_name":"grantcarthew/node-scalable-blob-store","owner":"grantcarthew","description":"A file system blob store that is designed to prevent conflicts when used with a distributed file system or storage area network","archived":false,"fork":false,"pushed_at":"2022-01-02T01:37:36.000Z","size":235,"stargazers_count":33,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-26T11:21:46.258Z","etag":null,"topics":["blob","blob-store","cuid","file","scalable","storage","store","streams","uuid","web-app"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grantcarthew.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-10-08T22:45:24.000Z","updated_at":"2025-02-14T17:08:09.000Z","dependencies_parsed_at":"2023-06-10T15:15:21.092Z","dependency_job_id":null,"html_url":"https://github.com/grantcarthew/node-scalable-blob-store","commit_stats":{"total_commits":175,"total_committers":2,"mean_commits":87.5,"dds":0.005714285714285672,"last_synced_commit":"98dfa08adf6950046d5135a36b8f2b04e67a155a"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grantcarthew%2Fnode-scalable-blob-store","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grantcarthew%2Fnode-scalable-blob-store/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grantcarthew%2Fnode-scalable-blob-store/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grantcarthew%2Fnode-scalable-blob-store/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grantcarthew","download_url":"https://codeload.github.com/grantcarthew/node-scalable-blob-store/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248602312,"owners_count":21131616,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blob","blob-store","cuid","file","scalable","storage","store","streams","uuid","web-app"],"created_at":"2024-10-14T10:15:10.750Z","updated_at":"2025-04-12T17:07:16.664Z","avatar_url":"https://github.com/grantcarthew.png","language":"JavaScript","funding_links":["https://www.patreon.com/grantcarthew"],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nA file system blob store that is designed to prevent conflicts when used with a distributed file system or storage area network.\n\n[![unit-tests](https://github.com/grantcarthew/node-scalable-blob-store/actions/workflows/unit-tests.yaml/badge.svg)][actions-url]\n[![Patreon Donation][patreon-image]][patreon-url]\n\n[![Mr Blobby][mrblobby-image]][sbs-url]\n\n[![NPM][nodei-npm-image]][nodei-npm-url]\n\nPlease **Star** on GitHub / NPM and **Watch** for updates.\n\n## Topics\n\n- [Features](#features)\n- [Quick Start](#quick-start)\n- [Rationale](#rationale)\n- [Function](#function)\n- [Performance](#performance)\n- [API](#api)\n- [Known Issues](#known-issues)\n- [Testing](#testing)\n- [About the Owner](#about-the-owner)\n- [Contributing](#contributing)\n- [History](#history)\n- [License](#license)\n\n## Features\n\n- Save binary large objects (blobs) locally in a scalable format.\n- Written using modern JavaScript language features.\n- No dependencies.\n- Blob file and directory names based on unique IDs.\n- Extensive read / write APIs in both stream and file format.\n- Promise based with no callbacks.\n\n## Quick Start\n\n### Installation\n\nNote: Requires Node.js v12 or later.\n\n```sh\n\nnpm install scalable-blob-store --save\n\n```\n\n### Create a writable stream\n\n```js\nconst os = require('os');\nconst ulid = require('ulid').ulid; // You need a unique ID generator function\nconst BlobStore = require('scalable-blob-store');\n\nconst options = {\n  blobStoreRoot: os.tmpdir() + '/blobs', // Change this!\n  idFunction: ulid,\n  dirDepth: 4,\n  dirWidth: 1000,\n};\n\n// Creating the blobStore Object\nconst blobStore = new BlobStore(options);\nconst result = await blobStore.createWriteStream();\n\nconsole.dir(result);\n// Logs the result object which contains the blobPath and writeStream.\n// Use the writeStream to save your blob.\n// Store the blobPath in your database.\n//\n// result object will be similar to this:\n// {\n//   blobPath: \"/01CTZRTWMAD153V20K26S4Y0BW/01CTZRTWMBZW4SPR4E5QGGJYSH/01CTZRTWMB3QXZK04SYFY8ZJVR/01CTZS3KJYFPRQ34S3T15Y798S\",\n//   writeStream: [WriteStream]\n// }\n//\n// In this example the full file path for the blob would be something like this:\n// /tmp/blobs/01CTZRTWMAD153V20K26S4Y0BW/01CTZRTWMBZW4SPR4E5QGGJYSH/01CTZRTWMB3QXZK04SYFY8ZJVR/01CTZS3KJYFPRQ34S3T15Y798S\n//\n// This is based on the blobStoreRoot + blobPath.\n```\n\nSee the Quick Start example files for more detail:\n\n- [quick-start-write.js](/examples/quick-start-write.js)\n- [quick-start-read.js](/examples/quick-start-read.js)\n- [quick-start-api.js](/examples/quick-start-api.js)\n\n## Rationale\n\nAfter researching user file storage, or blob storage, for a web application I was working on I discovered the most common solution used by web developers is to store files using a cloud service provider. After creating an account with such providers as [Amazon S3][amazones3-url], [Google Cloud Storage][googlecloud-url], or [Azure Storage][azurestorage-url], they just stash all their application files and blobs there.\n\nI researched the price of cloud storage and decided I wanted a free local version that would scale if needed.\n\nI looked at a number of existing solutions such as [filestorage][filestorage-url] but was unhappy with the scalability of these solutions. Most are only designed for a single server and would cause write conflicts if a distributed file system, cluster file system like [GlusterFS][glusterfs-url], or a storage area network was used as the backend file system.\n\nOn a long car trip I was thinking about a solution for my blob storage and came up with `scalable-blob-store`.\n\n## Function\n\nTo achieve scalability on a distributed or replicated file system, `scalable-blob-store` does not use index files or other databases to manage the files on the disk or storage system. Instead, the file system itself is used to find the latest storage path based on the file systems `birthtime` attribute (the directory creation date).\n\nOnce the latest path has been determined, the number of files within the directory are counted to ensure it remains under the configured value. This is to prevent disk performance issues when very large numbers of files are stored within a single directory. If the number of items within a directory becomes too large, a new storage path is determined.\n\nBecause there are no databases used to manage the files in the root path, it is up to you to maintain the returned `blobPath` value and metadata about the stored files in your own database.\n\nThe reason `scalable-blob-store` is scalable is due to the naming of the directories and files within your file system. Every directory and file saved to disk is named by a generated unique id based on a user defined funciton. You could use any unique id generator such as [ULID][ulid-url], [CUID][cuid-url], [UUID v4][uuid-url], or MongoDBs [ObjectIds][objectid-url] just to name a few. Check out my [Awesome Unique ID][awesome-url] repository for more examples. Merging directories between servers or disks should never cause file name collisions.\n\nIf a replicated or cluster file system is in use the only conflict that can occur is when one server is reading a file while another is removing the same file. `scalable-blob-store` does not try to manage this conflict, however it will raise the exception.\n\nBelow are examples of the directory structure created by `scalable-blob-store`.\n\nExample with CUID directory and file names:\n\n```sh\n\\blobs\\cij50xia200pzzph3we9r62bi // ← Directory    File ↓\n\\blobs\\cij50xia300q1zph3m4df4ypz\\..\\cij50xiae00qgzph3i0ms0l2w\n```\n\nExample with UUID directory and file names:\n\n```sh\n\\blobs\\846a291f-9864-40bb-aefe-f29bdc73a761 // ← Directory    File ↓\n\\blobs\\846a291f-9864-40bb-aefe-f29bdc73a761\\..\\8b86b6fe-6166-424c-aed9-8faf1e62689e\n```\n\n`scalable-blob-store` supports configuration options to give you control over the directory and file ids used, depth of the directory structure, and the width of the directories. The default options give 3 directories deep containing 1000 items giving a total storage of one billion files within the directory structure.\n\nOther operational points of interest:\n\n- Files are only stored at the bottom of the directory tree.\n- The directory used for writing files is determined by the latest creation time (file system birthtime attribute).\n- Once the number of files in a directory reaches the `dirWidth` value, the next directory is created.\n- Once the number of directories in any directory reaches the `dirWidth` value, the next parent directory is created.\n- If the number of directories in the highest directory, being the blob store root, has reached the `dirWidth` value, the `dirWidth` value is ignored.\n\n## Performance\n\n**Write**\n\nOn my laptop with an M.2 SSD disk, running the [test-fs.js](tests/test-fs.js) script produces the following results:\n\n```\n====================================================================================================\nTesting scalable-blob-store with the following options:\nblobStoreRoot: /tmp/blobs/test-fs\nidFunction: ulid\ndirDepth: 3\ndirWidth: 1000\nrepeat: 10000\n\nBeginning test...\n====================================================================================================\nTest complete.\n====================================================================================================\n{\n  blobStoreRoot: '/tmp/blobs/test-fs',\n  dirDepth: 3,\n  dirWidth: 1000,\n  runTimeMilliseconds: 83730,\n  totalDirectories: 12,\n  totalFiles: 10000,\n  totalBytes: 430000,\n  lastBlobPath: '/ckxwcwgwz0001lk9hgq8t9iup/ckxwcwgx00002lk9h6tbpdmq1/ckxwcy36m06yclk9hb0g92dwg/ckxwcy9ip07q4lk9h5uyl10k6'\n}\n====================================================================================================\nPlease remove /tmp/blobs/test-fs manually.\n====================================================================================================\n```\n\n**Read**\n\nRead performance will be close to, if not the same, as disk speed.\n\n## API\n\nAll the BlobStore methods within `scalable-blob-store` return a Promise. This is perfect for using with the async/await language features.\n\n| API                                                        | Type               | Returns               |\n| ---------------------------------------------------------- | ------------------ | --------------------- |\n| [new BlobStore(options)](#instantiation)                   | Constructor        | blobStore Instance    |\n| [blobStore.blobStoreRoot](#blobstoreroot)                  | Read Only Property | `String`              |\n| [blobStore.idFunction](#idfunction)                        | Read Only Property | `Function`            |\n| [blobStore.dirDepth](#dirdepth)                            | Read Only Property | `Number`              |\n| [blobStore.dirWidth](#dirwidth)                            | Read Only Property | `Number`              |\n| [blobStore.getCurrentBlobDir()](#getcurrentblobdir)        | Method             | `Promise\u003cString\u003e`     |\n| [blobStore.setCurrentBlobDir(blobDir)](#setcurrentblobdir) | Method             | `Promise\u003cundefined\u003e`  |\n| [blobStore.createWriteStream()](#createwritestream)        | Method             | `Promise\u003cObject\u003e`     |\n| [blobStore.write(data, writeOptions)](#write)              | Method             | `Promise\u003cString\u003e`     |\n| [blobStore.append(blobPath, data, appendOptions)](#append) | Method             | `Promise\u003cundefined\u003e`  |\n| [blobStore.copy(blobPath, flags)](#copy)                   | Method             | `Promise\u003cString\u003e`     |\n| [blobStore.createReadStream(blobPath)](#createreadstream)  | Method             | `Promise\u003cReadStream\u003e` |\n| [blobStore.read(blobPath, readOptions)](#read)             | Method             | `Promise\u003cdata\u003e`       |\n| [blobStore.open(blobPath, flags, mode)](#open)             | Method             | `Promise\u003cFileHandle\u003e` |\n| [blobStore.realPath(blobPath, realPathOptions)](#realpath) | Method             | `Promise\u003cString\u003e`     |\n| [blobStore.stat(blobPath)](#stat)                          | Method             | `Promise\u003cStats\u003e`      |\n| [blobStore.exists(blobPath)](#exists)                      | Method             | `Promise\u003cBoolean\u003e`    |\n| [blobStore.remove(blobPath)](#remove)                      | Method             | `Promise\u003cundefined\u003e`  |\n\n\u003ca name=\"instantiation\"\u003e\u003c/a\u003e\n\n### `new BlobStore(options)`\n\n**Type:** Constructor function.\n\n**Parameter:** `options` as an `Object`.\n\n- A JavaScript object with desired options set. See below.\n\n**Returns**: A new `BlobStore` object to be used to store data.\n\n**Description:**\n\nYou can call `new BlobStore(options)` multiple times to create more than one blob store.\n\nOptions are passed to the constructor function as a JavaScript `object`.\n\n| Key             | Description                                               | Defaults |\n| --------------- | --------------------------------------------------------- | -------- |\n| `blobStoreRoot` | Root directory to store blobs                             | Required |\n| `idFunction`    | Any ID function that returns a unique ID string           | Required |\n| `dirDepth`      | How deep you want the directories under the root          | 3        |\n| `dirWidth`      | The maximum number of files or directories in a directory | 1000     |\n\n**Example:**\n\n```js\n// Start by requiring the `scalable-blob-store` constructor function:\nconst BlobStore = require('scalable-blob-store');\n\n// You will need a unique ID function\nconst uuid = require('uuid');\n\n// Create the options object\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 4,\n  dirWidth: 2000,\n};\n\n// Create a blob store using the options `object`:\nconst blobStore = new BlobStore(options);\n```\n\nCreating multiple blob stores:\n\n```js\nconst userOptions = {\n  blobStoreRoot: '/app/blobs/user',\n  idFunction: uuid.v4,\n  dirDepth: 4,\n  dirWidth: 2000,\n};\n\nconst pdfOptions = {\n  blobStoreRoot: '/app/blobs/pdf',\n  idFunction: uuid.v4,\n  dirDepth: 2,\n  dirWidth: 300,\n};\n\nconst userFileStore = new BlobStore(userOptions);\nconst pdfDocumentStore = new BlobStore(pdfOptions);\n```\n\n\u003ca name=\"blobstoreroot\"\u003e\u003ca/\u003e\n\n### `blobStoreRoot`\n\n**Type:** Read only property.\n\n**Returns:** A `String` that matches your `options.blobStoreRoot` value.\n\n**Description:**\n\nThis is a convenience property to allow you to pass the blobStore object to a sub module and still have access to the configured properties.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 4,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\nconsole.log(blobStore.blobStoreRoot);\n// Outputs '/app/blobs' which you configured in the options\n```\n\n\u003ca name=\"idfunction\"\u003e\u003ca/\u003e\n\n### `idFunction`\n\n**Type:** Read only property.\n\n**Returns:** The unique ID function you configured in the `options.idFunction` value.\n\n**Description:**\n\nThis is a convenience property to allow you to pass the blobStore object to a sub module and still have access to the configured properties.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 4,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\nconsole.log(blobStore.idFunction());\n// Outputs 'bac00ab2-5e6d-4b77-bfa4-e9befc3e4279' which is a generated UUID from the idFunction.\n```\n\n\u003ca name=\"dirdepth\"\u003e\u003ca/\u003e\n\n### `dirDepth`\n\n**Type:** Read only property.\n\n**Returns:** A `Number` that matches your `options.dirDepth` value.\n\n**Description:**\n\nThis is a convenience property to allow you to pass the blobStore object to a sub module and still have access to the configured properties.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 4,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\nconsole.log(blobStore.dirDepth);\n// Outputs '4' which you configured in the options\n```\n\n\u003ca name=\"dirwidth\"\u003e\u003ca/\u003e\n\n### `dirWidth`\n\n**Type:** Read only property.\n\n**Returns:** A `Number` that matches your `options.dirWidth` value.\n\n**Description:**\n\nThis is a convenience property to allow you to pass the blobStore object to a sub module and still have access to the configured properties.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 4,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\nconsole.log(blobStore.dirWidth);\n// Outputs '2000' which you configured in the options\n```\n\n\u003ca name=\"getcurrentblobdir\"\u003e\u003ca/\u003e\n\n### `getCurrentBlobDir()`\n\n**Type:** Method.\n\n**Returns:** A `Promise` that resolves to a `String` that is the current active blob creation directory.\n\n**Description:**\n\nThis function is used internally by the `BlobStore` to determine the directory where the next blob file will be saved to disk.\n\nIf you ever need to store a blob file outside of the `BlobStore` you could use this method to locate the right place to put your file.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  try {\n    console.log(await blobStore.getCurrentBlobDir());\n    // The 'dirDepth' option above is set to 3 so the output will be similar to the following:\n    // '/e44d3b0d-b552-4257-8b64-a53331184c38/443061b9-bfa7-40fc-a5a9-d848bc52155e/4d818f4c-88b3-45fd-a104-a2fc3700e9de'\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n\u003ca name=\"setcurrentblobdir\"\u003e\u003ca/\u003e\n\n### `setCurrentBlobDir(blobDir)`\n\n**Type:** Method.\n\n**Parameters:** `blobDir` as a `String`.\n\n- Represents a file system directory path you desire to store blob files in that will be located under the `blobStoreRoot` path.\n\n**Returns:** A `Promise` that resolves to `undefined`.\n\n**Description:**\n\nThis function can be used to guide the `BlobStore` to save new blob files into a desired `blobPath`.\n\nOne issue with `scalable-blob-store` is that if you remove many blob files the directories the files were located in will not be removed.\nYou could either remove the directories yourself, or repopulate them with new blob files by setting the current active blob directory.\n\nThis function was added to enable consumers of this module to work around empty blob directories.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  try {\n    console.log(await blobStore.getCurrentBlobDir());\n    // The 'dirDepth' option above is set to 3 so the output will be similar to the following:\n    // '/e44d3b0d-b552-4257-8b64-a53331184c38/443061b9-bfa7-40fc-a5a9-d848bc52155e/4d818f4c-88b3-45fd-a104-a2fc3700e9de'\n\n    await blobStore.setCurrentBlobDir('/some/blob/path');\n\n    console.log(await blobStore.getCurrentBlobDir());\n    // Outputs '/some/blob/path' to the console.\n    // Any new blob files added to the blob store will go into this path until there are `dirWidth` or 2000 files within it.\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n\u003ca name=\"createWriteStream\"\u003e\u003ca/\u003e\n\n### `createWriteStream()`\n\n**Type:** Method.\n\n**Returns**: A `Promise` that resolves to an `Object` containing the child path to the file within the blob store root and a [WriteStream][writestream-url].\n\n**Description:**\n\nHere is an exampe of the returned object using UUID as the idFunction:\n\n```js\n\n{\n  blobPath: \"/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19\",\n  writeStream: stream.Writable\n}\n\n```\n\nUse the `writeStream` to save your blob or file.\nThe `blobPath` needs to be saved to your database for future access.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\n// The below readStream is simply to make this a complete example\nconst fs = require('fs');\nconst readStream = fs.createReadStream('/path/to/file');\n\nasync function main() {\n  let result;\n  try {\n    result = await blobStore.createWriteStream();\n  } catch (err) {\n    console.error(err);\n  }\n\n  console.dir(result);\n  // result object will be similar to this:\n  // {\n  //   blobPath: \"/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19\",\n  //   writeStream: [WriteStream]\n  // }\n\n  // Using a Promise to encapsulate the write asynchronous events.\n  await new Promise((resolve, reject) =\u003e {\n    result.writeStream.on('finish', () =\u003e {\n      resolve();\n    });\n    result.writeStream.on('error', reject);\n    readStream.pipe(result.writeStream);\n  });\n\n  console.log(blobPath);\n  // Logs the blobPath. Save this in your database.\n}\nmain();\n```\n\n\u003ca name=\"write\"\u003e\u003ca/\u003e\n\n### `write(data, writeOptions)`\n\n**Type:** Method.\n\n**Parameter:** `data` as either `String`, `Buffer`, `TypedArray`, or `DataView`.\n\n**Parameter:** `writeOptions` as an `Object`.\n\n- The `writeOptions` object supports an encoding, mode, and flag property.\n- See the [writeFile](https://nodejs.org/api/fs.html#fs_fs_writefile_file_data_options_callback) documentation for more detail.\n\n**Returns:** A `Promise` that resolves to a `String`.\n\n- The string contains the `blobPath` value which needs committing to your database.\n\n**Description:**\n\nIf you have simple data in memory rather than a stream of data you can use this method to store the data into a blob file.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  const data = 'The quick brown fox jumps over the lazy dog.';\n\n  try {\n    const blobPath = await blobStore.write(data);\n    // The returned blobPath will look something like this:\n    // '/e44d3b0d-b552-4257-8b64-a53331184c38/443061b9-bfa7-40fc-a5a9-d848bc52155e/4d818f4c-88b3-45fd-a104-a2fc3700e9de'\n    // Save it to your database.\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n\u003ca name=\"append\"\u003e\u003ca/\u003e\n\n### `append(blobPath, data, appendOptions)`\n\n**Type:** Method.\n\n**Parameter:** `blobPath` as a `String`.\n\n- Retrieve the `blobPath` from your application database.\n\n**Parameter:** `data` as either a `String` or `Buffer`.\n\n**Parameter:** `appendOptions` as an `Object`.\n\n- The `appendOptions` object supports an encoding, mode, and flag property.\n- See the [appendFile](https://nodejs.org/api/fs.html#fs_fs_appendfile_path_data_options_callback) documentation for more detail.\n\n**Returns:** A `Promise` that resolves to a `undefined`.\n\n**Description:**\n\nUse this method to add simple in memory data to the end of the blob file.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  const data = 'The quick brown fox jumps over the lazy dog.';\n\n  try {\n    await blobStore.append(data);\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n\u003ca name=\"copy\"\u003e\u003ca/\u003e\n\n### `copy(blobPath, flags)`\n\n**Type:** Method.\n\n**Parameter:** `blobPath` as a `String`.\n\n- Retrieve the `blobPath` from your application database.\n\n**Parameter:** `flags` as a `Number`.\n\n- See the [copyFile](https://nodejs.org/api/fs.html#fs_fspromises_copyfile_src_dest_flags) documentation for more detail.\n\n**Returns:** A `Promise` that resolves to a `String`.\n\n- The returned string is a new `blobPath` value for the copied blob file.\n\n**Description:**\n\nUse this method to create a copy of an existing blob file.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  try {\n    const blobPathSource =\n      '/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';\n\n    const blobPathDest = await blobStore.copy(blobPathSource);\n    // Store your new blobPath into your application database\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n\u003ca name=\"createreadstream\"\u003e\u003ca/\u003e\n\n### `createReadStream(blobPath)`\n\n**Type:** Method.\n\n**Parameter:** `blobPath` as a `String`.\n\n- Retrieve the `blobPath` from your application database.\n\n**Returns**: A `Promise` that resolves to a [`ReadStream`][readstream-url].\n\n**Description:**\n\nCreates a readable stream to the blob file located at the `blobPath`.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\nasync function main() {\n  // Get the blobPath value from your database.\n  const blobPath =\n    '/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19h';\n\n  let readStream;\n  try {\n    readStream = await blobStore.createReadStream(blobPath);\n  } catch (err) {\n    console.error(err);\n  }\n\n  readStream.on('error', (err) =\u003e {\n    console.error(err);\n  });\n\n  // Blob contents is piped to the console.\n  readStream.pipe(process.stdout);\n}\nmain();\n```\n\n\u003ca name=\"read\"\u003e\u003ca/\u003e\n\n### `read(blobPath, readOptions)`\n\n**Type:** Method.\n\n**Parameter:** `blobPath` as a `String`.\n\n- Retrieve the `blobPath` from your application database.\n\n**Parameter:** `readOptions` as an `Object`.\n\n- See the [readFile](https://nodejs.org/api/fs.html#fs_fspromises_readfile_path_options) documentation for more detail.\n\n**Returns:** A `Promise` that resolves to a the contents of the blob file.\n\n- The format of the file contents will depend on the readOptions passed.\n- `scalable-blob-store` sets the `readOptions.encoding` value to 'utf8' by default.\n\n**Description:**\n\nUse this method to read the content of a small blob file into memory.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  try {\n    // Retrieve the blobPath value from your database\n    const blobPath =\n      '/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';\n\n    const content = await blobStore.read(blobPath);\n    // Do something with the content\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n\u003ca name=\"open\"\u003e\u003ca/\u003e\n\n### `open(blobPath, flags, mode)`\n\n**Type:** Method.\n\n**Parameter:** `blobPath` as a `String`.\n\n- Retrieve the `blobPath` from your application database.\n\n**Parameter:** `flags` as an `String` or `Number`.\n\n- See the [open](https://nodejs.org/api/fs.html#fs_fspromises_open_path_flags_mode) documentation for more detail.\n\n**Returns:** A `Promise` that resolves to a [FileHandle](https://nodejs.org/api/fs.html#fs_class_filehandle) object.\n\n**Description:**\n\nThis is a more advanced method allowing you to carry out [many file operations](https://nodejs.org/api/fs.html#fs_class_filehandle) against the blob file.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  try {\n    // Retrieve the blobPath value from your database\n    const blobPath =\n      '/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';\n\n    const fileHandle = await blobStore.open(blobPath);\n    // Do something with the file handle object\n    // See the documentation for more detail\n    // The documentation link is in the description above\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n\u003ca name=\"realpath\"\u003e\u003ca/\u003e\n\n### `realPath(blobPath, realPathOptions)`\n\n**Type:** Method.\n\n**Parameter:** `blobPath` as a `String`.\n\n- Retrieve the `blobPath` from your application database.\n\n**Parameter:** `realPathOptions` as a `String` or `Object`.\n\n- See the [realPath](https://nodejs.org/api/fs.html#fs_fspromises_realpath_path_options) documentation for more detail.\n\n**Returns:** A `Promise` that resolves to a `String`.\n\n- The returned string will be the full file system path of the blob file.\n\n**Description:**\n\nUse this method to locate a blob file on the file system. This method should not really be needed because you can determine the full blob file path. Simply concatenate the blobStoreRoot and the blobPath values.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  try {\n    // Retrieve the blobPath value from your database\n    const blobPath =\n      '/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';\n\n    const fsPath = await blobStore.realPath(blobPath);\n    // With the above options the result will be similar to this:\n    // '/app/blobs/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n\u003ca name=\"stat\"\u003e\u003ca/\u003e\n\n### `stat(blobPath)`\n\n**Type:** Method.\n\n**Parameter:** `blobPath` as a `String`.\n\n**Returns:** A stats `Object`.\n\n**Description:**\n\nRather than parse the file system [`stats`][nodefs-url] object, `scalable-blob-store` returns the raw `stats` object.\n\nMore stat class details can be found on [Wikipedia][wikistat-url].\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  try {\n    // Retrieve the blobPath value from your database\n    const blobPath =\n      '/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';\n\n    const stats = await blobStore.stat(blobPath);\n    console.dir(stats);\n    // Console output will be similar to the following.\n    // { dev: 2050,\n    //   mode: 33188,\n    //   nlink: 1,\n    //   uid: 1000,\n    //   gid: 1000,\n    //   rdev: 0,\n    //   blksize: 4096,\n    //   ino: 6707277,\n    //   size: 44,\n    //   blocks: 8,\n    //   atime: Mon Oct 12 2015 08:51:29 GMT+1000 (AEST),\n    //   mtime: Mon Oct 12 2015 08:51:29 GMT+1000 (AEST),\n    //   ctime: Mon Oct 12 2015 08:51:29 GMT+1000 (AEST),\n    //   birthtime: Mon Oct 12 2015 08:51:29 GMT+1000 (AEST) }\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n\u003ca name=\"exists\"\u003e\u003ca/\u003e\n\n### `exists(blobPath)`\n\n**Type:** Method.\n\n**Parameter:** `blobPath` as a `String`.\n\n**Returns:** `Boolean`\n\n- `true` if the file exists, otherwise `false`.\n\n**Description:**\n\nUse this method for a simple blob file existence test.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  try {\n    // Retrieve the blobPath value from your database\n    const blobPath =\n      '/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';\n\n    const exists = await blobStore.exists(blobPath);\n    // The result will be either true or false depending if the blob file exists.\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n\u003ca name=\"remove\"\u003e\u003ca/\u003e\n\n### `remove(blobPath)`\n\n**Type:** Method.\n\n**Parameter:** `blobPath` as a `String`.\n\n**Returns**: `undefined` if nothing went wrong or the file did not exist.\n\n**Description:**\n\nUse this method to delete a blob file. This method can not be used to remove directories.\n\n**Example:**\n\n```js\nconst BlobStore = require('scalable-blob-store');\nconst uuid = require('uuid');\nconst options = {\n  blobStoreRoot: '/app/blobs',\n  idFunction: uuid.v4,\n  dirDepth: 3,\n  dirWidth: 2000,\n};\n\nconst blobStore = new BlobStore(options);\n\nasync function main() {\n  try {\n    // Retrieve the blobPath value from your database\n    const blobPath =\n      '/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';\n\n    await blobStore.remove(blobPath);\n    // The blob file will no longer exist\n  } catch (err) {\n    console.error(err);\n  }\n}\nmain();\n```\n\n## Known Issues\n\nThere is a minor issue in `scalable-blob-store`. If there are a large number of blob files added and then removed from the blob store, you may have empty directories or directories with a small number of files in them. These directories will never be removed and will not be populated.\n\nIf you wish to prevent empty or sparsely populated directories you will need to run a maintenance task against the `blobStoreRoot` directory. This maintenance task will need to look for empty or incomplete directories and call the [setCurrentBlobDir](#setcurrentblobdir) method passing in the empty `blobPath`.\n\nFor your application you may find you rarely remove large numbers of blob files. If this is the case then this issue can be ignored.\n\n## Testing\n\nThere are two methods for testing `scalable-blob-store`:\n\n1.  _Unit Testing_ which uses [tap][tap-url] and the local `os.tmpdir()` directory.\n2.  _Manual Testing_ which will create directories and files on your local disk.\n\n### Unit Testing\n\nAfter cloning `scalable-blob-store`, type the following into your console:\n\n```sh\n\nnpm install\nnpm test\n\n```\n\n### Manual Testing\n\nRunning the [test-fs.js](tests/test-fs.js) file will create a `~/blobs` directory in your temporary directory and then recursively fill it with lots of blobs.\n\nThe default options configured in the `test-fs.js` file are:\n\n```js\nconst opts = {\n  blobStoreRoot: os.tmpdir() + '/blobs',\n  idFunction: cuid,\n  dirDepth: 3,\n  dirWidth: 1000,\n};\n\nconst repeat = 10000;\n```\n\nChange the options if you wish to see different results.\n\nAfter cloning `scalable-blob-store`, type the following into your console:\n\n```sh\n\nnpm install\nnode ./tests/test-fs.js\n\n```\n\nOnce complete, inspect the `/tmp/blobs` directory. I suggest using the **tree** command which gives you a summary of directories and files within the target directory.\n\n```sh\n\ntree ~/blobs\ntree -d ~/blobs\n\n```\n\n## About the Owner\n\nI, Grant Carthew, am a technologist from Queensland, Australia. I work on code in a number of personal projects and when the need arises I build my own packages.\n\nThis project exists because I needed a local blob store that could scale.\n\nEverything I do in open source is done in my own time and as a contribution to the open source community.\n\nIf you are using my projects and would like to thank me or support me, please click the Patreon link below.\n\n[![Patreon Donation][patreon-image]][patreon-url]\n\nSee my [other projects on NPM](https://www.npmjs.com/~grantcarthew).\n\n## Contributing\n\n1. Fork it!\n1. Create your feature branch: `git checkout -b my-new-feature`\n1. Commit your changes: `git commit -am 'Add some feature'`\n1. Push to the branch: `git push origin my-new-feature`\n1. Submit a pull request :D\n\n## History\n\n- v5.0.1 [2022-01-02]: Updated README. New version to publish to npmjs.\n- v5.0.0 [2022-01-02]: Upgrade Node.js version and minor fixes:\n  - Node.js minimum version updated to v12 or later.\n  - Dependency packages updated.\n  - Replaced Jest with Tap for unit testing.\n  - Updated fs-blob-dir-latest sort function to include duplicate creation time handling.\n  - Converted all files from CRLF to LF line endings.\n- v4.0.0 [2018-10-29]: Major upgrade to modern syntax. See readme above.\n- v3.0.9 [2018-02-26]: Dependency packages updated.\n- v3.0.8 [2017-12-22]: Dependency packages updated.\n- v3.0.7 [2017-07-28]: Fixed test. Removed mock-fs (now uses /tmp). Dependency packages updated.\n- v3.0.6 [2017-05-17]: Dependency packages updated.\n- v3.0.5 [2017-03-20]: Dependency packages updated to support Node.js v7.7.3 and mock-fs v4.2.0.\n- v3.0.4 [2016-12-05]: Dependency packages updated.\n- v3.0.3 [2016-10-10]: Replaced `node-uuid` with `uuid`.\n- v3.0.2 [2016-09-20]: Dependency packages updated.\n- v3.0.1 [2016-05-05]: Packages updated and minor refactor.\n- v3.0.0 [2016-03-07]: Callback support added. createReadStream API changed.\n- v2.1.2 [2016-03-05]: Missed duplicate function in tests, removed.\n- v2.1.1 [2016-03-05]: Refactored duplicate function in tests.\n- v2.1.0 [2016-03-05]: Switched to using the `ES5` build code. Removed Nodejs engine requirements.\n- v2.0.10 [2016-03-03]: Dependency packages updated.\n- v2.0.9 [2016-02-09]: Added promisifyAll to the fsBlobStore instance. More `return null` statements.\n- v2.0.8 [2016-02-09]: Added `return null` after resolve/reject calls to prevent Bluebird warnings.\n- v2.0.7 [2016-02-09]: Added `es5dist` for older versions of node. Packages updated.\n- v2.0.6 [2016-01-28]: Added failure unit tests.\n- v2.0.5 [2016-01-26]: Refactor blob-store.js for minor performance improvement.\n- v2.0.4 [2016-01-24]: Minor performance improvements and bug fixes.\n- v2.0.3 [2016-01-22]: Added unit tests and minor fix.\n- v2.0.2 [2016-01-19]: Added [standard][js-standard-url] to package.json.\n- v2.0.1 [2016-01-12]: Minor performance improvements and bug fixes.\n- v2.0.0 [2016-01-08]: Added support for [CUID][cuid-url] or [UUID][uuid-url] directory and file names.\n- v1.0.1 [2016-01-07]: Last release of v1. Work on v2.0.0 to support cuid.\n- v1.0.0 [2016-01-05]: Minor delint and README updates. Bump to v1.0 for future changes.\n- v0.4.1 [2015-08-20]: Fix reference error.\n- v0.4.0 [2015-08-16]: Changed read and write to createReadStream and createWriteStream.\n- v0.3.1 [2015-08-16]: Fix write stream event order.\n- v0.3.0 [2015-08-16]: Removed file path function, change of plans.\n- v0.2.0 [2015-08-16]: Added file path function.\n- v0.1.0 [2015-09-30]: Initial release.\n\n## License\n\nMIT\n\n[sbs-url]: https://github.com/grantcarthew/node-scalable-blob-store\n[mrblobby-image]: https://cdn.rawgit.com/grantcarthew/node-scalable-blob-store/master/mrblobby.svg\n[amazones3-url]: https://aws.amazon.com/s3/\n[googlecloud-url]: https://cloud.google.com/storage/\n[azurestorage-url]: https://azure.microsoft.com/en-us/services/storage/\n[filestorage-url]: https://github.com/petersirka/node-filestorage\n[glusterfs-url]: http://www.gluster.org/\n[uuid-url]: https://www.npmjs.com/package/uuid\n[cuid-url]: https://github.com/ericelliott/cuid\n[objectid-url]: https://docs.mongodb.com/manual/reference/method/ObjectId/\n[ulid-url]: https://github.com/ulid/javascript\n[awesome-url]: https://github.com/grantcarthew/awesome-unique-id\n[nodefs-url]: https://nodejs.org/api/fs.html#fs_class_fs_stats\n[wikistat-url]: https://en.wikipedia.org/wiki/Stat_(system_call)\n[readstream-url]: https://nodejs.org/api/stream.html#stream_class_stream_readable\n[writestream-url]: https://nodejs.org/api/stream.html#stream_class_stream_writable\n[patreon-image]: https://img.shields.io/badge/patreon-donate-yellow.svg\n[patreon-url]: https://www.patreon.com/grantcarthew\n[nodei-npm-image]: https://nodei.co/npm/scalable-blob-store.png?downloads=true\u0026downloadRank=true\u0026stars=true\n[nodei-npm-url]: https://nodei.co/npm/scalable-blob-store/\n[cuid-discuss-url]: https://github.com/ericelliott/cuid/issues/22\n[actions-url]: https://github.com/grantcarthew/node-scalable-blob-store/actions\n[tap-url]: https://node-tap.org/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrantcarthew%2Fnode-scalable-blob-store","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrantcarthew%2Fnode-scalable-blob-store","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrantcarthew%2Fnode-scalable-blob-store/lists"}