{"id":17095607,"url":"https://github.com/meyfa/giantdb","last_synced_at":"2025-06-13T17:32:38.778Z","repository":{"id":32289201,"uuid":"131851244","full_name":"meyfa/giantdb","owner":"meyfa","description":"Large object database in TypeScript, with encryption support","archived":false,"fork":false,"pushed_at":"2025-06-06T19:16:27.000Z","size":761,"stargazers_count":3,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-06T20:26:38.124Z","etag":null,"topics":["database","encryption","files","javascript","nodejs","object-storage","typescript"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/giantdb","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/meyfa.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"meyfa"}},"created_at":"2018-05-02T13:02:23.000Z","updated_at":"2025-05-18T08:34:16.000Z","dependencies_parsed_at":"2024-03-18T23:25:19.375Z","dependency_job_id":"a50d3afa-81bc-4944-ae07-a57987e9548d","html_url":"https://github.com/meyfa/giantdb","commit_stats":{"total_commits":250,"total_committers":4,"mean_commits":62.5,"dds":0.392,"last_synced_commit":"33d260754d64ce4c8882461ba3e9c2e19d515ab7"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meyfa%2Fgiantdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meyfa%2Fgiantdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meyfa%2Fgiantdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meyfa%2Fgiantdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/meyfa","download_url":"https://codeload.github.com/meyfa/giantdb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meyfa%2Fgiantdb/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259267273,"owners_count":22831588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","encryption","files","javascript","nodejs","object-storage","typescript"],"created_at":"2024-10-14T14:43:18.492Z","updated_at":"2025-06-13T17:32:38.743Z","avatar_url":"https://github.com/meyfa.png","language":"TypeScript","funding_links":["https://github.com/sponsors/meyfa"],"categories":[],"sub_categories":[],"readme":"# GiantDB\n\n[![CI](https://github.com/meyfa/giantdb/actions/workflows/main.yml/badge.svg)](https://github.com/meyfa/giantdb/actions/workflows/main.yml)\n[![Test Coverage](https://api.codeclimate.com/v1/badges/39ebc35a4b32350a0191/test_coverage)](https://codeclimate.com/github/meyfa/giantdb/test_coverage)\n[![Maintainability](https://api.codeclimate.com/v1/badges/39ebc35a4b32350a0191/maintainability)](https://codeclimate.com/github/meyfa/giantdb/maintainability)\n\nGiantDB is a large object store, written entirely in TypeScript. It provides\nmanaged data storage with a minimal, yet powerful programming interface.\nPromises and streams are used for making this efficient.\n\nEncryption is realized as a middleware module,\n[giantdb-crypto](https://github.com/meyfa/giantdb-crypto). For usage\ninstructions please read giantdb-crypto's documentation.\n\n## Install\n\n```\nnpm i giantdb\n```\n\n## Setup\n\n```ts\nimport { GiantDB } from 'giantdb'\n\nconst db = new GiantDB(/* source */)\n```\n\nThe `source` argument is optional. If it is not provided, GiantDB will use an\nin-memory file store. If it is a string that denotes a directory on the file\nsystem, that directory will be used for file storage.\n\n## Usage\n\n### Class: DB\n\n#### Method: db.create()\n\nThis prepares a new item for storage. It returns a Promise that resolves to a\nspecial writable stream to which you can write data. Once done, you call\n`.commit()` to finalize the item and obtain an `Item` instance.\n\nExample:\n\n```ts\ndb.create().then((change) =\u003e {\n  change.write('hello world', 'utf8')\n  // you can also .pipe(change), for example\n\n  return change.commit()\n}).then((item) =\u003e {\n  // the item is now stored\n  console.log(item.id)\n})\n```\n\n#### Method: db.get(id)\n\nThis obtains the item with the given id. It returns a Promise that resolves to\nthe `Item` instance.\n\nExample:\n\n```ts\ndb.get('d54232abbf9e9dc4e6a8fd72a6e25585').then((item) =\u003e {\n  console.log(item.id) // 'd54232abbf9e9dc4e6a8fd72a6e25585'\n})\n```\n\n#### Method: db.remove(id)\n\nThis removes the item with the given id. It returns a Promise that resolves when\ndone. Any data associated with the item will be gone.\n\nExample:\n\n```ts\ndb.remove('d54232abbf9e9dc4e6a8fd72a6e25585').then(() =\u003e {\n  console.log('done')\n})\n```\n\n#### Method: db.each(callback)\n\nThis method iterates over all items. Iteration happens one after the other. If\nthe callback returns a Promise, it is awaited before continuing with the next\nitem.\n\nExample:\n\n```ts\ndb.each((item) =\u003e console.log(item.id))\n```\n\n### Class: Item\n\n#### Properties\n\n- `item.id`: The item's id string.\n- `item.metadata`: The item's metadata.\n\nItem metadata can be used by middleware, but is also available for other\npurposes. Call `item.saveMetadata()` to preserve changes.\n\n#### Method: item.getReadable()\n\nThis obtains a read stream for reading the item's data. It returns a Promise.\n\nExample:\n\n```ts\ndb.get('d54232abbf9e9dc4e6a8fd72a6e25585').then((item) =\u003e {\n  return item.getReadable().then((readable) =\u003e {\n    // read data from the stream\n  })\n})\n```\n\n#### Method: item.getWritable()\n\nThis obtains a write stream for writing data to the item, replacing any previous\ncontents. It returns a Promise.\n\nExample:\n\n```ts\ndb.get('d54232abbf9e9dc4e6a8fd72a6e25585').then((item) =\u003e {\n  return item.getWritable().then((writable) =\u003e {\n    // write data to the stream\n  })\n})\n```\n\n#### Method: item.saveMetadata()\n\nThis saves the item's metadata in its current state. It must be called for the\nmetadata to persist after modifications have been made.\n\nNote that the metadata may also be saved on other occurrences (e.g. when\nmodified by middleware), but that is not guaranteed.\n\nExample:\n\n```ts\ndb.get('d54232abbf9e9dc4e6a8fd72a6e25585').then((item) =\u003e {\n  item.metadata.lastRead = Date.now()\n  return item.saveMetadata().then(() =\u003e {\n    console.log('saved')\n  })\n})\n```\n\n## Writing Middleware\n\n### Basics\n\nEvery middleware is simply an object. At different stages, GiantDB will call\nspecific functions on that object, given that they are available. The functions\ncan operate on their inputs and provide some or no output.\n\nThrough this mechanic, it is possible to extend GiantDB with custom\nfunctionality. The encryption middleware is the best example for this.\n\n### Middleware Functions\n\n#### transformReadable\n\n```ts\nfunction transformReadable (stream, metadata, options, next)\n```\n\nThis function is called every time a *readable* stream to one of the items is\nconstructed. It enables the middleware to tap into the read stream or modify it.\nAn example for this would be piping the input through a decryption stream.\n\n- `stream: stream.Readable`: The input read stream.\n- `metadata: object`: The item's current metadata.\n- `options?: object`: An object provided by the user.\n- `next: (error?: Error, result?: object) =\u003e void`): A callback.\n\nCalling `next()` is mandatory, otherwise middleware processing cannot continue.\nIf you need to, pass an error as the first argument to the callback.\n\nWhen either `stream` or `metadata` changed as part of your middleware function,\nyou must pass a result object containing the changed properties to the callback,\nlike this:\n\n```ts\nnext(null, {\n  stream: myNewReadStream, // if stream changed\n  metadata: myNewMetadata // if metadata changed\n})\n```\n\n#### transformWritable\n\n```ts\nfunction transformWritable (stream, metadata, options, next)\n```\n\nThis function is called every time a *writable* stream to one of the items is\nconstructed. It enables the middleware to modify the stream.\nAn example for this would be adding an encryption layer.\n\n- `stream: stream.Writable`: The input write stream.\n- `metadata: object`: The item's current metadata.\n- `options?: object`: An object provided by the user.\n- `next: (error?: Error, result?: object) =\u003e void`): A callback.\n\nCalling `next()` is mandatory, otherwise middleware processing cannot continue.\nIf you need to, pass an `Error` as the first argument to the callback.\n\nWhen either `stream` or `metadata` changed as part of your middleware function,\nyou must pass a result object containing the changed properties to the callback,\nlike this:\n\n```ts\nnext(null, {\n  stream: myNewWriteStream, // if stream changed\n  metadata: myNewMetadata // if metadata changed\n})\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeyfa%2Fgiantdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmeyfa%2Fgiantdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeyfa%2Fgiantdb/lists"}