{"id":13555864,"url":"https://github.com/ether/ueberDB","last_synced_at":"2025-04-03T09:30:30.767Z","repository":{"id":1476116,"uuid":"1718892","full_name":"ether/ueberDB","owner":"ether","description":"Abstract your databases, make datababies.  Transforms every database into a object key value store.","archived":false,"fork":false,"pushed_at":"2024-10-28T13:53:36.000Z","size":2956,"stargazers_count":262,"open_issues_count":6,"forks_count":94,"subscribers_count":19,"default_branch":"main","last_synced_at":"2024-10-29T11:12:31.700Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ether.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2011-05-08T15:06:29.000Z","updated_at":"2024-10-18T16:40:35.000Z","dependencies_parsed_at":"2024-03-16T17:57:18.458Z","dependency_job_id":"964a68cc-0279-4fad-b561-143c3872c64c","html_url":"https://github.com/ether/ueberDB","commit_stats":{"total_commits":1180,"total_committers":50,"mean_commits":23.6,"dds":0.7203389830508475,"last_synced_commit":"2bfd6082144bc6b264ba80bf09e4f91728b8211f"},"previous_names":["pita/ueberdb"],"tags_count":231,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ether%2FueberDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ether%2FueberDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ether%2FueberDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ether%2FueberDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ether","download_url":"https://codeload.github.com/ether/ueberDB/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246752655,"owners_count":20827987,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T12:03:27.998Z","updated_at":"2025-04-03T09:30:30.751Z","avatar_url":"https://github.com/ether.png","language":"TypeScript","readme":"# UeberDB2: Abstract your databases\n\n## About\n\n✓ UeberDB turns every database into a simple key value store by providing a\nlayer of abstraction between your software and your database.\n\n✓ UeberDB uses a cache and buffer to make databases faster. Reads are cached and\nwrites are done in a bulk. This can be turned off.\n\n✓ UeberDB does bulk writing ergo reduces the overhead of database transactions.\n\n✓ UeberDB uses a simple and clean syntax ergo getting started is easy.\n\n## Database Support\n\n* Couch\n* Dirty\n* Elasticsearch\n* Maria\n* `memory`: An in-memory ephemeral database.\n* Mongo\n* MsSQL\n* MySQL\n* Postgres (single connection and with connection pool)\n* Redis\n* Rethink\n* `rustydb`\n* SQLite\n* Surrealdb\n* \n## Install\n\n```\nnpm install ueberdb2\n```\n\n## Examples\n\n### Basic\n\n```javascript\nconst ueberdb = require('ueberdb2');\n\n(async () =\u003e {\n  // mysql\n  const db = new ueberdb.Database('mysql', {\n    user: 'root',\n    host: 'localhost',\n    password: '',\n    database: 'store',\n    engine: 'InnoDB',\n  });\n  // dirty to file system\n  //const db = new ueberdb.Database('dirty', {filename: 'var/dirty.db'});\n\n  await db.init();\n  try {\n    await db.set('valueA', {a: 1, b: 2});\n    console.log('valueA is', await db.get('valueA'));\n  } finally {\n    await db.close();\n  }\n})();\n```\n\n### findKeys\n\n```javascript\nconst ueberdb = require('ueberdb2');\n\n(async () =\u003e {\n  const db = new ueberdb.Database('dirty', {filename: 'var/dirty.db'});\n  await db.init();\n  try {\n    await Promise.all([\n      db.set('valueA', {a: 1, b: 2}),\n      db.set('valueA:h1', {a: 1, b: 2}),\n      db.set('valueA:h2', {a: 3, b: 4}),\n    ]);\n    // prints [ 'valueA:h1', 'valueA:h2' ]\n    console.log(await db.findKeys('valueA:*', null));\n  } finally {\n    await db.close();\n  }\n})();\n```\n\n### Getting and setting subkeys\n\nueberDB can store complex JSON objects. Sometimes you only want to get or set a\nspecific (sub-)property of the stored object. The `.getSub()` and `.setSub()`\nmethods make this easier.\n\n#### `getSub`\n\n```javascript\nconst value = await db.getSub(key, propertyPath);\ndb.getSub(key, propertyPath, callback);\n```\n\nFetches the object stored at `key`, walks the property path given in\n`propertyPath`, and returns the value at that location. `propertyPath` must be\nan array. If `propertyPath` is an empty array then `getSub()` is equivalent to\n`get()`. Returns a nullish value (`null` or `undefined`) if the record does not\nexist or if the given property path does not exist.\n\nExamples:\n\n```javascript\n(async () =\u003e {\n  await db.set(key, {prop1: {prop2: ['value']}});\n\n  const val1 = await db.getSub(key, ['prop1', 'prop2', '0']);\n  console.log('1.', val1); // prints \"1. value\"\n\n  const val2 = await db.getSub(key, ['prop1', 'prop2']);\n  console.log('2.', val2); // prints \"2. [ 'value' ]\"\n\n  const val3 = await db.getSub(key, ['prop1']);\n  console.log('3.', val3); // prints \"3. { prop2: [ 'value' ] }\"\n\n  const val4 = await db.getSub(key, []);\n  console.log('4.', val4); // prints \"4. { prop1: { prop2: [ 'value' ] } }\"\n\n  const val5 = await db.getSub(key, ['does', 'not', 'exist']);\n  console.log('5.', val5); // prints \"5. null\" or \"5. undefined\"\n});\n```\n\n#### `setSub`\n\n```javascript\nawait db.setSub(key, propertyPath, value);\ndb.setSub(key, propertyPath, value, callback);\n```\n\nFetches the object stored at `key`, walks the property path given in\n`propertyPath`, and sets the value at that location to `value`. `propertyPath`\nmust be an array. If `propertyPath` is an empty array then `setSub()` is\nequivalent to `set()`. Empty objects are created as needed if the property path\ndoes not exist (including if `key` does not exist in the database). It is an\nerror to attempt to set a property on a non-object.\n\nExamples:\n\n```javascript\n// Assumption: db does not yet have any records.\n(async () =\u003e {\n  // Equivalent to db.set('key1', 'value'):\n  await db.setSub('key1', [], 'value');\n\n  // Equivalent to db.set('key2', {prop1: {prop2: {0: 'value'}}}):\n  await db.setSub('key2', ['prop1', 'prop2', '0'], 'value'):\n\n  await db.set('key3', {prop1: 'value'});\n\n  // Equivalent to db.set('key3', {prop1: 'value', prop2: 'other value'}):\n  await db.setSub('key3', ['prop2'], 'other value');\n\n  // TypeError: Cannot set property \"badProp\" on non-object \"value\":\n  await db.setSub('key3', ['prop1', 'badProp'], 'foo');\n});\n```\n\n### Disable the read cache\n\nSet the `cache` wrapper option to 0 to force every read operation to go directly\nto the database driver (except for reads of written values that have not yet\nbeen committed to the database):\n\n```javascript\nconst ueberdb = require('ueberdb2');\n\n(async () =\u003e {\n  const db = new ueberdb.Database(\n      'dirty', {filename: 'var/dirty.db'}, {cache: 0});\n  await db.init();\n  try {\n    await db.set('valueA', {a: 1, b: 2});\n    const value = await db.get('valueA');\n    console.log(JSON.stringify(value));\n  } finally {\n    await db.close();\n  }\n})();\n```\n\n### Disable write buffering\n\nSet the `writeInterval` wrapper option to 0 to force writes to go directly to\nthe database driver:\n\n```javascript\nconst ueberdb = require('ueberdb2');\n\n(async () =\u003e {\n  const db = new ueberdb.Database(\n      'dirty', {filename: 'var/dirty.db'}, {writeInterval: 0});\n  await db.init();\n  try {\n    await db.set('valueA', {a: 1, b: 2});\n    const value = await db.get('valueA');\n    console.log(JSON.stringify(value));\n  } finally {\n    await db.close();\n  }\n})();\n```\n\n## Feature support\n\n|               | Get | Set | findKeys | Remove | getSub | setSub | doBulk | CI Coverage |\n|---------------|-----|-----|----------|--------|--------|--------|--------|-------------|\n| cassandra     | ✓   | ✓   | *        | ✓      | ✓      | ✓      | ✓      | ✓           |\n| couchdb       | ✓   | ✓   | ✓        | ✓      | ✓      | ✓      | ✓      | ✓           |\n| dirty         | ✓   | ✓   | ✓        | ✓      | ✓      | ✓      |        | ✓           |\n| dirty_git     | ✓   | ✓   | ✓        | ✓      | ✓      | ✓      |        | ✓           |\n| elasticsearch | ✓   | ✓   | *        | ✓      | ✓      | ✓      | ✓      | ✓           |\n| maria         | ✓   | ✓   | ✓        | ✓      | ✓      | ✓      | ✓      | ✓           |\n| mysql         | ✓   | ✓   | ✓        | ✓      | ✓      | ✓      | ✓      | ✓           |\n| postgres      | ✓   | ✓   | ✓        | ✓      | ✓      | ✓      | ✓      | ✓           |\n| redis         | ✓   | ✓   | *        | ✓      | ✓      | ✓      | ✓      | ✓           |\n| rethinkdb     | ✓   | ✓   | *        | ✓      | ✓      | ✓      | ✓      | \n| rustydb       | ✓   | ✓   | ✓        | ✓      | ✓      | ✓      | ✓      | ✓           |\n| sqlite        | ✓   | ✓   | ✓        | ✓      | ✓      | ✓      | ✓      | ✓           |\n| surrealdb     | ✓   | ✓   | ✓        | ✓      | ✓      | ✓      | ✓      | ✓           |\n\n## Limitations\n\n### findKeys query support\n\nThe following characters should be avoided in keys `\\^$.|?*+()[{` as they will\ncause findKeys to fail.\n\n### findKeys database support*\n\nThe following have limitations on findKeys\n\n* redis (Only keys of the format \\*:\\*:\\*)\n* cassandra (Only keys of the format \\*:\\*:\\*)\n* elasticsearch (Only keys of the format \\*:\\*:\\*)\n* rethink (Currently doesn't work)\n\nFor details on how it works please refer to the wiki:\nhttps://github.com/ether/UeberDB/wiki/findKeys-functionality\n\n### Scaling, High availability and disaster recovery.\n\nTo scale UeberDB you should use sharding especially for real time applications.\nAn example of this is sharding given Pads within Etherpad based on their initial\npad authors geographical location. High availability and disaster recovery can\nbe provided through replication of your database however YMMV on passing\nSettings to your database library. Do not be under the illusion that UeberDB\nprovides any Stateless capabilities, it does not. An option is to use something\nlike rethinkdb and set cache to 0 but YMMV.\n\n### Key Length Restrictions\n\nYour Key Length will be limited by the database you chose to use but keep into\naccount portability within your application.\n\n### doBulk operations on .set out of memory\n\ndoBulk operations that chain IE a large number of .set without a pause to handle\nthe channel clearance can cause a `Javascript out of heap memory`. It's very\nrare this happens and is usually due to a bug in software causing a constant\nwrite to the database.\n\n## MySQL /MariaDB Advice\n\nYou should create your database as utf8mb4_bin.\n\n## Redis TLS communication\n\nIf you enabled TLS on your Redis database (available since Redis 6.0) you will\nneed to change your connections parameters, here is an example:\n\n```javascript\nconst db = new ueberdb.Database('redis', {url: 'rediss://localhost'});\n```\n\nDo not provide a `host` value.\n\nIf you don't provide a certificate on the client side, you need to add the\nenvironment variable `NODE_TLS_REJECT_UNAUTHORIZED = 0` and add the flag\n`--tls-auth-clients no` when launching the redis-server to accept connections.\n\n## How to add support for another database\n\n1. Add the database driver to `packages.json`, this will happen automatically if\n   you run `npm install %yourdatabase%`\n2. Create `databases/DATABASENAME_db.js` and have it export a `Database` class\n   that derives from `lib/AbstractDatabase.js`. Implement the required\n   functions.\n3. Add a service for the database to the test job in\n   `.github/workflows/npmpublish.yml`.\n4. Add an entry to `test/lib/databases.js` for your database and configure it to\n   work with the service added to the GitHub workflow.\n5. Install and start the database server and configure it to work with the\n   settings in your `test/lib/databases.js` entry.\n6. Run `npm test` to ensure that it works.\n\n## License\n\n[Apache License v2](http://www.apache.org/licenses/LICENSE-2.0.html)\n\n## What's changed from UeberDB?\n\n* Dropped broken databases: CrateDB, LevelDB, LMDB (probably a\n  breaking change for some people)\n* Introduced CI.\n* Introduced better testing.\n* Fixed broken database clients IE Redis.\n* Updated Depdendencies where possible.\n* Tidied file structure.\n* Improved documentation.\n* Sensible name for software makes it clear that it's maintained by The Etherpad\n  Foundation.\n* Make db.init await / async\n\n### Dirty_Git Easter Egg.\n\n* I suck at hiding Easter eggs..\n\nDirty_git will `commit` and `push` to Git on every `set`. To use `git init` or\n`git clone` within your dirty database location and then set your upstream IE\n`git remote add origin git://whztevz`.\n\nThe logic behind dirty git is that you can still use dirty, but you can also have\noffsite backups. It's noisy and spammy, but it can be useful.\n","funding_links":[],"categories":["JavaScript","others","TypeScript","\u003ca name=\"TypeScript\"\u003e\u003c/a\u003eTypeScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fether%2FueberDB","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fether%2FueberDB","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fether%2FueberDB/lists"}