{"id":22608999,"url":"https://github.com/bitfinexcom/wasteland","last_synced_at":"2025-04-11T06:18:48.477Z","repository":{"id":27018232,"uuid":"110756871","full_name":"bitfinexcom/wasteland","owner":"bitfinexcom","description":null,"archived":false,"fork":false,"pushed_at":"2022-12-10T11:03:59.000Z","size":139,"stargazers_count":4,"open_issues_count":5,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-25T04:13:00.710Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bitfinexcom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-14T23:28:30.000Z","updated_at":"2021-05-19T04:30:04.000Z","dependencies_parsed_at":"2023-01-14T05:49:14.837Z","dependency_job_id":null,"html_url":"https://github.com/bitfinexcom/wasteland","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitfinexcom%2Fwasteland","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitfinexcom%2Fwasteland/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitfinexcom%2Fwasteland/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitfinexcom%2Fwasteland/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bitfinexcom","download_url":"https://codeload.github.com/bitfinexcom/wasteland/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248351592,"owners_count":21089311,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-08T15:10:31.713Z","updated_at":"2025-04-11T06:18:48.459Z","avatar_url":"https://github.com/bitfinexcom.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# wasteland\n\nWasteland is a modular data layer for [storing data in DHT](http://www.bittorrent.org/beps/bep_0044.html) and other backends. Storage adapters are pluggable, right now Grenache (DHT) and in-memory backends are included.\n\nAs payload size is limited to [1000 bytes per spec](http://www.bittorrent.org/beps/bep_0044.html), DHT storage supports chunking with indirect pointers. We can store 996 chars in a 1000 byte buffer, and 22 pointers per 1000 bytes, so with just single indirect pointers we can store up to 21.912KB.\n\n## Mutable Data\n\nData updates and update conflicts: for data updates a `seq` property must be used, which increases monotonically. This way we can make sure that a client does not overwrite outdated data. Updates and conflicts have to get handled at the client side, specific to the use case (e.g. \"last-write-wins\" or manual conflict resolution).\n\nSalts are used to store different values with the same key pair. If the salt is omitted, Wasteland generates it. For data that is stored in chunks, the salt is generated by hashing the stored data plus a random modifier.\n\nData is signed using public/private keys (ed25519).\n\nYou can find examples for both mutable and immutable data in [the examples folder](/examples/).\n\n## Immutable Data\n\nImmutable data is not signed. It is just a hash of the stored content. That means there can be no duplicate content. In case of duplicate content the write succeed, and return the same hash as for the previous result. The hash that is also used as key / storage address is simply a hash of the stored content.\n\nYou can find examples for both mutable and immutable data in [the examples folder](/examples/).\n\n\n## Storage Backends\n\nWasteland comes with backends for Grenache and In-Memory storage. Plugging in new backends is no problem.\n\n### Example using Grenache:\n\nRun two Grapes:\n\n```\ngrape --dp 20001 --aph 30001 --bn '127.0.0.1:20002'\ngrape --dp 20002 --aph 40001 --bn '127.0.0.1:20001'\n```\n\n```js\nconst Wasteland = require('wasteland')\nconst GrenacheBackend = require('wasteland/backends/Grenache')\n\nconst Link = require('grenache-nodejs-link')\nconst ed = require('ed25519-supercop')\n\nconst link = new Link({\n  grape: 'http://127.0.0.1:30001'\n})\nlink.start()\n\nconst { publicKey, secretKey } = ed.createKeyPair(ed.createSeed())\n\nconst gb = new GrenacheBackend({\n  transport: link,\n  keys: { publicKey, secretKey }\n})\n\nconst wl = new Wasteland({ backend: gb })\n\nconst opts = { seq: 1, salt: 'pineapple-salt' }\nwl.put('unchunked-data', opts, (err, hash) =\u003e {\n  if (err) throw err\n\n  wl.get({ hash: hash, salt: 'pineapple-salt' }, {}, (err, data) =\u003e {\n    if (err) throw err\n\n    console.log(data)\n    cb(null)\n  })\n})\n\n```\n\n**Response:**\n\n```js\n{ id: '1da41d4e30c74fa5b0d9a7e4cf60c1bf9402b0f1',\n  token: null,\n  v: 'unchunked-data',\n  sig: 'f84062a412736d3219bcf7fd40fee911427d03391bd455a53545f971f2eac20b85aed3dac8baafee777905becbcfff312a47bd5aa50aa66f040e39b8e5739901',\n  k: 'e33a423dcbf900727afcaf16622ad3263104874dc3031260eb82825db3ae7c2a',\n  salt: 'pineapple-salt',\n  seq: 1 }\n```\n\n\n[grenache.js](./examples/grenache.js)\n\n### Custom Backends\n\n## Data Model\n\n### Wasteland pointers\n\nPointers to data or other pointers are stored in the following format:\n\n```\n{ wasteland_type: 'pointers',\n  p:\n   [ [ '8e65439da41c0263512d965af3b45e750379f808',\n       '109eb3bb524d9fce41af8b232b14353ec922f357' ],\n     [ '0533d23c50b8e9a33ab89dd9a2f1cb3323695375',\n       '9c5519b713a16afd0f411b7b53720f508dcf6319' ] ] }\n```\n\nIn this case the first chunk has the hash `8e65439da41c0263512d965af3b45e750379f808` and because it is stored as mutable, it has a salt of `109eb3bb524d9fce41af8b232b14353ec922f357`.\n\nThe second chunk has the hash `0533d23c50b8e9a33ab89dd9a2f1cb3323695375` and the salt `9c5519b713a16afd0f411b7b53720f508dcf6319`.\n\nImmutable data has no salt, so the second index is `null`:\n\n```\n{ wasteland_type: 'pointers',\n  p:\n   [ [ '3465439da41c0263512d965af3b45e750379f808',\n       null ],\n     [ 'ea33d23c50b8e9a33ab89dd9a2f1cb3323695375',\n       null ] ] }\n```\n\n\n### BEP 44 data models\n\nThese are the data models each storage-backend has to implement.\n\n**get Responses:**\n\n```\n{\n  \"id\": \u003c20 byte id of sending node/database (string)\u003e,\n  \"seq\": \u003cmonotonically increasing sequence number (integer), used for versioning\u003e,\n  \"sig\": \u003ced25519 signature (64 bytes string)\u003e,\n  \"v\": \u003cthe data payload\u003e,\n  \"k\": \u003ced25519 public key\u003e\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitfinexcom%2Fwasteland","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbitfinexcom%2Fwasteland","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitfinexcom%2Fwasteland/lists"}