{"id":15497361,"url":"https://github.com/heapwolf/level-replicator","last_synced_at":"2025-07-12T18:33:07.822Z","repository":{"id":11172131,"uuid":"13547272","full_name":"heapwolf/level-replicator","owner":"heapwolf","description":"WIP; Eventually consistent log-based multi-master replication for levelDB (@Level)","archived":false,"fork":false,"pushed_at":"2015-10-31T14:10:06.000Z","size":420,"stargazers_count":70,"open_issues_count":8,"forks_count":11,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-06-15T17:18:42.448Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/heapwolf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-10-13T21:48:47.000Z","updated_at":"2024-09-23T08:54:12.000Z","dependencies_parsed_at":"2022-09-01T20:41:04.302Z","dependency_job_id":null,"html_url":"https://github.com/heapwolf/level-replicator","commit_stats":null,"previous_names":["hij1nx/level-replicator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/heapwolf/level-replicator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heapwolf%2Flevel-replicator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heapwolf%2Flevel-replicator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heapwolf%2Flevel-replicator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heapwolf%2Flevel-replicator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/heapwolf","download_url":"https://codeload.github.com/heapwolf/level-replicator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heapwolf%2Flevel-replicator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265033132,"owners_count":23700951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T08:33:08.419Z","updated_at":"2025-07-12T18:33:07.762Z","avatar_url":"https://github.com/heapwolf.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SYNOPSIS\nEventually consistent log-based multi-master replication for leveldb.\n\n## MULTI MASTER EXAMPLE\n\n*example with UDP peer discovery (see below): peers object can be let empty*\n### Server 1 instance 1\n\n```js\nvar level = require('level')\nvar replicate = require('level-replicator')\n\nvar levelConfig = { // level configuration object }\n\n// default settings\nvar replicationConfig = { port: 9000, host: '127.0.0.1', peers: {} }\n\nvar db = replicate(level('/tmp/db', levelConfig), replicationConfig)\n\n// put something into the database\ndb.put('some-key', 'some-value', function(err) {\n})\n```\n\n### Server 1 instance 2\n\n```js\nvar level = require('level')\nvar replicate = require('level-replicator')\n\nvar levelConfig = { // level configuration object }\n\n// different port\nvar replicationConfig = { port: 9001, host: '127.0.0.1', peers: {} }\n\n// different db folder\nvar db = replicate(level('/tmp/db2', levelConfig), replicationConfig)\n\ndb.put('some-key', 'some-value', function(err) {\n})\n```\n\n### Server 2\n\n```js\nvar level = require('level')\nvar replicate = require('level-replicator')\n\nvar levelConfig = { // level configuration object }\n\n// any port\nvar replicationConfig = { port: 9000, host: '127.0.0.1', peers: {} }\n\nvar db = replicate(level('/tmp/db'))\n\ndb.put('some-key', 'some-value', function(err) {\n})\n\ndb.on('connect', function(host, port){ console.log('connect', host, port) })\ndb.on('connection', function(host, port){ console.log('connect', host, port) })\ndb.on('error', function(err){ console.log('error', err) })\n\n```\n\n### Server 3...\n\n```js\nvar level = require('level')\nvar replicate = require('level-replicator')\n\nvar levelConfig = { // level configuration object }\n\n// any port\nvar replicationConfig = { port: 9000, host: '127.0.0.1', peers: {} }\n\nvar db = replicate(level('/tmp/db'))\n\ndb.put('some-key', 'some-value', function(err) {\n})\n```\n\n## REPLICATION ALGORITHM\n- If a write operation (a put or delete) is committed to the local database\n  for the first time.\n\n  - A log is created that contains the type of operation and a logical clock\n    that is set at `0`. The log's key is the peer's id and a sequence.\n  - An index is created (a pointer to the log for faster lookups on writes).\n  - The new key/value, log and log-index are atomically committed to the local\n    database.\n\n- If an update operation (a put or delete) is committed to the local database.\n\n  - The index is used to look up the keys corresponding log\n    - The logical clock in the log is incremented\n    - The type of operation is updated\n    - A new log-index is created and the old one is deleted.\n  - The new key/value, log and log-index are atomically committed to the local\n    database.\n\n- The frequently at which the peer will try to replicate is determined by its\n  write-velocity.\n\n- When the database connects to a peer, it reads its remote logs in reverse\n  until it finds a locally known entry.\n\n  - The latest log for each key is placed into memory and then iterated\n    over to determine what should be added locally.\n\n    - If the log does not exist locally, the log and its corresponding\n      key/value is committed to the local database.\n    - If the log exists locally (with an earlier clock), the remote log\n      and key/value are both atomically committed.\n\n## REPLICATON CONFLICTS\nBefore a local database can accept writes, it must attempt to replicate. This\nwill reduce the possibility for conflicts. However, in the eventual consistency\nmodel, there is a case in which conflicts can occur. Conflicts happen when two\nor more writes with the same `key` and `logical clock value` are written to two\nor more servers, for example...\n\n- `Server A` writes `foo` and a coresponding log with a logical clock of `0`.\n- At a different time, without knowing about the data on `Server A`, `Server B`\n  writes `foo` and a log with a logical clock of `0`.\n\nWhich write happened first? There is no reliable way to know. If this is a\npossibility for you, a resolver can be used to determine which write should be\naccepted. A resolver is a function can be passed into the configuration. The\nresolver function should return `true` to accept the remote value or `false` to\nreject it.\n\n```js\n{ resolver: function(a, b) { return a.timestamp \u003e b.timestamp; } }\n```\n\n## PEER DISCOVERY\nServer lists are a suck to maintain. They also don't work well in auto-scaling\nscenarios. `level-replicator` can use UDP multicast to discover peers that it\nwill replicate with.\n\nHowever not all VPCs support multicast and not all replication scenarios will\nbe within the same subnet, you may want to add known servers to a configuration,\nfor instance...\n\n```js\n{ peers: ['100.2.14.104:8000', '100.2.14.105:8000'] }\n```\n\nOtherwise you can use a service registry like [`seaport`]() or an module\nlike [`aws-instances`]() to feed the peers member of the options object.\n\n## FAQ\n\nQ. Why not expose the tcp connection so I can pool it / manage it myself?\n\nA. Connections are not long lived, a server connects, has a conversation and\nthen disconnects.\n\nQ. Why not allow me to manage the connection protocol?\n\nA. Give me a good use case.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheapwolf%2Flevel-replicator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fheapwolf%2Flevel-replicator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheapwolf%2Flevel-replicator/lists"}