Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mafintosh/hyperlog
Merkle DAG that replicates based on scuttlebutt logs and causal linking
https://github.com/mafintosh/hyperlog
Last synced: 9 days ago
JSON representation
Merkle DAG that replicates based on scuttlebutt logs and causal linking
- Host: GitHub
- URL: https://github.com/mafintosh/hyperlog
- Owner: mafintosh
- License: mit
- Created: 2014-11-23T17:15:25.000Z (almost 10 years ago)
- Default Branch: master
- Last Pushed: 2017-01-24T06:23:21.000Z (almost 8 years ago)
- Last Synced: 2024-10-22T16:40:16.302Z (17 days ago)
- Language: JavaScript
- Homepage:
- Size: 137 KB
- Stars: 470
- Watchers: 19
- Forks: 33
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-peer-to-peer - hyperlog
- awesome-peer-to-peer - hyperlog
- awesome-starred - mafintosh/hyperlog - Merkle DAG that replicates based on scuttlebutt logs and causal linking (others)
README
# hyperlog
[Merkle DAG](https://github.com/jbenet/random-ideas/issues/20) that replicates based on scuttlebutt logs and causal linking
```
npm install hyperlog
```
[](http://travis-ci.org/mafintosh/hyperlog)
``` js
var hyperlog = require('hyperlog')
var log = hyperlog(db) // where db is a levelup instance
// add a node with value 'hello' and no links
log.add(null, 'hello', function(err, node) {
console.log('inserted node', node)
// insert 'world' with a link back to the above node
log.add([node.key], 'world', function(err, node) {
console.log('inserted new node', node)
})
})
```
## Replicate graph
To replicate this log with another one simply use `log.replicate()` and pipe it together with a replication stream from another log.
``` js
var l1 = hyperlog(db1)
var l2 = hyperlog(db2)
var s1 = l1.replicate()
var s2 = l2.replicate()
s1.pipe(s2).pipe(s1)
s1.on('end', function() {
console.log('replication ended')
})
```
A detailed write-up on how this replication protocol works will be added to this repo in the near
future. For now see the source code.
## API
#### `log = hyperlog(db, opts={})`
Create a new log instance. Valid keys for `opts` include:
- `id` - some (ideally globally unique) string identifier for the log.
- `valueEncoding` - a [levelup-style](https://github.com/Level/levelup#options)
encoding string or object (e.g. `"json"`)
- `hash(links, value)` - a hash function that runs synchronously. Defaults to a
SHA-256 implementation.
- `asyncHash(links, value, cb)` - an asynchronous hash function with node-style
callback (`cb(err, hash)`).
- `identity`, `sign`, `verify` - values for creating a cryptographically signed
feed. See below.
You can also pass in an `identity` and `sign`/`verify` functions which can be
used to create a signed log:
``` js
{
identity: aPublicKeyBuffer, // will be added to all nodes you insert
sign: function (node, cb) {
// will be called with all nodes you add
var signatureBuffer = someCrypto.sign(node.key, mySecretKey)
cb(null, signatureBuffer)
},
verify: function (node, cb) {
// will be called with all nodes you receive
if (!node.signature) return cb(null, false)
cb(null, someCrypto.verify(node.key, node.signature. node.identity))
}
}
```
#### `log.add(links, value, opts={}, [cb])`
Add a new node to the graph. `links` should be an array of node keys that this node links to.
If it doesn't link to any nodes use `null` or an empty array. `value` is the value that you want to store
in the node. This should be a string or a buffer. The callback is called with the inserted node:
``` js
log.add([link], value, function(err, node) {
// node looks like this
{
change: ... // the change number for this node in the local log
key: ... // the hash of the node. this is also the key of the node
value: ... // the value (as the valueEncoding type, default buffer) you inserted
log: ... // the peer log this node was appended to
seq: ... // the peer log seq number
links: ['hash-of-link-1', ...]
}
})
```
Optionally supply an `opts.valueEncoding`.
#### `log.append(value, opts={}, [cb])`
Add a value that links all the current heads.
Optionally supply an `opts.valueEncoding`.
#### `log.batch(docs, opts={}, [cb])`
Add many documents atomically to the log at once: either all the docs are
inserted successfully or nothing is inserted.
`docs` is an array of objects where each object looks like:
``` js
{
links: [...] // array of ancestor node keys
value: ... // the value to insert
}
```
The callback `cb(err, nodes)` is called with an array of `nodes`. Each `node` is
of the form described in the `log.add()` section.
You may specify an `opts.valueEncoding`.
#### `log.get(hash, opts={}, cb)`
Lookup a node by its hash. Returns a node similar to `.add` above.
Optionally supply an `opts.valueEncoding`.
#### `log.heads(opts={}, cb)`
Get the heads of the graph as a list. A head is node that no other node
links to.
``` js
log.heads(function(err, heads) {
console.log(heads) // prints an array of nodes
})
```
The method also returns a stream of heads which is useful
if, for some reason, your graph has A LOT of heads
``` js
var headsStream = log.heads()
headsStream.on('data', function(node) {
console.log('head:', node)
})
headsStream.on('end', function() {
console.log('(no more heads)')
})
```
Optionally supply an `opts.valueEncoding`.
#### `changesStream = log.createReadStream([options])`
Tail the changes feed from the log. Everytime you add a node to the graph
the changes feed is updated with that node.
``` js
var changesStream = log.createReadStream({live:true})
changesStream.on('data', function(node) {
console.log('change:', node)
})
```
Options include:
``` js
{
since: changeNumber // only returns changes AFTER the number
live: false // never close the change stream
tail: false // since = lastChange
limit: number // only return up to `limit` changes
until: number // (for non-live streams) only returns changes BEFORE the number
valueEncoding: 'binary'
}
```
#### `replicationStream = log.replicate([options])`
Replicate the log to another one using a replication stream.
Simply pipe your replication stream together with another log's replication stream.
``` js
var l1 = hyperlog(db1)
var l2 = hyperlog(db2)
var s1 = l1.createReplicationStream()
var s2 = l2.createReplicationStream()
s1.pipe(s2).pipe(s1)
s1.on('end', function() {
console.log('replication ended')
})
```
Options include:
``` js
{
mode: 'push' | 'pull' | 'sync', // set replication mode. defaults to sync
live: true, // keep the replication stream open. defaults to false
metadata: someBuffer, // send optional metadata as part of the handshake
frame: true // frame the data with length prefixes. defaults to true
}
```
If you send `metadata` it will be emitted as an `metadata` event on the stream.
A detailed write up on how the graph replicates will be added later.
#### log.on('preadd', function (node) {})
On the same tick as `log.add()` is called, this event fires with the `node`
about to be inserted into the log. At this stage of the add process, node has
these properties:
* `node.log`
* `node.key`
* `node.value`
* `node.links`
#### log.on('add', function (node) {})
After a node has been successfully added to the log, this event fires with the
full `node` object that the callback to `.add()` gets.
#### log.on('reject', function (node) {})
When a node is rejected, this event fires. Otherwise the `add` event will fire.
You can track `preadd` events against both `add` and `reject` events in
combination to know when the log is completely caught up.
## Hyperlog Hygiene
A hyperlog will refer to potentially *many* different logs as it replicates with
others, each with its own ID. Bear in mind that each hyperlog's underlying
leveldb contains a notion of what its *own* local ID is. If you make a copy of a
hyperlog's leveldb and write different data to each copy, the results are
unpredictable and likely disastrous. Always only use the included replication
mechanism for making hyperlog copies!
## License
MIT