Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mafintosh/hyperdrive

Hyperdrive is a secure, real time distributed file system
https://github.com/mafintosh/hyperdrive

Last synced: 3 months ago
JSON representation

Hyperdrive is a secure, real time distributed file system

Awesome Lists containing this project

README 

        
# Hyperdrive



[See API docs at docs.holepunch.to](https://docs.holepunch.to/building-blocks/hyperdrive)



Hyperdrive is a secure, real-time distributed file system



## Install



```sh

npm install hyperdrive

```



## Usage



```js

const Hyperdrive = require('hyperdrive')

const Corestore = require('corestore')



const store = new Corestore('./storage')

const drive = new Hyperdrive(store)



await drive.put('/blob.txt', Buffer.from('example'))

await drive.put('/images/logo.png', Buffer.from('..'))

await drive.put('/images/old-logo.png', Buffer.from('..'))



const buffer = await drive.get('/blob.txt')

console.log(buffer) // =>  "example"



const entry = await drive.entry('/blob.txt')

console.log(entry) // => { seq, key, value: { executable, linkname, blob, metadata } }



await drive.del('/images/old-logo.png')



await drive.symlink('/images/logo.shortcut', '/images/logo.png')



for await (const file of drive.list('/images')) {

  console.log('list', file) // => { key, value }

}



const rs = drive.createReadStream('/blob.txt')

for await (const chunk of rs) {

  console.log('rs', chunk) // => 

}



const ws = drive.createWriteStream('/blob.txt')

ws.write('new example')

ws.end()

ws.once('close', () => console.log('file saved'))

```



## API



#### `const drive = new Hyperdrive(store, [key])`



Creates a new Hyperdrive instance. `store` must be an instance of `Corestore`.



By default it uses the core at `{ name: 'db' }` from `store`, unless you set the public `key`.



#### `await drive.ready()`



Waits until internal state is loaded.



Use it once before reading synchronous properties like `drive.discoveryKey`, unless you called any of the other APIs.



#### `await drive.close()`



Fully close this drive, including its underlying Hypercore backed datastructures.



#### `drive.corestore`



The Corestore instance used as storage.



#### `drive.db`



The underlying Hyperbee backing the drive file structure.



#### `drive.core`



The Hypercore used for `drive.db`.



#### `drive.id`



String containing the id (z-base-32 of the public key) identifying this drive.



#### `drive.key`



The public key of the Hypercore backing the drive.



#### `drive.discoveryKey`



The hash of the public key of the Hypercore backing the drive.



Can be used as a `topic` to seed the drive using Hyperswarm.



#### `drive.contentKey`



The public key of the [Hyperblobs](https://github.com/holepunchto/hyperblobs) instance holding blobs associated with entries in the drive.



#### `drive.writable`



Boolean indicating if we can write or delete data in this drive.



#### `drive.readable`



Boolean indicating if we can read from this drive. After closing the drive this will be `false`.



#### `drive.version`



Number that indicates how many modifications were made, useful as a version identifier.



#### `drive.supportsMetadata`



Boolean indicating if the drive handles or not metadata. Always `true`.



#### `await drive.put(path, buffer, [options])`



Creates a file at `path` in the drive. `options` are the same as in `createWriteStream`.



#### `const buffer = await drive.get(path, [options])`



Returns the blob at `path` in the drive. If no blob exists, returns `null`.



It also returns `null` for symbolic links.



`options` include:

```js

{

  wait: true, // Wait for block to be downloaded

  timeout: 0 // Wait at max some milliseconds (0 means no timeout)

}

```



#### `const entry = await drive.entry(path, [options])`



Returns the entry at `path` in the drive. It looks like this:

```js

{

  seq: Number,

  key: String,

  value: {

    executable: Boolean, // Whether the blob at path is an executable

    linkname: null, // If entry not symlink, otherwise a string to the entry this links to

    blob: { // Hyperblobs id that can be used to fetch the blob associated with this entry

      blockOffset: Number,

      blockLength: Number,

      byteOffset: Number,

      byteLength: Number

    },

    metadata: null

  }

}

```



`options` include:

```js

{

  follow: false, // Follow symlinks, 16 max or throws an error

  wait: true, // Wait for block to be downloaded

  timeout: 0 // Wait at max some milliseconds (0 means no timeout)

}

```



#### `const exists = await drive.exists(path)`



Returns `true` if the entry at `path` does exists, otherwise `false`.



#### `await drive.del(path)`



Deletes the file at `path` from the drive.



#### `const comparison = drive.compare(entryA, entryB)`



Returns `0` if entries are the same, `1` if `entryA` is older, and `-1` if `entryB` is older.



#### `const cleared = await drive.clear(path, [options])`



Deletes the blob from storage to free up space, but the file structure reference is kept.



`options` include:

```js

{

  diff: false // Returned `cleared` bytes object is null unless you enable this

}

```



#### `const cleared = await drive.clearAll([options])`



Deletes all the blobs from storage to free up space, similar to how `drive.clear()` works.



`options` include:

```js

{

  diff: false // Returned `cleared` bytes object is null unless you enable this

}

```



#### `await drive.purge()`



Purge both cores (db and blobs) from your storage, completely removing all the drive's data.



#### `await drive.symlink(path, linkname)`



Creates an entry in drive at `path` that points to the entry at `linkname`.



If a blob entry currently exists at `path` then it will get overwritten and `drive.get(key)` will return `null`, while `drive.entry(key)` will return the entry with symlink information.



#### `const batch = drive.batch()`



Useful for atomically mutate the drive, has the same interface as Hyperdrive.



#### `await batch.flush()`



Commit a batch of mutations to the underlying drive.



#### `const stream = drive.list(folder, [options])`



Returns a stream of all entries in the drive at paths prefixed with `folder`.



`options` include:

```js

{

  recursive: true | false // Whether to descend into all subfolders or not

}

```



#### `const stream = drive.readdir(folder)`



Returns a stream of all subpaths of entries in drive stored at paths prefixed by `folder`.



#### `const stream = await drive.entries([range], [options])`



Returns a read stream of entries in the drive.



`options` are the same as `Hyperbee().createReadStream([range], [options])`.



#### `const mirror = drive.mirror(out, [options])`



Efficiently mirror this drive into another. Returns a [`MirrorDrive`](https://github.com/holepunchto/mirror-drive#api) instance constructed with `options`.



Call `await mirror.done()` to wait for the mirroring to finish.



#### `const watcher = drive.watch([folder])`



Returns an iterator that listens on `folder` to yield changes, by default on `/`.



Usage example:

```js

for await (const [current, previous] of watcher) {

  console.log(current.version)

  console.log(previous.version)

}

```



Those `current` and `previous` are snapshots that are auto-closed before next value.



Don't close those snapshots yourself because they're used internally, let them be auto-closed.



`await watcher.ready()`



Waits until the watcher is loaded and detecting changes.



`await watcher.destroy()`



Stops the watcher. You could also stop it by using `break` in the loop.



#### `const rs = drive.createReadStream(path, [options])`



Returns a stream to read out the blob stored in the drive at `path`.



`options` include:

```js

{

  start: Number, // `start` and `end` are inclusive

  end: Number,

  length: Number, // `length` overrides `end`, they're not meant to be used together

  wait: true, // Wait for blocks to be downloaded

  timeout: 0 // Wait at max some milliseconds (0 means no timeout)

}

```



#### `const ws = drive.createWriteStream(path, [options])`



Stream a blob into the drive at `path`.



`options` include:

```js

{

  executable: Boolean,

  metadata: null // Extended file information i.e. arbitrary JSON value

}

```



#### `await drive.download(folder, [options])`



Downloads the blobs corresponding to all entries in the drive at paths prefixed with `folder`.



`options` are the same as those for `drive.list(folder, [options])`.



#### `const snapshot = drive.checkout(version)`



Get a read-only snapshot of a previous version.



#### `const stream = drive.diff(version, folder, [options])`



Efficiently create a stream of the shallow changes to `folder` between `version` and `drive.version`.



Each entry is sorted by key and looks like this:

```js

{

  left: Object, // Entry in folder at drive.version for some path

  right: Object, // Entry in folder at drive.checkout(version) for some path

}

```



If an entry exists in `drive.version` of the `folder` but not in `version`, then `left` is set and `right` will be `null`, and vice versa.



#### `await drive.downloadDiff(version, folder, [options])`



Downloads all the blobs in `folder` corresponding to entries in `drive.checkout(version)` that are not in `drive.version`.



In other words, downloads all the blobs added to `folder` up to `version` of the drive.



#### `await drive.downloadRange(dbRanges, blobRanges)`



Downloads the entries and blobs stored in the [ranges][core-range-docs] `dbRanges` and `blobRanges`.



#### `const done = drive.findingPeers()`



Indicate to Hyperdrive that you're finding peers in the background, requests will be on hold until this is done.



Call `done()` when your current discovery iteration is done, i.e. after `swarm.flush()` finishes.



#### `const stream = drive.replicate(isInitiatorOrStream)`



Usage example:

```js

const swarm = new Hyperswarm()

const done = drive.findingPeers()

swarm.on('connection', (socket) => drive.replicate(socket))

swarm.join(drive.discoveryKey)

swarm.flush().then(done, done)

```



See more about how replicate works at [corestore.replicate][store-replicate-docs].



#### `const updated = await drive.update([options])`



Waits for initial proof of the new drive version until all `findingPeers` are done.



`options` include:

```js

{

  wait: false

}

```



Use `drive.findingPeers()` or `{ wait: true }` to make await `drive.update()` blocking.



#### `const blobs = await drive.getBlobs()`



Returns the [Hyperblobs](https://github.com/holepunchto/hyperblobs) instance storing the blobs indexed by drive entries.



```js

await drive.put('/file.txt', Buffer.from('hi'))



const buffer1 = await drive.get('/file.txt')



const blobs = await drive.getBlobs()

const entry = await drive.entry('/file.txt')

const buffer2 = await blobs.get(entry.value.blob)



// => buffer1 and buffer2 are equals

```



[core-range-docs]: https://github.com/holepunchto/hypercore#const-range--coredownloadrange

[store-replicate-docs]: https://github.com/holepunchto/corestore#const-stream--storereplicateoptsorstream



## License



Apache-2.0