Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/ozymandiasthegreat/autodrive

Dumb, permissionless, but secure, real-time, multi-writer, distributed file system
https://github.com/ozymandiasthegreat/autodrive

autobase distributed-file-system hyperdrive multi-writer p2p

Last synced: 7 months ago
JSON representation

Dumb, permissionless, but secure, real-time, multi-writer, distributed file system

Awesome Lists containing this project

README 

          
# autodrive



Dumb, permissionless, but secure, real-time, multi-writer, distributed file system



## Install



`npm i autodrive`



## Usage



```javascript

import Autodrive from "autodrive"

import b4a from "b4a"

import Corestore from "corestore"

import { once } from "bare-events"

import fs from "fs"



const driveA = new Autodrive(new Corestore("./storageA"))

await driveA.ready()

const driveB = new Autodrive(new Corestore("./storageB"), driveA.key)

await driveB.ready()



await driveA.put("/example.txt", b4a.from("Hello, World!"))

const data = await driveB.get("/example.txt")



const ws = driveB.createWriteStream("/example.txt")

const rs = fs.createReadStream("./file.txt")

rs.pipe(ws)

await once(ws, "finish")



for await (const chunk of driveA.createReadStream("/example.txt")) {

  console.log(b4a.toString(chunk))

}



await driveA.del("/example.txt")

```



## API



#### `const drive = new Autodrive(store, key, options)`



Create new Autodrive. `store` must be an instance of Corestore.

If you provide a key you get read access to the drive at key.

Options are passed-through to underlying autobase.



#### `await drive.ready()`



Wait until internal state is loaded. Use it once before accessing synchronous properties like `key`.



#### `await drive.close()`



Fully close this drive and all internal hypercores used by it.



#### `drive.id`



The ID of the drive, this consists of base-z-32 encoded key of the drive.



#### `drive.key`



The binary hash of the drive's signers. Giving this someone grants full read access to the drive.



#### `drive.discoveryKey`



Hash of the `drive.key`. Use this to e.g. replicate drive over Hyperswarm without leaking read access.



#### `drive.keyPair`



For writable drives this is the key pair used to authenticate local writer. `null` otherwise.



#### `drive.encrypted`



Boolean indicating whether this drive uses encryption.



#### `drive.encryptionKey`



Buffer containing the key used to encrypt this drive. `null` otherwise.



#### `drive.local`



Local writer hypercore. If the drive is not writable this is null.



#### `drive.db`



The underlying file database, an instance of Hyperbee.



#### `drive.store`



The Corestore used to instantiate this drive.



#### `drive.corestore`



The Corestore used to instantiate this drive. Hyperdrive compat.



#### `drive.version`



The current version of the drive's file database.



#### `drive.writable`



Boolean indicating whether this drive has write access.



#### `drive.readable`



Should always be true.



#### `drive.supportsMetadata`



Whether this drive supports file metadata. Should always be true.



#### `const blobs = await drive.getBlobs(forWriter?: Uint8Array)`



Get the Hyperblobs instance for the given writer. If `forWriter` is omitted, returns Hyperblobs for the local writer.



#### `await drive.addWriter(key: Uint8Array, { indexer?: boolean })`



Add writer to this drive. `key` can be obtained by accessing `drive.local.key` on the writer to be added. Can only be called by existing writer.



#### `await drive.removeWriter(key: Uint8Array)`



Remove a writer from this drive. `key` should be `drive.local.key` of the writer to be removed. Can only be called by existing writer.



#### `await drive.update()`



Try to download the latest version of the files database.



#### `const stream = drive.replicate(isInitiator: boolean)`



See [`Corestore.replicate()`](https://github.com/holepunchto/corestore#const-stream--storereplicateoptsorstream)



#### `const done = drive.findingPeers()`



Indicating that you're finding peers in the background, operations will be on hold until `done()` is called.

Call `done()` when current iteration finishes, e.g. after `swarm.flush()`.



#### `const checkout = drive.checkout(version: number)`



Checkout an earlier database version. This allows to go back in time to an earlier filesystem state.



#### `const mirror = drive.mirror(dest: Drive, options?: MirrorDrive Options)`



Mirror this drive into an instance of either another Autodrive, Hyperdrive, or Localdrive. Returns an instance of MirrorDrive.



#### `const item = await drive.entry(path: string | Node, options?: { follow?: boolean, wait?: boolean, timeout?: number })`



Get an entry in the files database.



- If `follow` is true and the entry is a symlink, this will resolve the destination.

- `wait: false` returns immediately, if the entry is not locally available, it will return `null`.

- `timeout` specifies how long to wait for the entry data to download. The default timeout of 0 means wait indefinitely.



#### `const buffer = await drive.get(path: string | Node, options?: { follow?: boolean, wait?: boolean, timeout?: number })`



Get the file contents. Takes either a path, or a Node object returned by e.g. `drive.entry()`.

Options are the same as `drive.entry()`.



#### `await drive.put(path: string, data: Uint8Array, options?: { executable?: boolean, metadata?: JSON })`



Write data to the filesystem, either creating or updating a file. For large data consider `drive.createWriteStream()`.



#### `await drive.del(path: string)`



Delete a file from filesystem.



#### `for await (const chunk of drive.createReadStream(path: string, options?: { start?: number, end?: number, length?: number, follow?: boolean, wait?: boolean, timeout?: number }))`



Get a readable stream of file data at path. Options are same as `drive.entry()` with the addition of range options:



- `start`: byte offset at which to start reading

- `end`: byte offset at which to stop

- `length`: how many bytes to read



#### `const ws = drive.createWriteStream(path: string, options?: { executable?: boolean, metadata?: JSON })`



Efficiently write data at given path, returns a writable stream. Options are the same as `drive.put()`.



#### `await symlink(path: string, dest: string, options?: { executable?: boolean, metadata?: JSON })`



Create a symlink to another location in the filesystem. Options are the same as `drive.put()`.



#### `await drive.exists(path: string, options?: { wait?: boolean, timeout?: number })`



Check if the file at given path exists. Options are the same as `drive.entry()` without the `follow`.



#### `for await (const item of drive.list(folder?: string, options?: { recursive?: boolean }))`



List file entries at a given folder. If `recursive: true` also descends into the subdirectories.



#### `for await (const filename of drive.readdir(folder?: string))`



List filenames of entries at given folder.



#### `const int = drive.compare(a: Node, b: Node)`



Returns `0` if entries are the same, `1` if `a` is older, and `-1` if `b` is older.



#### `for await (const { left: Node, right: Node } of drive.diff(version: number, folder?: string, options?: { limit?: number }))`



Efficiently create a stream of the shallow changes to folder between `version` and `drive.version`.

Each entry is sorted by key.

If an entry exists in `drive.version` of the folder but not in `version`, then `left` is set and `right` will be `null`, and vice versa.



#### `const watcher = drive.watch(folder?: string)`



Returns an iterator that listens on `folder` to yield changes, by default on `/`.



Usage example:



```javascript

for await (const [current, previous] of watcher) {

  console.log(current.version)

  console.log(previous.version)

}

```



Those current and previous are Autodrive snapshots that are auto-closed before next value.

Don't close those snapshots yourself because they're used internally, let them be auto-closed.



##### `await watcher.ready()`



Waits until the watcher is loaded and detecting changes.



##### `await watcher.destroy()`



Stops the watcher. You could also stop it by using break in the loop.



##### `await drive.download(folder?: string, options?: { recursive?: boolean })`



Downloads the blobs corresponding to all entries in the drive at paths prefixed with `folder`.



#### `await drive.downloadDiff(version: number, folder?: string, options?: { limit?: number })`



Downloads all the blobs in `folder` corresponding to entries in `drive.checkout(version)` that are not in `drive.version`.

In other words, downloads all the blobs added to `folder` up to `version` of the drive.



#### `const { blocks: number } = await drive.clear(path: string | Node, options?: { diff?: boolean })`



Deletes the blob from storage to free up space, but the file structure reference is kept.

Pass `{ diff: true }` to get stats about cleared blocks. Otherwise returns void.



#### `const { blocks: number } = await drive.clearAll(options?: { diff?: boolean })`



Clear all blobs from storage, freeing up space. Options are the same as `drive.clear()`



#### `await drive.purge()`



Delete all data used by this drive from the local filesystem, effectively erasing the drive.



## Types



### Blob



The ID object identifying data store in blobs hypercores,

paired with a source this allows to retrieve the actual binary data.



```typescript

interface Blob {

  blockOffset: number

  blockLength: number

  byteOffset: number

  /** The size of this file in bytes */

  byteLength: number

}

```



### Entry



The main file object Autodrive deals with.



```typescript

interface Entry {

  /** The writer core key that produced this version of a file */

  source: Uint8Array

  /** Blob object pointing to the actual file data */

  blob?: Blob | null

  executable: boolean

  /** If this is a symlink, linkname will point to the destination */

  linkname?: string | null

  /** Any additional data you want to associate with the file, stored as JSON */

  metadata?: JsonValue

}

```



### Node



Carry over from hyperbee, Node wraps entry with additional metadata.



```typescript

interface Item {

  /** The database version when this entry was written */

  seq: number

  /** The path at which this entry is stored */

  key: string

  /** The Entry object */

  value: Entry

}

```