{"id":13930101,"url":"https://github.com/Telios-org/nebula-drive","last_synced_at":"2025-07-19T12:31:34.740Z","repository":{"id":45635062,"uuid":"401697618","full_name":"Telios-org/nebula-drive","owner":"Telios-org","description":"Real-time distributed storage for files and key value databases built on top of Hypercore Protocol","archived":false,"fork":false,"pushed_at":"2022-01-07T15:25:35.000Z","size":1245,"stargazers_count":10,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-25T02:50:02.509Z","etag":null,"topics":["hypercore-protocol","p2p"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Telios-org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":["Telios-org"]}},"created_at":"2021-08-31T12:37:53.000Z","updated_at":"2023-07-04T22:43:42.000Z","dependencies_parsed_at":"2022-09-15T17:51:10.316Z","dependency_job_id":null,"html_url":"https://github.com/Telios-org/nebula-drive","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Telios-org%2Fnebula-drive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Telios-org%2Fnebula-drive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Telios-org%2Fnebula-drive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Telios-org%2Fnebula-drive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Telios-org","download_url":"https://codeload.github.com/Telios-org/nebula-drive/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226607592,"owners_count":17658481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hypercore-protocol","p2p"],"created_at":"2024-08-07T18:04:42.662Z","updated_at":"2024-11-26T19:30:49.402Z","avatar_url":"https://github.com/Telios-org.png","language":"JavaScript","funding_links":["https://github.com/sponsors/Telios-org"],"categories":["p2p"],"sub_categories":[],"readme":"# Nebula Drive\n\n![Build Status](https://github.com/Telios-org/nebula-drive/actions/workflows/node.js.yml/badge.svg)\n\n#### ⚠️ This version has been deprecated and is no longer supported. Please visit the latest version [here](https://github.com/Telios-org/nebula) ⚠️\n\nNebula drives are real-time distributed storage for files and key value databases built on top of [Hypercore Protocol](https://hypercore-protocol.org/). This project exists because the [Telios](https://telios.io) email client needed a way to distribute and store encrypted emails on user's local file systems over a peer-to-peer (P2P) network. A lot of inspiration was taken from [Hyperdrive](https://github.com/hypercore-protocol/hyperdrive), but Hyperdrive didn't have options for fine-grain access control, multiple writers, and the ability to delete files from disk once added to the drives.\n\nNebula drives come with a handful of useful features like:\n- __Shareable over company firewalls and mobile networks__: The P2P network runs on [Hyperswarm](https://github.com/hyperswarm/hyperswarm) which has the ability to hole-punch through most company firewalls and mobile connections.\n- __Access Control__: Control access to each file by sharing the file's hash and the drive's discovery key.\n- __Multiwriter__: Drives can have multiple peers with write access by sharing and adding eachother's diff keys.\n- __Collections__: Along with files, drives can create and share simple key value btree databases built on [Hyperbee](https://github.com/hypercore-protocol/hyperbee). Collections also have the option to be encrypted with a secret key.\n\n### TODOs:\n- [x] Connect to drives behind corporate firewalls and mobile networks\n- [x] Create and share key value databases between peers\n- [x] Upgrade multiwriter to Hypercore v10\n- [ ] Share files by only their hash much like [IPFS](https://docs.ipfs.io/concepts/how-ipfs-works/)\n- [ ] Upgrade access control to limit sharing by a peer's public key\n- [ ] Turn an existing directory into a drive and watch for changes\n- [ ] Upgrade collections to be closer to MongoDB with [Hyperbeedeebee](https://github.com/RangerMauve/hyperbeedeebee)\n\n## Installation\n\n```js\nnpm i @telios/nebula-drive\n```\n\n## Usage\n\n```js\nconst Drive = require('nebula-drive')\n\nconst encryptionKey = Buffer.alloc(32, 'hello world')\n\nconst localDrive = new Drive(__dirname + \"/drive\", null, { \n  keyPair,\n  encryptionKey,\n  swarmOpts: {\n    server: true,\n    client: true\n  }\n})\n\nawait localDrive.ready()\n\n// Key to be shared with other devices or services that want to seed this drive\nconst drivePubKey = localDrive.publicKey\n\n// Clone a remote drive\nconst remoteDrive = new Drive(__dirname + \"/drive_remote\", drivePubKey, { \n  keyPair,\n  swarmOpts: {\n    server: true,\n    client: true\n  }\n})\n\nawait remoteDrive.ready()\n\n\nlocalDrive.on('file-sync', file =\u003e {\n  // Local drive has synced somefile.json from remote drive\n})\n\nawait remoteDrive.writeFile('/dest/path/on/drive/somefile.json', readableStream)\n\n```\n\n## API / Examples\n\n#### `const drive = new Drive(storagePath, [key], [options])`\n\nCreate a drive to be shared over the network which can be replicated and seeded by other peers.\n\n- `storagePath`: The directory where you want the drive to be created.\n- `key`: The public key of the remote drive you want to clone\n\nOptions include:\n\n```js\n{\n  storage, // Override Hypercore's default random-access-file storage with a different random-access-storage module\n  encryptionKey,  // optionally pass an encryption key to encrypt the drive's database\n  keyPair: { // ed25519 keypair\n    publicKey, \n    secretKey\n  },\n  joinSwarm: true | false // Optionally set whether or not to join hyperswarm when starting the drive. Defaults to true.\n  swarmOpts: { // Set server to true to start this drive as a server and announce its public key to the network\n    server: true | false,\n    client: true | false\n  },\n  checkNetworkStatus: true | false // Listen for when the drive's network status changes\n}\n```\n\n```js\nconst Drive = require('nebula-drive')\n\n// Create a new local drive.\nconst localDrive = new Drive(__dirname + \"/drive\", null, { \n  keyPair,\n  swarmOpts: {\n    server: true,\n    client: true\n  }\n})\n\nawait localDrive.ready()\n\n// Key to be shared with other devices or services that want to seed this drive\nconst drivePubKey = localDrive.publicKey\n\n// Clone a remote drive\nconst remoteDrive = new Drive(__dirname + \"/drive_remote\", drivePubKey, { \n  keyPair,\n  swarmOpts: {\n    server: true,\n    client: true\n  }\n})\n\nawait remoteDrive.ready()\n```\n\n#### `await drive.ready()`\n\nInitialize the drive and all resources needed.\n\n#### `await drive.addPeer(publicKey)`\n\nAdds a remote drive as a new writer. After a peer has been added, the drive will automatically try to reconnect to this peer after every restart.\n\nExample Usage:\n\n```js\n// Local drive on Device A\nconst drive1 = new Drive(__dirname + \"/drive\", null, { \n  keyPair,\n  swarmOpts: {\n    server: true,\n    client: true\n  }\n})\n\n// Local drive on Device B\nconst drive2 = new Drive(__dirname + \"/drive\", null, { \n  keyPair,\n  swarmOpts: {\n    server: true,\n    client: true\n  }\n})\n\n\nawait drive2.addPeer(drive1.publicKey)\n```\n\n#### `await drive.removePeer(publicKey)`\n\nStop replicating with another drive peer.\n\n\n#### `const file = await drive.writeFile(path, readableStream, [opts])`\n\nWrite a file from a readable stream. When choosing to encrypt a file, the encryption key will be passed back in the response. Each file is encrypted with a unique key which should be stored spearately.\n\n- `path`: Full path where the file resides on the local drive `dir/to/my/file.jpg`\n- `readableStream`: Any readableStream `fs.createReadableStream()`\n\nOptions include:\n```js\n// When encrypted is true a key and header value will be returned after the file has been written\n{ \n  encrypted: true \n}\n```\n\n#### `const stream = await drive.readFile(path)`\n\nCreates a readable stream of data from the requested file path.\n\n- `path`: Full path where the file resides on the local drive `dir/to/my/file.jpg`\n\n\u003c!-- #### `const readableStream = await drive.fetchFileByHash(fileHash)`\n\nSimilar to how IPFS uses (content addressing)[https://proto.school/content-addressing/03], a drive can fetch a file by the hash of it's contents. If another drive is announcing the file hash you're looking for then a readable stream will be returned.\n\n- `fileHash`: Hash of the file's contents. --\u003e\n\n#### `const stream = await drive.fetchFileByDriveHash(discoveryKey, fileHash, [opts])`\n\nDrives with many files may not want to announce every file by it's hash due to network bandwidth limits. In this case, a drive has the option of sharing it's `discoveryKey` which peers can use to connect to the drive and then make a request file hash request.\n\n- `discoveryKey`: Remote drive's discovery key `drive.discoveryKey` which is used by peers to request resources from the drive.\n- `fileHash`: Hash of the file being requested on the remote drive.\n- `opts`: If a key and header are passed in then the return stream will be the deciphered data\n  - `key`: Encryption key used for deciphering the encrypted stream. This key is returned from the `drive.writeFile` method.\n  - `header`: Needed for validating the encrypted stream. This gets returned from `drive.writeFile()`.\n\n#### `const stream = drive.decryptFileStream(stream, key, header)`\n\nIf `drive.fetchFileByDriveHash` is returning encrypted data, then `decryptFileStream` will transform that stream and return a new stream of deciphered data.\n\n- `stream`: Readable stream of encrypted data\n- `key`: Encryption key used for deciphering the encrypted stream. This key is returned from the `drive.writeFile` method.\n- `header`: Needed for validating the encrypted stream. This gets returned from `drive.writeFile()`.\n\n#### `await drive.fetchFileBatch(files, cb)`\n\nFetching files as a batch automatically chunks parallel requests in a fixed batch size so a drive can request as many files as it needs without impacting performance.\n\n- `files`: Array of file objects with the following structure\n  - `discovery_key`:  Remote drive's discovery key `drive.discoveryKey` which is used by peers to request resources from the drive.\n  - `hash`: Hash of the file being requested on the remote drive.\n  - `key`: Encryption key used for deciphering the encrypted stream. This key is returned from the `drive.writeFile` method.\n  - `header`: Needed for validating the encrypted stream. This gets returned from `drive.writeFile()`.\n- `cb`: Callback method that runs after every file stream has been initialized. Use this for handling what to do with the individual file streams. Note that this should return a promise.\n\nExample Usage:\n\n```js\n\nawait drive.fetchFileBatch(files, (stream, file) =\u003e {\n  return new Promise((resolve, reject) =\u003e {\n    const writeStream = fs.createWriteStream(`./${file.path}`)\n    pump(stream, writeStream, (err) =\u003e {\n      resolve()\n    })\n  })\n})\n\n```\n\n#### `await drive.close()`\n\nFully close the drive and all of it's resources.\n\n#### `drive.on('message', (peerPubKey, socket) =\u003e {})`\n\nEmitted when the drive has recieved a message from a peer.\n\n- `peerPubKey`: Public key of the peer that sent the message\n- `socket`: The socket returned on this event can be used as a duplex stream for bi-directional communication with the connecting peer. `socket.write` `socket.on('data, data =\u003e {})`\n\n#### `drive.on('file-add', (file, enc) =\u003e {})`\n\nEmitted when a new file has been added to a local drive.\n\n- `file`: A file object\n  - `path`: drive path the file was saved to\n  - `hash`: Hash of the file\n- `enc`: Passes back properties needed to decrypt the file\n  - `key`: Key needed to decrypt the file\n  - `header`: Needed for validating the encrypted stream\n\n#### `drive.on('sync', () =\u003e {})` \n\nEmitted when the drive has synced any remote data.\n\n#### `drive.on('file-sync', (file) =\u003e {})`\n\nEmitted when the drive has synced remote a remote file.\n\n#### `drive.on('file-unlink', (file) =\u003e {})`\n\nEmitted when a file has been deleted on the drive.\n\n#### `drive.on('fetch-error', (err) =\u003e {})`\n\nEmitted when there has been an error downloading from the remote drive\n\n#### `drive.on('network-updated', (network) =\u003e {})`\n\nEmitted when either the internet connection or the drive's connection to Hyperswarm has changed. The drive option `checkNetworkStatus` must be set to true in order for these events to be emitted.\n\nReturns:\n- `network`\n  - `internet`: true|false\n  - `drive`: true|false\n\n#### `const collection = await drive.db.collection(name)`\n\nCreates a new key value collection. Collections are encrypted with the drive's `encryptionKey` (`drive.encryptionKey`) when the key is passed in during initialization.\n\n#### `await collection.put(key, value)`\n\nInserts a new document into the collection. Value should be a JSON object.\n\n#### `await collection.get(key)`\n\nGet a document by it's key\n\n#### `await collection.del(key)`\n\nDeletes a document by it's key\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTelios-org%2Fnebula-drive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTelios-org%2Fnebula-drive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTelios-org%2Fnebula-drive/lists"}