{"id":13447113,"url":"https://github.com/dat-ecosystem-archive/dat-node","last_synced_at":"2025-03-21T17:30:45.860Z","repository":{"id":57121328,"uuid":"62181610","full_name":"dat-ecosystem-archive/dat-node","owner":"dat-ecosystem-archive","description":"Node module for creating dat compatible tools on file systems [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ] ","archived":true,"fork":false,"pushed_at":"2022-01-06T16:02:56.000Z","size":797,"stargazers_count":503,"open_issues_count":35,"forks_count":64,"subscribers_count":28,"default_branch":"master","last_synced_at":"2024-05-22T19:09:51.467Z","etag":null,"topics":["dat","dat-protocol","hyperdrive"],"latest_commit_sha":null,"homepage":"https://dat.foundation","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dat-ecosystem-archive.png","metadata":{"files":{"readme":"readme.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-06-28T23:54:23.000Z","updated_at":"2024-04-08T12:22:29.000Z","dependencies_parsed_at":"2022-08-22T21:30:41.698Z","dependency_job_id":null,"html_url":"https://github.com/dat-ecosystem-archive/dat-node","commit_stats":null,"previous_names":["datproject/dat-node"],"tags_count":82,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dat-ecosystem-archive%2Fdat-node","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dat-ecosystem-archive%2Fdat-node/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dat-ecosystem-archive%2Fdat-node/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dat-ecosystem-archive%2Fdat-node/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dat-ecosystem-archive","download_url":"https://codeload.github.com/dat-ecosystem-archive/dat-node/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244153316,"owners_count":20406996,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dat","dat-protocol","hyperdrive"],"created_at":"2024-07-31T05:01:08.550Z","updated_at":"2025-03-21T17:30:45.562Z","avatar_url":"https://github.com/dat-ecosystem-archive.png","language":"JavaScript","readme":"[![deprecated](http://badges.github.io/stability-badges/dist/deprecated.svg)](https://dat-ecosystem.org/) \n\nMore info on active projects and modules at [dat-ecosystem.org](https://dat-ecosystem.org/) \u003cimg src=\"https://i.imgur.com/qZWlO1y.jpg\" width=\"30\" height=\"30\" /\u003e \n\n---\n\n# dat-node\n\n\u003e **dat-node** is a high-level module for building Dat applications on the file system.\n\n[![npm][0]][1] [![Travis][2]][3] [![Test coverage][4]][5] [![Greenkeeper badge](https://badges.greenkeeper.io/datproject/dat-node.svg)](https://greenkeeper.io/)\n\n**For a lower-level API for building your own applications, use the [Dat SDK](https://github.com/datproject/sdk) which works in Node and the Web**\n\n#### Compatibility\n\nNote: Version 4 of `dat-node` is not compatible with earlier versions (3.5.15\nand below). \n\n#### Dat Project Documentation \u0026 Resources\n\n* [dat project Docs](http://docs.datproject.org/)\n* [dat protocol](https://www.datprotocol.com/)\n* Chat on [Gitter](https://gitter.im/datproject/discussions) or [#dat on IRC](http://webchat.freenode.net/?channels=dat)\n\n### Features\n\n* High-level glue for common dat:// and [hyperdrive](https://github.com/mafintosh/hyperdrive) modules.\n* Sane defaults and consistent management of storage \u0026 secret keys across applications, using [dat-storage](https://github.com/datproject/dat-storage).\n* Easily connect to the dat:// network with holepunching, using [hyperswarm](https://github.com/hyperswarm)\n* Import files from the file system, using [mirror-folder](https://github.com/mafintosh/mirror-folder/)\n* Serve dats over http with [hyperdrive-http](https://github.com/joehand/hyperdrive-http)\n* Access APIs to lower level modules with a single `require`!\n\n#### Browser Support\n\nMany of our dependencies work in the browser, but `dat-node` is tailored for file system applications. See [dat-sdk](https://github.com/datproject/sdk) if you want to build browser-friendly applications.\n\n## Example\n\nTo send files via dat:\n\n1. Tell dat-node where the files are.\n2. Import the files.\n3. Share the files on the dat network! (And share the link)\n\n```js\nvar Dat = require('dat-node')\n\n// 1. My files are in /joe/cat-pic-analysis\nDat('/joe/cat-pic-analysis', function (err, dat) {\n  if (err) throw err\n\n  // 2. Import the files\n  dat.importFiles()\n\n  // 3. Share the files on the network!\n  dat.joinNetwork()\n  // (And share the link)\n  console.log('My Dat link is: dat://' + dat.key.toString('hex'))\n})\n```\n\nThese files are now available to share over the dat network via the key printed in the console.\n\nTo download the files, you can make another dat-node instance in a different folder. This time we also have three steps:\n\n1. Tell dat where I want to download the files.\n2. Tell dat what the link is.\n3. Join the network and download!\n\n```js\nvar Dat = require('dat-node')\n\n// 1. Tell Dat where to download the files\nDat('/download/cat-analysis', {\n  // 2. Tell Dat what link I want\n  key: '\u003cdat-key\u003e' // (a 64 character hash from above)\n}, function (err, dat) {\n  if (err) throw err\n\n  // 3. Join the network \u0026 download (files are automatically downloaded)\n  dat.joinNetwork()\n})\n```\n\nThat's it! By default, all files are automatically downloaded when you connect to the other users.\n\nDig into more use cases below and please let us know if you have questions! You can [open a new issue](https://github.com/datproject/dat-node/issues) or talk to nice humans in [our chat room](https://gitter.im/datproject/discussions).\n\n### Example Applications\n\n* [CLI](https://github.com/datproject/dat): We use dat-node in the dat CLI.\n* [Desktop](https://github.com/datproject/dat-desktop): The Dat Desktop application manages multiple dat-node instances via [dat-worker](https://github.com/juliangruber/dat-worker).\n* See the [examples folder](examples) for a minimal share + download usage.\n* And more! Let us know if you have a neat dat-node application to add here.\n\n## Usage\n\nAll dat-node applications have a similar structure around three main elements:\n\n1. **Storage** - where the files and metadata are stored.\n2. **Network** - connecting to other users to upload or download data.\n3. **Adding Files** - adding files from the file system to the hyperdrive archive.\n\nWe'll go through what these are for and a few of the common usages of each element.\n\n### Storage\n\nEvery dat archive has **storage**, this is the required first argument for dat-node. By default, we use [dat-storage](http://github.com/datproject/dat-storage) which stores the secret key in `~/.dat/` and the rest of the data in `dir/.dat`. Other common options are:\n\n* **Persistent storage**: Stored files in `/my-dir` and metadata in `my-dir/.dat` by passing `/my-dir` as the first argument.\n* **Temporary Storage**: Use the `temp: true` option to keep metadata stored in memory.\n\n```js\n// Permanent Storage\nDat('/my-dir', function (err, dat) {\n  // Do Dat Stuff\n})\n\n// Temporary Storage\nDat('/my-dir', {temp: true}, function (err, dat) {\n  // Do Dat Stuff\n})\n```\n\nBoth of these will import files from `/my-dir` when doing `dat.importFiles()` but only the first will make a `.dat` folder and keep the metadata on disk.\n\nThe storage argument can also be passed through to hyperdrive for more advanced storage use cases.\n\n### Network\n\nDat is all about the network! You'll almost always want to join the network right after you create your Dat:\n\n```js\nDat('/my-dir', function (err, dat) {\n  dat.joinNetwork()\n  dat.network.on('connection', function () {\n    console.log('I connected to someone!')\n  })\n})\n```\n\n#### Downloading Files\n\nRemember, if you are downloading - metadata and file downloads will happen automatically once you join the network!\n\ndat runs on a peer to peer network, sometimes there may not be anyone online for a particular key. You can make your application more user friendly by using the callback in `joinNetwork`:\n\n```js\n// Downloading \u003ckey\u003e with joinNetwork callback\nDat('/my-dir', {key: '\u003ckey\u003e'}, function (err, dat) {\n  dat.joinNetwork(function (err) {\n    if (err) throw err\n\n    // After the first round of network checks, the callback is called\n    // If no one is online, you can exit and let the user know.\n    if (!dat.network.connected || !dat.network.connecting) {\n      console.error('No users currently online for that key.')\n      process.exit(1)\n    }\n  })\n})\n```\n\n##### Download on Demand\n\nIf you want to control what files and metadata are downloaded, you can use the sparse option:\n\n```js\n// Downloading \u003ckey\u003e with sparse option\nDat('/my-dir', {key: '\u003ckey\u003e', sparse: true}, function (err, dat) {\n  dat.joinNetwork()\n\n  // Manually download files via the hyperdrive API:\n  dat.archive.readFile('/cat-locations.txt', function (err, content) {\n    console.log(content) // prints cat-locations.txt file!\n  })\n})\n```\n\nDat will only download metadata and content for the parts you request with `sparse` mode!\n\n### Importing Files\n\nThere are many ways to get files imported into an archive! Dat node provides a few basic methods. If you need more advanced imports, you can use the `archive.createWriteStream()` methods directly.\n\nBy default, just call `dat.importFiles()` to import from the directory you initialized with. You can watch that folder for changes by setting the watch option:\n\n```js\nDat('/my-data', function (err, dat) {\n  if (err) throw err\n\n  var progress = dat.importFiles({watch: true}) // with watch: true, there is no callback\n  progress.on('put', function (src, dest) {\n    console.log('Importing ', src.name, ' into archive')\n  })\n})\n```\n\nYou can also import from another directory:\n\n```js\nDat('/my-data', function (err, dat) {\n  if (err) throw err\n\n  dat.importFiles('/another-dir', function (err) {\n    console.log('done importing another-dir')\n  })\n})\n```\n\nThat covers some of the common use cases, let us know if there are more to add! Keep reading for the full API docs.\n\n## API\n\n### `Dat(dir|storage, [opts], callback(err, dat))`\n\nInitialize a Dat Archive in `dir`. If there is an existing Dat Archive, the archive will be resumed.\n\n#### Storage\n\n* `dir` (Default) - Use [dat-storage](https://github.com/datproject/dat-storage) inside `dir`. This stores files as files, sleep files inside `.dat`, and the secret key in the user's home directory.\n* `dir` with `opts.latest: false` - Store as SLEEP files, including storing the content as a `content.data` file. This is useful for storing all history in a single flat file.\n* `dir` with `opts.temp: true` - Store everything in memory (including files).\n* `storage` function - pass a custom storage function along to hyperdrive, see dat-storage for an example.\n\nMost options are passed directly to the module you're using (e.g. `dat.importFiles(opts)`. However, there are also some initial `opts` can include:\n\n```js\nopts = {\n  key: '\u003cdat-key\u003e', // existing key to create archive with or resume\n  temp: false, // Use random-access-memory as the storage.\n\n  // Hyperdrive options\n  sparse: false // download only files you request\n}\n```\n\nThe callback, `cb(err, dat)`, includes a `dat` object that has the following properties:\n\n* `dat.key`: key of the dat (this will be set later for non-live archives)\n* `dat.archive`: Hyperdrive archive instance.\n* `dat.path`: Path of the Dat Archive\n* `dat.live`: `archive.live`\n* `dat.writable`: Is the `archive` writable?\n* `dat.resumed`: `true` if the archive was resumed from an existing database\n* `dat.options`: All options passed to Dat and the other submodules\n\n### Module Interfaces\n\n**`dat-node` provides an easy interface to common Dat modules for the created Dat Archive on the `dat` object provided in the callback:**\n\n#### `var network = dat.joinNetwork([opts], [cb])`\n\nJoin the network to start transferring data for `dat.key`, using [discovery-swarm](https://github.com/mafintosh/discovery-swarm). You can also use `dat.join([opts], [cb])`.\n\nIf you specify `cb`, it will be called *when the first round* of discovery has completed. This is helpful to check immediately if peers are available and if not fail gracefully, more similar to http requests.\n\nReturns a `network` object with properties:\n\n* `network.connected` - number of peers connected\n* `network.on('listening')` - emitted with network is listening\n* `network.on('connection', connection, info)` - Emitted when you connect to another peer. Info is an object that contains info about the connection\n\n##### Network Options\n\n`opts` are passed to discovery-swarm, which can include:\n\n```js\nopts = {\n  upload: true, // announce and upload data to other peers\n  download: true, // download data from other peers\n  port: 3282, // port for discovery swarm\n  utp: true, // use utp in discovery swarm\n  tcp: true // use tcp in discovery swarm\n}\n\n//Defaults from datland-swarm-defaults can also be overwritten:\n\nopts = {\n  dns: {\n    server: // DNS server\n    domain: // DNS domain\n  }\n  dht: {\n    bootstrap: // distributed hash table bootstrapping nodes\n  }\n}\n```\n\nReturns a [discovery-swarm](https://github.com/mafintosh/discovery-swarm) instance.\n\n#### `dat.leaveNetwork()` or `dat.leave()`\n\nLeaves the network for the archive.\n\n#### `var importer = dat.importFiles([src], [opts], [cb])`\n\n**Archive must be writable to import.**\n\nImport files to your Dat Archive from the directory using [mirror-folder](https://github.com/mafintosh/mirror-folder/).\n\n* `src` - By default, files will be imported from the folder where the archive was initiated. Import files from another directory by specifying `src`.\n* `opts` - options passed to mirror-folder (see below).\n* `cb` - called when import is finished.\n\nReturns a `importer` object with properties:\n\n* `importer.on('error', err)`\n* `importer.on('put', src, dest)` - file put started. `src.live` is true if file was added by file watch event.\n* `importer.on('put-data', chunk)` - chunk of file added\n* `importer.on('put-end', src, dest)` - end of file write stream\n* `importer.on('del', dest)` - file deleted from dest\n* `importer.on('end')` - Emits when mirror is done (not emitted in watch mode)\n* If `opts.count` is true:\n  * `importer.on('count', {files, bytes})` - Emitted after initial scan of src directory. See import progress section for details.\n  * `importer.count` will be `{files, bytes}` to import after initial scan.\n  * `importer.putDone` will track `{files, bytes}` for imported files.\n\n##### Importer Options\n\nOptions include:\n\n```js\nvar opts = {\n  count: true, // do an initial dry run import for rendering progress\n  ignoreHidden: true, // ignore hidden files  (if false, .dat will still be ignored)\n  ignoreDirs: true, // do not import directories (hyperdrive does not need them and it pollutes metadata)\n  useDatIgnore: true, // ignore entries in the `.datignore` file from import dir target.\n  ignore: // (see below for default info) anymatch expression to ignore files\n  watch: false, // watch files for changes \u0026 import on change (archive must be live)\n}\n```\n\n##### Ignoring Files\n\nYou can use a `.datignore` file in the imported directory, `src`, to ignore any the user specifies. This is done by default.\n\n`dat-node` uses [dat-ignore](https://github.com/joehand/dat-ignore) to provide a default ignore option, ignoring the `.dat` folder and all hidden files or directories. Use `opts.ignoreHidden = false` to import hidden files or folders, except the `.dat` directory.\n\n*It's important that the `.dat` folder is not imported because it contains a private key that allows the owner to write to the archive.*\n\n#### `var stats = dat.trackStats()`\n\n##### `stats.on('update')`\n\nEmitted when archive stats are updated. Get new stats with `stats.get()`.\n\n##### `var st = stats.get()`\n\n`dat.trackStats()` adds a `stats` object to `dat`.  Get general archive stats for the latest version:\n\n```js\n{\n  files: 12,\n  byteLength: 1234,\n  length: 4, // number of blocks for latest files\n  version: 6, // archive.version for these stats\n  downloaded: 4 // number of downloaded blocks for latest\n}\n```\n\n##### `stats.network`\n\nGet upload and download speeds: `stats.network.uploadSpeed` or `stats.network.downloadSpeed`. Transfer speeds are tracked using [hyperdrive-network-speed](https://github.com/joehand/hyperdrive-network-speed/).\n\n##### `var peers = stats.peers`\n\n* `peers.total` - total number of connected peers\n* `peers.complete` - connected peers with all the content data\n\n#### `var server = dat.serveHttp(opts)`\n\nServe files over http via [hyperdrive-http](https://github.com/joehand/hyperdrive-http). Returns a node http server instance.\n\n```js\nopts = {\n  port: 8080, // http port\n  live: true, // live update directory index listing\n  footer: 'Served via Dat.', // Set a footer for the index listing\n  exposeHeaders: false // expose dat key in headers\n}\n```\n\n#### `dat.pause()`\n\nPause all upload \u0026 downloads. Currently, this is the same as `dat.leaveNetwork()`, which leaves the network and destroys the swarm. Discovery will happen again on `resume()`.\n\n#### `dat.resume()`\n\nResume network activity. Current, this is the same as `dat.joinNetwork()`.\n\n#### `dat.close(cb)`\n\nStops replication and closes all the things opened for dat-node, including:\n\n* `dat.archive.close(cb)`\n* `dat.network.close(cb)`\n* `dat.importer.destroy()` (file watcher)\n\n## License\n\nMIT\n\n[0]: https://img.shields.io/npm/v/dat-node.svg?style=flat-square\n[1]: https://npmjs.org/package/dat-node\n[2]: https://img.shields.io/travis/datproject/dat-node/master.svg?style=flat-square\n[3]: https://travis-ci.org/datproject/dat-node\n[4]: https://img.shields.io/codecov/c/github/datproject/dat-node/master.svg?style=flat-square\n[5]: https://codecov.io/github/datproject/dat-node\n","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdat-ecosystem-archive%2Fdat-node","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdat-ecosystem-archive%2Fdat-node","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdat-ecosystem-archive%2Fdat-node/lists"}