{"id":15679847,"url":"https://github.com/danielgtaylor/node-desktop-uploader","last_synced_at":"2025-05-07T10:44:27.593Z","repository":{"id":20459462,"uuid":"23736742","full_name":"danielgtaylor/node-desktop-uploader","owner":"danielgtaylor","description":"Recursively watch directories and upload new/updated files","archived":false,"fork":false,"pushed_at":"2014-09-14T21:47:34.000Z","size":280,"stargazers_count":11,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-28T07:48:26.434Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"CoffeeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danielgtaylor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-09-06T14:42:56.000Z","updated_at":"2017-01-20T02:50:15.000Z","dependencies_parsed_at":"2022-07-31T20:38:06.276Z","dependency_job_id":null,"html_url":"https://github.com/danielgtaylor/node-desktop-uploader","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielgtaylor%2Fnode-desktop-uploader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielgtaylor%2Fnode-desktop-uploader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielgtaylor%2Fnode-desktop-uploader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielgtaylor%2Fnode-desktop-uploader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danielgtaylor","download_url":"https://codeload.github.com/danielgtaylor/node-desktop-uploader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252862593,"owners_count":21815883,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T16:37:53.188Z","updated_at":"2025-05-07T10:44:27.573Z","avatar_url":"https://github.com/danielgtaylor.png","language":"CoffeeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Desktop Uploader\n\n[![Dependency Status](http://img.shields.io/david/danielgtaylor/node-desktop-uploader.svg?style=flat)](https://david-dm.org/danielgtaylor/node-desktop-uploader) [![Build Status](http://img.shields.io/travis/danielgtaylor/node-desktop-uploader.svg?style=flat)](https://travis-ci.org/danielgtaylor/node-desktop-uploader) [![Coverage Status](http://img.shields.io/coveralls/danielgtaylor/node-desktop-uploader.svg?style=flat)](https://coveralls.io/r/danielgtaylor/node-desktop-uploader) [![NPM version](http://img.shields.io/npm/v/desktop-uploader.svg?style=flat)](https://www.npmjs.org/package/desktop-uploader) [![License](http://img.shields.io/npm/l/desktop-uploader.svg?style=flat)](http://dgt.mit-license.org/)\n\n\nThe `desktop-uploader` module lets you easily write a desktop uploader for a remote service such as Dropbox, S3, Google Storage, or your own company using Node.js. You define directories to watch and a function that uploads a file entry, and `desktop-uploader` handles the rest!\n\n#### Features\n\n* Recursively watch folders and files for changes\n  * Uses native events (fsevents, inotify, ReadDirectoryChangesW)\n* Whitelist file extensions you care about\n* Persistent custom configuration values\n* Persistent per-folder custom configuration\n* Determine when a file is no longer being modified\n* Upload files using custom business logic\n* Concurrently upload many files\n* Automatically handle retries of failures\n* Keep a cache of info on already-uploaded files\n* Throttle aggregate uploads to a set bandwidth (e.g. 100Kbytes/sec)\n* Works with [atom-shell](https://github.com/atom/atom-shell) and [node-webkit](https://github.com/rogerwang/node-webkit) for a cross-platform user interface\n\n#### Improvement Ideas\n\n* Custom cache and ignore strategies (e.g. file hash instead of last modified time)\n* Upload bandwidth auto-detection for throttling\n* Allow manually adding items to the queue\n\n## Installation\nInstall like any other Node.js package, with NPM:\n\n```bash\n$ npm install --save desktop-uploader\n```\n\n## Basic Example\n\nCreate a new desktop uploader instance. It takes an optional options object where you can set initial paths to watch and a few options, like the number of concurrent uploads and how many times to retry failures. **Note**: the uploader is created in a paused state.\n\n```javascript\nvar DesktopUploader = require('desktop-uploader').DesktopUploader;\nvar request = require('request');\n\nvar uploader = new DesktopUploader({\n  name: 'my-cool-app',\n  paths: ['/home/daniel/Pictures'],\n  concurrency: 3,\n  retries: 2\n});\n```\n\nNow we need to tell the uploader how to actually upload a file when that file is no longer being modified, since this is specific to your service and API. Here we are assuming that we are going to do an HTTP POST to `api.myservice.com` to the `items` collection using an OAuth bearer token for authentication. We'll be using the [request](https://github.com/mikeal/request) library to make this easier.\n\n```javascript\nuploader.on('upload', function (entry, done) {\n  var url = 'https://api.myservice.com/items';\n  var headers = {\n    authorization: 'bearer abc123'\n  };\n\n  // Create the HTTP POST request\n  var req = request.post {url: url, headers: headers}, function (err, res) {\n    if (err) return done(err);\n    console.log(entry.path + ' uploaded!');\n    done();\n  });\n\n  // Pipe the file into the request\n  entry.stream.pipe(req)\n});\n```\n\nNotice that you are piping `entry.stream` into the request rather than reading it all into memory first. All that's left is to start the uploader:\n\n```javascript\nuploader.resume();\n```\n\nAt this point, the uploader is running. It is recursively watching all paths that you have configured and uploading new files.\n\n## Adjusting Paths\nYou can dynamically add or remove paths, as well as path-specific custom configuration.\n\n```javascript\n// Add a new path to watch, with a custom configuration which sets\n// an owner. Your `upload` method can use this configuration via\n// the `entry.config` attribute.\nuploader.watch('/home/daniel/Documents', {owner: 'Kari'});\n\n// Edit an existing watched path\nvar config = uploader.get('/home/daniel/Pictures');\nconfig.owner = 'Daniel';\n\n// Remove a watched path and its configuration\nuploader.unwatch('/home/daniel/Documents');\n```\n\nYou can then access the custom config during the upload process:\n\n```javascript\nuploader.on('upload', function (entry, done) {\n  console.log(entry.config.owner);\n\n  // Do your upload\n  done()\n});\n```\n\n## Upload Throttling\nIt's possible to automatically throttle uploads, or set throttling to a specific value. If you use the `entry.stream` to pipe data to an HTTP request then all concurrent reads will be throttled to the aggregate global throttle value. For example, if three concurrent uploads are being performed, then the combined bandwidth they consume is the throttle limit.\n\n```javascript\n// Throttle to 100 kbytes per second\nuploader.throttle = 100 * 1024;\n\n// Disable throttling\nuploader.throttle = false;\n```\n\n## Advanced Example\nYou can find an advanced, real-world example that uploads files to S3 in [examples/example.litcoffee](https://github.com/danielgtaylor/node-desktop-uploader/blob/master/example/example.litcoffee).\n\n# API Reference\nThe `DesktopUploader` class is an `EventEmitter` and has the following events, properties, and methods, as well as those [inherited from EventEmitter](http://nodejs.org/api/events.html#events_class_events_eventemitter).\n\n### Events\n\n#### Event: `drain`\nEmitted when the last item in the queue has finished uploading (or failed). At this point, the queue is empty and no items are being processed.\n\n```javascript\nuploader.on('drain', function () {\n  console.log('We are finished!');\n});\n```\n\n#### Event: `error`\nEmitted when an error occurs. The second argument, if present, is the filename which was being processed when the error occured.\n\n```javascript\nuploader.on('error', function (err, filename) {\n  console.error('Error processing ' + filename + ':', err);\n});\n```\n\n#### Event: `ignore`\nEmitted when a file has been ignored (e.g. incorrect extension, no longer being watched, etc).\n\n```javascript\nuploader.on('ignore', function (filename) {\n  console.log('Ignoring ' + filename);\n});\n```\n\n#### Event: `log`\nLog a debug message from the uploader.\n\n```javascript\nuploader.on('log', function (message) {\n  console.log(message);\n});\n```\n\n#### Event: `pause`\nEmitted when the uploader has been paused. The `type` argument will be either `'queue'` or `'watcher'` depending on which was paused.\n\n```javascript\nuploader.on('pause', function (type) {\n  console.log('Uploader ' + type + ' has been paused!');\n});\n```\n\n#### Event: `processed`\nEmitted after an entry is finished uploading (including retries) and is going to be removed from the queue. Parameters are the enty and whether the upload was successful.\n\n```javascript\nuploader.on('processed', function (entry, success) {\n  if (success) {\n    console.log(entry.path + ' successully uploaded!');\n  } else {\n    console.log(entry.path + ' failed to upload!');\n  }\n});\n```\n\n#### Event: `queue`\nEmitted when an item is added to the queue. This event is fired after the item has been added or changed on disk and after a reasonable effort has been made to ensure it is no longer being written.\n\n```javascript\nuploader.on('queue', function (filename, root) {\n  console.log('File: ' + filename);\n  console.log('Watch path: ' + root);\n});\n```\n\n#### Event: `resume`\nEmitted when the uploader has resumed uploading after being created or paused.\n\n```javascript\nuploader.on('resume', function () {\n  console.log('Uploader has resumed!');\n});\n```\n\n#### Event: `upload`\nEmitted when a file is ready to be uploaded. This is where you implement custom logic to asyncronously upload the file. The `entry` argument has the following fields:\n\nName   | Description                              | Example\n------ | ---------------------------------------- | -------\nconfig | Custom configuration set on `root`       | `{owner: 'daniel'}`\npath   | The full path to the file                | `'/home/daniel/Pictures/2014/IMG_8088.jpg'`\nroot   | The watched directory path               | `'/home/daniel/Pictures'`\nsize   | Approximate stream length in bytes       | `102483`\nstream | Read stream to pipe into an HTTP request | `ReadableStream`\n\nIf throttling is enabled, then `stream` will produce data to keep within your bandwidth limit. This event may be fired multiple times before the first upload has finished.\n\nYou **must** call the `done` function to let the uploader know that it can process the next item in the queue.\n\n```javascript\nuploader.on('upload', function (entry, done) {\n  console.log('Uploading ' + entry.path);\n\n  // Create an HTTP POST request\n  var req = http.request({\n    method: 'POST',\n    hostname: 'your-server.com',\n    path: '/widgets',\n    headers: {\n      authorization: 'bearer abc123def456'\n    }\n  });\n\n  // Ensure we call `done` in all cases!\n  req.on('error', done);\n\n  req.on('response', function (res) {\n    if (res.statusCode == 200) {\n      done()\n    } else {\n      done(new Error('Bad response!'));\n    }\n  });\n\n  // Pipe the file data into the request\n  entry.stream.pipe(req);\n});\n```\n\n#### Event: `unwatch`\nEmitted when a folder is unwatched.\n\n```javascript\nuploader.on('unwatch', function (paths) {\n  console.log('Unwatching:\\n' + path.join('\\n'));\n});\n\nuploader.unwatch('/some/path');\n```\n\n#### Event: `watch`\nEmitted when a folder is watched.\n\n```javascript\nuploader.on('watch', function (path, config) {\n  console.log('Watching ' + path);\n});\n\nuploader.watch('/some/path', {my: 'config'});\n```\n\n### Properties\n\n#### Property: `concurrency = 2`\nThis value determines the number of concurrent uploads. If throttling is enabled, then all uploads are throttled to the aggregate bandwidth limit. Setting a concurrency limit of `1` means only one upload at a time.\n\n```javascript\nuploader.concurrency = 5;\n```\n\n#### Property: `modifyInterval = 5000`\nThis value determines how often in milliseconds a file is checked to see if it has been modified. If a file has not been modified between checks, then it is eligible to be uploaded and an `upload` event will be fired. Defaults to **5 seconds**.\n\n#### Property `retries = 0`\nThis value determines the automatic retry count. Anytime the `done` function is called with an error during the `upload` event handler it is considered for a retry. The `upload` event will be emitted again up to the number of retries. Set to zero to disable retry logic.\n\n```javascript\n# Retry up to two times (total of three upload requests)\nuploader.retries = 2;\n\n# Disable retries\nuploader.retries = 1;\n```\n\n#### Property: `tasks`\nA **read-only** array of tasks in the queue. Each task has the following fields:\n\nName | Description                | Example\n---- | -------------------------- | -------\npath | The full path to the file  | `'/home/daniel/Pictures/2014/IMG_8088.jpg'`\nroot | The watched directory path | `'/home/daniel/Pictures'`\n\n#### Property: `throttle = false`\nThis value determines the bandwidth throttling limit in bytes per second. Setting to `null`, `false`, or no options will disable bandwidth throttling.\n\n```javascript\n// Throttle to 10 Kbytes per second\nuploader.throttle = 10240;\n\n// Disable throttling\nuploader.throttle = false;\n```\n\n### Methods\n\n#### Method: Constructor\nCreate a new `DesktopUploader` instance in a paused state. Takes the following optional parameters:\n\nParameter      | Description                          | Default\n-------------- | ------------------------------------ | -------\nconcurrency    | Number of concurrent uploads         | `2`\nconfigPath     | Directory to store configuration     | `null`\nextensions     | File extensions to watch             | `null`\nmodifyInterval | Duration in ms to check file writes  | `5000`\nname           | Unique name used for configuration   | `'desktop-uploader'`\npaths          | List of paths to watch               | `[]`\nretries        | Number of retries for failures       | `0`\nsaveInterval   | Duration in ms to save configuration | `10000`\nthrottle       | Limit bandwidth in bytes per second  | `null`\n\nNote: extensions are not case-sensitive. You should always supply them in lowercase.\n\n```javascript\nvar uploader = new DesktopUploader({\n  name: 'my-cool-uploader',\n  configPath: process.env.HOME,\n  paths: ['/some/path', '/another/path'],\n  extensions: ['jpg', 'png'],\n  throttle: 250 * 1024,\n  retries: 1\n});\n```\n\n#### Method: `get`\nGet the configuration for a particular watched directory by its path. You may modify the returned object. If not path is passed, then it returns an object where the keys are paths and the values are configs.\n\n```javascript\nvar config = uploader.get('/some/path');\nconfig.foo = 3;\n\nvar paths = uploader.get();\nconsole.log(paths['/some/path'].foo); // Prints out 3\n```\n\n#### Method: `pause`\nTemporarily stop the uploader from firing `upload` events. Existing in-flight items will complete, but no new items will be processed until `resume` has been called. File system events will continue to add items on to the queue.\n\n```javascript\nuploader.pause();\n```\n\n#### Method: `pauseWatcher`\nTemporarily ignore all file system events. No new or changed items will be added to the queue until `resume` has been called. The `upload` event will continue to be called for existing items in the queue. See the `pause` method to prevent items already in the queue from being processed.\n\n```javascript\nuploader.pauseWatcher();\n```\n\n#### Method: `resume`\nStart or resume the uploader and watcher. Since the uploader and watcher are created in a paused state, you **must** call this method to begin watching and uploading. Until this method is called, *no* items will be added to the queue and *no* `upload` events are fired.\n\n```javascript\nuploader.resume();\n```\n\n#### Method: `save`\nGive a hint that the uploader should save its configuration to disk in the near future. If `immediate` is `true`, then save to disk right now. If `immediate` is `false`, then at most `saveInterval` milliseconds (see the constructor method) will pass before the file is saved. When your app is about to exit, you must remember to force an immediate save, otherwise data may be lost.\n\n```javascript\n// Save in the near future, when convenient\nuploader.save();\n\n// Useful to call on app exit\nuploader.save(true);\n```\n\n#### Method: `unwatch`\nRemove a directory from being watched.\n\n```javascript\nuploader.unwatch('/some/path');\n```\n\n#### Method: `watch`\nAdd a new directory to recursively watch, with an optional config. The config will be saved between runs and is accessible during the `upload` event via `entry.config`.\n\n```javascript\nuploader.watch('/some/path', {\n  some: 'optional configuration goes here',\n  foo: 2\n});\n```\n\n## Development\nThis project uses [Gulp](http://gulpjs.com/) and is written using [CoffeeScript](http://coffeescript.org/). That means that you do not edit the `.js` files in the `lib` folder - those are generated by the build system. Instead, you work on the `src` folder. You can get started like so:\n\n```bash\n$ sudo npm install -g gulp\n$ git clone https://github.com/danielgtaylor/node-desktop-uploader\n$ cd node-desktop-uploader\n$ npm install\n```\n\nYou can edit and then compile the source via:\n\n```bash\n$ gulp compile\n...\n```\n\nYou can test in a Node shell via:\n\n```javascript\n\u003e var DesktopUploader = require('./lib/main').DesktopUploader;\n\u003e uploader = new DesktopUploader();\n\u003e ...\n```\n\nYou can run the unit tests via:\n\n```bash\n$ gulp test\n...\n```\n\nPull requests are welcome, so please fork the project and submit one! Please keep in mind that any new features should include unit tests coverage, or they may be rejected.\n\n## License\nhttp://dgt.mit-license.org/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanielgtaylor%2Fnode-desktop-uploader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanielgtaylor%2Fnode-desktop-uploader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanielgtaylor%2Fnode-desktop-uploader/lists"}