Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/tusharmath/Multi-threaded-downloader

A http file downloader made in nodejs
https://github.com/tusharmath/Multi-threaded-downloader
Last synced: 13 days ago
JSON representation
A http file downloader made in nodejs
Host: GitHub
URL: https://github.com/tusharmath/Multi-threaded-downloader
Owner: tusharmath
License: mit
Created: 2013-05-01T07:43:37.000Z (over 11 years ago)
Default Branch: master
Last Pushed: 2018-03-18T03:56:24.000Z (over 6 years ago)
Last Synced: 2024-10-01T10:48:53.657Z (about 1 month ago)
Language: JavaScript
Homepage:
Size: 10.1 MB
Stars: 194
Watchers: 14
Forks: 90
Open Issues: 39
Metadata Files:
- Readme: README.md
- Contributing: contributing.md
- License: LICENSE.md
Awesome Lists containing this project

README

        # mt-downloader

[![Join the chat at https://gitter.im/tusharmath/Multi-threaded-downloader](https://badges.gitter.im/tusharmath/Multi-threaded-downloader.svg)](https://gitter.im/tusharmath/Multi-threaded-downloader?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

[![Build Status](https://travis-ci.org/tusharmath/Multi-threaded-downloader.svg?branch=master)](https://travis-ci.org/tusharmath/Multi-threaded-downloader)

[![npm](https://img.shields.io/npm/v/mt-downloader.svg)](https://www.npmjs.com/package/mt-downloader)

[![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](http://commitizen.github.io/cz-cli/)

[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)

[![js-standard-style](https://img.shields.io/badge/code%20style-standard-brightgreen.svg)](http://standardjs.com/)

[![Coverage Status](https://coveralls.io/repos/github/tusharmath/Multi-threaded-downloader/badge.svg)](https://coveralls.io/github/tusharmath/Multi-threaded-downloader)

This is a nodejs based utility library that helps in performing **resumable**, **multi-threaded** downloads over Http. The module is highly inspired by Speedbit's — [Download Accelerator Plus](http://www.speedbit.com/dap/).

## Features

0. **Multi connection downloads:** In a conventional download we don't completely utilize our bandwidth. With multiple connections there is always one thread which is getting data thus minimizing the wait period between data packets. **This doesn't mean that we will be able to download faster than what our ISP allows**.

0. **Stop and start from the last downloaded byte:** You don't have to worry about internet getting disconnected or your computer shutting down while downloading. You can quite easily start from the last byte that was downloaded.

0. **Console application:** If installed globally, `mtd` command would be available.

## Installation

The conventional npm installation process needs to be followed.

```bash

npm install mt-downloader --save

```

## CLI Installation

```bash

$ npm install -g mt-downloader

$ mtd --help

```

## Objects

* [FILE](#FILE) : object

    * [.open(params$)](#FILE.open) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.fstat(params$)](#FILE.fstat) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.read(params$)](#FILE.read) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.write(params$)](#FILE.write) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.close(params$)](#FILE.close) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.truncate(params$)](#FILE.truncate) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.rename(params$)](#FILE.rename) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

* [HTTP](#HTTP) : object

    * [.request(params)](#HTTP.request) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

## Functions

* [CreateMTDFile(options)](#CreateMTDFile) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

* [DownloadFromMTDFile(mtdPath, [meta])](#DownloadFromMTDFile) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

* [FinalizeDownload(params)](#FinalizeDownload) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

* [Completion(meta$)](#Completion) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)



## FILE : object

Provides wrappers over the async utils inside the

[fs module](https://nodejs.org/api/fs.html).

The wrappers take in an input stream of arguments

and returns the result of function call as another stream.

**Kind**: global namespace  

* [FILE](#FILE) : object

    * [.open(params$)](#FILE.open) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.fstat(params$)](#FILE.fstat) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.read(params$)](#FILE.read) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.write(params$)](#FILE.write) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.close(params$)](#FILE.close) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.truncate(params$)](#FILE.truncate) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

    * [.rename(params$)](#FILE.rename) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)



### FILE.open(params$) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

**Kind**: static method of [FILE](#FILE)  

| Param | Type |

| --- | --- |

| params$ | [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) | 



### FILE.fstat(params$) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

[https://nodejs.org/api/fs.html#fs_fs_open_path_flags_mode_callback](https://nodejs.org/api/fs.html#fs_fs_open_path_flags_mode_callback)

**Kind**: static method of [FILE](#FILE)  

| Param | Type |

| --- | --- |

| params$ | [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) | 



### FILE.read(params$) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

[https://nodejs.org/api/fs.html#fs_fs_read_fd_buffer_offset_length_position_callback](https://nodejs.org/api/fs.html#fs_fs_read_fd_buffer_offset_length_position_callback)

**Kind**: static method of [FILE](#FILE)  

| Param | Type |

| --- | --- |

| params$ | [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) | 



### FILE.write(params$) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

[https://nodejs.org/api/fs.html#fs_fs_write_fd_buffer_offset_length_position_callback](https://nodejs.org/api/fs.html#fs_fs_write_fd_buffer_offset_length_position_callback)

**Kind**: static method of [FILE](#FILE)  

| Param | Type |

| --- | --- |

| params$ | [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) | 



### FILE.close(params$) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

[https://nodejs.org/api/fs.html#fs_fs_close_fd_callback](https://nodejs.org/api/fs.html#fs_fs_close_fd_callback)

**Kind**: static method of [FILE](#FILE)  

| Param | Type |

| --- | --- |

| params$ | [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) | 



### FILE.truncate(params$) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

[https://nodejs.org/api/fs.html#fs_fs_truncate_path_len_callback](https://nodejs.org/api/fs.html#fs_fs_truncate_path_len_callback)

**Kind**: static method of [FILE](#FILE)  

| Param | Type |

| --- | --- |

| params$ | [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) | 



### FILE.rename(params$) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

[https://nodejs.org/api/fs.html#fs_fs_rename_oldpath_newpath_callback](https://nodejs.org/api/fs.html#fs_fs_rename_oldpath_newpath_callback)

**Kind**: static method of [FILE](#FILE)  

| Param | Type |

| --- | --- |

| params$ | [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) | 



## HTTP : object

**Kind**: global namespace  



### HTTP.request(params) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

Stream based wrapper over [npm/request](https://www.npmjs.com/package/request)

**Kind**: static method of [HTTP](#HTTP)  

**Returns**: [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) - multiplex stream  

| Param | Type | Description |

| --- | --- | --- |

| params | object | [ request](https://www.npmjs.com/package/request) module params. |



## CreateMTDFile(options) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

Creates a new .mtd file that is a little larger in size than the original

file. The file is initially empty and has all the relevant meta

information regarding the download appended to the end.

**Kind**: global function  

**Returns**: [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) - A [multiplexed stream](https://github.com/tusharmath/muxer) containing ~

- `written$` - Bytes being saved on disk.

- `meta$` - Meta information about the download.

- `remoteFileSize$` - Size of the content that is to be downloaded.

- `fdW$` - File descriptor in `w` mode.  

| Param | Type | Default | Description |

| --- | --- | --- | --- |

| options | object |  | The `options` must have `mtdPath` and `url`. |

| options.url | string |  | Download url. |

| options.path | string |  | Relative path where the file needs to be saved. |

| [options.range] | number | 3 | Number of concurrent downloads. |

| [options.metaWrite] | number | 300 | Throttles the write frequency of meta data. |



## DownloadFromMTDFile(mtdPath, [meta]) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

Reads a `.mtd` file and resumes the download from the last successfully saved

byte.

**Kind**: global function  

**Returns**: [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) - A [multiplexed stream](https://github.com/tusharmath/muxer) containing ~

- `metaWritten$` - Meta data buffer stream.

- `response$` - HTTP response object.

- `responses$` - List of all the HTTP response objects.

- `localFileSize$` - Size of the `.mtd` file on disk.

- `fdR$` - File Descriptor in `r+` mode.

- `meta$` - Download meta information.  

| Param | Type | Description |

| --- | --- | --- |

| mtdPath | String | Relative path to the `.mtd` file. |

| [meta] | Object | Optional meta data to override the one that's being loaded from the `.mtd` file. |



## FinalizeDownload(params) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

Removes the meta information and the `.mtd` extension from the file once the

download is successfully completed.

**Kind**: global function  

**Returns**: [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) - A [multiplexed stream](https://github.com/tusharmath/muxer) containing ~

- `truncated$` - Fired when the meta data is removed.

- `renamed$` - Fired when the `.mtd` extension is removed.  

| Param | Type | Description |

| --- | --- | --- |

| params | object | `{fd$, meta$}` |

| params.fd$ | [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) | File descriptor Observable |

| params.meta$ | [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) | Download meta information |



## Completion(meta$) ⇒ [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md)

Util method that calculates the total completion percentage (between 0-100).

**Kind**: global function  

**Returns**: [Observable](https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/observable.md) - Value between 0-100  

| Param | Type | Description |

| --- | --- | --- |

| meta$ | Observable | Meta data stream ie. exposed by [DownloadFromMTDFile](#DownloadFromMTDFile) |

## .mtd file

Once the download starts the library will create a file with a **.mtd** extension. This file contains some meta information related to the download and is a little bigger *(around 4kb)* than the original file size. The **.mtd** file can be used later to restart downloads from where the last byte that was downloaded. After the download is completed the downloader will truncate the `.mtd` file to remove that meta information.

## Releases

There are two release channels viz. — `latest` and `next`. The `next` is unstable and must be used with caution.

**Latest**

```bash

npm i mt-downloader

```

**Next**

```bash

npm i mt-downloader@next

```