https://github.com/bendrucker/azure-blob-to-s3
Batch copy files from Azure Blob Storage to Amazon S3
https://github.com/bendrucker/azure-blob-to-s3
Last synced: over 1 year ago
JSON representation
Batch copy files from Azure Blob Storage to Amazon S3
- Host: GitHub
- URL: https://github.com/bendrucker/azure-blob-to-s3
- Owner: bendrucker
- License: mit
- Created: 2016-10-20T01:33:53.000Z (almost 10 years ago)
- Default Branch: master
- Last Pushed: 2023-05-15T13:56:56.000Z (about 3 years ago)
- Last Synced: 2025-04-15T00:02:49.641Z (over 1 year ago)
- Language: JavaScript
- Homepage:
- Size: 86.9 KB
- Stars: 46
- Watchers: 4
- Forks: 19
- Open Issues: 4
-
Metadata Files:
- Readme: readme.md
- License: license
Awesome Lists containing this project
README
# azure-blob-to-s3 [](https://github.com/bendrucker/azure-blob-to-s3/actions?query=workflow%3Atests)
> Batch copy files from Azure Blob Storage to Amazon S3
* Fully streaming
* Lists files from Azure Blob storage only as needed
* Uploads Azure binary data to S3 streamingly
* Skips unnecessary uploads (files with a matching key and `Content-Length` already on S3)
* Retries on (frequent) failed downloads from Azure
* Generates [ndjson](http://ndjson.org/) logs for each network operation
For large workloads (either number of files or bytes), you should run this from AWS in the same region where your bucket is located. This will minimize cost and offer reliable/fast/cheap uploads to S3. You will be billed per byte by Azure for [outbound data transfer](https://azure.microsoft.com/en-us/pricing/details/bandwidth/).
## Install
```
$ npm install --save azure-blob-to-s3
```
## Usage
### API
```js
var toS3 = require('azure-blob-to-s3')
toS3({
azure: {
connection: '',
container: 'my-container'
},
aws: {
region: 'us-west-2',
bucket: 'my-bucket',
prefix: 'my-prefix'
}
})
```
### CLI
```sh
azure-s3 \
--concurrency 10 \
--azure-connection "..." \
--azure-container my-container \
--aws-bucket my-bucket \
--aws-prefix my-prefix
```
## API
#### `toS3(options)` -> `output`
Copys files from Azure Blob Storage to AWS S3.
##### options
*Required*
Type: `object`
Options for configuring the copy.
###### concurrency
Type: `number`
Default: `100`
The maximum number of files to concurrently stream from Azure and into S3. This is the `highWaterMark` of the file upload stream.
##### azure
Type: `object`
###### connection
*Required*
Type: `string`
Azure Blob Storage connection string.
###### container
*Required*
Type: `string`
Azure Blob Storage container name.
###### token
Type: `string`
Default: `undefined`
A page token where the file list operation will begin.
##### aws
Type: `object`
## License
MIT © [Ben Drucker](http://bendrucker.me)