{"id":17420109,"url":"https://github.com/bendrucker/azure-blob-to-s3","last_synced_at":"2025-04-15T00:03:39.210Z","repository":{"id":12274713,"uuid":"71414317","full_name":"bendrucker/azure-blob-to-s3","owner":"bendrucker","description":"Batch copy files from Azure Blob Storage to Amazon S3","archived":false,"fork":false,"pushed_at":"2023-05-15T13:56:56.000Z","size":89,"stargazers_count":46,"open_issues_count":4,"forks_count":19,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-15T00:02:49.641Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bendrucker.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-20T01:33:53.000Z","updated_at":"2025-04-03T09:35:20.000Z","dependencies_parsed_at":"2024-10-19T01:07:24.276Z","dependency_job_id":null,"html_url":"https://github.com/bendrucker/azure-blob-to-s3","commit_stats":{"total_commits":69,"total_committers":8,"mean_commits":8.625,"dds":"0.42028985507246375","last_synced_commit":"8c55ef5ffdf132d695969aedf7cea0a52088a0e5"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bendrucker%2Fazure-blob-to-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bendrucker%2Fazure-blob-to-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bendrucker%2Fazure-blob-to-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bendrucker%2Fazure-blob-to-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bendrucker","download_url":"https://codeload.github.com/bendrucker/azure-blob-to-s3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248981261,"owners_count":21193146,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-17T02:41:11.061Z","updated_at":"2025-04-15T00:03:39.184Z","avatar_url":"https://github.com/bendrucker.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# azure-blob-to-s3 [![tests](https://github.com/bendrucker/azure-blob-to-s3/workflows/tests/badge.svg)](https://github.com/bendrucker/azure-blob-to-s3/actions?query=workflow%3Atests)\n\n\u003e Batch copy files from Azure Blob Storage to Amazon S3\n\n* Fully streaming\n  * Lists files from Azure Blob storage only as needed\n  * Uploads Azure binary data to S3 streamingly\n* Skips unnecessary uploads (files with a matching key and `Content-Length` already on S3)\n* Retries on (frequent) failed downloads from Azure\n* Generates [ndjson](http://ndjson.org/) logs for each network operation\n\nFor large workloads (either number of files or bytes), you should run this from AWS in the same region where your bucket is located. This will minimize cost and offer reliable/fast/cheap uploads to S3. You will be billed per byte by Azure for [outbound data transfer](https://azure.microsoft.com/en-us/pricing/details/bandwidth/).\n\n## Install\n\n```\n$ npm install --save azure-blob-to-s3\n```\n\n\n## Usage\n\n### API\n\n```js\nvar toS3 = require('azure-blob-to-s3')\n\ntoS3({\n  azure: {\n    connection: '',\n    container: 'my-container'\n  },\n  aws: {\n    region: 'us-west-2',\n    bucket: 'my-bucket',\n    prefix: 'my-prefix'\n  }\n})\n```\n\n### CLI\n\n```sh\nazure-s3 \\\n  --concurrency 10 \\\n  --azure-connection \"...\" \\\n  --azure-container my-container \\\n  --aws-bucket my-bucket \\\n  --aws-prefix my-prefix\n```\n\n## API\n\n#### `toS3(options)` -\u003e `output`\n\nCopys files from Azure Blob Storage to AWS S3.\n\n##### options\n\n*Required*  \nType: `object`\n\nOptions for configuring the copy.\n\n###### concurrency\n\nType: `number`  \nDefault: `100`\n\nThe maximum number of files to concurrently stream from Azure and into S3. This is the `highWaterMark` of the file upload stream.\n\n##### azure\n\nType: `object`  \n\n###### connection\n\n*Required*  \nType: `string`\n\nAzure Blob Storage connection string.\n\n###### container\n\n*Required*  \nType: `string`\n\nAzure Blob Storage container name.\n\n###### token\n  \nType: `string`  \nDefault: `undefined`\n\nA page token where the file list operation will begin.\n\n##### aws\n\nType: `object`\n\n## License\n\nMIT © [Ben Drucker](http://bendrucker.me)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbendrucker%2Fazure-blob-to-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbendrucker%2Fazure-blob-to-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbendrucker%2Fazure-blob-to-s3/lists"}