{"id":13526891,"url":"https://github.com/apiaryio/s3-streaming-upload","last_synced_at":"2025-04-01T08:30:31.745Z","repository":{"id":8006221,"uuid":"9414092","full_name":"apiaryio/s3-streaming-upload","owner":"apiaryio","description":"s3-streaming-upload is node.js library that listens to your stream and upload its data to Amazon S3 using ManagedUpload API.","archived":true,"fork":false,"pushed_at":"2023-04-15T12:02:02.000Z","size":282,"stargazers_count":123,"open_issues_count":7,"forks_count":13,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-03-08T06:48:13.901Z","etag":null,"topics":["sre"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apiaryio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2013-04-13T13:33:49.000Z","updated_at":"2024-11-08T09:04:08.000Z","dependencies_parsed_at":"2024-01-13T22:25:32.257Z","dependency_job_id":"3ce23761-fac8-40fa-813e-4aecde0e9e38","html_url":"https://github.com/apiaryio/s3-streaming-upload","commit_stats":{"total_commits":119,"total_committers":13,"mean_commits":9.153846153846153,"dds":0.7058823529411764,"last_synced_commit":"4461c4c1faa014f3b2fa4858338546b4345fdb64"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apiaryio%2Fs3-streaming-upload","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apiaryio%2Fs3-streaming-upload/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apiaryio%2Fs3-streaming-upload/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apiaryio%2Fs3-streaming-upload/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apiaryio","download_url":"https://codeload.github.com/apiaryio/s3-streaming-upload/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246607076,"owners_count":20804513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["sre"],"created_at":"2024-08-01T06:01:36.863Z","updated_at":"2025-04-01T08:30:31.448Z","avatar_url":"https://github.com/apiaryio.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"## s3-streaming-upload [![s3-streaming-upload](https://github.com/apiaryio/s3-streaming-upload/workflows/s3-streaming-upload%20CI/badge.svg)](https://github.com/apiaryio/s3-streaming-upload/actions?query=workflow%3A%22s3-streaming-upload+CI%22)\n\n[s3-streaming-upload](https://github.com/apiaryio/s3-streaming-upload) is [node.js](http://nodejs.org) library that listens to your [stream](http://nodejs.org/docs/v0.8.9/api/stream.html) and upload its data to Amazon S3 and OCI Bucket Store.\n\nIt is heavily inspired by [knox-mpu](https://github.com/nathanoehlman/knox-mpu), but unlike it, it does not buffer data to disk and is build on top of [official AWS SDK](https://github.com/aws/aws-sdk-js) instead of knox.\n\n### Changes\n\n- Version 0.3.2 NodeJS 12+ supported.\n- Version 0.3.x Change from Coffee-script to Javascript. NodeJS 6 and 8 supported.\n\n- Version 0.2.x using [ManagedUpload API](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html). NodeJS 0.10 and 0.12 supported.\n\n- Version 0.1.x using [MultiPartUpload API](http://docs.amazonwebservices.com/AmazonS3/latest/dev/sdksupportformpu.html). NodeJS 0.8 and 0.10 supported.\n\n### Installation\n\nInstallation is done via NPM, by running `npm install s3-streaming-upload`\n\n### Features\n\n- Super easy to use\n- No need to know data size beforehand\n- Stream is buffered up to specified size (default 5MBs) and then uploaded to S3\n- Segments are not written to disk and memory is freed as soon as possible after upload\n- Uploading is asynchronous\n- You can react to upload status through events\n\n### Quick example\n\n```javascript\nvar Uploader = require('s3-streaming-upload').Uploader,\n  upload = null,\n  stream = require('fs').createReadStream('/etc/resolv.conf');\n\nupload = new Uploader({\n  // credentials to access AWS\n  accessKey: process.env.AWS_S3_ACCESS_KEY,\n  secretKey: process.env.AWS_S3_SECRET_KEY,\n  bucket: process.env.AWS_S3_TEST_BUCKET,\n  objectName: 'myUploadedFile',\n  stream: stream,\n  debug: true,\n});\n\nupload.send(function(err) {\n  if (err) {\n    console.error('Upload error' + err);\n  }\n});\n```\n\n### Setting up ACL\n\nPass it in `objectParams` to the `Uploader`:\n\n```javascript\nupload = new Uploader({\n  // credentials to access AWS\n  accessKey: process.env.AWS_API_KEY,\n  secretKey: process.env.AWS_SECRET,\n  bucket: process.env.AWS_S3_TRAFFIC_BACKUP_BUCKET,\n  objectName: 'myUploadedFile',\n  stream: stream,\n  objectParams: {\n    ACL: 'public-read',\n  },\n});\n```\n\n### Example usage with Oracle Cloud (OCI) compatible S3 API\n\n```javascript\nregion = process.env.OCI_REGION;\ntenancy = process.env.OCI_TENANCY;\n// define custom service\nservice = new aws.S3({\n  apiVersion: '2006-03-01',\n  credentials: {\n    accessKeyId: process.env.BUCKET_ACCESS_KEY,\n    secretAccessKey: process.env.BUCKET_SECRET_KEY,\n  },\n  params: { Bucket: process.env.BUCKET_NAME },\n  endpoint: `${tenancy}.compat.objectstorage.${region}.oraclecloud.com`,\n  region: region,\n  signatureVersion: 'v4',\n  s3ForcePathStyle: true,\n});\n\nuploader = new Uploader({\n  accessKey: process.env.BUCKET_ACCESS_KEY,\n  secretKey: process.env.BUCKET_SECRET_KEY,\n  bucket: process.env.BUCKET_NAME,\n  objectName: filename,\n  stream: source,\n  service: service,\n  objectParams: {\n    ContentType: 'text/csv',\n  },\n  debug: true,\n});\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapiaryio%2Fs3-streaming-upload","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapiaryio%2Fs3-streaming-upload","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapiaryio%2Fs3-streaming-upload/lists"}