{"id":14982099,"url":"https://github.com/pgherveou/gulp-awspublish","last_synced_at":"2025-05-15T17:06:24.798Z","repository":{"id":13798106,"uuid":"16493650","full_name":"pgherveou/gulp-awspublish","owner":"pgherveou","description":"gulp plugin to publish files to amazon s3","archived":false,"fork":false,"pushed_at":"2023-12-19T17:48:05.000Z","size":711,"stargazers_count":398,"open_issues_count":23,"forks_count":84,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-15T17:06:18.844Z","etag":null,"topics":["gulp","s3","stream"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pgherveou.png","metadata":{"files":{"readme":"README.md","changelog":"History.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2014-02-03T21:32:09.000Z","updated_at":"2025-01-30T15:06:06.000Z","dependencies_parsed_at":"2024-01-05T20:46:08.272Z","dependency_job_id":"bfc0b894-6842-473a-a1cf-23df95f1492a","html_url":"https://github.com/pgherveou/gulp-awspublish","commit_stats":{"total_commits":254,"total_committers":48,"mean_commits":5.291666666666667,"dds":0.6259842519685039,"last_synced_commit":"ece6e67b9cbbbacfe5edf29886be39942f0c65bc"},"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgherveou%2Fgulp-awspublish","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgherveou%2Fgulp-awspublish/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgherveou%2Fgulp-awspublish/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgherveou%2Fgulp-awspublish/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pgherveou","download_url":"https://codeload.github.com/pgherveou/gulp-awspublish/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254384988,"owners_count":22062422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gulp","s3","stream"],"created_at":"2024-09-24T14:04:47.380Z","updated_at":"2025-05-15T17:06:19.788Z","avatar_url":"https://github.com/pgherveou.png","language":"JavaScript","funding_links":[],"categories":["Open Source Repos"],"sub_categories":["S3"],"readme":"# gulp-awspublish\n\n[![NPM version][npm-image]][npm-url] ![Dependency Status][depstat-image] [![Install size][packagephobia-image]][packagephobia-url]\n\n\u003e awspublish plugin for [gulp](https://github.com/wearefractal/gulp)\n\n## Usage\n\nFirst, install `gulp-awspublish` as a development dependency:\n\n```shell\nnpm install --save-dev gulp-awspublish\n```\n\nThen, add it to your `gulpfile.js`:\n\n```javascript\nvar awspublish = require('gulp-awspublish');\n\ngulp.task('publish', function () {\n  // create a new publisher using S3 options\n  // http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#constructor-property\n  var publisher = awspublish.create(\n    {\n      region: 'your-region-id',\n      params: {\n        Bucket: '...',\n      },\n    },\n    {\n      cacheFileName: 'your-cache-location',\n    }\n  );\n\n  // define custom headers\n  var headers = {\n    'Cache-Control': 'max-age=315360000, no-transform, public',\n    // ...\n  };\n\n  return (\n    gulp\n      .src('./public/*.js')\n      // gzip, Set Content-Encoding headers and add .gz extension\n      .pipe(awspublish.gzip({ ext: '.gz' }))\n\n      // publisher will add Content-Length, Content-Type and headers specified above\n      .pipe(publisher.publish(headers))\n\n      // create a cache file to speed up consecutive uploads\n      .pipe(publisher.cache())\n\n      // print upload updates to console\n      .pipe(awspublish.reporter())\n  );\n});\n\n// output\n// [gulp] [create] file1.js.gz\n// [gulp] [create] file2.js.gz\n// [gulp] [update] file3.js.gz\n// [gulp] [cache]  file3.js.gz\n// ...\n```\n\n- Note: If you follow the [aws-sdk suggestions](http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html) for\n  providing your credentials you don't need to pass them in to create the publisher.\n\n- Note: In order for publish to work on S3, your policy has to allow the following S3 actions:\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\"s3:ListBucket\"],\n      \"Resource\": [\"arn:aws:s3:::BUCKETNAME\"]\n    },\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"s3:PutObject\",\n        \"s3:PutObjectAcl\",\n        \"s3:GetObject\",\n        \"s3:GetObjectAcl\",\n        \"s3:DeleteObject\",\n        \"s3:ListMultipartUploadParts\",\n        \"s3:AbortMultipartUpload\"\n      ],\n      \"Resource\": [\"arn:aws:s3:::BUCKETNAME/*\"]\n    }\n  ]\n}\n```\n\n### Bucket permissions\n\nBy default, if no `x-amz-acl` header is passed, the uploaded object will inherit from the bucket setting. If you have specific requirements for the uploaded object, make sure to pass a value for the `x-amz-acl` header:\n\n```js\npublisher.publish({ 'x-amz-acl': 'something' });\n```\n\nSee [canned ACL on AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#CannedACL).\n\n## Testing\n\n1.  Create an S3 bucket which will be used for the tests. Optionally create an IAM user for running the tests.\n2.  Set the buckets Permission, so it can be edited by the IAM user who will run the tests.\n3.  Add an aws-credentials.json file to the project directory with the name of your testing buckets\n    and the credentials of the user who will run the tests.\n4.  Run `npm test`\n\n```json\n{\n  \"params\": {\n    \"Bucket\": \"\u003ctest-bucket-name\u003e\"\n  },\n  \"credentials\": {\n    \"accessKeyId\": \"\u003cyour-access-key-id\u003e\",\n    \"secretAccessKey\": \"\u003cyour-secret-access-key\u003e\",\n    \"signatureVersion\": \"v3\"\n  }\n}\n```\n\n## API\n\n### awspublish.gzip(options)\n\ncreate a through stream, that gzip file and add Content-Encoding header.\n\n- Note: Node version 0.12.x or later is required in order to use `awspublish.gzip`. If you need an older node engine to work with gzipping, you can use [v2.0.2](https://github.com/pgherveou/gulp-awspublish/tree/v2.0.2).\n\nAvailable options:\n\n- ext: file extension to add to gzipped file (eg: { ext: '.gz' })\n- smaller: gzip files only when result is smaller\n- Any options that can be passed to [zlib.gzip](https://nodejs.org/api/zlib.html#zlib_options)\n\n### awspublish.create(AWSConfig, cacheOptions)\n\nCreate a Publisher.\nThe AWSConfig object is used to create an `aws-sdk` S3 client. At a minimum you must pass a `Bucket` key, to define the site bucket. You can find all available options in the [AWS SDK documentation](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#constructor-property).\n\nThe cacheOptions object allows you to define the location of the cached hash digests. By default, they will be saved in your projects root folder in a hidden file called '.awspublish-' + 'name-of-your-bucket'.\n\n#### Adjusting upload timeout\n\nThe AWS client has a default timeout which may be too low when pushing large files (\u003e 50mb).\nTo adjust timeout, add `httpOptions: { timeout: 300000 }` to the AWSConfig object.\n\n#### Credentials\n\nBy default, gulp-awspublish uses the credential chain specified in the AWS [docs](http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html).\n\nHere are some example credential configurations:\n\nHardcoded credentials (**Note**: We recommend you **not** hard-code credentials inside an application. Use this method only for small personal scripts or for testing purposes.):\n\n```javascript\nvar publisher = awspublish.create({\n  region: 'your-region-id',\n  params: {\n    Bucket: '...',\n  },\n  credentials: {\n    accessKeyId: 'akid',\n    secretAccessKey: 'secret',\n  },\n});\n```\n\nUsing a profile by name from `~/.aws/credentials`:\n\n```javascript\nvar AWS = require('aws-sdk');\n\nvar publisher = awspublish.create({\n  region: 'your-region-id',\n  params: {\n    Bucket: '...',\n  },\n  credentials: new AWS.SharedIniFileCredentials({ profile: 'myprofile' }),\n});\n```\n\nInstead of putting anything in the configuration object, you can also provide the following environment variables: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_PROFILE`. You can also define a `[default]` profile in `~/.aws/credentials` which the SDK will use transparently without needing to set anything.\n\n#### Publisher.publish([headers], [options])\n\nCreate a through stream, that push files to s3.\n\n- header: hash of headers to add or override to existing s3 headers.\n- options: optional additional publishing options\n  - force: bypass cache / skip\n  - putOnly: bypass cache and head request (overrides `force`)\n  - simulate: debugging option to simulate s3 upload\n  - createOnly: skip file updates\n\nFiles that go through the stream receive extra properties:\n\n- s3.path: s3 path\n- s3.etag: file etag\n- s3.date: file last modified date\n- s3.state: publication state (create, update, put, delete, cache or skip)\n- s3.headers: s3 headers for this file. Defaults headers are:\n  - Content-Type\n  - Content-Length\n\n\u003e Note: `publish` will never delete files remotely. To clean up unused remote files use `sync`.\n\n#### publisher.cache()\n\nCreate a through stream that create or update a cache file using file s3 path and file etag.\nConsecutive runs of publish will use this file to avoid reuploading identical files.\n\nCache file is save in the current working dir and is named `.awspublish-\u003cbucket\u003e`. The cache file is flushed to disk every 10 files just to be safe.\n\n#### Publisher.sync([prefix], [whitelistedFiles])\n\ncreate a transform stream that delete old files from the bucket.\n\n- prefix: prefix to sync a specific directory\n- whitelistedFiles: array that can contain regular expressions or strings that match against filenames that\n  should never be deleted from the bucket.\n\ne.g.\n\n```js\n// only directory bar will be synced\n// files in folder /foo/bar and file baz.txt will not be removed from the bucket despite not being in your local folder\ngulp\n  .src('./public/*')\n  .pipe(publisher.publish())\n  .pipe(publisher.sync('bar', [/^foo\\/bar/, 'baz.txt']))\n  .pipe(awspublish.reporter());\n```\n\n\u003e **warning** `sync` will delete files in your bucket that are not in your local folder unless they're whitelisted.\n\n```js\n// this will publish and sync bucket files with the one in your public directory\ngulp\n  .src('./public/*')\n  .pipe(publisher.publish())\n  .pipe(publisher.sync())\n  .pipe(awspublish.reporter());\n\n// output\n// [gulp] [create] file1.js\n// [gulp] [update] file2.js\n// [gulp] [delete] file3.js\n// ...\n```\n\n#### Publisher.client\n\nThe `aws-sdk` S3 client is exposed to let you do other s3 operations.\n\n### awspublish.reporter([options])\n\nCreate a reporter that logs s3.path and s3.state (delete, create, update, put, cache, skip).\n\nAvailable options:\n\n- states: list of state to log (default to all)\n\n```js\n// this will publish,sync bucket files and print created, updated and deleted files\ngulp\n  .src('./public/*')\n  .pipe(publisher.publish())\n  .pipe(publisher.sync())\n  .pipe(\n    awspublish.reporter({\n      states: ['create', 'update', 'delete'],\n    })\n  );\n```\n\n## Examples\n\n### [Rename file \u0026 directory](examples/rename.js)\n\nYou can use `gulp-rename` to rename your files on s3\n\n```js\n// see examples/rename.js\n\ngulp\n  .src('examples/fixtures/*.js')\n  .pipe(\n    rename(function (path) {\n      path.dirname += '/s3-examples';\n      path.basename += '-s3';\n    })\n  )\n  .pipe(publisher.publish())\n  .pipe(awspublish.reporter());\n\n// output\n// [gulp] [create] s3-examples/bar-s3.js\n// [gulp] [create] s3-examples/foo-s3.js\n```\n\n### [Upload file in parallel](examples/concurrent.js)\n\nYou can use `concurrent-transform` to upload files in parallel to your amazon bucket\n\n```js\nvar parallelize = require('concurrent-transform');\n\ngulp\n  .src('examples/fixtures/*.js')\n  .pipe(parallelize(publisher.publish(), 10))\n  .pipe(awspublish.reporter());\n```\n\n### Upload both gzipped and plain files in one stream\n\nYou can use the [`merge-stream`](https://github.com/grncdr/merge-stream) plugin\nto upload two streams in parallel, allowing `sync` to work with mixed file\ntypes\n\n```js\nvar merge = require('merge-stream');\nvar gzip = gulp.src('public/**/*.js').pipe(awspublish.gzip());\nvar plain = gulp.src(['public/**/*', '!public/**/*.js']);\n\nmerge(gzip, plain)\n  .pipe(publisher.publish())\n  .pipe(publisher.sync())\n  .pipe(awspublish.reporter());\n```\n\n## Plugins\n\n### gulp-awspublish-router\n\nA router for defining file-specific rules\nhttps://www.npmjs.org/package/gulp-awspublish-router\n\n### gulp-cloudfront-invalidate-aws-publish\n\nInvalidate cloudfront cache based on output from awspublish\nhttps://www.npmjs.com/package/gulp-cloudfront-invalidate-aws-publish\n\n## License\n\n[MIT License](http://en.wikipedia.org/wiki/MIT_License)\n\n[npm-url]: https://npmjs.org/package/gulp-awspublish\n[npm-image]: https://badge.fury.io/js/gulp-awspublish.svg\n[depstat-image]: https://img.shields.io/librariesio/release/npm/gulp-awspublish?style=flat\n[packagephobia-image]: https://packagephobia.now.sh/badge?p=gulp-awspublish\n[packagephobia-url]: https://packagephobia.now.sh/result?p=gulp-awspublish\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpgherveou%2Fgulp-awspublish","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpgherveou%2Fgulp-awspublish","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpgherveou%2Fgulp-awspublish/lists"}