{"id":20763087,"url":"https://github.com/nickpisacane/websync","last_synced_at":"2025-04-30T07:42:58.147Z","repository":{"id":23576409,"uuid":"99379116","full_name":"nickpisacane/websync","owner":"nickpisacane","description":"Like `aws s3 sync` with automatic CloudFront invalidations and more.","archived":false,"fork":false,"pushed_at":"2022-04-10T06:04:50.000Z","size":248,"stargazers_count":45,"open_issues_count":7,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-23T01:37:55.706Z","etag":null,"topics":["aws","cli","deployment","s3","static-site"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nickpisacane.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-04T20:57:25.000Z","updated_at":"2023-07-20T23:22:12.000Z","dependencies_parsed_at":"2022-08-07T11:00:38.011Z","dependency_job_id":null,"html_url":"https://github.com/nickpisacane/websync","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickpisacane%2Fwebsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickpisacane%2Fwebsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickpisacane%2Fwebsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickpisacane%2Fwebsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nickpisacane","download_url":"https://codeload.github.com/nickpisacane/websync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251663422,"owners_count":21623900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cli","deployment","s3","static-site"],"created_at":"2024-11-17T10:42:33.511Z","updated_at":"2025-04-30T07:42:58.089Z","avatar_url":"https://github.com/nickpisacane.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Websync\n![travis](https://travis-ci.org/nickpisacane/websync.svg?branch=master)\n\nWebsync is like `aws s3 sync` with automatic CloudFront invalidation and more.\n\n![alt text](https://raw.githubusercontent.com/nickpisacane/websync/master/media/example.gif)\n\n\nWebsync sync is meant to be a replacement for `aws s3 sync`. Websync, like the AWS cli, syncs local directories with s3 prefixes, and visa-versa. Websync expands on these features by automatically creating *optimized* invalidations on any associated CloudFront distributions, and exposing an expressive configuration system (on top of the CLI interface) with JSON or JavaScript, and a programmatic API.\n\n# Table Of Contents\n* [Installation](#installation)\n* [Usage](#usage)\n* [Configuration Files](#configuration-files)\n* [Item API](#item-api)\n* [Invalidation System](#invalidation-system)\n* [Wildcard Policies](#wildcard-policies)\n* [Roadmap](#roadmap)\n\n\n# Installation\n```sh\n# Install global cli, the `websync` command\nnpm i -g websync\n# Install local\nnpm i websync\n```\n\n# Usage\n## websync command\n```sh\n# Parse configuration from `websync.json` or `websync.js`\nwebsync\n# Parse configuration explicitly\nwebsync --config ./myConfig.js\n# With command line options\nwebsync ./www s3://mybucket.io\n# General\nwebsync [source] [target] [...options]\n```\n### Options\n* `source` Source container (local directory or S3 bucket): `./myDirectory`\n* `target` Target container (S3 bucket or local directory): `s3://my-bucket`\n* `config` Explicit configuration file (JSON or JavaScript): `--config ./myConfig.json`\n* `include` Glob pattern to filter files (from source) to include: `--include **/*.ext`\n* `exclude` Glob pattern to filter files (from source) to exclude: `--exclude **/*.ext`\n* `diffBy`  Override property by which items are diffed (`modtime`, or `size` with default: `modtime`): `--diffBy size`\n* `wildcardPolicy` Override the wildcard policy (`majority`, `unanimous`, or `minority` with default: `majority`): `--wildcardPolicy unanimous`\n* `wildcardAll` Append wildcard to _all_ invalidation paths (NOTE: this does not change invalidation path resolution), useful for invalidating querystring paths: `--wildcardAll`\n* `invalidateDeletes` Invalidate paths for items being _deleted_ from target. Useful for situations where you _do not_ want users to be able to access the items anymore: `--invalidateDeletes`\n* `distribution` One or more CloudFront distribution IDs (NOTE: this overrides discovery of distributions): `--distribution \u003cDIST ID\u003e --distribution \u003cANOTHER DIST ID\u003e`\n* `yes` Skip all prompts with a \"yes\" response (NOTE: websync will warn you if more than 500 invalidations are being made, as this will require a payment): `--yes`\n\n**NOTE**: More options are available in the [Configuration Files](#configuration-files)\n\n**NOTE**: All command line arguments _OVERRIDE_ configuration file options. Additionally, `source` and `target` are _required_, but can be provided by CLI or Configuration File\n\n\n# Configuration Files\nConfiguration files can provide all of the options available from the CLI with the addition of `modifiers`, a flexible system to provide explicit arguments to S3 put operations.\n## Modifiers Object\nThe modifier object of the configuration file is an object in which the `keys` are Glob Patterns, and the `values` are [`S3.putObject Params`](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property), or a function that returns either an `S3.putObject Params`, or a Promise which resolves `S3.putObject Params`. Note, if a function is provided (async or not) it will be called with a single [`Item`](#item-api) argument that will represent the file or object from the _SOURCE_ container.\n**NOTE**: Source files can match **multiple** modifiers, allowing one to keep things DRY.\n### Examples\nJavaScript configuration. See the [example](https://github.com/nickpisacane/websync/tree/master/examples/complex) for context.\n```js\nconst Path = require('path')\n\nconst DOWNLOAD_CONTENT_TYPE = 'application/octet-stream'\nconst DOWNLOAD_TAG = 'Downloadable'\nconst REDIRECT_TAG = 'Redirectable'\n\nconst makeDispositionName = fileName =\u003e\n  `${Path.basename(fileName).split('.')[0]}-${Date.now()}${Path.extname(fileName)}`\n\nmodule.exports = {\n  source: './public',\n  target: 's3://websync-complex-example-bucket',\n  modifiers: {\n    // This matches all files, provides Plain Object\n    '**/*': {\n      Metadata: {\n        'source-user': process.env.USER,\n      },\n    },\n    // Matches all files in downloads, provides a synchronous function\n    'downloads/**/*': item =\u003e ({\n      ContentType: DOWNLOAD_CONTENT_TYPE,\n      ContentDisposition: `attachment; filename=\"${makeDispositionName(item.key)}\"`,\n      Tagging: DOWNLOAD_TAG,\n    }),\n    // Matches any file with the `.redirect` extension, provides an asynchronous funcion\n    '*.redirect': async item =\u003e ({\n      WebsiteRedirectLocation: (await item.read()).toString().trim(),\n      ContentType: 'text/html',\n      Tagging: REDIRECT_TAG,\n    }),\n  },\n}\n```\nJSON configuration. See the [example](https://github.com/nickpisacane/websync/tree/master/examples/basic) for context. In the example below, the `!*.*` pattern matches any item with _no extension_, i.e. \"another-page\", and overrides the implied `Content-Type` with `text/html` to have clean paths for a simple static website.\n```json\n{\n    \"source\": \"./public\",\n    \"target\": \"s3://websync-basic-example-bucket\",\n    \"exclude\": \"*.exclude\",\n    \"modifiers\": {\n        \"!*.*\": {\n            \"ContentType\": \"text/html\"\n        }\n    }\n}\n```\n\n# Item API\nWebsync's `Item` object is an interface that abstractly represents either a local file, or an `S3` Object. With regards to the [`Configuration File`](#configuration-files), the `Item` object passed to a `modifier` function is always from the __source__ container (local directory, or `S3` Bucket). All `Item`s adhere to the following interface:\n```ts\ninterface Item {\n  // The \"key\" (path or S3 Object key)\n  key: string\n  // Last modification time\n  modtime: Date\n  // Size in bytes of the Item\n  size: number\n  // Whether item is a symbolic link (always false for S3)\n  isSymbolicLink: boolean\n  // Read the *entire* body of the item\n  read(): Promise\u003cBuffer\u003e\n}\n```\n\n# Invalidation System\nWebsync's invalidation system automatically creates the minimal amount of invalidation paths required to accommodate the provided `wildcard` policy. It does this by creating a `diff` of the target and the `source`, and two trees: one of the items in the `diff` and all of the items in the `target`. It then walks the `diff` (starting at the root) tree and compares the number of children that are being invalidated with those that are not -- this is where the `wildcard` policy makes all the difference. Additionally, websync will detect when a given path that is being wildcarded should invalidate all of its children, or only its direct children, thereby producing the most optimal invalidation paths.\n\n***NOTE***: the `wildcardAll` option _DOES NOT_ change the invalidation path generation, rather, wildcards are appended to every path generated. This is useful for invalidating querystring paths for a given object, etc.\n\nFor more information on how invalidations work on CloudFront, please refer to the [AWS Documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html#invalidation-specifying-objects).\n\n# Wildcard Policies\nWildcard policies determine when a given path will be _wildcarded_, thereby invalidating, all or only its direct, children to reduce the number of invalidation paths generated. The three policies available from least _strict_ to most _strict_ include `minority`, `majority`, and `unanimous`. \n\n## minority\nA given path is wildcarded when a _minority_ of its children are being invalidated. **NOTE**: This always results in a the `/*` invalidation path, when invalidations are required.\n#### Example:\nAll Target Items:\n* `/`\n  * `/css`\n    * `main.css`\n    * `vendor.css`\n  * `/js`\n    * `main.js`\n  * `index.html`\n\nInvalidated Items:\n  * `/`\n    * `index.html`\n\nInvalidation Paths:\n  * `/*`\n## majority\nA given path is wildcarded when a _majority_ of its children are being invalidated.\n#### Example:\nAll Target Items:\n* `/`\n  * `/css`\n    * `main.css`\n    * `vendor.css`\n  * `/js`\n    * `main.js`\n  * `index.html`\n\nInvalidated Items:\n  * `/`\n    * `/css`\n      * `main.css`\n      * `vendor.css`\n    * `index.html`\n\nInvalidation Paths:\n  * `/css/*`\n  * `/index.html`\n## unanimous\nA given path is wildcarded when a _all_ of its children are being invalidated.\n#### Example:\nAll Target Items:\n* `/`\n  * `/css`\n    * `main.css`\n    * `vendor.css`\n  * `/js`\n    * `main.js`\n  * `index.html`\n\nInvalidated Items:\n  * `/`\n    * `/css`\n      * `main.css`\n    * `/js`\n      * `/main.js`\n    * `index.html`\n\nInvalidation Paths:\n  * `/css/main.css`\n  * `/js/*`\n  * `/index.html`\n\n# Roadmap\n- [x] Initial CLI\n- [ ] Programmatic API Documentation\n- [ ] Better Roadmap\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickpisacane%2Fwebsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnickpisacane%2Fwebsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickpisacane%2Fwebsync/lists"}