{"id":27335060,"url":"https://github.com/sloshy/dumb-sync","last_synced_at":"2026-04-30T09:36:12.152Z","repository":{"id":175372581,"uuid":"617283452","full_name":"sloshy/dumb-sync","owner":"sloshy","description":"A microframework for syncing and transforming files","archived":false,"fork":false,"pushed_at":"2023-06-15T20:10:01.000Z","size":148,"stargazers_count":2,"open_issues_count":6,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-12T14:51:38.646Z","etag":null,"topics":["bash","jq","rsync"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-2.1","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sloshy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-03-22T03:59:51.000Z","updated_at":"2023-07-17T21:03:25.000Z","dependencies_parsed_at":"2023-06-29T09:15:41.451Z","dependency_job_id":null,"html_url":"https://github.com/sloshy/dumb-sync","commit_stats":null,"previous_names":["sloshy/dumb-sync"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sloshy/dumb-sync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloshy%2Fdumb-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloshy%2Fdumb-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloshy%2Fdumb-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloshy%2Fdumb-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sloshy","download_url":"https://codeload.github.com/sloshy/dumb-sync/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloshy%2Fdumb-sync/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32460781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T22:27:22.272Z","status":"online","status_checked_at":"2026-04-30T02:00:05.929Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","jq","rsync"],"created_at":"2025-04-12T14:46:59.126Z","updated_at":"2026-04-30T09:36:12.145Z","avatar_url":"https://github.com/sloshy.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dumb Sync\nDumb Sync is a sort of micro-framework for using `rsync` to synchronize files across multiple directories and machines.\nThis project was born out of a need to have slightly-varying synchronizations across multiple folders, including a desire to transform the synchronized files in some cases while still keeping them in sync.\n\n## Main Features\n* Unidirectional synchronizing folders via `rsync`\n* Per-file and per-folder transformation scripts\n* Custom script argument support per-config\n* Program variables (such as current file/time/output directory) support for scripts\n* `sudo` support for transformations to pass your password along to scripts as necessary\n* Date-time-based synchronization of transformed files using a stored time offset\n* Optional removal of deleted files from remote based on file listing\n\nA good use-case for using this sync script is if you have multiple target folders to sync, each with slightly varying transformations you wish to apply to some or each of them.\nWhen you have a file sync + transformation use-case that spans more than a few folders and shared scripts between them, things can get a little complicated without a script framework in place like this one.\n\n## Install \u0026 Requirements\nBefore using, make sure you have a modern version of Bash as well as `rsync` and `jq` installed on your system.\n```bash\nhash rsync || echo \"rsync not installed\"\nhash jq || echo \"jq not installed\"\n```\n\nAlso manually check that you have a Bash version \u003e= 4.3\n\n```bash\nbash --version\n```\n\nDownload the latest `sync.sh` from this repository and make sure it is marked as executible\n\n```bash\nwget https://raw.githubusercontent.com/sloshy/dumb-sync/main/sync.sh\nchmod +x sync.sh\n```\n\nOr, you can click the big green \"Clone\" button and \"download as ZIP\" a copy of the entire repository to get the latest version of every file.\nThen, make sure that every `.sh` file you would like to use is marked executible using a command similar to the above.\n\nNext, create a file in the same directory named `sync.json`.\nYou can follow the template in the included [`sync.json`](/sync.json) or you can create one yourself.\n\nFor a more fleshed-out example, see the [example_transformations](/example_transformations/) directory which has a more motivated example of how you can chain transformation scripts together while keeping the resulting files in sync.\n\n## Configuration\nTo use the sync script, you need a configuration file with the proper settings.\nRequired settings are as follows:\n* `remote_url` or `remote_urls` - One or more rsync-compatible locations for files. Can be another directory on this local machine, or an rsync-compatible URL such as `rsync://some-host/` or `me@example.com:/some/dir/`. For syncing folders, **be sure this ends in a trailing slash**. If you have a single URL, specify it as `remote_url` and by default all of your configs will use this url for syncing. For multiple URLs, you can create an object in an array named `remote_urls` to designate the URL with a name.\n* `log_dir` - The directory for storing log files. Defaults to the current directory if not set. Trailing slashes are stripped and implied at runtime.\n* `file_list_dir` - The directory for storing rsync file lists. Defaults to `log_dir` if not set. Trailing slashes are stripped and implied at runtime.\n* `sync_offset_seconds` - A number of seconds to offset date-time checking for transformed files. This defaults to 0, so the main use case for ever modifying this is if you want to pretend you are further in the future than your current stored time offset.\n* `configs` - An array of individual sync configs, described below.\n\nEach URL defined in `remote_urls` has two required parameters:\n* `name` - A friendly name for the URL, for selecting in each of your configs.\n* `url` - The rsync URL of the location you are syncing from.\n\nEach sync config in the `configs` array requires the following options:\n* `remote` - An additional suffix to append to the base `remote_url` specified above. For example, if you are syncing from `rsync://some.remote.server/files/`, you could add `some/dir/` to make the final sync URL equal `rsync://some.remote.server/files/some/dir/`.\n* `local` - The local directory to sync files to. It is recommended to **not** use the current directory as some settings can cause the sync program to inadvertedly remove program files. Also, any trailing slashes are stripped and implied at runtime.\n* `disabled` - Whether to run the current config. Defaults to `false`, but can be set to `true` to skip the current config.\n\nYou can also supply these optional settings:\n* `url_name` - A specific remote URL to use. If specified, uses the URL as named in the `remote_urls` array in the root of the config. If not specified, your config will use the default URL specified in `remote_url` if defined. Otherwise, it will fail.\n* `exclude` - An array of rsync-compatible exclusions for the current sync config. For example, a setting of `[\"Something*\", \"*Something\"]` will exclude files that start and end with the string `Something` in their name.\n* `include` - An array of rsync-compatible includions for the current sync config, for usage in conjunction with exclusions above. These are applied before exclusions.\n* `transforms` - An array of the names of transformation scripts to apply in-order. These are ran on a per-file basis.\n* `cleanup` - An array of the names of cleanup transformation scripts to apply in-order. These are ran once at the end of synchronizing files.\n* `comparison` - The name of a `comparison` script to run for determining whether or not a file is `missing`, `updated`, or `current`. An example is defined in this repository for keeping files with modified extensions in sync with the originals, defined in more detail below.\n* `max_size_bytes` - The upper limit in file size for syncing. Files over this size in bytes will not be synced.\n* `min_size_bytes` - The lower limit in file size for syncing. Files under this size in bytes will not be synced.\n* `rm_missing` - A boolean flag for whether or not to remove files that are available locally but missing from the remote sync directory. Defaults to false, but should usually be set to `true` for cases where you want to keep a clean sync directory.\n\nFor the `max_size_bytes`/`min_size_bytes` options, this only affects syncing and it does not affect the file lists used for comparisons.\nThis means, for example, if you have a file locally that has the same name as a file being excluded due to the size limit, it will not be deleted if you also specify `rm_missing`.\n\nYou can also supply arbitrary properties for usage as transformation and comparison parameters, described below.\n\n### Transformations\n\nTransformations are listed in two places in the root of the config:\n* `transformations` - Definitions of scripts to run on a per-file basis.\n* `cleanup_transformations` - Definitions of scripts to run once after syncing is completed.\n\nNone of these transformations are ran without being explicitly invoked by a config object definition.\nFor example, if you have a config that does not specify any transformations, a traditional 1:1 sync will be performed.\nIf instead, you define your transformations array in the config as `[\"script-a\", \"script-b\"]`, then for each file that is synchronized it will run the transformation named `script-a` followed by `script-b`, once per-file.\nIf these scripts are set for the `cleanup` array instead, the scripts will be ran once.\n\nEach transformation is defined as follows:\n* `name` - A friendly name for the transformation, separate from the actual script name. This is the name you supply in the sync configs above.\n* `script` - The location of the executible file or script to run when this is invoked. Files in the same directory can be invoked as `./script-name.sh`.\n* `params` - An ordered list of arguments that the script accepts, described below.\n* `sudo` - A boolean flag as to whether or not the script expects to be ran with `sudo`. Defaults to false. If set to true, your `sudo` password will be supplied to the script via standard input, so you should make sure your script handles this appropriately. For an example of handling `sudo` passwords from standard input, see the included [`chd.sh`](/example_transformations/chd.sh) script that uses Docker, which is usually limited to root users for security reasons.\n\nThe `params` section defines the list of parameters that will be supplied in order to the script.\nYou can supply arbitrary strings, or you can use special keywords to pass runtime variables and your own config settings to the script.\nThese keywords available for transformations are as follows:\n* `\u003coutdir\u003e` - Passes the configured output directory (`local`) for the current config.\n* `\u003coutdir_abs\u003e` - The absolute path of the configured output directory. This is determined by prefixing the directory with the results of the `pwd` command, so if your output directory is already absolute and not relative, you should not use this.\n* `\u003cfile_list\u003e` - A file containing the most recent list of files from the remote directory, synchronized by rsync. This list is saved *before commencing the sync* so it is possible for it to be outdated in later steps. This is primarily used for checking existing files before syncing, and is mostly useful in comparisons, rather than transformations.\n* `\u003cfilename_remote\u003e` - The filename of the remote file being synchronized, prefixed by the current output directory. **Not valid for cleanup scripts.** If the current file is `my-file.txt` and the currently-set `local` output directory is `some/dir/`, the parameter will be rendered as `some/dir/my-file.txt`.\n* `\u003cfilename_remote_base\u003e` - The base name of the current file, including its extension. **Not valid for cleanup scripts.** Does not include any path information.\n* `\u003clast_sync_time_secs\u003e` - The UNIX epoch timestamp in seconds-since-1970-01-01 that the last sync was performed. Useful for keeping transformed files in sync with remote ones when local file timestamps might not be reliable (for example, if you are syncing a file that might become modified for unrelated reasons, that you would prefer to sync from scratch each time).\n* `\u003ccurrent_time_secs\u003e` - The current timestamp (same format as above).\n* `\u003csync_offset_secs\u003e` - The number of seconds of the sync offset defined in your root config. Typically 0, but could be different, so for some scripts you want to make sure you include this parameter any time you are dealing with time offsets.\n* `\u003carg:some_arg\u003e` - The literal value of the property `some_arg` in the current config. Can be used to pass per-config settings into the script that might vary based on your use-cases. For example, to get the expected starting file extension (if defined), use `\u003carg:ext_remote\u003e` to copy the value of `ext_remote` as an argument to the script.\n* You can also supply literal text to have it passed verbatim as an argument.\n\nFor some ideas on how you can configure the transformations, see the [example transformations](/example_transformations/) directory.\n\n### Comparisons\nComparisons are a way to keep your files in sync even if they are transformed.\nThey work in conjunction with the `rm_missing` option to help determine whether or not a file exists in the state you desire on your local machine, and also with transformations by limiting the scope of transformations to updated or freshly-synchronized files only.\n\nThe way that `rsync` usually works is that it compares the file names and modified times across hosts.\nIf you are transforming the files, or deleting the original files as part of transformation, this will no longer work, so to keep them in sync the script will make use of one or more comparison techniques that you can customize yourself with scripts.\n\nThe default comparison method uses the built-in `rsync` file name and modification checking, but to opt into a different form of comparison, you can specify a dedicated comparison script in the `comparisons` array in the root of your config.\nThe included example config specifies [one such script](/comparisons/expected_ext.sh) that takes a variety of parameters in order to determine whether or not a file in the output directory counts as `missing`, `current`, or `updated`. Each `current` file is excluded from the upcoming sync, while `updated` files are removed and redownloaded and `missing` files are either explicitly ignored as a \"preexisting file\" that should not be transformed, or removed outright if the `rm_missing` option is enabled.\n\nTo configure a comparison, simply add an object much like a `transformation` or `cleanup_transformation` object inside the `comparisons` array, and configure it in the same sort of way, with the same options, except for `sudo` which is not supported for comparisons intentionally.\n\nA single comparison can be set for each sync configuration, unlike transformations or cleanup transformations.\nThis single script will be ran if defined, and its return value must be either `missing`, `transform`, `current`, or `updated` for every file in the output directory.\nEach case has certain properties:\n\n* `missing` - This file is present locally, but not on the remote destination. Will be removed if `rm_missing` is enabled.\n* `transform` - This file is present locally, but it is still in its initial state. Will not exclude the file from syncing if it's updated and will transform as normal.\n* `current` - This file is derived/transformed from a remote file, and it is up-to-date. It is excluded from sync and per-file transformations.\n* `updated` - This file is derived/transformed from a remote file, but it is not up-to-date with the remote source. It will be deleted, redownloaded, and transformed again.\n\nFor the `current` case, it actually needs to be specified as `current \u003cremote_file_name\u003e`, where `\u003cremote_file_name\u003e` is the name of the file you are deriving your final, local file from after all your transformations.\nThis is so that the sync script can explicitly exclude this file from sync.\n\nIf a comparison is not specified, only removing nonexistent files is taken into account if specified, so be sure to specify a comparison if your local output directory is likely not a 1:1 match to your remote folder.\n\nComparisons support some special keywords as parameters, and do not support some keywords that are supported for transformations.\nHere is the list of keywords that are **only valid for comparisons**:\n\n* `\u003cfilename_local\u003e` - The local filename that is being checked for comparison. Is prefixed by the local path specified in the current config.\n* `\u003cfilename_local_base\u003e` - The local filename that is being checked for comparison, without any path information.\n\nThe following keywords are **not valid for comparisons**:\n\n* `\u003cfilename_remote\u003e`\n* `\u003cfilename_remote_base\u003e`\n\nThis is due to the internal logic at the time of comparison only being over the files in the local output directory, and not over the list of files from the remote source.\nFor the list of remote files, use the `\u003cfile_list\u003e` keyword instead, which you can search and interpret as you see fit.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsloshy%2Fdumb-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsloshy%2Fdumb-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsloshy%2Fdumb-sync/lists"}