{"id":13587851,"url":"https://github.com/openbridge/ob_bulkstash","last_synced_at":"2025-04-10T11:53:25.038Z","repository":{"id":73272213,"uuid":"112799574","full_name":"openbridge/ob_bulkstash","owner":"openbridge","description":"Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage. ","archived":false,"fork":false,"pushed_at":"2020-09-21T17:49:45.000Z","size":179,"stargazers_count":114,"open_issues_count":3,"forks_count":16,"subscribers_count":11,"default_branch":"develop","last_synced_at":"2025-03-24T10:47:38.077Z","etag":null,"topics":["amazon-web-services","data-pipeline","docker","docker-image","docker-rclone","docker-service","google-cloud","google-cloud-storage","oracle-cloud","rclone","s3","sftp-synchronisation","storage-service","sync"],"latest_commit_sha":null,"homepage":"https://www.openbridge.com","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openbridge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-02T00:25:22.000Z","updated_at":"2025-02-07T20:14:56.000Z","dependencies_parsed_at":"2023-03-05T21:15:39.664Z","dependency_job_id":null,"html_url":"https://github.com/openbridge/ob_bulkstash","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openbridge%2Fob_bulkstash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openbridge%2Fob_bulkstash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openbridge%2Fob_bulkstash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openbridge%2Fob_bulkstash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openbridge","download_url":"https://codeload.github.com/openbridge/ob_bulkstash/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248215193,"owners_count":21066622,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-web-services","data-pipeline","docker","docker-image","docker-rclone","docker-service","google-cloud","google-cloud-storage","oracle-cloud","rclone","s3","sftp-synchronisation","storage-service","sync"],"created_at":"2024-08-01T15:06:23.377Z","updated_at":"2025-04-10T11:53:25.020Z","avatar_url":"https://github.com/openbridge.png","language":"Shell","funding_links":[],"categories":["Shell","HarmonyOS","Docker Images"],"sub_categories":["Windows Manager","NFS"],"readme":"# Bulk Stash - Batch processing for cloud storage\nBulk Stash is an `rclone` service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage. Bulk Stash is a dockerized version of rclone.\n\nYou can also use this for copying or syncing files locally to a remote SFTP server or between two remote SFTP servers.\n\n![rclone](https://github.com/openbridge/ob_bulkstash/raw/develop/rclone.png \"How It Works\")\n\nThere are advanced use cases where you can actually transfer a certain class of files (CSV) to Amazon S3 for import into Amazon Redshift, BigQuery or Amazon Athena. If you are interested in learning check out this [blog post](https://blog.openbridge.com/how-to-setup-a-batch-data-pipeline-for-csv-files-8c4d0cd7394b).\n\n# Features\n* **Dockerized**: `rclone` is neatly packed into a Docker image that can be run anywhere Docker can be installed.\n* **Alpine Linux**: The container uses Alpine Linux which makes it light and efficient. The image size is \u003c 26mb.\n* **Environment Variables**: The image is designed to take advantage of recent support in `rclone` to utilize environment variables. This means you don't have to step through the typical config initialization process.\n* **Process Monitoring and Management**: The container also uses `Monit` to ensure that long running processes are monitored under process management. For example, `Monit` will make sure `crond` is running in the background and will restart it if it crashed. You can extend Monit to monitor folders, check for file sizes and many other.\n* **Configuration**: Configuration can be stored and managed outside the container. Configurations can also be inserted at runtime manually or via a controller script/app\n* **Runtime**: You can run a collection of containers running independent tasks via config files. This means you can wrap the Docker service with other apps like bash, python and so forth on your host\n* **Deployment**: You can setup things like Amazon Lambda and ECS tasks to control the runtime tasks. Configurations can be encrypted and stored in a service like AWS KMS. Configuration attributes can then be provided by an end user via a front-end web app. For example, you can have a form that collects all the S3 or Google OAuth tokens. A front end is not include :)\n* **Notifications**: You can uses services like cronalarm, Cronitor.io, healthchecks.io...to set status and alerts. Notifications via Slack, Hipchat... are also possible, but not enabled.\n* **Folder Size Monitoring**: Local source folders can be monitored. If the size of the contents in the source folder exceed a value you set an rclone copy operation will occur\n\n### Features\n* MD5/SHA1 hashes checked at all times for file integrity\n* Timestamps preserved on files\n* Partial syncs supported on a whole file basis\n* Copy mode to just copy new/changed files\n* Sync (one way) mode to make a directory identical\n* Check mode to check for file hash equality\n* Can sync to and from network, eg two different cloud accounts\n* Optional encryption (Crypt)\n* Optional FUSE mount (rclone mount)\n* Optional database loading\n\n## Supported Services\n What services are supported?\n* Google drive\n* **Amazon S3**\n* Swift / Rackspace Cloudfiles / Memset Memstore\n* Dropbox\n* **Google Cloud Storage**\n* Local filesystem\n* Amazon Drive\n* Backblaze B2\n* Hubic\n* Microsoft OneDrive\n* Yandex Disk\n* **SFTP**\n\nFor the items in **bold** there are sample commands and configurations in the readme.\n\n# Getting Started\n\nFirst, pull the latest docker image:\n\n```bash\ndocker pull openbridge/ob_bulkstash\n```\nThis will pull the latest version by default. However, as part of the `hooks/build` process we publish a number of older versions of rclone. If you want to see the available versions, check out Docker Hub [`openbridge/ob_bulkstash`](https://hub.docker.com/r/openbridge/ob_bulkstash/tags/). For example, if you wanted to run version `1.48`, then pull that version like this:\n\n```bash\ndocker pull openbridge/ob_bulkstash:1.48.0\n```\nAdditional pre-built versions are tagged and available for use: https://hub.docker.com/r/openbridge/ob_bulkstash/tags/\n\nIf you want to build your own image, you need to pass the version you want to use:\n```bash\ndocker build --build-arg RCLONE_VERSION=1.48.0 -t openbridge/ob_bulkstash:1.48 .\ndocker build --build-arg RCLONE_VERSION=1.48.0 -t openbridge/ob_bulkstash:latest .\n```\nYou may also pass a different architecture: `--build-arg RCLONE_TYPE=arm`\n\nGot your version setup? Great. Next, we need to define a configuration for remote storage locations. The following demonstrates how to sync Amazon and Google cloud storages.\n\n### Testing Your Build\nThe validate that your build worked correctly, you can run a simple check which will have rclone echo the version back to you.\n\nIf you run this `docker run openbridge/ob_bulkstash rclone -V` you should see the version displayed like this:\n```Bash\nrclone v1.47.0\n- os/arch: linux/amd64\n- go version: go1.12.4\n```\nIf you see this, success! Your image is ready to go!\n\n## Amazon and Google Examples\nIn our example we have a source of files at Amazon S3 and destination for those files at Google Cloud Storage location. This means we will need to set the configuration ENV variables for source and destination.\n\n### Amazon S3\nHere is an Amazon Simple Storage Service (aka \"S3\") config:\n\n```bash\nRCLONE_CONFIG_MYS3_TYPE=s3\nRCLONE_CONFIG_MYS3_ACCESS_KEY_ID=enter your access key\nRCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=enter your secret key\nRCLONE_CONFIG_MYS3_SERVER_SIDE_ENCRYPTION=AES256\nRCLONE_CONFIG_MYS3_STORAGE_CLASS=REDUCED_REDUNDANCY\n```\n\nThe first thing is to define the `type` as s3: `RCLONE_CONFIG_MYS3_TYPE=s3`\n\nNext, you need to provide credentials to access (pull data from or copy data to) S3:\n`RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=enter your access key` and\n`RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=enter your secret key`\n\nYou will want to set the encryption for the data you send to S3: `RCLONE_CONFIG_MYS3_SERVER_SIDE_ENCRYPTION=AES256`\n\nYou can also set your storage class in the event you want to use lower cost options: `RCLONE_CONFIG_MYS3_STORAGE_CLASS=REDUCED_REDUNDANCY`. Set this to your preference.\n\n\n### Google\nLike S3 we can use Google Cloud Storage as a remote location.\n\n```bash\nRCLONE_CONFIG_MYGS_TYPE=google cloud storage\nRCLONE_CONFIG_MYGS_CLIENT_ID=\nRCLONE_CONFIG_MYGS_CLIENT_SECRET=\nRCLONE_CONFIG_MYGS_PROJECT_NUMBER=foo-mighty-139217\nRCLONE_CONFIG_MYGS_SERVICE_ACCOUNT_FILE=/auth.json\n```\n\nThe first step is to set the `type` for Google: `RCLONE_CONFIG_MYGS_TYPE=google cloud storage`\n\nNext, make sure you set your credentials like you did for S3: `RCLONE_CONFIG_MYGS_CLIENT_ID=enter your client key` and `RCLONE_CONFIG_MYGS_CLIENT_SECRET=enter your secret key`\n\nThe project number is only needed only for list/create/delete buckets:\n `RCLONE_CONFIG_MYGS_PROJECT_NUMBER=foo-mighty-139217`\n\nIf you are using a Google Account Credentials JSON file you would leave the client ID and secret blank and enter the path to your file:\n `RCLONE_CONFIG_MYGS_SERVICE_ACCOUNT_FILE=/auth.json`\n\n You will need to make sure that the volume that contains the auth file is mounted and the path is passed via `RCLONE_CONFIG_MYGS_SERVICE_ACCOUNT_FILE`.\n\n**Thats it!** You defined two remote locations, one for Amazon and one for Google. You can start to transfer files.\n\n# How To Run\nWith your config setup, now you can run `rclone`!\n\nThis is an example Docker `RUN` command\n```bash\ndocker run openbridge/ob_bulkstash \\\n  --env-file env/sample.env \\\n  rclone copy MYS3:myawsbucket/path/to/file/ MYGS:mygooglebucket/path/to/files/\n```\nCheck out the [docker run docs](https://docs.docker.com/engine/reference/commandline/run/) for the latest syntax.\n\nLastly, you can use Docker Compose:\n```bash\ndocker-compose up -d\n```\nor\n\n```bash\n/usr/local/bin/docker-compose -f prod.yml up -d --remove-orphans\n```\nNote: You will need to put the appropriate command in the compose YAML file you want docker rclone to run.\n\n\n## Examples Running Docker On Your Host\nThese are a couple of simple examples around wrapping the Docker image with Bash on your host.\n\n### Run multiple config files\nThis example will go through all the env files and run the image with a `COPY` command:\n```bash\nfor i in ./env/*.env; do\ndocker run -v /my/volume:/data -it --env-file ${i} openbridge/ob_bulkstash rclone copy MYS3:myawsbucket/path/to/file/ MYGS:mygooglebucket/path/to/files/\ndone\n```\nEarlier we showed a simple rclone command to echo the version number:\n```bash\ndocker run openbridge/ob_bulkstash rclone -V\n```\n\n### Using the Google AUTH file\nHere is an example that mounts the Google auth file needed for service level accounts:\n```bash\nfor i in ./env/prod/*.env; do\n  echo \"working on $i\"\n  bash -c \"docker run -it -v /auth/prod/auth.json:/auth.json --env-file ${i} openbridge/ob_bulkstash rclone copy MYS3:myawsbucket/path/to/file/ MYGS:mygooglebucket/path/to/files/\"\n  if [[ $? = 0 ]]; then echo \"OK: \"; else echo \"ERROR: \"; fi\ndone\n```\n# Using `crond` Inside Docker\nIf you want to persist your container you can set it up to always be running with `crond` as a background process. While most everything is automated there are a few configuration items you need to set.\n\n**IMPORTANT**: This assumes you have a basic understanding of Docker and background processes. If you do not know what `--detach , -d` means then please review the Docker docs about running in detached mode (hint: this is how you run things in the background)\n\n## Runtime Environment\nDepending on your use of `CROND`, it may not have access to the OS defined `ENV` variables. As a convenience, the image will output these to a file:\n\n```bash\nprintenv | sed 's/^\\([a-zA-Z0-9_]*\\)=\\(.*\\)$/export \\1=\"\\2\"/g' | grep -E \"^export RCLONE\" \u003e /cron/rclone.env\n```\nIf needed, you can then import these variables into any scripts that you want to run in the container such as using something like `source /cron/rclone.env`.\n\n## Option 1: Bring Your Own Crontab Configuration\n\n### Step 1: Setup your `crontab.conf` config\nRunning `crond` requires a proper configuration file. You can easily add a crontab config file and have the container use it. A `crontab.conf` should look something like this:\n\n```bash\nSHELL=/bin/bash\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n*/15 * * * * /usr/bin/env bash -c /rclone.sh run 2\u003e\u00261\n```\nThere is a sample of this in `cron/crontab.conf`. You can use this as a starting point for your own config. Once you have your config ready, we can move to Step 2.\n\n### Step 2: Mount your `crontab.conf` config\nNext, you will want to mount your config file. The config is on your host, not in the container. To get it into your container you need to mount it from your host into the container.\n\nThe basic template is this:\n\n```docker\n-v /path/to/the/file/on/your/host/crontab.conf:/where/we/mount/in/container/crontab.conf\n```\n\nThis example shows the mount your config in docker compose format:\n```docker\nvolumes:\n  - /Github/ob_bulkstash/cron/crontab.conf:/cron/crontab.conf\n```\nIt will look the same if you are doing it via Docker run:\n```docker\n-v /Github/ob_bulkstash/cron/crontab.conf:/cron/crontab.conf\n```\nIn those examples, the `crontab.conf` located in my local GitHub folder will get mounted inside the container at `/cron/crontab.conf`\n\nMounting your config makes it available to the startup service within your container. If you are unfamiliar with `-v` or `volumes`, check the docs from Docker.\n\n\n### Step 3: Set environment variable `RCLONE_CRONFILE`\nIn your `ENV` make sure to set the path to the location you are mounting your `crontab.conf` file. In our example above we are using `/cron/crontab.conf`. This means you set the `ENV` path like this:\n```\nRCLONE_CRONFILE=/cron/crontab.conf\n```\n\n**This is the location in your container, not the host**\n\n\n## Option 2: Automatic Generation `RCLONE_CRONFILE`\n\nYou can let the image generate and run a command for you under `CROND`.\nThis is geared to running a single `CROND` task. If you want to run multiple tasks, it is best to choose **Option 1** which allows you more control over the number of tasks run.\n\n### Setting your `CROND` command\nIn your ENV, you need to set the desired command via `RCLONE_SYNC_COMMAND`. Here is an example command:\n\n```bash\ndocker run -d -e RCLONE_SYNC_COMMAND=\"*/15 * * * * /usr/bin/env bash -c /foo run\" openbridge/ob_bulkstash crond -f\n```\nThis will result in your container running in the detached mode (in the background) with a `CROND` entry like this:\n```bash\nSHELL=/bin/bash\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n*/15 * * * * /usr/bin/env bash -c /foo run\n```\nThis is just an example command, it will likely vary according to what you are looking too run.\n\n\n## IMPORTANT `crontab.conf` NOTE\nPlease note that if you set your own crontab config file via `RCLONE_CRONFILE=/cron/crontab.conf` it will take precedent over anything you pass via `-e` or set other environment variables.\n\n## Understanding How To Run Docker and `CROND`\nHere are a few examples of running Docker and `CROND` in the background. You can accomplish the same using `docker-compose`\n\nRunning in detached mode:\n```bash\ndocker run -d -e RCLONE_SYNC_COMMAND=\"*/15 * * * * /usr/bin/env bash -c /foo run\" openbridge/ob_bulkstash crond -f\n```\nYou can see the process running in the background:\n```bash\nPID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND\n 31     0 root     R     1528   0%   3   0% top\n  1     0 root     S     1516   0%   1   0% crond -f\n```\nRunning in detached mode using the `rcron.sh` helper script. This will use Monit has the background process monitor to make sure `CROND` is always running:\n```bash\ndocker run -d -e RCLONE_SYNC_COMMAND=\"*/15 * * * * /usr/bin/env bash -c /foo run\" openbridge/ob_bulkstash rcron start\n```\nYou can see the process running in the background:\n```bash\nPID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND\n 21     1 root     S     4788   0%   2   0% monit -Iv -c /etc/monitrc -l /dev/null\n  1     0 root     S     2172   0%   3   0% bash /usr/bin/rcron start\n 23     0 root     R     1524   0%   2   0% top\n 20     1 root     S     1516   0%   0   0% crond -b\n ```\nHere is another example running Monit as the controlling process:\n```bash\ndocker run -d -e RCLONE_SYNC_COMMAND=\"*/15 * * * * /usr/bin/env bash -c /foo\" openbridge/ob_bulkstash monit -Iv -c /etc/monitrc -l /dev/null\n```\nYou can see the process running in the background:\n```bash\nPID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND\n  1     0 root     S     4788   0%   2   0% monit -Iv -c /etc/monitrc -l /dev/null\n 22     0 root     R     1528   0%   1   0% top\n 21     1 root     S     1516   0%   3   0% crond -b\n ```\n\n Hopefully you get the point on how to do this. You have options, just make sure you understand the basics on how to run Docker in various contexts.\n\n# Creating Your Own Scripts: The `/rclone.sh` Example\n\nIncluded in the image is a utility script. You can use this as a robust example for creating your own. It highlights the potential to mount scripts like this into your container to run different types of operations.\n\n**Note**: `rclone.sh` is provided as-is and has not been fully tested. Think of it as a proof-of-concept, not something you should blindly use.\n\n## Overview\nThe script shows an example of how to run `rclone copy` and `rclone move`. It will also has a `foldersize` check in the event you want to trigger a `rclone move`. How is this helpful? If your disk is getting full this can trigger what amounts to be a cleanup task.  \n\n### Getting Started with `/rclone.sh`\nWhile the image contains `rclone.sh`, you will likely want to mount your own version of the script. For example;\n```\n-v /path/to/the/file/on/your/host/script.sh:/path/in/container/script.sh\n```\n\nAlso, you need to make sure the image can use `/rclone.sh`. This means you need to make sure any required environment variables are set correctly. For example, in your `ENV`, you need to set the following for `rclone.sh`:\n\n* `RCLONE_CROND_SCHEDULE` crontab schedule `* * * * *` to perform sync every midnight\n* `RCLONE_CROND_SOURCE_PATH` source location for `rclone copy` command\n* `RCLONE_CROND_DESTINATION_PATH` destination location for `rclone copy` command\n\n#### Setting Source and Destination\nIn your environment file you need to make sure your source and destination remotes are set. You need to put the full statement (remote names, buckets, paths...) for the source and destination for each variable.\n```bash\nRCLONE_CROND_SOURCE_PATH=\"/temp\"\nRCLONE_CROND_DESTINATION_PATH=\"MYS3:myawsbucket/path/to/file/\"\n```\nThis will ensure that `rclone` knows where to look for the files and where you want them delivered.\n\n\nThis is what it would look like in your config:\n```bash\nRCLONE_CROND_SCHEDULE=*/5 * * * *\nRCLONE_CROND_SOURCE_PATH=\"/tmp\"\nRCLONE_CROND_DESTINATION_PATH=\"MYS3:ob-testing/ebs\"\n```\n\n### Optional Settings\n\n#### Health check service\n\n`RCLONE_CROND_HEALTHCHECK_URL`\nIf you want to use a cron healthcheck service, set the environment variable:\n* `RCLONE_CROND_HEALTHCHECK_URL` used for health check services to ping status like cronalarm, Cronitor.io, healthchecks.io...\n\nThis is what it would look like in your config:\n```bash\nRCLONE_CROND_SCHEDULE=*/5 * * * *\nRCLONE_CROND_SOURCE_PATH=\"/tmp\"\nRCLONE_CROND_DESTINATION_PATH=\"MYS3:ob-testing/ebs\"\nRCLONE_CROND_HEALTHCHECK_URL=https://hchk.io/asads-aa12-ee23-qqw1-543e4c2ddv54385\n```\n#### How to monitor the size of your source directory\nYou can use a `foldersize` check to monitor your source path. To do this set the environment variable `RCLONE_CROND_SOURCE_SIZE` to a number in megabytes. For example, if you want to monitor your source path for 1 GB of files, you would set `RCLONE_CROND_SOURCE_SIZE=1000`. The 1000 megabytes = 1 GB.\n\n\n```bash\nRCLONE_CROND_SCHEDULE=*/5 * * * *\nRCLONE_CROND_SOURCE_PATH=\"/tmp\"\nRCLONE_CROND_DESTINATION_PATH=\"MYS3:ob-testing/ebs\"\nRCLONE_CROND_SOURCE_SIZE=\"1000\"\n```\n\n\n# Setting Up SFTP Remotes\nYou can setup SFTP remotes. This allows you to upload or download files from an SFTP server. You can also do server to server transfers between two remotes.\n\nLastly, if you want to pipeline data to Redshift, BigQuery, Athena or Spectrum via rclone take a look at the following batch data pipeline workflow:\n* https://blog.openbridge.com/how-to-setup-a-batch-data-pipeline-for-csv-files-8c4d0cd7394b\n\nIf you have any questions about how to pipeline data to a warehouse let me know.\n\nSample config\n* `RCLONE_CONFIG_MYSFTP_TYPE=sftp`\n* `RCLONE_CONFIG_MYSFTP_HOST=pipeline.openbridge.io`\n* `RCLONE_CONFIG_MYSFTP_USER=user`\n* `RCLONE_CONFIG_MYSFTP_PORT=443`\n* `RCLONE_CONFIG_MYSFTP_PASS=34232424234234234`\n\nHere is a sample command that copies data from a remote SFTP server locally with a `60s` timeout:\n`rclone copy sftp:/folder /tmp --contimeout 60s`\n\n# Sample Commands\nHere are a few sample commands you can use use for testing or general usage\n\n## List Remote Directories\nList a remote drive  like this: `rclone lsd {remote name}:`\n**Note:** The command has `:` at the after the remote. If you do not include `:` things wont work\n\nReplace `{remote name}` with your actual remote name. Using our Amazon example it would look like this\n`rclone lsd MYS3:`\n\nUsing Docker this is a possible way to run the command:\n```bash\ndocker run -env-file /env/my.env openbridge/ob_bulkstash rclone lsd MYS3:\n```\n\nThis will output your remote buckets like this:\n\n```bash\n          -1 2017-04-11 16:38:05        -1 athena-lambda\n          -1 2016-12-04 14:32:54        -1 aws-athena-query\n          -1 2016-12-17 14:19:23        -1 aws-logs\n          -1 2015-03-31 21:48:40        -1 cf-templates\n          -1 2016-11-17 15:56:56        -1 chat000\n          -1 2016-10-06 17:09:10        -1 chat001\n          -1 2016-10-24 05:28:09        -1 chat002\n          -1 2014-02-11 14:55:25        -1 prod001\n          -1 2014-02-11 14:55:43        -1 prod002\n          -1 2016-06-05 21:59:28        -1 temp002\n```\n\n## List Remote Files\nList remote files of a certain type:\n`rclone --include \"*.jpg\" ls {remote name}:{aws bucket name}/{folder}`\n\nUsing our AWS remote:\n`rclone --include \"*.jpg\" ls MYS3:mybucket/files`\n\n## Make Remote Location (Bucket)\nMake a new bucket sample:\n`rclone mkdir {remote name}:{aws bucket name}`\n\nUsing our AWS remote:\n`rclone mkdir MYS3:mynewbucket`\n\n## Sync\nSync file sample:\n`rclone sync /home/local/directory {remote name}:{aws bucket name}/{folder}`\n\nUsing our AWS remote:\n`rclone sync /tmp MYS3:mynewbucket/temp [--drive-use-trash]`\n\n## Copy\nThe basic structure of the `rclone` command looks like this for `COPY`:\n```\nrclone copy {{source_config}}:{{bucket}}/{{path/to/file/}} {{dest_config}}:{{bucket}}/{{path/to/file/}}\n```\nYou assign your source and destinations according to your configs. For example, in this case we have `MYS3` and `MYGS` where we assing one as the source and the other as the destination:\n```\nrclone copy MYS3:myawsbucket/path/to/file/ MYGS:mygooglebucket/path/to/files/\n```\n\n\n# Why Use ENV variables?\nThis docker image uses rclone and is focused on separating configuration from the runtime. This does not preclude using a traditional config file. Feel free to go down that path if it makes sense to you. The image would support it.\n\n## Config Syntax\n* Each config statement has `RCLONE_`. It is the prefix for each variable.\n* In each variable you define the name you block by setting `RCLONE_CONFIG_{{NAME}}_`. In our S3 example we use `RCLONE_CONFIG_MYS3` for Amazon and `RCLONE_CONFIG_MYGS` for Google.\n* The last part is the formal configuration attribute. For example, `TYPE`, `ACCESS_KEY_ID` or `SERVER_SIDE_ENCRYPTION` are standard config elements for s3 rclone. Normally be prefixed like this: `--type`\n* You need to make sure `{{NAME}}` is unique to avoid any collisions in your configs. For example, you cant have multiple `RCLONE_CONFIG_MYS3` statements. If you have multiple S3 locations do something like `RCLONE_CONFIG_MYS3-01`, `RCLONE_CONFIG_MYS3-02` and `RCLONE_CONFIG_MYS3-03`\n\n## Using Docker Secrets\nEnvironment variables can be formed to point at the content of Docker secrets\nfiles, so as to avoid giving away sensitive information. Any environment\nvariable which value looks like the following `DOCKER-SECRET::\u003cpath\u003e` (note the\nleading `DOCKER-SECRET` keyword and the double colon `::`) will be replaced by\nthe content of the file at `\u003cpath\u003e` if it exists. Relative paths are\nautomatically resolved to `/run/secrets` (the default path for Docker secrets),\nbut absolute paths can also be used.\n\n# Performance Tips\nThese tips come from  http://moo.nac.uci.edu/~hjm/HOWTO-rclone-to-Gdrive.html\n\nTo obtain good transfer rates, you have to increase the number and size of files you transfer at one time, as well as the number of simultaneous streams and the checkers. So, for rclone to transfer files efficiently, there has to be a large payload per transfer and a number of simultaneous streams. It works best if there are large, identically sized files, but regardless, larger files are better, because of the initiation overhead.\n\n`rclone --transfers=32 --checkers=16 --drive-chunk-size=16384k \\\n--drive-upload-cutoff=16384k  copy /my/folder MYGS:mybucket/myfiles`\n\nCopy files from remote location locally:\n\n`rclone --transfers=12  copy MYGS:mybucket/myfiles /my/local/dir`\n\n# Versioning\n\nDocker Tag | Git Hub Release | rclone | Alpine Version\n---------- | --------------- | -------- | --------------\nlatest*     | develop         | latest   | 3.12.x\n\nAdditional versions are tagged and available for use: [https://hub.docker.com/r/openbridge/ob_bulkstash/tags/](https://hub.docker.com/r/openbridge/ob_bulkstash/tags/)\n\n# Docs\nFor more examples on configuration and rclone commands please refer to the docs:\nhttps://rclone.org/docs/\n\nThis images is using Docker. If you don't know what Docker is read \"[What is Docker?](https://www.docker.com/what-docker)\". Once you have a sense of what Docker is, you can then install the software. It is free: \"[Get Docker](https://www.docker.com/products/docker)\". Select the Docker package that aligns with your environment (ie. OS X, Linux or Windows). If you have not used Docker before, take a look at the guides:\n\n- [Engine: Get Started](https://docs.docker.com/engine/getstarted/)\n- [Docker Mac](https://docs.docker.com/docker-for-mac/)\n- [Docker Windows](https://docs.docker.com/docker-for-windows/)\n\n# Issues\n\nIf you have any problems with or questions about this docker rclone image, please contact us through a GitHub issue.\n\n# Contributing\n\nYou are invited to contribute new features, fixes, or updates, large or small; we are always thrilled to receive pull requests, and do our best to process them as fast as we can.\n\nBefore you start to code, we recommend discussing your plans through a GitHub issue, especially for more ambitious contributions. This gives other contributors a chance to point you in the right direction, give you feedback on your design, and help you find out if someone else is working on the same thing.\n\n# License\n\nThis project is licensed under the MIT License\n\n\n# References\n* https://hub.docker.com/r/tynor88/rclone/\n* https://forum.rclone.org/t/request-official-docker-container/1659\n* https://github.com/kevineye/docker-rclone\n* https://github.com/valentine/docker-rclone-sh\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenbridge%2Fob_bulkstash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenbridge%2Fob_bulkstash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenbridge%2Fob_bulkstash/lists"}