{"id":20263639,"url":"https://github.com/springerpe/cf-rclone-buildpack","last_synced_at":"2025-07-05T05:37:27.863Z","repository":{"id":80987071,"uuid":"217556694","full_name":"SpringerPE/cf-rclone-buildpack","owner":"SpringerPE","description":"A buildpack to provide rclone","archived":false,"fork":false,"pushed_at":"2020-11-24T09:42:54.000Z","size":69,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-03T19:59:39.319Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SpringerPE.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-25T14:54:48.000Z","updated_at":"2022-02-24T00:42:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"dc1a5de2-cff5-4045-b8d6-391e2d5768f4","html_url":"https://github.com/SpringerPE/cf-rclone-buildpack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SpringerPE/cf-rclone-buildpack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fcf-rclone-buildpack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fcf-rclone-buildpack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fcf-rclone-buildpack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fcf-rclone-buildpack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SpringerPE","download_url":"https://codeload.github.com/SpringerPE/cf-rclone-buildpack/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fcf-rclone-buildpack/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261618123,"owners_count":23185094,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T11:35:48.017Z","updated_at":"2025-06-24T06:06:54.764Z","avatar_url":"https://github.com/SpringerPE.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cf-rclone-buildpack\n\nCloudfoundry buildpack to manage buckets S3, GCS ... based on [rclone](https://rclone.org/)\n\nFunctionalities of this buildpack:\n\n* Automatically configure Rclone from AWS and GCP service brokers services\n* Provide a web interface to explore the contents of the buckets\n* Enable serving of remote objects via HTTP\n* Clone data from one bucket to another keeping it sync periodically\n* Use a Rclone server with a HTTP API\n\n\n## Using it\n\nExample `manifest.yml`: \n\n```manifest.yml\n---\napplications:\n- name: rclone\n  memory: 512M\n  instances: 1\n  stack: cflinuxfs3\n  random-route: true\n  buildpacks:\n  - https://github.com/SpringerPE/cf-rclone-buildpack.git\n  services:\n  - jose-rclone-gcs\n  - jose-rclone-aws\n  env:\n    AUTH_USER: \"admin\"\n    AUTH_PASSWORD: \"admin\"\n    CLONE_SOURCE_SERVICE: \"jose-rclone-aws\"\n    CLONE_DESTINATION_SERVICE: \"jose-rclone-gcs\"\n    CLONE_MODE: sync\n    CLONE_TIMER: 600\n```\n\nWith this configuration, the program will run [rclone sync](https://rclone.org/commands/rclone_sync/)\nto synchronize data from the bucket `jose-rclone-aws` to `jose-rclone-gcs`\nevery 10 minutes. As each service offers only one bucket, you do not need to\nknow the bucket name.\n\n\n### Environment variables\n\nThe web service always requires authentication. If **AUTH_USER** is not defined,\nit defaults to `admin` and **AUTH_PASSWORD** will be autogenerated and printed\nin stdout (you can see it with `cf logs`) and stored in\n`/home/vcap/auth/${AUTH_USER}.password`\n\n**GCS_PROJECT_NUMBER** is predefined, but if you have your own project in GCP you will need to redefine it\n\n**CLONE_SOURCE_SERVICE** and **CLONE_DESTINATION_SERVICE** should match the\nname of the services bound to the application and both need to be set in\norder to run the clone operation.\n\n**RCLONE_MODE** is one option of:\n* `copy` (default): copies data from one bucket to another, just adding files to the new bucket. It does not delete files in source neither in destination buckets. See [rclone copy](https://rclone.org/commands/rclone_copy/)\n* `sync`: synchronizes data from source to destination, making both identical, modifying destination only. **Destination is updated to match source, including deleting files if necessary**. See [rclone sync](https://rclone.org/commands/rclone_sync/)\n* `move`: Moves the contents of the source bucket to the destination bucket. **Source contents will deleted as soon as they are copied to destination**, [rclone move](https://rclone.org/commands/rclone_move/)\n\n\u003e Be careful with `CLONE_MODE=sync` or `CLONE_MODE=move`, **those are destructive options**\n\n\n**CLONE_TIMER** specifies amount of seconds to wait to re-run the clone\noperation, by default is `0`, so the clone process will not run periodically,\njust once after the program starts. The process will wait after the previous\nrun has finished, it is not queuing jobs, so if the clone process takes\none hour, the next run will be in 10 minutes (see previous manifest).\n\nExtra rclone parameters can be defined via environment variables.\nSee https://rclone.org/docs/#environment-variables, but be aware that the \nautomatic CLONE process uses the rclone API so most likely those environment\nvariables will be ignored.\n\n\u003e **This buildpack does not allow more than one instance**, deploying more\n\u003e than one, will cause the extra intances will fail.\n\n\n### What if ...\n\n#### ... my service(bucket) is not defined/available in the current platform\n\nJust copy the environment variable `VCAP_SERVICES` from the other CF platform\nand create a file called `VCAP_SERVICES` in the root of the application with\nthe contents of the variable. When start, the buildpack will merge the contents\nof the file with the environment variable and setup the rclone configuration.\n\n\n#### ... my bucket is not provided by CF service brokers, there is no VCAP_SERVICES variable\n\nCreate a rclone configuration file `rclone.conf` with the parameters of the\nbucket, something like:\n\n```\n# S3 example, please fill the access key and key id\n[s3-service]\ntype = s3\nprovider = AWS\naccess_key_id = \u003cS3-KEY-ID\u003e\nsecret_access_key = \u003cS3-ACCESS-KEY\u003e\nregion = eu-central-1\nlocation_constraint = eu-central-1\nacl = private\nenv_auth = false\n\n# GCS Example. Please put the `auth.json` file in the app folder\n[gcs-service]\ntype = google cloud storage\nclient_id =\nclient_secret =\nproject_number =\nservice_account_file = /home/vcap/app/auth.json\nstorage_class = REGIONAL\nlocation = europe-west4\n```\n\nNote that the bucket names are not defined in this configuration, you\nhave to define them in the environment variables **CLONE_SOURCE_BUCKET** or\n**CLONE_DESTINATION_BUCKET** and set the variable **CLONE_SOURCE_SERVICE**\nor **CLONE_DESTINATION_SERVICE** to the name of the entry between brackets\n(`s3-service` or `gcs-service` -no brackets- in this example).\n\n\n#### ... I need something else, other actions or more\n\nJust create a file `post-start.sh`, like this:\n```\n#!/bin/bash\n# $RCLONE is defined env variable, just use it to execute commands\n\n# Example command\n$RCLONE rc core/version\n\n# Sync these 2 buckets\n$RCLONE -vv rc sync/sync srcFs=s3-service:bucket1 dstFs=gcs-service:bucket2\n\n# alternative way to do it (aysnc == true)\n$RCLONE rc sync/sync  --json '{ \"srcFs\": \"s3-service:bucket1\", \"dstFs\": \"gcs-service:bucket2\", \"_async\": true }'\n```\n\nThe variables `CLONE_SOURCE_BUCKET` and `CLONE_DESTINATION_BUCKET` are \nautomatically defined if the counterparts SERVICE variables are provided.\n\nIf a `post-start.sh` file is found, no automatic clone operation will be performed.\nYou can define all kind of logic in this file, sync or async operations, it\ndoes not matter, the file will be executed in background automatically at\nstartup.\n\n\n## Remote objects via HTTP\n\nJust open in a browser `https://rclone.example.app/[SERVICE_NAME:]BUCKET_NAME/`\nchanging `SERVICE_NAME` and `BUCKET_NAME` to the correct values.\n\nOr using curl:\n\n```\n# Note the square brackets are escaped with \\\n# curl -u admin:password 'https://rclone.example.app/\\[SERVICE_NAME:\\]BUCKET_NAME/'\ncurl -u admin:password 'https://rclone.example.app/\\[s3-service:\\]bucket1/'\n```\n\n## Use rclone as server\n\nYou can define a lot of buckets and use rclone API to trigger actions to those\nbuckets (also retrieve them using HTTP). All calls must made using POST.\n\nhttps://rclone.org/rc/#accessing-the-remote-control-via-http\n\n```\ncurl -u admin:password -H \"Content-Type: application/json\" -X POST -d '{\"potato\":2,\"sausage\":1}' http://rclone.example.com/rc/noop\n```\n\nReal world example, perform a sync between 2 buckets:\n\n```\ncurl -u admin:password -H \"Content-Type: application/json\" -X POST -d '{ \"srcFs\": \"s3-service:bucket1\", \"dstFs\": \"gcs-service:bucket2\", \"_async\": true }'  http://rclone.example.com/sync/sync\n```\n\n# Known issues\n\nAll issues found are regarding the new web ui. It is quite new piece of software\n(Aug 2019) and currently is in alpha.\n\n\n* When log in, you have to introduce twice the auth settings. The second time,\nin the program interface, click first on *Verify* and then *Login*.\n\n* It allows you to visualize the contents of the buckets, see current \noperations and view/delete objects. In order to see the contents of a bucket,\ngo to *Explorer* and type `\u003cname-of-service\u003e:\u003cname-of-bucket\u003e` and click *Open*\n(yep, you need to now the name of the bucket!).\n\n* Graph does not get refreshed after the transation is done.\n\n\n# Development\n\nBuildpack implemented using bash scripts to make it easy to understand and change.\n\nhttps://docs.cloudfoundry.org/buildpacks/understand-buildpacks.html\n\nThe builpack uses the `deps` and `cache` folders according the implementation purposes,\nso, the first time the buildpack is used it will download all resources, next times \nit will use the cached resources.\n\n\n# Author\n\n(c) 2019 Jose Riguera Lopez  \u003cjose.riguera@springernature.com\u003e\nSpringernature Engineering Enablement\n\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspringerpe%2Fcf-rclone-buildpack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspringerpe%2Fcf-rclone-buildpack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspringerpe%2Fcf-rclone-buildpack/lists"}