{"id":13542681,"url":"https://github.com/madslundt/docker-cloud-media-scripts","last_synced_at":"2025-08-19T21:32:39.793Z","repository":{"id":146746304,"uuid":"99616207","full_name":"madslundt/docker-cloud-media-scripts","owner":"madslundt","description":"Upload and stream media from the cloud with or without encryption. Cache all new and recently streamed media locally to access quickly and reduce API calls","archived":false,"fork":false,"pushed_at":"2019-07-28T22:38:29.000Z","size":213,"stargazers_count":100,"open_issues_count":12,"forks_count":34,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-11-03T08:33:55.763Z","etag":null,"topics":["cloud-data","cloud-storage","google-drive","mount","plex","plex-media-server","plexdrive","rclone","union"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/madslundt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-08-07T20:14:08.000Z","updated_at":"2024-06-17T17:56:37.000Z","dependencies_parsed_at":"2023-04-13T12:38:20.688Z","dependency_job_id":null,"html_url":"https://github.com/madslundt/docker-cloud-media-scripts","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madslundt%2Fdocker-cloud-media-scripts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madslundt%2Fdocker-cloud-media-scripts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madslundt%2Fdocker-cloud-media-scripts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madslundt%2Fdocker-cloud-media-scripts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/madslundt","download_url":"https://codeload.github.com/madslundt/docker-cloud-media-scripts/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230371842,"owners_count":18215801,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-data","cloud-storage","google-drive","mount","plex","plex-media-server","plexdrive","rclone","union"],"created_at":"2024-08-01T10:01:15.251Z","updated_at":"2024-12-19T03:29:13.927Z","avatar_url":"https://github.com/madslundt.png","language":"Shell","funding_links":["https://www.paypal.me/madslundt"],"categories":["others","Shell"],"sub_categories":[],"readme":"# Usage\r\n\r\nDefault settings use ~100GB for local media, remove atleast 80 GB and Plexdrive chunks and cache are removed after 24 hours:\r\n```\r\ndocker create \\\r\n\t--name cloud-media-scripts \\\r\n\t-v /media:/local-media:shared \\\r\n\t-v /mnt/external/media:/local-decrypt:shared \\\r\n\t-v /configurations:/config \\\r\n\t-v /mnt/external/plexdrive:/chunks \\\r\n\t-v /logs:/log \\\r\n\t--privileged --cap-add=MKNOD --cap-add=SYS_ADMIN --device=/dev/fuse \\\r\n\tmadslundt/cloud-media-scripts\r\n```\r\n\r\nIf you have more space you can increase `REMOVE_LOCAL_FILES_WHEN_SPACE_EXCEEDS_GB`, `FREEUP_ATLEAST_GB` and either increase `CLEAR_CHUNK_AGE` or add `CLEAR_CHUNK_MAX_SIZE`.\r\n\r\nExample of having `REMOVE_LOCAL_FILES_WHEN_SPACE_EXCEEDS_GB` set to 2TB, `FREEUP_ATLEAST_GB` to 1TB and `CLEAR_CHUNK_MAX_SIZE` to 1TB:\r\n```\r\ndocker create \\\r\n\t--name cloud-media-scripts \\\r\n\t-v /media:/local-media:shared \\\r\n\t-v /mnt/external/media:/local-decrypt:shared \\\r\n\t-v /configurations:/config \\\r\n\t-v /mnt/external/plexdrive:/chunks \\\r\n\t-v /logs:/log \\\r\n\t-e CLEAR_CHUNK_MAX_SIZE=\"1000G\" \\\r\n\t-e REMOVE_LOCAL_FILES_WHEN_SPACE_EXCEEDS_GB=\"2000\" \\\r\n\t-e FREEUP_ATLEAST_GB=\"1000\" \\\r\n\t--privileged --cap-add=MKNOD --cap-add=SYS_ADMIN --device=/dev/fuse \\\r\n\tmadslundt/cloud-media-scripts\r\n```\r\n\r\n\r\n# Parameters\r\nThe parameters are split into two halves, separated by a colon, the left hand side representing the host and the right the container side.\r\nFor example with a volume `-v external:internal` - what this shows is the volume mapping from internal to external of the container.\r\nExample `-v /media:/local-media` would expose directory **/local-media** from inside the container to be accessible from the host's directory **/media**.\r\n\r\nOBS: Some of the volumes need to have **:shared** appended to it for it to work. This is needed to have the files visible for the host.\r\nExample `-v /media:/local-media:shared`.\r\n\r\n**:shared** is also needed on if you mount these folders to your other Docker containers.\r\n\r\nBefore creating the docker container (with the :shared appends), run the command `sudo mount --make-shared /volume1` (remember to change /volume1 to match your setup). *Thanks to freakshock88 for pointing this out*\r\n\r\nVolumes:\r\n* `-v /local-media` - Union of all files stored on cloud and local - Append **:shared**\r\n* `-v /local-decrypt` - Local files stored on disk - Append **:shared**\r\n* `-v /config` - Rclone and plexdrive configurations\r\n* `-v /chunks` - Plexdrive cache chunks\r\n* `-v /data/db` - MongoDB database\r\n* `-v /log` - Log files from mount, cloudupload and rmlocal\r\n* `-v /cloud-encrypt` - Cloud files encrypted synced with Plexdrive. This is empty if `ENCRYPT_MEDIA` is 0. - Append **:shared**\r\n* `-v /cloud-decrypt` - Cloud files decrypted with Rclone - Append **:shared**\r\n\r\nEnvironment variables:\r\n* `-e ENCRYPT_MEDIA` - If media is or should be encrypted. 0 means no encryption and 1 means encryption (default **1**)\r\n* `-e BUFFER_SIZE` - Rclone: Buffer size when copying files (default **500M**)\r\n* `-e MAX_READ_AHEAD` - Rclone: The number of bytes that can be prefetched for sequential reads (default **30G**)\r\n* `-e CHECKERS` - Rclone: Number of checkers to run in parallel (default **16**)\r\n* `-e RCLONE_CLOUD_ENDPOINT` - Rclone: Cloud endpoint (default **gd-crypt:**)\r\n* `-e RCLONE_LOCAL_ENDPOINT` - Rclone: Local endpoint (default **local-crypt:**) - this is ignored when `ENCRYPT_MEDIA` is 0.\r\n* `-e CHUNK_SIZE` - Plexdrive: The size of each chunk that is downloaded (default **10M**)\r\n* `-e CLEAR_CHUNK_MAX_SIZE` - Plexdrive: The maximum size of the temporary chunk directory (empty as default)\r\n* `-e CLEAR_CHUNK_AGE` - Plexdrive: The maximum age of a cached chunk file (default **24h**) - this is ignored if `CLEAR_CHUNK_MAX_SIZE` is set\r\n* `-e MONGO_DATABASE` - Mongo database used for Plexdrive (default **plexdrive**)\r\n* `-e DATE_FORMAT` - Date format for loggin (default **+%F@%T**)\r\n* `-e REMOVE_LOCAL_FILES_BASED_ON` - Remove local files based on `space`, `time` or `instant` (default **space**)\r\n* `-e REMOVE_LOCAL_FILES_WHEN_SPACE_EXCEEDS_GB` - Remove local files when local storage exceeds this value in GB (default **100**) - this is ignored if `REMOVE_LOCAL_FILES_BASED_ON` is set to time or instant\r\n* `-e FREEUP_ATLEAST_GB` - Remove atleast this value in GB on removal (default **80**) - this is ignored if `REMOVE_LOCAL_FILES_BASED_ON` is set to time or instant\r\n* `-e REMOVE_LOCAL_FILES_AFTER_DAYS` Remove local files older than this value in days (default **10**) - this is ignored if `REMOVE_LOCAL_FILES_BASED_ON` is set to space or instant\r\n* `-e READ_ONLY` If Rclone and Plexdrive should be read only or not. 0 means writeable and 1 means read only (default **1**)\r\n* `-e PLEX_URL` If you want to use empty trash script you have to provide the url for your Plex Media Server (default empty).\r\n* `-e PLEX_TOKEN` If you want to use empty trash script you have to provide the plex token for your Plex Media Server (default empty).\r\n* `-e PGID` Group id\r\n* `-e PUID` User id\r\n* `-e CLOUDUPLOADTIME` - When to run cloudupload using cron expression (default 0 1 * * * ) set to \"0 0 31 2 0\" to disable\r\n* `-e RMDELETETIME` -- When to run cloudupload using cron expression (default 0 6 * * *) set to \"0 0 31 2 0\" to disable\r\n\r\n`--privileged --cap-add=MKNOD --cap-add=SYS_ADMIN --device=/dev/fuse` must be there for fuse to work within the container.\r\n\r\nIf using docker-compose:\r\n```\r\nprivileged: true\r\n   devices:\r\n     - /dev/fuse\r\n   cap_add:\r\n     - MKNOD\r\n     - SYS_ADMIN\r\n```\r\n\r\n# Setup\r\nAfter the docker image has been setup and running, Rclone and Plexdrive need to be configured.\r\n\r\n## Rclone\r\nSetup Rclone run `docker exec -ti \u003cDOCKER_CONTAINER\u003e rclone_setup`\r\n\r\n### With encryption\r\n3 remotes are needed when using encryption:\r\n1. First one is for the Google drive connection\r\n2. Second one is for the Google drive on-the-fly encryption/decryption\r\n3. Third and last one is for the local encryption/decryption\r\n\r\n - Endpoint to your cloud storage.\r\n\t- Create new remote [**Press N**]\r\n\t- Give it a name example gd\r\n\t- Choose Google Drive [**Press 8**]\r\n\t- If you have a client id paste it here or leave it blank\r\n\t- Choose headless machine [**Press N**]\r\n\t- Open the url in your browser and enter the verification code\r\n - Encryption and decryption for your cloud storage.\r\n\t- Create new remote [**Press N**]\r\n\t- Give it the same name as specified in the environment variable `RCLONE_CLOUD_ENDPOINT` but without colon (:) (*default gd-crypt*)\r\n\t- Choose Encrypt/Decrypt a remote [**Press 5**]\r\n\t- Enter the name of the endpoint created in cloud-storage appended with a colon (:) and the subfolder on your cloud. Example `gd:/Media` or just `gd:` if you have your files in root in the cloud.\r\n\t- Choose how to encrypt filenames. I prefer option 2 Encrypt the filenames\r\n\t- Choose to either generate your own or random password. I prefer to enter my own.\r\n\t- Choose to enter pass phrase for the salt or leave it blank. I prefer to enter my own.\r\n - Encryption and decryption for your local storage.\r\n\t- Create new remote [**Press N**]\r\n\t- Give it the same name as specified in the environment variable `RCLONE_LOCAL_ENDPOINT` but without colon (:) (*default local-crypt*)\r\n\t- Choose Encrypt/Decrypt a remote [**Press 5**]\r\n\t- Enter the encrypted folder: **/cloud-encrypt**. If you are using subdirectory append it to it. Example /cloud-encrypt/Media\r\n\t- Choose the same filename encrypted as you did with the cloud storage.\r\n\t- Enter the same password as you did with the cloud storage.\r\n\t- Enter the same pass phrase as you did with the cloud storage.\r\n\r\n\r\n### Without encryption\r\n1 remote is needed to connect rclone to Google drive:\r\n - Endpoint to your cloud storage.\r\n\t- Create new remote [**Press N**]\r\n\t- Give it the same name as specified in the environment variable `RCLONE_CLOUD_ENDPOINT` but without the colon (:)\r\n\t- Choose Google Drive [**Press 7**]\r\n\t- If you have a client id paste it here or leave it blank\r\n\t- Choose headless machine [**Press N**]\r\n\t- Open the url in your browser and enter the verification code\r\n\r\nRclone documentation if needed [click here](https://rclone.org/docs/)\r\n\r\n## Plexdrive\r\nSetup Plexdrive to the cloud. Run the command `docker exec -ti \u003cDOCKER_CONTAINER\u003e plexdrive_setup`\r\n\r\nPlexdrive documentation if needed [click here](https://github.com/dweidenfeld/plexdrive/tree/4.0.0)\r\n\r\n# Commands\r\nUpload local files to cloud run: `docker exec \u003cDOCKER_CONTAINER\u003e cloudupload`\r\n\r\nRemove local files run `docker exec \u003cDOCKER_CONTAINER\u003e rmlocal`\r\n\r\nCheck if everything is running `docker exec \u003cDOCKER_CONTAINER\u003e check`\r\n\r\nEmpty trash on Plex Media Server but only if mount is up `docker exec \u003cDOCKER_CONTAINER\u003e emptytrash`\r\n\r\n`cloudupload` and `rmlocal` can be ran with arguments. All arguments are passed to rclone.\r\nFor example it is possible to run `docker exec \u003cDOCKER_CONTAINER\u003e cloudupload -v` to get verbose on the rclone operations in cloudupload.\r\n\r\n# Cron jobs\r\nSetup cron jobs to upload and remove local files:\r\n - `@daily docker exec \u003cDOCKER_CONTAINER\u003e cloudupload`\r\n - `@weekly docker exec \u003cDOCKER_CONTAINER\u003e rmlocal`\r\n\r\n\r\n# How this works?\r\nFollowing services are used to sync, encrypt/decrypt and mount media:\r\n - Plexdrive\r\n - Rclone\r\n - UnionFS\r\n\r\nWhen using encryption this gives us a total of 5 directories:\r\n - /cloud-encrypt: Cloud data encrypted (Mounted with Plexdrive)\r\n - /cloud-decrypt: Cloud data decrypted (Mounted with Rclone)\r\n - /local-decrypt: Local data decrypted that is yet to be uploaded to the cloud\r\n - /chunks: Plexdrive temporary files and caching\r\n - /local-media: Union of decrypted cloud data and local data (Mounted with Union-FS)\r\n\r\nWhen NOT using encryption this gives us a total of 4 directories:\r\n - /cloud-decrypt: Cloud data decrypted (Mounted with Plexdrive)\r\n - /local-decrypt: Local data decrypted that is yet to be uploaded to the cloud\r\n - /chunks: Plexdrive temporary files and caching\r\n - /local-media: Union of decrypted cloud data and local data (Mounted with Union-FS)\r\n\r\n\r\nAll Cloud data is mounted to `/cloud-encrypt`. This folder is then decrypted and mounted to `/cloud-decrypt`. If `ENCRYPT_MEDIA` is turned off cloud data is mounted directly to `/cloud-decrypt`.\r\nd\r\nA local folder (`/local-decrypt`) containing local media that is yet to be uploaded to the cloud.\r\n`/local-decrypt` and `/cloud-decrypt` is then mounted to a third folder (`/local-media`) with certain permissions - `/local-decrypt` with Read/Write permissions and `/cloud-decrypt` with Read-only permission.\r\n\r\nEverytime new media is retrieved it should be added to `/local-media`. By adding files to `/local-media` it is added to `/local-decrypt` because of the Read/Write permissions. That is why a cronjob is needed to upload local files from `/local-decrypt`.\r\n\r\nBy having a cronjob to rmlocal it will sooner or later move media from `/local-decrypt` depending on the `REMOVE_LOCAL_FILES_BASED_ON` setting. Media is only removed from `/local-decrypt` and still appears in `/local-media` because it is still be accessable from the cloud.\r\n\r\nIf `REMOVE_LOCAL_FILES_BASED_ON` is set to **space** it will only remove content (if local media size has exceeded `REMOVE_LOCAL_FILES_WHEN_SPACE_EXCEEDS_GB`) starting from the oldest accessed file and will only free up atleast `FREEUP_ATLEAST_GB`. If **time** is set it will only remove files older than `REMOVE_LOCAL_FILES_AFTER_DAYS`. If **instant** is set it will remove all files when running.\r\n\r\n*Media is never deleted locally before being uploaded successful to the cloud.*\r\n\r\n![UML diagram](uml.png)\r\n\r\n## Rclone\r\nRclone 1.39 is currently used and tested.\r\n\r\nRclone is used to encrypt, decrypt and upload files to the cloud. It mounts and decrypts Plexdrive to a different folder (`/cloud-decrypt`) and later encrypts and uploads from a local folder (`/local-decrypt`) to the cloud.\r\n\r\nRclone creates one config file in `/config`: `config.json`. This is used to stored Google Drive api keys and encryption/decryption keys.\r\n\r\n## Plexdrive\r\nPlexdrive 4.0.0 is currently used and tested.\r\n\r\nPlexdrive is used to mount Google Drive to a local folder (`/cloud-encrypt`).\r\n\r\nPlexdrive create two files in `/config`: `config.json` and `token.json`. These are used to store Google Drive api keys.\r\n\r\n## UnionFS\r\nUnionFS is used to mount both cloud and local media to a local folder (`/local-media`).\r\n\r\n - Cloud storage `/cloud-decrypt` is mounted with Read-only permissions.\r\n - Local storage `/local-decrypt` is mounted with Read/Write permissions.\r\n\r\nThe reason for these permissions are that when writing to the local folder (`/local-media`) it will not try to write it directly to the cloud storage `/cloud-decrypt`, but instead to the local storage (`/local-decrypt`). Later this will be encrypted and uploaded to the cloud by Rclone.\r\n\r\n\r\n# Build Dockerfile\r\n## Build\r\n`docker build -t cloud-media-scripts .`\r\n\r\n## Test run\r\n`docker run --name cloud-media-scripts -d cloud-media-scripts`\r\n\r\n\r\nIf you want to support the project or just buy me a beer I accept Paypal and bitcoins.\r\n\r\n[![paypal](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.me/madslundt)\r\n\r\nBitCoin address: 18fXu7Ty9RB4prZCpD8CDD1AyhHaRS1ef3\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmadslundt%2Fdocker-cloud-media-scripts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmadslundt%2Fdocker-cloud-media-scripts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmadslundt%2Fdocker-cloud-media-scripts/lists"}