{"id":18929122,"url":"https://github.com/jwillikers/autoupload","last_synced_at":"2025-04-15T14:31:57.075Z","repository":{"id":223006128,"uuid":"758957166","full_name":"jwillikers/autoupload","owner":"jwillikers","description":"A systemd unit to autoupload a directory's contents to S3-compatible object storage with Rclone}or to {mmich with the Immich CLI.","archived":true,"fork":false,"pushed_at":"2024-10-02T11:43:18.000Z","size":142,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-06T21:52:12.631Z","etag":null,"topics":["autoupload","immich","immich-cli","minio","rclone","s3","systemd","systemd-path","tailscale","upload"],"latest_commit_sha":null,"homepage":"https://codeberg.org/jwillikers/autoupload","language":"Nushell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jwillikers.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.adoc","code_of_conduct":"CODE_OF_CONDUCT.adoc","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-17T15:18:01.000Z","updated_at":"2024-10-02T11:47:22.000Z","dependencies_parsed_at":"2024-05-04T15:27:15.842Z","dependency_job_id":"d640e38d-81ef-4040-beb9-788a3002211e","html_url":"https://github.com/jwillikers/autoupload","commit_stats":null,"previous_names":["jwillikers/autoupload"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwillikers%2Fautoupload","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwillikers%2Fautoupload/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwillikers%2Fautoupload/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwillikers%2Fautoupload/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jwillikers","download_url":"https://codeload.github.com/jwillikers/autoupload/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249088891,"owners_count":21210878,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoupload","immich","immich-cli","minio","rclone","s3","systemd","systemd-path","tailscale","upload"],"created_at":"2024-11-08T11:30:12.222Z","updated_at":"2025-04-15T14:31:56.750Z","avatar_url":"https://github.com/jwillikers.png","language":"Nushell","funding_links":[],"categories":[],"sub_categories":[],"readme":"= AutoUpload\nJordan Williams \u003cjordan@jwillikers.com\u003e\n:experimental:\n:icons: font\n:keywords: autoupload file immich minio path rclone s3 systemd tailscale upload watcher\nifdef::env-github[]\n:tip-caption: :bulb:\n:note-caption: :information_source:\n:important-caption: :heavy_exclamation_mark:\n:caution-caption: :fire:\n:warning-caption: :warning:\nendif::[]\n:Immich: https://immich.app/[Immich]\n:Immich-CLI: https://immich.app/docs/features/command-line-interface/[Immich CLI]\n:just: https://github.com/casey/just[just]\n:MinIO: https://min.io/[MinIO]\n:Nushell: http://www.nushell.sh/[Nushell]\n:podman: https://podman.io/[podman]\n:Rclone: https://rclone.org/[Rclone]\n:systemd: https://systemd.io/[systemd]\n:Tailscale: https://tailscale.com/[Tailscale]\n:Tailscale-Online-Target: https://codeberg.org/jwillikers/tailscale-online-target[Tailscale Online Target]\n\nA {systemd} unit to autoupload a directory's contents to S3-compatible object storage with {Rclone} or to {Immich} with the {Immich-CLI}.\nThis is handy for automatically uploading files like photos or sound clips.\n\n== Overview\n\nWhen a file appears in the designated directory, its presence is automatically detected by a {systemd} service.\nWhen an internet connection is next available, systemd uploads the files to either S3-compatible object storage with {Rclone} or to {Immich} with the {Immich-CLI}.\nThe files are then removed from local storage.\n\n== Storage\n\nThe files are uploaded somewhere, which requires configuration.\nFiles can be uploaded to S3-compatible object storage or to {Immich}.\nFor S3-compatible object storage, this documentation specifically refers to {MinIO}.\nBoth my MinIO and Immich setups are documented in my https://codeberg.org/jwillikers/home-lab-helm[Home Lab Helm] repository.\nI use {Tailscale} to access these services.\nUsing MinIO is more secure as an API token can be scoped with access that is limited to uploading files.\nFor Immich, it is possible to use the local API token to delete uploaded files or to do even worse if the API token belongs to an admin user.\n\n=== MinIO\n\nThe instructions here are for setting up S3-compatible object storage using a a self-hosted MinIO instance with a dedicated `pi-camera` bucket for the files.\nThese instructions describe how to create the `pi-camera` bucket and generate an access key with the required write-access necessary to upload the files with Rclone.\nThe instructions assume that {Podman} is installed.\n\n. Create a configuration to access the MinIO server.\n+\n[,sh]\n----\npodman run \\\n  --interactive \\\n  --name minio-client \\\n  --rm \\\n  --tty \\\n  --user $(id -u):$(id -g) \\\n  --userns keep-id \\\n  --volume minio-client-config:/.mc:Z \\\n  quay.io/minio/mc:latest \\\n  alias set jwillikers https://minio.jwillikers.io\nmc: Configuration written to `/.mc/config.json`. Please update your access credentials.\nmc: Successfully created `/.mc/share`.\nmc: Initialized share uploads `/.mc/share/uploads.json` file.\nmc: Initialized share downloads `/.mc/share/downloads.json` file.\nEnter Access Key: abcde123\nEnter Secret Key:\nAdded `jwillikers` successfully.\n----\n\n. Create a `pi-camera` bucket in MinIO to store the files.\n+\n[,sh]\n----\npodman run \\\n  --interactive \\\n  --name minio-client \\\n  --rm \\\n  --tty \\\n  --user $(id -u):$(id -g) \\\n  --userns keep-id \\\n  --volume minio-client-config:/.mc:Z \\\n  quay.io/minio/mc:latest \\\n  mb jwillikers/pi-camera\n----\n\n. Place a quota on the `pi-camera` bucket to prevent uploading too much data.\n+\n[,sh]\n----\npodman run \\\n  --interactive \\\n  --name minio-client \\\n  --rm \\\n  --tty \\\n  --user $(id -u):$(id -g) \\\n  --userns keep-id \\\n  --volume minio-client-config:/.mc:Z \\\n  quay.io/minio/mc:latest \\\n  mc quota set jwillikers/pi-camera --size 200gi\n----\n\n. Generate an access token for the Minio server which uses the `pi-camera-minio-policy.json` policy.\nThis policy allows only the minimal access necessary for Rclone to upload files to the bucket.\n+\n[,sh]\n----\npodman run \\\n  --interactive \\\n  --name minio-client \\\n  --rm \\\n  --tty \\\n  --user $(id -u):$(id -g) \\\n  --userns keep-id \\\n  --volume minio-client-config:/.mc:Z \\\n  --volume ./pi-camera-minio-policy.json:/pi-camera-minio-policy.json:Z \\\n  quay.io/minio/mc:latest \\\n  admin user svcacct add --description \"Pi Camera\" --name \"Pi Camera\" --policy \"pi-camera-minio-policy.json\" jwillikers core\nAccess Key: XXXXXXXXXXXXXXXXXXXX\nSecret Key: ****************************************\nExpiration: no-expiry\n----\n\n=== Immich\n\nUploading to {Immich} is pretty straightforward using the {Immich-CLI} utility.\nFiles will be uploaded to a user account.\nFor security, create a dedicated user account to prevent any potential misuse of the API key.\nUnfortunately, since it isn't possible to limit the scope of an API key, the key can still be abused to delete photos for this user's account.\nWith that in mind, there are important limitations with partner sharing at the moment which, is probably how you'll want to access the pictures from a dedicated account.\nRight now, facial recognition and favorites won't work across partner sharing, although these are requested features.\nIt's also not possible to import shared photos to your account either.\nThe following instructions walkthrough how to create this account using the Immich API.\nThe WebUI can be used to accomplish the same tasks.\nMy Immich server resides at `https://immich.jwillikers.io`.\nReplace it with the URL of your Immich instance.\n\n. Login as an admin user and obtain an access token.\nSubstitute your user's username and password.\n+\n[TIP]\n====\nPrecede the following commands that use the sensitive data such as your password and access token with a space to omit them from your shell's history.\n====\n+\n--\n[,sh]\n----\n curl --location 'https://immich.jwillikers.io/api/auth/login' \\\n  --header 'Content-Type: application/json' \\\n  --header 'Accept: application/json' \\\n  --data-raw '{\n    \"email\": \"jordan@jwillikers.com\",\n    \"password\": \"password\"\n  }'\n----\n\nThe successful response from this command includes the access token as the value for the `accessToken` key.\n--\n\n. Create a dedicated user account for the Pi Camera with a quota.\nThe following command creates the `Pi Camera` user with a quota of 5 GiB.\nSubstitute the API key generated in the previous step for `\u003caccess_token\u003e` and use a good password for the Pi Camera user, not _password_.\n+\n--\n[,sh]\n----\n curl --location 'https://immich.jwillikers.io/api/user' \\\n  --header 'Authorization: Bearer \u003caccess_token\u003e' \\\n  --header 'Content-Type: application/json' \\\n  --header 'Accept: application/json' \\\n  --data '{\n    \"email\": \"pi-camera@jwillikers.com\",\n    \"name\": \"Pi Camera\",\n    \"password\": \"password\",\n    \"quotaSizeInBytes\": 5368709120\n  }'\n----\n\nThe response will return the user's `id` which will be used in the following command.\n--\n\n. Disable the need to change the user's password.\nUse the Pi Camera user's id for `id` in this command.\n+\n[,sh]\n----\n curl --location --request PUT 'https://immich.jwillikers.io/api/user' \\\n  --header 'Authorization: Bearer \u003caccess_token\u003e' \\\n  --header 'Content-Type: application/json' \\\n  --header 'Accept: application/json' \\\n  --data '{\n    \"id\": \"3fa85f64-5717-4562-b3fc-2c963f66afa6\",\n    \"shouldChangePassword\": false\n  }'\n----\n\n. Log out from the admin user's account.\n+\n[,sh]\n----\n curl --location --request POST 'https://immich.jwillikers.io/api/auth/logout' \\\n  --header 'Authorization: Bearer \u003caccess_token\u003e' \\\n  --header 'Accept: application/json'\n----\n\n. Now log in as the Pi Camera user.\n+\n--\n[,sh]\n----\n curl --location 'https://immich.jwillikers.io/api/auth/login' \\\n  --header 'Content-Type: application/json' \\\n  --header 'Accept: application/json' \\\n  --data-raw '{\n    \"email\": \"pi-camera@jwillikers.com\",\n    \"password\": \"password\"\n  }'\n----\n\nThe successful response from this command includes the access token as the value for the `accessToken` key.\n--\n\n. Find the id for any users you want to add as partners to the Pi Camera account.\nObtain all of the users with this command.\n+\n[,sh]\n----\n curl --location 'https://immich.jwillikers.io/api/user' \\\n  --header 'Authorization: Bearer \u003caccess_token\u003e' \\\n  --header 'Accept: application/json'\n----\n\n. For each desired user, run the following command to add them as a partner.\nUse the user's id as the last path in the URL, replacing `:id`.\n+\n[,sh]\n----\n curl --location --request POST 'https://immich.jwillikers.io/api/partner/:id' \\\n  --header 'Accept: application/json' \\\n  --header 'Authorization: Bearer \u003caccess_token\u003e'\n----\n\n. Generate an API key for the Pi Camera user.\nGetting an API key is documented in the https://immich.app/docs/features/command-line-interface/#obtain-the-api-key[Obtain the API Key] section of the Immich CLI page.\n+\n--\n[,sh]\n----\ncurl --location 'https://immich.jwillikers.io/api/api-key' \\\n  --header 'Authorization: Bearer \u003caccess_token\u003e' \\\n  --header 'Content-Type: application/json' \\\n  --header 'Accept: application/json' \\\n  --data '{\n    \"name\": \"Pi Camera Upload\"\n  }'\n----\n\nThe value for the `secret` key in the response is the API token that will be used later on the Pi Camera.\nDon't lose it or you'll have to generate another one.\n--\n\n. Log out from the Pi Camera user's account.\n+\n[,sh]\n----\n curl --location --request POST 'https://immich.jwillikers.io/api/auth/logout' \\\n  --header 'Authorization: Bearer \u003caccess_token\u003e' \\\n  --header 'Accept: application/json'\n----\n\n== Install\n\nThese instructions install the autoupload systemd units and their dependencies.\nThe instructions here are for both Debian and Fedora Atomic.\n\n. Install {just} by following the instructions in the https://github.com/casey/just?tab=readme-ov-file#installation[installation section].\n. Follow the instructions to setup the {Tailscale-Online-Target} for the node where files will be uploaded.\nThese instructions also document how to install Tailscale itself.\nAfter installing the necessary files, the following command enables the Tailscale Online Target for the `meerkat` node where my files will be uploaded.\nThe Tailscale Online Target for this specific instance is added as a dependency for the autoupload systemd service.\n+\nSystem::\n+\n[,sh]\n----\nsudo systemctl enable --now tailscale-dispatcher@meerkat.timer\n----\n\nUser::\n+\n[,sh]\n----\nsystemctl --user enable --now tailscale-dispatcher@meerkat.timer\n----\n\n. Clone this project's repository.\n+\n[,sh]\n----\ngit clone https://codeberg.org/jwillikers/autoupload.git\n----\n\n. Change to the project's root directory.\n+\n[,sh]\n----\ncd autoupload\n----\n\n=== Immich CLI\n\n. Install {Immich-CLI} by running `just install-immich-cli`.\n+\n[,sh]\n----\njust install-immich-cli\n----\n\n. Set up the API key for authenticating with the Immich instance.\nThis can be done with the `just login` command which will run the `immich login-key` command.\nThe last argument should be the API key generated for the Pi Camera user in the \u003c\u003cImmich\u003e\u003e section.\nRun this command under the same user that will be running the systemd unit.\n+\nSystem::\n+\n[,sh]\n----\n printf '%s' '\u003cthe API key\u003e' | sudo podman secret create immich_api_key\n----\n\nUser::\n+\n[,sh]\n----\n printf '%s' '\u003cthe API key\u003e' | podman secret create immich_api_key -\n----\n\n. Enable the `update-immich-container.timer` systemd timer to automatically pull the latest Immich container image.\n+\nSystem::\n+\n[,sh]\n----\nsudo systemctl enable update-immich-container.timer\n----\n\nUser::\n+\n[,sh]\n----\nsystemctl --user enable update-immich-container.timer\n----\n\n. Modify the `autoupload-immich@.service` file as needed.\nAdditionally, the `Requires` and `After` should be updated appropriately for the node where the Immich server resides.\n+\nSystem::\n+\n./etc/systemd/system/autoupload-immich@.service\n[,systemd]\n----\ninclude::systemd/system/autoupload-immich@.service[]\n----\n\nUser::\n+\n./etc/systemd/user/autoupload-immich@.service\n[,systemd]\n----\ninclude::systemd/user/autoupload-immich@.service[]\n----\n\n. Enable and start the systemd unit.\nThis unit uses a {Nushell} script to watch the directory for new files and upload them.\nUse the `systemd-escape --path` command to escape the directory's path.\n+\nSystem:::\n+\n[,sh]\n----\nsudo systemctl enable --now autoupload-immich@$(systemd-escape --path ~/Pictures).service\n----\n\nUser:::\n+\n[,sh]\n----\nsystemctl --user enable --now autoupload-immich@$(systemd-escape --path ~/Pictures).service\n----\n\n. When running under a user, make sure to enable linger for that user account.\n+\n[,sh]\n----\nsudo loginctl enable-linger $USER\n----\n\n=== Rclone\n\n. Install Rclone with `just install-rclone`.\n+\n[,sh]\n----\njust install-rclone\n----\n\n. Create the Rclone configuration directory which is `/etc/rclone/` for the system and `~/.config/rclone` for a user.\n+\nSystem::\n+\n[,sh]\n----\nsudo mkdir --parents /etc/rclone/\n----\n\nUser::\n+\n[,sh]\n----\nmkdir --parents ~/.config/rclone/\n----\n\n. Configure the Rclone credentials in the `rclone.conf` file in the corresponding configuration directory.\n+\n./etc/rclone/rclone.conf\n[,ini]\n----\n[minio]\ntype = s3\nprovider = Minio\naccess_key_id = ********************\nsecret_access_key = ****************************************\nregion = us-east-1\nendpoint = https://minio.jwillikers.io\nacl = private\n----\n\n. Ensure that only the owner can read and write the `rclone.conf` file.\n+\nSystem::\n+\n[,sh]\n----\nsudo chmod 0600 /etc/rclone/rclone.conf\n----\n\nUser::\n+\n[,sh]\n----\nchmod 0600 ~/.config/rclone/rclone.conf\n----\n\n. Modify the `autoupload-rclone@.service` file as needed.\nAdditionally, the `Requires` and `After` should be updated appropriately for the node where the MinIO server resides.\n+\nSystem::\n+\n./etc/systemd/system/autoupload-rclone@.service\n[,systemd]\n----\ninclude::systemd/system/autoupload-rclone@.service[]\n----\n\nUser::\n+\n./etc/systemd/user/autoupload-rclone@.service\n[,systemd]\n----\ninclude::systemd/user/autoupload-rclone@.service[]\n----\n\n. Enable and start the instantiable `autoupload-rclone@.path` systemd unit passing the properly escaped path of the directory to monitor.\nUse the `systemd-escape --path` command to escape the directory's path.\n+\nSystem::\n+\n[,sh]\n----\nsudo systemctl enable --now autoupload-rclone@$(systemd-escape --path ~/pictures).path\n----\n\nUser::\n+\n[,sh]\n----\nsystemctl --user enable --now autoupload-rclone@$(systemd-escape --path ~/Pictures).path\n----\n\n== Code of Conduct\n\nThe project's Code of Conduct is available in the link:CODE_OF_CONDUCT.adoc[Code of Conduct] file.\n\n== License\n\nThis repository is licensed under the https://www.gnu.org/licenses/gpl-3.0.html[GPLv3].\nPlease refer to the bundled link:LICENSE.adoc[license].\n\n© 2024 Jordan Williams\n\n== Authors\n\nmailto:{email}[{author}]\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwillikers%2Fautoupload","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjwillikers%2Fautoupload","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwillikers%2Fautoupload/lists"}