Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/jwillikers/autoupload

A systemd unit to autoupload a directory's contents to S3-compatible object storage with Rclone}or to {mmich with the Immich CLI.
https://github.com/jwillikers/autoupload

autoupload immich immich-cli minio rclone s3 systemd systemd-path tailscale upload

Last synced: about 1 month ago
JSON representation

A systemd unit to autoupload a directory's contents to S3-compatible object storage with Rclone}or to {mmich with the Immich CLI.

Host: GitHub
URL: https://github.com/jwillikers/autoupload
Owner: jwillikers
License: other
Created: 2024-02-17T15:18:01.000Z (10 months ago)
Default Branch: main
Last Pushed: 2024-06-10T15:37:20.000Z (6 months ago)
Last Synced: 2024-06-11T14:37:00.483Z (6 months ago)
Topics: autoupload, immich, immich-cli, minio, rclone, s3, systemd, systemd-path, tailscale, upload
Language: Nushell
Homepage:
Size: 138 KB
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.adoc
- License: LICENSE.adoc
- Code of conduct: CODE_OF_CONDUCT.adoc

Awesome Lists containing this project

README

= AutoUpload
Jordan Williams
:experimental:
:icons: font
:keywords: autoupload file immich minio path rclone s3 systemd tailscale upload watcher
ifdef::env-github[]
:tip-caption: :bulb:
:note-caption: :information_source:
:important-caption: :heavy_exclamation_mark:
:caution-caption: :fire:
:warning-caption: :warning:
endif::[]
:Immich: https://immich.app/[Immich]
:Immich-CLI: https://immich.app/docs/features/command-line-interface/[Immich CLI]
:just: https://github.com/casey/just[just]
:MinIO: https://min.io/[MinIO]
:Nushell: http://www.nushell.sh/[Nushell]
:podman: https://podman.io/[podman]
:Rclone: https://rclone.org/[Rclone]
:systemd: https://systemd.io/[systemd]
:Tailscale: https://tailscale.com/[Tailscale]
:Tailscale-Online-Target: https://codeberg.org/jwillikers/tailscale-online-target[Tailscale Online Target]

A {systemd} unit to autoupload a directory's contents to S3-compatible object storage with {Rclone} or to {Immich} with the {Immich-CLI}.
This is handy for automatically uploading files like photos or sound clips.

== Overview

When a file appears in the designated directory, its presence is automatically detected by a {systemd} service.
When an internet connection is next available, systemd uploads the files to either S3-compatible object storage with {Rclone} or to {Immich} with the {Immich-CLI}.
The files are then removed from local storage.

== Storage

The files are uploaded somewhere, which requires configuration.
Files can be uploaded to S3-compatible object storage or to {Immich}.
For S3-compatible object storage, this documentation specifically refers to {MinIO}.
Both my MinIO and Immich setups are documented in my https://codeberg.org/jwillikers/home-lab-helm[Home Lab Helm] repository.
I use {Tailscale} to access these services.
Using MinIO is more secure as an API token can be scoped with access that is limited to uploading files.
For Immich, it is possible to use the local API token to delete uploaded files or to do even worse if the API token belongs to an admin user.

=== MinIO

The instructions here are for setting up S3-compatible object storage using a a self-hosted MinIO instance with a dedicated `pi-camera` bucket for the files.
These instructions describe how to create the `pi-camera` bucket and generate an access key with the required write-access necessary to upload the files with Rclone.
The instructions assume that {Podman} is installed.

. Create a configuration to access the MinIO server.
+
[,sh]
----
podman run \
--interactive \
--name minio-client \
--rm \
--tty \
--user $(id -u):$(id -g) \
--userns keep-id \
--volume minio-client-config:/.mc:Z \
quay.io/minio/mc:latest \
alias set jwillikers https://minio.jwillikers.io
mc: Configuration written to `/.mc/config.json`. Please update your access credentials.
mc: Successfully created `/.mc/share`.
mc: Initialized share uploads `/.mc/share/uploads.json` file.
mc: Initialized share downloads `/.mc/share/downloads.json` file.
Enter Access Key: abcde123
Enter Secret Key:
Added `jwillikers` successfully.
----

. Create a `pi-camera` bucket in MinIO to store the files.
+
[,sh]
----
podman run \
--interactive \
--name minio-client \
--rm \
--tty \
--user $(id -u):$(id -g) \
--userns keep-id \
--volume minio-client-config:/.mc:Z \
quay.io/minio/mc:latest \
mb jwillikers/pi-camera
----

. Place a quota on the `pi-camera` bucket to prevent uploading too much data.
+
[,sh]
----
podman run \
--interactive \
--name minio-client \
--rm \
--tty \
--user $(id -u):$(id -g) \
--userns keep-id \
--volume minio-client-config:/.mc:Z \
quay.io/minio/mc:latest \
mc quota set jwillikers/pi-camera --size 200gi
----

. Generate an access token for the Minio server which uses the `pi-camera-minio-policy.json` policy.
This policy allows only the minimal access necessary for Rclone to upload files to the bucket.
+
[,sh]
----
podman run \
--interactive \
--name minio-client \
--rm \
--tty \
--user $(id -u):$(id -g) \
--userns keep-id \
--volume minio-client-config:/.mc:Z \
--volume ./pi-camera-minio-policy.json:/pi-camera-minio-policy.json:Z \
quay.io/minio/mc:latest \
admin user svcacct add --description "Pi Camera" --name "Pi Camera" --policy "pi-camera-minio-policy.json" jwillikers core
Access Key: XXXXXXXXXXXXXXXXXXXX
Secret Key: ****************************************
Expiration: no-expiry
----

=== Immich

Uploading to {Immich} is pretty straightforward using the {Immich-CLI} utility.
Files will be uploaded to a user account.
For security, create a dedicated user account to prevent any potential misuse of the API key.
Unfortunately, since it isn't possible to limit the scope of an API key, the key can still be abused to delete photos for this user's account.
With that in mind, there are important limitations with partner sharing at the moment which, is probably how you'll want to access the pictures from a dedicated account.
Right now, facial recognition and favorites won't work across partner sharing, although these are requested features.
It's also not possible to import shared photos to your account either.
The following instructions walkthrough how to create this account using the Immich API.
The WebUI can be used to accomplish the same tasks.
My Immich server resides at `https://immich.jwillikers.io`.
Replace it with the URL of your Immich instance.

. Login as an admin user and obtain an access token.
Substitute your user's username and password.
+
[TIP]
====
Precede the following commands that use the sensitive data such as your password and access token with a space to omit them from your shell's history.
====
+
--
[,sh]
----
curl --location 'https://immich.jwillikers.io/api/auth/login' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data-raw '{
"email": "[email protected]",
"password": "password"
}'
----

The successful response from this command includes the access token as the value for the `accessToken` key.
--

. Create a dedicated user account for the Pi Camera with a quota.
The following command creates the `Pi Camera` user with a quota of 5 GiB.
Substitute the API key generated in the previous step for `` and use a good password for the Pi Camera user, not _password_.
+
--
[,sh]
----
curl --location 'https://immich.jwillikers.io/api/user' \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"email": "[email protected]",
"name": "Pi Camera",
"password": "password",
"quotaSizeInBytes": 5368709120
}'
----

The response will return the user's `id` which will be used in the following command.
--

. Disable the need to change the user's password.
Use the Pi Camera user's id for `id` in this command.
+
[,sh]
----
curl --location --request PUT 'https://immich.jwillikers.io/api/user' \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"shouldChangePassword": false
}'
----

. Log out from the admin user's account.
+
[,sh]
----
curl --location --request POST 'https://immich.jwillikers.io/api/auth/logout' \
--header 'Authorization: Bearer ' \
--header 'Accept: application/json'
----

. Now log in as the Pi Camera user.
+
--
[,sh]
----
curl --location 'https://immich.jwillikers.io/api/auth/login' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data-raw '{
"email": "[email protected]",
"password": "password"
}'
----

The successful response from this command includes the access token as the value for the `accessToken` key.
--

. Find the id for any users you want to add as partners to the Pi Camera account.
Obtain all of the users with this command.
+
[,sh]
----
curl --location 'https://immich.jwillikers.io/api/user' \
--header 'Authorization: Bearer ' \
--header 'Accept: application/json'
----

. For each desired user, run the following command to add them as a partner.
Use the user's id as the last path in the URL, replacing `:id`.
+
[,sh]
----
curl --location --request POST 'https://immich.jwillikers.io/api/partner/:id' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer '
----

. Generate an API key for the Pi Camera user.
Getting an API key is documented in the https://immich.app/docs/features/command-line-interface/#obtain-the-api-key[Obtain the API Key] section of the Immich CLI page.
+
--
[,sh]
----
curl --location 'https://immich.jwillikers.io/api/api-key' \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"name": "Pi Camera Upload"
}'
----

The value for the `secret` key in the response is the API token that will be used later on the Pi Camera.
Don't lose it or you'll have to generate another one.
--

. Log out from the Pi Camera user's account.
+
[,sh]
----
curl --location --request POST 'https://immich.jwillikers.io/api/auth/logout' \
--header 'Authorization: Bearer ' \
--header 'Accept: application/json'
----

== Install

These instructions install the autoupload systemd units and their dependencies.
The instructions here are for both Debian and Fedora Atomic.

. Install {just} by following the instructions in the https://github.com/casey/just?tab=readme-ov-file#installation[installation section].
. Follow the instructions to setup the {Tailscale-Online-Target} for the node where files will be uploaded.
These instructions also document how to install Tailscale itself.
After installing the necessary files, the following command enables the Tailscale Online Target for the `meerkat` node where my files will be uploaded.
The Tailscale Online Target for this specific instance is added as a dependency for the autoupload systemd service.
+
System::
+
[,sh]
----
sudo systemctl enable --now [email protected]
----

User::
+
[,sh]
----
systemctl --user enable --now [email protected]
----

. Clone this project's repository.
+
[,sh]
----
git clone https://codeberg.org/jwillikers/autoupload.git
----

. Change to the project's root directory.
+
[,sh]
----
cd autoupload
----

=== Immich CLI

. Install {Immich-CLI} by running `just install-immich-cli`.
+
[,sh]
----
just install-immich-cli
----

. Set up the API key for authenticating with the Immich instance.
This can be done with the `just login` command which will run the `immich login-key` command.
The last argument should be the API key generated for the Pi Camera user in the <> section.
Run this command under the same user that will be running the systemd unit.
+
System::
+
[,sh]
----
printf '%s' '' | sudo podman secret create immich_api_key
----

User::
+
[,sh]
----
printf '%s' '' | podman secret create immich_api_key -
----

. Enable the `update-immich-container.timer` systemd timer to automatically pull the latest Immich container image.
+
System::
+
[,sh]
----
sudo systemctl enable update-immich-container.timer
----

User::
+
[,sh]
----
systemctl --user enable update-immich-container.timer
----

. Modify the `[email protected]` file as needed.
Additionally, the `Requires` and `After` should be updated appropriately for the node where the Immich server resides.
+
System::
+
./etc/systemd/system/[email protected]
[,systemd]
----
include::systemd/system/[email protected][]
----

User::
+
./etc/systemd/user/[email protected]
[,systemd]
----
include::systemd/user/[email protected][]
----

. Enable and start the systemd unit.
This unit uses a {Nushell} script to watch the directory for new files and upload them.
Use the `systemd-escape --path` command to escape the directory's path.
+
System:::
+
[,sh]
----
sudo systemctl enable --now autoupload-immich@$(systemd-escape --path ~/Pictures).service
----

User:::
+
[,sh]
----
systemctl --user enable --now autoupload-immich@$(systemd-escape --path ~/Pictures).service
----

. When running under a user, make sure to enable linger for that user account.
+
[,sh]
----
sudo loginctl enable-linger $USER
----

=== Rclone

. Install Rclone with `just install-rclone`.
+
[,sh]
----
just install-rclone
----

. Create the Rclone configuration directory which is `/etc/rclone/` for the system and `~/.config/rclone` for a user.
+
System::
+
[,sh]
----
sudo mkdir --parents /etc/rclone/
----

User::
+
[,sh]
----
mkdir --parents ~/.config/rclone/
----

. Configure the Rclone credentials in the `rclone.conf` file in the corresponding configuration directory.
+
./etc/rclone/rclone.conf
[,ini]
----
[minio]
type = s3
provider = Minio
access_key_id = ********************
secret_access_key = ****************************************
region = us-east-1
endpoint = https://minio.jwillikers.io
acl = private
----

. Ensure that only the owner can read and write the `rclone.conf` file.
+
System::
+
[,sh]
----
sudo chmod 0600 /etc/rclone/rclone.conf
----

User::
+
[,sh]
----
chmod 0600 ~/.config/rclone/rclone.conf
----

. Modify the `[email protected]` file as needed.
Additionally, the `Requires` and `After` should be updated appropriately for the node where the MinIO server resides.
+
System::
+
./etc/systemd/system/[email protected]
[,systemd]
----
include::systemd/system/[email protected][]
----

User::
+
./etc/systemd/user/[email protected]
[,systemd]
----
include::systemd/user/[email protected][]
----

. Enable and start the instantiable `[email protected]` systemd unit passing the properly escaped path of the directory to monitor.
Use the `systemd-escape --path` command to escape the directory's path.
+
System::
+
[,sh]
----
sudo systemctl enable --now autoupload-rclone@$(systemd-escape --path ~/pictures).path
----

User::
+
[,sh]
----
systemctl --user enable --now autoupload-rclone@$(systemd-escape --path ~/Pictures).path
----

== Code of Conduct

The project's Code of Conduct is available in the link:CODE_OF_CONDUCT.adoc[Code of Conduct] file.

== License

This repository is licensed under the https://www.gnu.org/licenses/gpl-3.0.html[GPLv3].
Please refer to the bundled link:LICENSE.adoc[license].

== Authors

mailto:{email}[{author}]