{"id":25814717,"url":"https://github.com/morriz/itsup","last_synced_at":"2025-02-28T03:34:52.770Z","repository":{"id":221257741,"uuid":"753657316","full_name":"Morriz/itsUP","owner":"Morriz","description":"Lean, automated, poor man's infra for lightweight services running in docker.","archived":false,"fork":false,"pushed_at":"2025-01-29T13:33:26.000Z","size":1283,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-29T14:31:15.234Z","etag":null,"topics":["automation","docker","docker-compose","ias","infra","infrastructure-as-code","openapi","zero-downtime"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Morriz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-06T14:54:17.000Z","updated_at":"2025-01-29T13:33:30.000Z","dependencies_parsed_at":"2024-02-21T10:45:06.419Z","dependency_job_id":"63194e4b-ae97-4061-89ac-7ec72b0e9c03","html_url":"https://github.com/Morriz/itsUP","commit_stats":null,"previous_names":["morriz/drup","morriz/doup","morriz/uptid","morriz/itsup"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Morriz%2FitsUP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Morriz%2FitsUP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Morriz%2FitsUP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Morriz%2FitsUP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Morriz","download_url":"https://codeload.github.com/Morriz/itsUP/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241095449,"owners_count":19908814,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","docker","docker-compose","ias","infra","infrastructure-as-code","openapi","zero-downtime"],"created_at":"2025-02-28T03:34:52.139Z","updated_at":"2025-02-28T03:34:52.759Z","avatar_url":"https://github.com/Morriz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# itsUP \u003c!-- omit in toc --\u003e\n\n_Lean, secure, automated, zero downtime\u003csup\u003e\\*\u003c/sup\u003e, poor man's infra for services running in docker._\n\n\u003cimg align=\"center\" src=\"assets/freight-conductor.png\"\u003e\n\u003cp\u003e\u003c/p\u003e\n\u003cp\u003e\nRunning a home network? Then you may already have a custom setup, probably using docker compose. You might enjoy all the maintenance and tinkering, but you are surely aware of the pitfalls and potential downtime. If you think that is ok, or if you don't want automation, then this stack is probably not for you.\nStill interested? Then read on...\n\u003c/p\u003e\n\n**Table of contents:**\n\n- [Key concepts](#key-concepts)\n  - [Single source of truth](#single-source-of-truth)\n  - [Managed proxy setup](#managed-proxy-setup)\n  - [Managed service deployments \\\u0026 updates](#managed-service-deployments--updates)\n  - [\\*Zero downtime?](#zero-downtime)\n- [Apps included](#apps-included)\n- [Prerequisites](#prerequisites)\n- [Dev/ops tools](#devops-tools)\n  - [utility functions](#utility-functions)\n  - [Utility scripts](#utility-scripts)\n- [Howto](#howto)\n  - [Install \\\u0026 run](#install--run)\n  - [Configure services](#configure-services)\n    - [Scenario 1: Adding an upstream service that will be deployed and managed](#scenario-1-adding-an-upstream-service-that-will-be-deployed-and-managed)\n    - [Scenario 2: Adding a TLS passthrough endpoint](#scenario-2-adding-a-tls-passthrough-endpoint)\n    - [Scenario 3: Adding a TCP endpoint](#scenario-3-adding-a-tcp-endpoint)\n    - [Scenario 4: Adding a local (host) endpoint](#scenario-4-adding-a-local-host-endpoint)\n    - [Additional docker properties](#additional-docker-properties)\n  - [Configure plugins](#configure-plugins)\n    - [CrowdSec](#crowdsec)\n  - [Using the Api \\\u0026 OpenApi spec](#using-the-api--openapi-spec)\n  - [Webhooks](#webhooks)\n  - [OpenVPN server with SSH access](#openvpn-server-with-ssh-access)\n    - [1. Initialize the configuration files and certificates](#1-initialize-the-configuration-files-and-certificates)\n    - [2. Create a client file](#2-create-a-client-file)\n    - [3. Retrieve the client configuration with embedded certificates and place in github workflow folder](#3-retrieve-the-client-configuration-with-embedded-certificates-and-place-in-github-workflow-folder)\n    - [4. SSH access](#4-ssh-access)\n    - [5. Make sure port 1194 is portforwarding the UDP protocol.](#5-make-sure-port-1194-is-portforwarding-the-udp-protocol)\n- [Questions one might have](#questions-one-might-have)\n  - [What about Nginx?](#what-about-nginx)\n  - [Does this scale to more machines?](#does-this-scale-to-more-machines)\n- [Disclaimer](#disclaimer)\n\n## Key concepts\n\n### Single source of truth\n\nOne file (`db.yml`) is used for all the infra and workloads it creates and manages, to ensure a predictable and reliable automated workflow.\nThis means abstractions are used which means a trade off between flexibility and reliability, but the stack is easily modified and enhanced to meet your needs. We strive to mirror docker compose functionality, which means no concessions are necessary from a docker compose enthusiast's perspective.\n\n### Managed proxy setup\n\nitsUP generates and manages `proxy/docker-compose.yml` which operates traefik to be able to do all one wants from a routing solution:\n\n1. Terminate TLS and forward tcp/udp traffic over an encrypted network to listening endpoints.\n   ææ2. Passthrough TLS to endpoints (most people have secure Home Assistant setups already).\n2. Open host ports if needed to choose a new port (openvpn service does exactly that)\n\n### Managed service deployments \u0026 updates\n\nitsUP generates and manages `upstream/{project}/docker-compose.yml` files to deploy container workloads as defined as a service in `db.yml`.\nThis centralizes and abstracts away the plethora of custom docker compose setups that are mostly uniform in their approach anyway, so controlling their artifacts from one source of truth makes a lot of sense.\n\n### \u003csup\u003e\\*\u003c/sup\u003eZero downtime?\n\nLike with all docker orchestration platforms (even Kubernetes) this is dependent on the containers:\n\n- are healthchecks correctly implemented?\n- Are SIGHUP signals respected to shutdown within an acceptable time frame?\n- Are the containers stateless?\n\nitsUP will rollout changes by:\n\n1. bringing up a new container and wait till it is healthy (if it has a health check then max 60s, otherwise assumes it is healthy after 10s)\n2. kill the old container and wait for it to drain, then removes it\n\n_What about stateful services?_\n\nIt is surely possible to deploy stateful services but beware that those might not be good candidates for the `docker rollout` automation. In order to update those services it is strongly advised to first read the upgrade documentation for the newer version and follow the prescribed steps. More mature databases might have integrated these steps in the runtime, but expect that to be an exception. So, to garner correct results you are on your own and will have to read up on your chosen solutions.\n\n## Apps included\n\n- [traefik/traefik](https://github.com/traefik/traefik): the famous L7 routing proxy that manages letsencrypt certificates\n- [minio/minio](https://github.com/minio/minio): S3 storage\n- [nubacuk/docker-openvpn](https://github.com/nuBacuk/docker-openvpn): vpn access to the host running this stack\n- [traefik/whoami](https://github.com/traefik/whoami): to demonstrate that headers are correctly passed along\n\n## Prerequisites\n\n**Tools:**\n\n- [docker](https://www.docker.com) daemon and client\n- docker [rollout](https://github.com/Wowu/docker-rollout) plugin\n- [openvpn](https://openvpn.net): for testing vpn access (optional)\n\n**Infra:**\n\n- Portforwarding of port `80` and `443` to the machine running this stack. This stack MUST overtake whatever routing you now have, but don't worry, as it supports your home assistant setup and forwards any traffic it expects to it (if you finish the pre-configured `home-assistant` project in `db.yml`)\n- A wildcard dns domain like `*.itsup.example.com` that points to your home ip. This allows to choose whatever subdomain for your services. You may of course choose and manage any domain in a similar fashion for a public service, but I suggest not going through such trouble for anything private.\n\n## Dev/ops tools\n\n### utility functions\n\nSource `lib/functions.sh` to get:\n\n- `dcp`: run a `docker compose` command targeting the proxy stack (`proxy` + `terminate` services): `dcp logs -f`\n- `dcu`: run a `docker compose` command targeting a specific upstream: `dcu test up`\n- `dca`: run a `docker compose` command targeting all upstreams: `dca ps`\n- `dcpx`: execute a command in one of the proxy containers: `dcpx traefik 'rm -rf /etc/acme/acme.json \u0026\u0026 shutdown' \u0026\u0026 dcp up`\n- `dcux`: execute a command in one of the upstream containers: `dcux test test-informant env`\n\nIn effect these wrapper commands achieve the same as when going into an `upstream/\\*`folder and running`docker compose` there.\nI don't want to switch folders/terminals all the time and want to keep a \"project root\" history of my commands so I choose this approach.\n\n### Utility scripts\n\n- ~~`bin/update-certs.py`: pull certs and reload the proxy if any certs were created or updated. You could run this in a crontab every week if you want to stay up to date.~~ (Obsolete since migration to Treaefik)\n- `bin/write-artifacts.py`: after updating `db.yml` you can run this script to generate new artifacts.\n- `bin/validate-db.py`: also ran from `bin/write-artifacts.py`\n- `bin/requirements-update.sh`: You may want to update requirements once in a while ;)\n\n## Howto\n\n### Install \u0026 run\n\nThese are the scripts to install everything and start the proxy and api so that we can receive incoming challenge webhooks:\n\n1. `bin/install.sh`: creates a local `.venv` and installs all python project deps.\n2. `bin/start-all.sh`: starts the proxy (docker compose) and the api server (uvicorn).\n3. `bin/apply.py`: applies all of `db.yml`.\n4. `bin/api-logs.sh`: tails the output of the api server.\n\nBut before doing so please configure your stuff:\n\n### Configure services\n\n1. Copy `.env.sample` to `.env` and set the correct info (comments should be self explanatory).\n2. Copy `db.yml.sample` to `db.yml` and edit your project and their services (see explanations below).\n\nProject and service configuration is explained below with the following scenarios. Please also check `db.yml.sample` as it contains more examples.\n\n#### Scenario 1: Adding an upstream service that will be deployed and managed\n\nEdit `db.yml` and add your projects with their service(s). Any service that is given an `image: ` prop will be deployed with `docker compose`.\nExample:\n\n```yaml\nprojects:\n  ...\n  - description: whoami service\n    name: whoami\n    services:\n      - image: traefik/whoami:latest\n        ingress:\n          - domain: whoami.example.com\n        host: web\n```\n\nRun `bin/apply.py` to write all artifacts and deploy/update relevant docker stacks.\n\n#### Scenario 2: Adding a TLS passthrough endpoint\n\nAdd a service with ingress and set `passthrough: true`.\nExample:\n\n```yaml\nprojects:\n  ...\n  - description: Home Assistant passthrough\n    enabled: true\n    name: home-assistant\n    services:\n      - ingress:\n        - domain: home.example.com\n          passthrough: true\n          port: 443\n        host: 192.168.1.111\n```\n\nIf you also need port 80 to listen for http challenges for your endpoint (home-assistant may do its own), then you may also add:\n\n```yaml\n  ...\n      - ingress:\n        ...\n        - domain: home.example.com\n          passthrough: true\n          path_prefix: /.well-known/acme-challenge/\n          port: 80\n```\n\n(Port 80 is disallowed for any other other cases.)\n\n#### Scenario 3: Adding a TCP endpoint\n\nAdd a service with ingress and set `router: tcp`.\nExample:\n\n```yaml\nprojects:\n  ...\n  - description: Minio service\n    name: minio\n    services:\n      - command: server --console-address \":9001\" /data\n        env:\n          MINIO_ROOT_USER: root\n          MINIO_ROOT_PASSWORD: xx\n        host: app\n        image: minio/minio:latest\n        ingress:\n          - domain: minio-api.example.com\n            port: 9000\n            router: tcp\n          - domain: minio-ui.example.com\n            port: 9001\n        volumes:\n          - /data\n```\n\n#### Scenario 4: Adding a local (host) endpoint\n\nYou can expose an existing service that is already running on the host by creating a service:\n\n- without an `image` prop\n- targeting the host from within docker\n- configuring it's ingress\n\nExample:\n\n```yaml\nprojects:\n  ...\n  - description: itsUP API running on the host\n    name: itsUP\n    services:\n      - ingress:\n          - domain: itsup.example.com\n            port: 8888\n        host: 172.17.0.1 # change this to host.docker.internal when on Docker Desktop\n```\n\n#### Additional docker properties\n\nOne can add additional docker properties to a service by adding them to the `additional_properties` dictionary:\n\n```yaml\nadditional_properties:\n  cpus: 0.1\n```\n\nThe following docker service properties exist at the service root level and MUST NOT be added via `additional_properties`:\n\n- command\n- depends_on\n- env\n- image\n- port\n- name\n- restart\n- volumes\n\n(Also see `lib/models.py`)\n\n### Configure plugins\n\nYou can enable and configure plugins in `db.yml`. Right now we support the following:\n\n#### CrowdSec\n\n[CrowdSec](https://www.crowdsec.net) can run as a container via plugin [crowdsec-bouncer-traefik-plugin](https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin).\n\n**Step 1: generate api key**\n\nFirst set `enable: true`, run `bin/write-artifacts.py`, and bring up the `crowdsec` container:\n\n```\ndocker compose up -d crowdsec\n```\n\nNow we can execute the command to get the key:\n\n```\ndocker compose exec crowdsec cscli bouncers add crowdsecBouncer\n```\n\nPut the resulting api key in the `plugins.crowdsec.apikey` configuration in `db.yml` and apply with `bin/apply.py`.\nCrowdsec is now running and wired up, but does not use any blocklists yet. Those can be managed manually, but preferable is to become part of the community by creating an account with CrowdSec to get access and contribute to the community blocklists, as well as view results in your account's dashboards.\n\n**Step 2: connect your instance with the CrowdSec console**\n\nAfter creating an account create a machine instance in the console, and register the enrollment key in your stack:\n\n```\ndocker compose exec crowdsec cscli console enroll ${enrollment key}\n```\n\n**Step 3: subscribe to 3rd party blocklists**\n\nIn the [security-engines](https://app.crowdsec.net/security-engines) section select the \"Blocklists\" of your engine and choose some blocklists of interest.\nExample:\n\n- Free proxies list\n- Firehol SSL proxies list\n- Firehol cruzit.com list\n\n### Using the Api \u0026 OpenApi spec\n\nThe API allows openapi compatible clients to do management on this stack (ChatGPT works wonders).\n\nGenerate the spec with `api/extract-openapi.py`.\n\nAll endpoints do auth and expect an incoming Bearer token to be set to `.env/API_KEY`.\n\nException: Only github webhook endpoints (check for annotation `@app.hooks.register(...`) get it from the `github_secret` header.\n\n### Webhooks\n\nWebhooks are used for the following:\n\n1. to receive updates to this repo, which will result in a `git pull` and `bin/apply.py` to update any changes in the code. The provided project with `name: itsUP` is used for that, so DON'T delete it if you care about automated updates to this repo.\n2. to receive incoming github webhooks (or GET requests to `/update-upstream?project=bla\u0026service=dida`) that result in rolling up of a project or specific service only.\n\nOne GitHub webhook listening to `workflow_job`s is provided, which needs:\n\n- the hook you will register in the github project to end with `/hook?project=bla\u0026service=dida` (`service` optional), and the `github_secret` set to `.env/API_KEY`.\n\nI mainly use GitHub workflows and created webhooks for my individual projects, so I can just manage all webhooks in one place.\n\n**NOTE:**\n\nWhen using crowdsec this webhook is probably not coming in as it exits the Azure cloud (public IP range), which is also host to many malicious actors that spin up ephemeral intrusion tools. To still receive signals from github you can use a vpn setup as the one used in this repo (check `.github/workflows/test.yml`).\n\n### OpenVPN server with SSH access\n\nThis setup contains a project called \"vpn\" which runs an openvpn service that gives ssh access. To bootstrap it:\n\n#### 1. Initialize the configuration files and certificates\n\n```\ndcu vpn run vpn-openvpn ovpn_genconfig -u udp4://vpn.itsup.example.com\ndcu vpn run vpn-openvpn ovpn_initpki\n```\n\nSave the signing passphrase you created.\n\n#### 2. Create a client file\n\n```\nexport CLIENTNAME='github'\ndcu vpn run vpn-openvpn easyrsa build-client-full $CLIENTNAME\n```\n\nSave the client passphrase you created as it will be used for `OVPN_PASSWORD` below.\n\n#### 3. Retrieve the client configuration with embedded certificates and place in github workflow folder\n\n```\ndcu vpn run vpn-openvpn ovpn_getclient $CLIENTNAME combined \u003e .github/workflows/client.ovpn\n```\n\n**IMPORTANT:** Now change `udp` to `udp4` in the `remote: ...` line to target UDP with IPv4 as docker is still not there.\n\nTest access (expects local `openvpn` installed):\n\n```\nsudo openvpn .github/workflows/client.ovpn\n```\n\nNow save the `$OVPN_USER_KEY` from `client.ovpn`'s `\u003ckey\u003e$OVPN_USER_KEY\u003c/key\u003e` and remove the `\u003ckey\u003e...\u003c/key\u003e`.\nAlso save the `$OVPN_TLS_AUTH_KEY` from `\u003ctls-auth...` section and remove it.\n\nAdd the secrets to your github repo\n\n- `OVPN_USERNAME`: `github`\n- `OVPN_PASSWORD`: the client passphrase\n- `OVPN_USER_KEY`\n- `OVPN_TLS_AUTH_KEY`\n\n#### 4. SSH access\n\nIn order for ssh access by github, create a private key and add the pub part to the `authorized_keys` on the host:\n\n```\n\nssh-keygen -t ed25519 -C \"your_email@example.com\"\ncat ~/.ssh/id_ed25519.pub \u003e\u003e ~/.ssh/authorized_keys\n\n```\n\nAdd the secrets to GitHub:\n\n- `SERVER_HOST`: the hostname of this repo's api server\n- `SERVER_USERNAME`: the username that has access to your host's ssh server\n- `SSH_PRIVATE_KEY`: the private key of the user\n\n#### 5. Make sure port 1194 is portforwarding the UDP protocol.\n\nNow we can start the server and expect all to work ok.\n\nIf you wish to revoke a cert or do something else, please visit this page: [kylemanna/docker-openvpn/blob/master/docs/docker-compose.md](https://github.com/kylemanna/docker-openvpn/blob/master/docs/docker-compose.md)\n\n## Questions one might have\n\n### What about Nginx?\n\nAs you may have noted there is a lot of functionality based on Nginx in this repo. I started out using their proxy, but later on ran into the problem of their engine not picking up upstream changes, learning that only the paid Nginx+ does that. I heavily relied on kubernetes in the past years and such was not an issue in their `ingress-NGINX` controller. When I found that Traefik does not suffer this, AND manages letsencrypt certs gracefully, AND gives us label based L7 functionality (like in Kubernetes), I decided to integrate that instead. Weary about its performance though, I intended to keep both approaches side by side. The Nginx part is not working anymore, but I left the code for others to see how one can overcome certain problems in that ecosystem. If one would like to use Nginx for some reason (it is about 40% faster), it is very easy to switch back. But be aware it implies hooking up the hacky `bin/update-certs.py` script to a cron tab for automatic cert rotation.\n\n### Does this scale to more machines?\n\nIn the future we might consider expanding this setup to use docker swarm, as it should be easy to do. For now we like to keep it simple.\n\n## Disclaimer\n\n**Don't blame this infra automation tooling for anything going wrong inside your containers!**\n\nI suggest you repeat that mantra now and then and question yourself when things go wrong: where lies the problem?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmorriz%2Fitsup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmorriz%2Fitsup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmorriz%2Fitsup/lists"}