{"id":13510382,"url":"https://github.com/archlinux/infrastructure","last_synced_at":"2025-06-28T06:06:37.959Z","repository":{"id":38838740,"uuid":"210748484","full_name":"archlinux/infrastructure","owner":"archlinux","description":"  Official Arch Linux Infrastructure Repository (read-only mirror)","archived":false,"fork":false,"pushed_at":"2025-06-20T07:20:49.000Z","size":15186,"stargazers_count":103,"open_issues_count":0,"forks_count":22,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-06-20T08:29:52.316Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://gitlab.archlinux.org/archlinux/infrastructure","language":"Jinja","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/archlinux.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-09-25T03:34:37.000Z","updated_at":"2025-06-17T22:53:15.000Z","dependencies_parsed_at":"2023-10-21T19:25:54.112Z","dependency_job_id":"dddec88b-244d-4bb6-8b0a-865c8ed11c85","html_url":"https://github.com/archlinux/infrastructure","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/archlinux/infrastructure","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archlinux%2Finfrastructure","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archlinux%2Finfrastructure/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archlinux%2Finfrastructure/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archlinux%2Finfrastructure/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/archlinux","download_url":"https://codeload.github.com/archlinux/infrastructure/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archlinux%2Finfrastructure/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262382744,"owners_count":23302298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T02:01:36.704Z","updated_at":"2025-06-28T06:06:37.951Z","avatar_url":"https://github.com/archlinux.png","language":"Jinja","funding_links":[],"categories":["Jinja","others"],"sub_categories":[],"readme":"# Arch Infrastructure\n\nThis repository contains the complete collection of ansible playbooks and roles for the Arch Linux infrastructure.\n\n## Table of contents\n[[_TOC_]]\n\n## Requirements\n\nInstall these packages:\n  - terraform\n  - python-click\n  - python-jmespath\n  - moreutils (for playbooks/tasks/reencrypt-vault-key.yml)\n\n### Instructions\n\nAll systems are set up the same way. For the first time setup in the Hetzner rescue system,\nrun the provisioning script: `ansible-playbook playbooks/tasks/install_arch.yml -l $host`.\nThe provisioning script configures a sane basic systemd with sshd. By design, it is NOT idempotent.\nAfter the provisioning script has run, it is safe to reboot.\n\nOnce in the new system, run the regular playbook: `HCLOUD_TOKEN=$(misc/get_key.py misc/vaults/vault_hetzner.yml hetzner_cloud_api_key) ansible-playbook playbooks/$hostname.yml`.\nThis playbook is the one regularity used for administrating the server and is entirely idempotent.\n\nWhen adding a new machine you should also deploy our SSH known_hosts file and update the SSH hostkeys file in this git repo.\nFor this you can simply run the `playbooks/tasks/sync-ssh-hostkeys.yml` playbook and commit the changes it makes to this git repository.\nIt will also deploy any new SSH host keys to all our machines.\n\n#### Note about GPG keys\n\nThe `root_access.yml` file contains the `vault_default_pgpkeys` variable which\ndetermines the users that have access to the `default` vault, as well as the\nborg backup keys. A separate `super` vault exists for storing highly sensitive\nsecrets like Hetzner credentials; access to the `super` vault is controlled by\nthe `vault_super_pgpkeys` variable.\n\nAll the keys should be on the local user gpg keyring and at **minimum** be\nlocally signed with `--lsign-key` (or if you use TOFU, have `--tofu-policy\ngood`). This is necessary for running any of the `reencrypt-vault-default-key`,\n`reencrypt-vault-super-key `or `fetch-borg-keys` tasks.\n\n#### Note about packer\n\nWe use packer to build snapshots on hcloud to use as server base images.\nIn order to use this, you need to install packer and then run\n\n    packer build -var $(misc/get_key.py misc/vaults/vault_hetzner.yml hetzner_cloud_api_key --format env) packer/archlinux.pkr.hcl\n\nThis will take some time after which a new snapshot will have been created on the primary hcloud archlinux project.\n\nFor the sandbox project please run\n\n    packer build -var $(misc/get_key.py misc/vaults/vault_hetzner.yml hetzner_cloud_sandbox_infrastructure_api_key --format env | sed 's/_sandbox_infrastructure//') -var install_ec2_public_keys_service=true packer/archlinux.pkr.hcl\n\n#### Note about terraform\n\nWe use terraform in two ways:\n\n1. To provision a part of the infrastructure on hcloud (and possibly other service providers in the future)\n2. To declaratively configure applications\n\nFor both of these, we have set up a separate terraform script. The reason for that is that sadly terraform can't have\nproviders depend on other providers so we can't declaratively state that we want to configure software on a server which\nitself needs to be provisioned first. Therefore, we use a two-stage process. Generally speaking, scenario 1. is configured in\n`tf-stage1` and 2. is in `tf-stage2`. Maybe in the future, we can just have a single terraform script for everything\nbut for the time being, this is what we're stuck with.\n\nThe very first time you run terraform on your system, you'll have to init it:\n\n    cd tf-stage1  # and also tf-stage2\n    terraform init -backend-config=\"conn_str=postgres://terraform:$(../misc/get_key.py ../group_vars/all/vault_terraform.yml vault_terraform_db_password)@state.archlinux.org?sslmode=verify-full\"\n\nAfter making changes to the infrastructure in `tf-stage1/archlinux.tf`, run\n\n    terraform plan\n\nThis will show you planned changes between the current infrastructure and the desired infrastructure.\nYou can then run\n\n    terraform apply\n\nto actually apply your changes.\n\nThe same applies to changed application configuration in which case you'd run\nit inside of `tf-stage2` instead of `tf-stage1`.\n\nWe store terraform state on a special server that is the only hcloud server NOT\nmanaged by terraform so that we do not run into a chicken-egg problem. The\nstate server is assumed to just exist so in an unlikely case where we have to\nentirely redo this infrastructure, the state server would have to be manually\nset up.\n\n#### SMTP Configuration\n\nAll hosts should be relaying email through our primary mx host (currently 'mail.archlinux.org'). See [docs/email.md](./docs/email.md) for full details.\n\n### Putting a service in maintenance mode\n\nMost web services with a nginx configuration, can be put into a maintenance mode, by running the playbook with a maintenance variable:\n\n    ansible-playbook -e maintenance=true playbooks/\u003cplaybook.yml\u003e\n\nThis also works with a tag:\n\n    ansible-playbook -t \u003ctag\u003e -e maintenance=true playbooks/\u003cplaybook.yml\u003e\n\nAs long as you pass the maintenance variable to the playbook run, the web service will stay in maintenance mode. As soon as you stop\npassing it on the command line and run the playbook again, the regular nginx configuration should resume and the service should accept\nrequests by the end of the run.\n\nPassing maintenance=false, will also prevent the regular nginx configuration from resuming, but will not put the service into maintenance\nmode.\n\nKeep in mind that passing the maintenance variable to the whole playbook, without any tag, will make all the web services that have the\nmaintenance mode in them, to be put in maintenance mode. Use tags to affect only the services you want.\n\nDocumentation on how to add the maintenance mode to a web service is inside [docs/maintenance.md](./docs/maintenance.md).\n\n### Finding servers requiring security updates\n\nArch-audit can be used to find servers in need of updates for security issues.\n\n    ansible all -a \"arch-audit -u\"\n\n### Semi-automated server upgrades\n\nFor updating all servers in a mostly unattented manner, the following playbook\ncan be used:\n\n    ansible-playbook playbooks/tasks/upgrade-servers.yml [-l SUBSET]\n\nIt runs `pacman -Syu` on the targeted hosts in batches and then reboots them.\nIf any server fails to reboot successfully, the rolling update stops and\nfurther batches are cancelled. To display the packages updated on each host,\nyou can pass the `--diff` option to ansible-playbook.\n\nUsing this update method, `.pacnew` files are left unmerged which is OK for\nmost configuration files that are managed by Ansible. However, care must be\ntaken with updates that require manual intervention (e.g. major PostgreSQL\nreleases).\n\n## Servers\n\nThis section has been moved to [docs/servers.md](docs/servers.md).\n\n## Ansible repo workflows\n\n### Fetching the borg keys for local storage\n\n  - Make sure you have all the GPG keys **at least** locally signed\n  - Run the `playbooks/tasks/fetch-borg-keys.yml` playbook\n  - Make sure the playbook runs successfully and check the keys under the borg-keys directory\n\n### Re-encrypting the vaults after adding a new PGP key\n\nFollow the instructions in [group_vars/all/root_access.yml](group_vars/all/root_access.yml).\n\n### Changing the vault password on encrypted files\n\nSee [docs/vault-rekeying.md](docs/vault-rekeying.md).\n\n## Backup documentation\n\nWe use BorgBackup for all of our backup needs. We have a primary backup storage as well as an\nadditional offsite backup.\n\nSee [docs/backups.md](./docs/backups.md) for detailed backup information.\n\n## Updating Gitlab\n\nOur Gitlab installation uses [Omnibus](https://docs.gitlab.com/omnibus/) to run Gitlab on Docker. Updating Gitlab is as simple as running the ansible gitlab playbook:\n\n    ansible-playbook playbooks/gitlab.archlinux.org.yml --diff -t gitlab\n\nTo view the current Gitlab version visit [this url](https://gitlab.archlinux.org/help/)\n\n## One-shots\n\nA bunch of once-only admin task scripts can be found in `one-shots/`.\nWe try to minimize the amount of manual one-shot admin work we have to do but sometimes for some migrations it might be necessary to have such scripts.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchlinux%2Finfrastructure","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farchlinux%2Finfrastructure","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchlinux%2Finfrastructure/lists"}