{"id":13583850,"url":"https://github.com/ooni/sysadmin","last_synced_at":"2025-03-05T09:21:39.764Z","repository":{"id":19563757,"uuid":"22812838","full_name":"ooni/sysadmin","owner":"ooni","description":"System administration tools","archived":false,"fork":false,"pushed_at":"2024-04-08T10:07:04.000Z","size":3029,"stargazers_count":60,"open_issues_count":21,"forks_count":26,"subscribers_count":23,"default_branch":"master","last_synced_at":"2024-04-14T00:59:40.270Z","etag":null,"topics":["ooni-infrastructure","sysadmin"],"latest_commit_sha":null,"homepage":"https://ooni.org","language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ooni.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2014-08-10T16:09:38.000Z","updated_at":"2024-04-15T21:57:05.948Z","dependencies_parsed_at":"2023-10-16T04:22:48.725Z","dependency_job_id":"b6a38086-df2c-4b8c-8cc8-e4a90ce3dc6e","html_url":"https://github.com/ooni/sysadmin","commit_stats":{"total_commits":2142,"total_committers":22,"mean_commits":97.36363636363636,"dds":0.5672268907563025,"last_synced_commit":"a0a9a942be7abfdea5f11e9f042842de16a23ae5"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2Fsysadmin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2Fsysadmin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2Fsysadmin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2Fsysadmin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ooni","download_url":"https://codeload.github.com/ooni/sysadmin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241131198,"owners_count":19914862,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ooni-infrastructure","sysadmin"],"created_at":"2024-08-01T15:03:50.867Z","updated_at":"2025-03-05T09:21:39.529Z","avatar_url":"https://github.com/ooni.png","language":"Jinja","readme":"# OONI sysadmin\n\nIn here live all the tools and scripts related to administering the\ninfrastructure that are part of OONI.\n\n# Getting started\n\nIt is recommended you use a [python virtualenv](https://virtualenv.pypa.io/en/latest/) to install ansible and dnspython.\n\nOnce you activate the virtualenv you can install the correct version of ansible and dnspython with:\n\n```\npip install ansible==2.9.16\npip install requests\npip install requests[socks]\n```\n\n\n# Ansible roles\n\nIt is required for all OONI team to run the same ansible version to minimise\ncompatibility issues.\nAll playbooks should include `ansible-version.yml` at the beginning of the\nplaybook, by including:\n```\n---\n- import_playbook: ansible-version.yml\n```\n\nIf you need some python packages **only** for ansible module to\nwork and don't need it in system-wide pip repository, then you should put these\nmodules in separate virtual environment and set proper\n`ansible_python_interpreter` for the play. See `docker_py` role and grep for\n`/root/venv` for examples.\n\nIf you need to store secrets in repository then store them as vaults using\n`ansible/vault` script as a wrapper for `ansible-vault`. Store encrypted\nvariables with `vault_` prefix to make world [a more grepable place](http://docs.ansible.com/ansible/playbooks_best_practices.html#best-practices-for-variables-and-vaults)\nand link location of the variable using same name without prefix in corresponding `vars.yml`.\n`scripts/ansible-syntax-check` checks links between vaults and plaintext files during Travis build.\n`ansible/play` wrapper for `ansible-playbook` will execute a playbook with\nproper vault secret and inventory.\n\nIn order to access secrets stored inside of the vault, you will need a copy of\nthe vault password encrypted with your PGP key. This file should be stored\ninside of `~/.ssh/ooni-sysadmin.vaultpw.gpg`.\n\n# SSH Config\n\nYou should configure your `~/.ssh/config` with the following:\n\n```\nCiphers aes128-ctr,aes256-ctr,aes128-cbc,aes256-cbc\nIdentitiesOnly yes\nServerAliveInterval 120\nUserKnownHostsFile ~/.ssh/known_hosts ~/PATH/TO//ooni/sysadmin/ext/known_hosts\n\n```\n\nReplace `~/PATH/TO//ooni/sysadmin/ext/known_hosts` to where you have cloned the\n`ooni/sysadmin` repo. This will ensure you use the host key fingeprints from\nthis repo instead of just relying on TOFU.\n\nYou probably also want to add:\n\n```\nhost *.ooni.io\n  user USERNAME\n  identityfile ~/.ssh/id_rsa\n\nhost *.ooni.nu\n  user USERNAME\n  identityfile ~/.ssh/id_rsa\n```\n\nYou should replace `USERNAME` with your username from `adm_login`.\n\nOn macOS you may want to also add:\n```\nhost *\n    UseKeychain yes\n```\n\nTo use the Keychain to store passwords.\n\n# M-Lab deployment\n\nM-Lab [deployment process](https://github.com/m-lab/ooni-support/#m-lab-deployment-process).\n\n# Upgrading OONI infrastructure\n\n## ooni-backend pitfalls\n\n* Ensure that the HS private keys of bouncer and collector are in right PATH\n(collector/private_key , bouncer/private_key).\n* Set the bouncer address in bouncer.yaml to the correct HS address.\n* ooni-backend will *not* generate missing directories and fail to start\n\nRunning a short ooni-probe test will ensure that the backend has been\nsuccessfully upgraded, an example test:\n\n```\nooniprobe --collector httpo://CollectorAddress.onion blocking/http_requests \\\n--url http://ooni.io/\n```\n\n# New host HOWTO\n\n- come up with a name for $name.ooni.tld using DNS name policy\n- create a VM to allocate IP address\n- create `A` record for the domain name in namecheap web UI (API is hell)\n- fetch external inventory with `./play ext-inventory.yml`, it'll create a git commit\n- add $name.ooni.tld to _location tags_ section of `inventory` file, git-commit it\n- write firewall rules to `templates/iptables.filter.part/$name.ooni.tld` if needed, git-commit it\n- bootstrap VM with `./play dom0-bootstrap.yml -l $name.ooni.tld`\n- update Prometheus with `./play deploy-prometheus.yml -t prometheus-conf`\n- check `inventory` sanity with `./play inventory-check.yml` (everything should be `ok`, no changes, no failures), update `inventory-check.yml` with new checksum, git-commit it\n- `git push` those commits\n\n## DNS name policy\n\nPublic HTTP services are named `${service}.ooni.org` or `${service}.ooni.io`\n(legacy). _Public_ means that it's part of some external system we can't\ncontrol: published APP or MK versions, web URLs and so on. Public names should\nnot be used as an `inventory_hostname` to ease migration.\n\nVMs should have FQDN like `${location}-${name}-${number}.ooni.org`.\nVMs can provide one or more public-facing services that can change over time.\nThe name should be as descriptive as possible e.g. the type of services\nor the most important service being run. \n\nVarious legacy names should be cleaned up during re-deploying VMs with newer\nbase OS version.\n\n# Rename host HOWTO\n\nFirst, try hard to avoid renaming hosts. It's pain:\n\n- inventory_hostname is stamped in Prometheus SSL certificates\n- inventory_hostname is stamped as FQDN inside of firewall rules\n- inventory_hostname is stamped as filename for firewall rules\n- hostname is stamped in `/etc/hosts` on the host\n- hostname is stamped as `kernel.hostname` on the host\n- some applications use hostname as configuration value, e.g. [MongoDB](https://docs.mongodb.com/manual/tutorial/change-hostnames-in-a-replica-set/)\n\nBut re-deploying is not also an options sometimes due to GH platform limitations.  So...\n\n- use _New host HOWTO_ as a checklist to keep in mind\n- `on-rename` tag can save some time while running `dom0-bootstrap`: `./play dom0-bootstrap.yml -t on-rename -l $newname.ooni.tld`\n- grep ooni/sysadmin for `inventory_hostname`, $oldname, $newname (NB: not just oldname.ooni.tld, short name may be used somewhere as well)\n\n# PostgreSQL replica bootstrap\n\n[`pg_basebackup`](https://www.postgresql.org/docs/current/static/app-pgbasebackup.html)\nis nice, but does not support network traffic compression out of box and has no\nobvious way to resume interrupted backup. `rsync` solves that issue, but it\nneeds either WAL archiving (to external storage) to be configured or\n`wal_keep_segments` to be non-zero, because otherwise WAL logs are rotated ASAP\n(`min_wal_size` and `max_wal_size` do not set amount of WAL log available to\nreader, these options set amount of disk space allocated to writer!).\nAlso [replication slot](https://www.postgresql.org/docs/current/static/functions-admin.html#FUNCTIONS-REPLICATION)\nmay reserve WAL on creation, but beware, it postpones WAL reservation till replica connection by default.\n\n[`pg_start_backup()`](https://www.postgresql.org/docs/current/static/continuous-archiving.html#BACKUP-LOWLEVEL-BASE-BACKUP) +\n`rsync -az --exclude pg_replslot --exclude postmaster.pid --exclude postmaster.opts` is a way to go.\nAnd, obviously, don't exclude `pg_wal` (aka `pg_xlog`) if neither WAL archiving nor replication slot is not set up.\n\nAnd don't forget to revoke `authorized_keys` if SSH was used for `rsync`!\n\nYou may also want to bootstrap the replica [from MetaDB snapshot in S3](docs/metadb-sharing.md)\nand switch it to streaming replication afterwards.\n\n# Updating firewall rules\n\nIf you need to update the firewalling rules, because you added a new host to\nthe `have_fw` group or you changed the hostname of a host, you should edit the\nfile `templates/iptables.filter.part/$HOSTNAME` and then run:\n\n```\n./play dom0-bootstrap.yml -t fw\n```\n","funding_links":[],"categories":["Jinja"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fooni%2Fsysadmin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fooni%2Fsysadmin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fooni%2Fsysadmin/lists"}