{"id":21351759,"url":"https://github.com/serverfarmer/sm-backup-collector","last_synced_at":"2026-05-18T22:03:52.949Z","repository":{"id":92623512,"uuid":"47940879","full_name":"serverfarmer/sm-backup-collector","owner":"serverfarmer","description":"sf-backup-collector extension provides central backup functionality for small and medium server farms.","archived":false,"fork":false,"pushed_at":"2021-11-28T10:28:16.000Z","size":21,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-22T17:13:27.296Z","etag":null,"topics":["backup","backup-manager","backup-process","backup-script","backup-solution","backup-tool","data-storage","debian","devops","devops-tools","storage","ubuntu"],"latest_commit_sha":null,"homepage":"http://serverfarmer.org/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/serverfarmer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-12-13T23:25:57.000Z","updated_at":"2024-06-23T22:34:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"a65eb16a-2afd-4097-ab6e-f1b3b1cec936","html_url":"https://github.com/serverfarmer/sm-backup-collector","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serverfarmer%2Fsm-backup-collector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serverfarmer%2Fsm-backup-collector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serverfarmer%2Fsm-backup-collector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serverfarmer%2Fsm-backup-collector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/serverfarmer","download_url":"https://codeload.github.com/serverfarmer/sm-backup-collector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243826783,"owners_count":20354220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","backup-manager","backup-process","backup-script","backup-solution","backup-tool","data-storage","debian","devops","devops-tools","storage","ubuntu"],"created_at":"2024-11-22T03:11:00.182Z","updated_at":"2026-05-18T22:03:52.895Z","avatar_url":"https://github.com/serverfarmer.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Server Farmer backup collector\n\n### Overview\n\nDistributed backup is one of the key functionalities of Server Farmer.\n\nAll hosts with installed Server Farmer, unless explicitly disabled, are doing their own backup (to specified directory on local drive, using `sf-backup` extension), which is then pulled using `scp` by *backup collector* host (which is responsible for long-term backup storage).\n\n*This part of backup documentation describes only collector/storage aspects. See `sf-backup` repository for details about the backup process itself.*\n\n\n### Adding new servers to backup collector\n\nNew servers are added using `add-backup-host.sh` script, which requires that added server has already installed dedicated ssh key for `backup` system user.\n\nUsually all this process is wrapped by `add-managed-host.sh` script from `sm-farm-manager` extension, which is the primary script for adding new servers to the farm.\n\n\n### Types of backup collectors\n\nThere are 4 types of backup collectors:\n\n1. Standalone - in this default scenario, *farm manager* is also the only backup collector (except possibly blind slaves). It is responsible for:\n- managing `backup.hosts` file (and associated ssh keys)\n- fetching backup archives from managed hosts\n- all possible further actions on fetched backups (statistics, transformations etc. - out of scope)\n\n2. Master - responsible only for the management part, while `backup.hosts` and ssh keys are sent to slave collector(s) after each update, using `sync-remote-collectors.sh` script.\n\n3. Slave - responsible only for actual fetching backup archives and further processing. Registered in `collector.hosts` file on master collector. Typically there is only one slave collector (all of them share the same `backup.hosts` file and would fetch the same data, so multiple slaves are mostly used in tricky and very rare network setups).\n\n4. Blind slave - like slave, but not registered in `collector.hosts` file, instead updated manually, and responsible for backups of some group of servers only. Such configuration is used mainly for commercial customers, who want to have all their backup stored in separate place.\n\n\n### `backup.hosts` file format\n\n`backup.hosts` (as well as `collector.hosts`) is simply a text file with list of hostnames, possibly including port numbers, one per line:\n\n```\nhostname1.internal:22\nhostname2.internal\notherserver.domain.com:3322\n```\n\nEmpty lines and lines starting with `#` character are ignored.\n\n\n### How backup collector backups itself\n\nBackup collector, just like any other host, is responsible for its own backup.\n\nThen, if its hostname is found in `backup.hosts` file, its backup is moved locally (instead of `scp`) to storage directory tree.\n\nIn some special scenarios, you can remove backup collector hostname from `backup.hosts` file (this is used mostly in high-compliance setups, in combination with `push-to-collector` (where *farm manager* and backup collector are sending their own encrypted backups to another host).\n\n\n### Backup storage directory tree format\n\nAll fetched backup archives are stored inside `/srv/mounts/backup/remote` directory (you can't change it, but it can be a symlink). Default directory tree structure looks like this:\n\n```\n/srv/mounts/backup/remote/otherserver.domain.com/20180902/etc.tar.gpg\n                          ^                      ^        ^\n                          hostname (no port)     date     file\n```\n\nIf this is not enough, since you eg. need to store backups from each hour, you can extend this structure using `/etc/local/.config/backup.index` file, which should contain the date mask, eg.:\n\n```\n%Y%m%d/%H\n```\n\n\n### Backup transfer process\n\nBackup collector fetches backup files from remote hosts using `scp` and dedicated ssh keys (separate key for each host), for `backup` system user.\n\n\n### Push-to-collector mode\n\nIn default scenario, backup collector has full shell access to managed hosts, as `backup` system user, including to *farm manager*. In high-compliance setups, eg. PCI DSS, this is not acceptable, as this is just one step away from having access to management ssh keys located on *farm manager*.\n\nBecause of that, script `cron/push-to-collector.sh` in `sf-backup` extension allows transferring farm manager encrypted backups in other direction: from farm manager to backup collector. This way, backup collector no longer needs ssh access to farm manager, which .\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserverfarmer%2Fsm-backup-collector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserverfarmer%2Fsm-backup-collector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserverfarmer%2Fsm-backup-collector/lists"}