Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/serverfarmer/sm-backup-collector

sf-backup-collector extension provides central backup functionality for small and medium server farms.
https://github.com/serverfarmer/sm-backup-collector

backup backup-manager backup-process backup-script backup-solution backup-tool data-storage debian devops devops-tools storage ubuntu

Last synced: 14 days ago
JSON representation

sf-backup-collector extension provides central backup functionality for small and medium server farms.

Awesome Lists containing this project

README 

        
# Server Farmer backup collector



### Overview



Distributed backup is one of the key functionalities of Server Farmer.



All hosts with installed Server Farmer, unless explicitly disabled, are doing their own backup (to specified directory on local drive, using `sf-backup` extension), which is then pulled using `scp` by *backup collector* host (which is responsible for long-term backup storage).



*This part of backup documentation describes only collector/storage aspects. See `sf-backup` repository for details about the backup process itself.*



### Adding new servers to backup collector



New servers are added using `add-backup-host.sh` script, which requires that added server has already installed dedicated ssh key for `backup` system user.



Usually all this process is wrapped by `add-managed-host.sh` script from `sm-farm-manager` extension, which is the primary script for adding new servers to the farm.



### Types of backup collectors



There are 4 types of backup collectors:



1. Standalone - in this default scenario, *farm manager* is also the only backup collector (except possibly blind slaves). It is responsible for:

- managing `backup.hosts` file (and associated ssh keys)

- fetching backup archives from managed hosts

- all possible further actions on fetched backups (statistics, transformations etc. - out of scope)



2. Master - responsible only for the management part, while `backup.hosts` and ssh keys are sent to slave collector(s) after each update, using `sync-remote-collectors.sh` script.



3. Slave - responsible only for actual fetching backup archives and further processing. Registered in `collector.hosts` file on master collector. Typically there is only one slave collector (all of them share the same `backup.hosts` file and would fetch the same data, so multiple slaves are mostly used in tricky and very rare network setups).



4. Blind slave - like slave, but not registered in `collector.hosts` file, instead updated manually, and responsible for backups of some group of servers only. Such configuration is used mainly for commercial customers, who want to have all their backup stored in separate place.



### `backup.hosts` file format



`backup.hosts` (as well as `collector.hosts`) is simply a text file with list of hostnames, possibly including port numbers, one per line:



```

hostname1.internal:22

hostname2.internal

otherserver.domain.com:3322

```



Empty lines and lines starting with `#` character are ignored.



### How backup collector backups itself



Backup collector, just like any other host, is responsible for its own backup.



Then, if its hostname is found in `backup.hosts` file, its backup is moved locally (instead of `scp`) to storage directory tree.



In some special scenarios, you can remove backup collector hostname from `backup.hosts` file (this is used mostly in high-compliance setups, in combination with `push-to-collector` (where *farm manager* and backup collector are sending their own encrypted backups to another host).



### Backup storage directory tree format



All fetched backup archives are stored inside `/srv/mounts/backup/remote` directory (you can't change it, but it can be a symlink). Default directory tree structure looks like this:



```

/srv/mounts/backup/remote/otherserver.domain.com/20180902/etc.tar.gpg

                          ^                      ^        ^

                          hostname (no port)     date     file

```



If this is not enough, since you eg. need to store backups from each hour, you can extend this structure using `/etc/local/.config/backup.index` file, which should contain the date mask, eg.:



```

%Y%m%d/%H

```



### Backup transfer process



Backup collector fetches backup files from remote hosts using `scp` and dedicated ssh keys (separate key for each host), for `backup` system user.



### Push-to-collector mode



In default scenario, backup collector has full shell access to managed hosts, as `backup` system user, including to *farm manager*. In high-compliance setups, eg. PCI DSS, this is not acceptable, as this is just one step away from having access to management ssh keys located on *farm manager*.



Because of that, script `cron/push-to-collector.sh` in `sf-backup` extension allows transferring farm manager encrypted backups in other direction: from farm manager to backup collector. This way, backup collector no longer needs ssh access to farm manager, which .