{"id":19979460,"url":"https://github.com/msteinhoff/backup-tools","last_synced_at":"2025-10-29T16:34:46.697Z","repository":{"id":32495650,"uuid":"36076303","full_name":"msteinhoff/backup-tools","owner":"msteinhoff","description":"Shell scripts to create backups for our internal systems","archived":false,"fork":false,"pushed_at":"2015-09-28T20:23:18.000Z","size":200,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-12T10:28:50.399Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/msteinhoff.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-05-22T14:23:24.000Z","updated_at":"2017-07-06T16:31:52.000Z","dependencies_parsed_at":"2022-09-05T11:21:53.714Z","dependency_job_id":null,"html_url":"https://github.com/msteinhoff/backup-tools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msteinhoff%2Fbackup-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msteinhoff%2Fbackup-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msteinhoff%2Fbackup-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msteinhoff%2Fbackup-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/msteinhoff","download_url":"https://codeload.github.com/msteinhoff/backup-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241412958,"owners_count":19958995,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T03:37:51.279Z","updated_at":"2025-10-29T16:34:41.662Z","avatar_url":"https://github.com/msteinhoff.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# catapullt backup tools\n\n**WARNING: WORK IN PROGRESS, DO NOT USE YET**\n\nThis repository contains a set of backup scripts to backup user generated\napplication data of all applications currently used at catapullt. Being a small\ncompany with \u003c 10 employees, our system setup is very simple and so are the\nbackup scripts.\n\n# Backup design\n\nAt catapullt, we currently run a bunch of virtualized applications to support\nour project management and developers.\n\nTo avoid having to backup large amounts of data, we decided on a hybrid approach\nof classic system administration and infrastructure as code:\n\nIaaC:\n\n- Host systems are provisioned semi-automatic using a custom system.\n- Host systems are configured using ansible roles.\n- All provisioning information and configuration is version controlled.\n- Any configuration that is on the server but not in an ansible role is seen as\nnon-existent.\n- Ansible application roles only prepare a runtime environment (e.g.\ndependencies and special system configuration) for the applications.\n\nClassic system administration:\n\n- Applications themselves are installed and upgraded manually, installation and\nupgrade procedures are documented in simple text files.\n\nWith this system we only need to backup actual application data. as all host VMs\ncan be re-instanciated from configuration/code in a very short timeframe. Full\nVM backups are not necessary as we do not have high DR requirements.\n\n## Application data backup\n\nAn application data backup usually includes a database schema that must be\ndumped to disk and optional some files not stored in the database (e.g.\nConfluence attachments) that must be copied somewhere. To avoid version problems\non restore, we also write the application's current version to the backup\narchive.\n\nAll backup jobs create a backup archive which is stored in a temporary work\ndirectory. The default is to use a directory on the backed up host but in a\ndifferent place than the application's home directory. The location is\nconfigurable and its possible to also write directly to e.g. an NFS folder.\n\nIt is expected that the applications run on hosts with ECC RAM to prevent memory\ncorruption. If the backup is written to the local disk and the disk corrupts\nyour files, you are screwed. ¯\\_(ツ)_/¯\n\nAfter a backup archive has been created, a SHA1 checksum of the backup archive\nis calculated and stored beneath the file.\n\nAs the backup is worthless when it stays on the backed up host, it should be\npost-processed afterwards. Possible options include:\n\n- A script on the backed up host copies the data to another backup host, then\nprunes the local backup directory\n\n- A cron job on some backup host regularly polls all to-be-backed-up systems,\nnew backup archives are then pulled and removed\n\n# Components\n\n- `/agents`\n\n    For each application, a backup agent is installed on the application host\n    local backup. The agent only perform a backup to disk and log to stdout or\n    stderr. Configuration is done via config file and/or environment variables.\n\n- `/notifier`\n\n    The notifier is installed on all backed up hosts. It wraps the execution of\n    a backup agent and notifies a third party about successful and failed\n    backups. This way, an administrator can see if the backup was successful,\n    failed, or did not run at all.\n\n- `/archiver`\n\n    The archiver is installed on a backup host and is used to pull data off the\n    application hosts.\n\n# Wishlist\n\n- Support other DBMS than MySQL for Atlassian applications\n- Other notification transports than e-mail (e.g. HipChat)\n- Time-series logging of backup sizes, notification on variations\n- GPG signed and/or encrypted backup archives\n\n# Contributing\n\nYou have found a bug or want to improvement something? Awesome! :)\n\nTo avoid frustration, please do not just fork the repository and start sending\nPRs. Of course nobody prevents you from doing this, but I might reject it.\n\nInstead, create an issue to start a discussion on what you'd like to achieve and\nthen we can see if it makes sense to implement here. :)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsteinhoff%2Fbackup-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmsteinhoff%2Fbackup-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsteinhoff%2Fbackup-tools/lists"}