{"id":20547707,"url":"https://github.com/corpusops/db_smart_backup","last_synced_at":"2025-05-09T15:30:45.855Z","repository":{"id":13497694,"uuid":"16188369","full_name":"corpusops/db_smart_backup","owner":"corpusops","description":"***UNMAINTAINED/ABANDONED CODE / DO NOT USE *** Another database(db) smart backup tool (mysql/postgresql/slapd/mongo/es/redis)","archived":true,"fork":false,"pushed_at":"2024-12-28T21:48:22.000Z","size":197,"stargazers_count":26,"open_issues_count":1,"forks_count":8,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-15T11:30:46.370Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://www.makina-corpus.com","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/corpusops.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-01-23T22:51:28.000Z","updated_at":"2025-02-28T03:01:51.000Z","dependencies_parsed_at":"2025-01-07T14:50:39.877Z","dependency_job_id":"06167d1c-94fc-4c66-85c4-b8b4d5b2cd45","html_url":"https://github.com/corpusops/db_smart_backup","commit_stats":null,"previous_names":["corpusops/db_smart_backup"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/corpusops%2Fdb_smart_backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/corpusops%2Fdb_smart_backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/corpusops%2Fdb_smart_backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/corpusops%2Fdb_smart_backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/corpusops","download_url":"https://codeload.github.com/corpusops/db_smart_backup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253275576,"owners_count":21882326,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T02:09:47.322Z","updated_at":"2025-05-09T15:30:45.816Z","avatar_url":"https://github.com/corpusops.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"=====================================================\nBackup Script for various databases:\n=====================================================\nSimple dump based backup with intelligent rotation and hooks.\nSupports with battery included for mysql, mongodb, slapd \u0026 postgresql\n\n\nDISCLAIMER - ABANDONED/UNMAINTAINED CODE / DO NOT USE\n=======================================================\nminitage project was terminated in 2013. Consequently, this repository and all associated resources (including related projects, code, documentation, and distributed packages such as Docker images, PyPI packages, etc.) are now explicitly declared **unmaintained** and **abandoned**.\n\nI would like to remind everyone that this project’s free license has always been based on the principle that the software is provided \"AS-IS\", without any warranty or expectation of liability or maintenance from the maintainer.\nAs such, it is used solely at the user's own risk, with no warranty or liability from the maintainer, including but not limited to any damages arising from its use.\n\nDue to the enactment of the Cyber Resilience Act (EU Regulation 2024/2847), which significantly alters the regulatory framework, including penalties of up to €15M, combined with its demands for **unpaid** and **indefinite** liability, it has become untenable for me to continue maintaining all my Open Source Projects as a natural person.\nThe new regulations impose personal liability risks and create an unacceptable burden, regardless of my personal situation now or in the future, particularly when the work is done voluntarily and without compensation.\n\n**No further technical support, updates (including security patches), or maintenance, of any kind, will be provided.**\n\nThese resources may remain online, but solely for public archiving, documentation, and educational purposes.\n\nUsers are strongly advised not to use these resources in any active or production-related projects, and to seek alternative solutions that comply with the new legal requirements (EU CRA).\n\n**Using these resources outside of these contexts is strictly prohibited and is done at your own risk.**\n\nRegarding the potential transfer of the project to another entity, discussions are ongoing, but no final decision has been made yet. As a last resort, if the project and its associated resources are not transferred, I may begin removing any published resources related to this project (e.g., from PyPI, Docker Hub, GitHub, etc.) starting **March 15, 2025**, especially if the CRA’s risks remain disproportionate.\n\n\n.. contents::\n\n\nBadges\n------\n\n.. image:: https://travis-ci.org/kiorky/db_smart_backup.png\n    :target: http://travis-ci.org/kiorky/db_smart_backup\n\nSupported databases\n-------------------\n    - Mongodb\n    - PostGRESQL\n    - Redis\n    - Elasticsearch\n    - MySQL\n    - slapd (OpenLDAP)\n\nWhy another tool ?\n--------------------\n- There are great tools out there, but they are not fitting our needs and\n  knowledge, and some of them did not have much tests, sorry.\n- We just wanted a simple bash script, and using **dumps** (even in custom format\n  for postgres) but just snapshots. So for example, postgreSQL PITR wal were not an\n  option eliminating btw *barman* \u0026 *pg_rman*. All the other shell scripts including\n  *automysqlbackup*/*autopostgresql* were not fitting exactly all the features we\n  wanted and some were just too bash complicated for our little own brains.\n- We wanted hooks to react on each backup stage, those hooks can be in another\n  language, this is up to the user (very usefull for monitoring stuff).\n- We want a generic script for any database, providing you add support on\n  it, this consists just on writing a 'global' and a 'dump' function. For more\n  information, read the sources luke.\n\n- **WARNING**\n  DO NOT PUT DATA UNDER THE DATADIR ELSE THAN WHAT DBSMARTBACKUP\n\nSo main features/requirements are:\n\n    - Posix shell compliant (goal, but not that tested, the really tested one\n      is bash in posix mode)\n    - **PostgreSQL / MySQL support** for simple database and privileges\n      dumps\n    - Enougthly unit **tested**\n    - XZ **compression** if available\n    - Easily **extensible** to add another backup type / Generic backups methods\n    - **Optional hooks** at each stage of the process addable via configuration\n      (bash functions to uncomment)\n    - **Keep a fixed number of dumps**, recent ones, old ones, and in a smart way.\n      More on that later on this document. But for example the default is to keep\n      the last 24 dumps, then 14 days (1 per day), 8 weeks (1 per week) and 12\n      months (1 per month).\n\n\nInstallation\n------------\n::\n\n    curl -OJLs https://raw.githubusercontent.com/kiorky/db_smart_backup/master/db_smart_backup.sh\n    curl -OJLs https://raw.githubusercontent.com/kiorky/db_smart_backup/master/run_dbsmartbackups.sh\n    chmod +x db_smart_backup.sh run_dbsmartbackups.sh\n\nGenerate a config file::\n\n    ./db_smart_backup.sh --gen-config /path/to/config\n    vim /path/to/config\n\nBackup::\n\n    ./db_smart_backup.sh /path/to/config\n\n\n\nBackup all found databases in cron\n-----------------------------------\nWe also bundle a script named **run_dbsmartbackups.sh** which search in /etc/dbsmartbackup for any database configuration:\n\n    -  pg: /etc/dbsmartbackup/postgresql.conf\n    -  mysql: /etc/dbsmartbackup/mysql.conf\n    -  mongodb: /etc/dbsmartbackup/mongod.conf\n    -  slapd /etc/dbsmartbackup/slapd.conf\n    -  redis /etc/dbsmartbackup/redis.conf\n    -  elasticsearch /etc/dbsmartbackup/elasticsearch.conf\n\nbe sure to have the scripts in your path::\n\n    curl -OJLs https://raw.githubusercontent.com/kiorky/db_smart_backup/master/db_smart_backup.sh\n    curl -OJLs https://raw.githubusercontent.com/kiorky/db_smart_backup/master/run_dbsmartbackups.sh\n    chmod +x db_smart_backup.sh run_dbsmartbackups.sh\n    mkdir /etc/dbsmartbackup\n\nIn /etc/dbsmartbackup, generate a config file (either: mysql.conf, mongod.conf, slapd.conf, postgresql.conf)::\n\n    ./db_smart_backup.sh --gen-config /etc/dbsmartbackup/\u003cdb_type\u003e.conf\n    vim /path/to/configa\n\nTesting the backup::\n\n    ./db_smart_backup.sh /etc/dbsmartbackup/\u003cdb_type\u003e.conf\n\nOnly execute the pruning policy::\n\n    ./db_smart_backup.sh -p /etc/dbsmartbackup/\u003cdb_type\u003e.conf\n\nTest the cron that search for all possible things to backups::\n\n    run_dbsmartbackups.sh\n\nAdd it to cron::\n\n    0 0 * * * root /usr/bin/run_dbsmartbackups.sh --no-colors --quiet\n\n\nFor postgresql, you can configure the path to your postgresql.conf(s) PATH(s) by\nexporting \"PG_CONFS\" that is a space separated absolute paths to\npostgresql.conf's.\nNote, that for redhat or debian based, PG_CONFS should be OK by default.\n\nChangelog\n----------\n\nCredits\n-------------\n  - by Mathieu Le Marec - Pasquet / kiorky@cryptelium.net\n  - inspired by automysqlbackup/autopostgresqlbackup\n\nThe great things\n-----------------\n- Hooks support for each stage, those are bash functions acting as entry point\n  for you to customize the backup upon what will happen during execution\n- Smart idiot and simple retention policies\n  Idea is to have a directory with all the sql for all days of the year\n  and then hard links in subdirs to those files for easy access\n  but also to triage what to rotate and what to prune::\n\n    POSTGRESQL/\n     DBNAME/\n      dumps/\n        DBNAME_20xx0101_01-01-01.sql.compressed  \u003c- 01/01/20xx\n        DBNAME_20xx0102_01-01-01.sql.compressed\n        DBNAME_20xx0103_01-01-01.sql.compressed\n        DBNAME_20xx0107_01-01-01.sql.compressed\n        DBNAME_20xx0108_01-01-01.sql.compressed\n        DBNAME_20xx3101_01-01-01.sql.compressed\n        DBNAME_20xx0202_01-01-01.sql.compressed\n      lastsnapshots/\n        DBNAME_20xx0101_01-01-01.sql.compressed\n        DBNAME_20xx0102_01-01-01.sql.compressed\n        DBNAME_20xx0202_01-01-01.sql.compressed\n      monthly/\n        20xx_01_DBNAME_20xx0101.sql.compressed -\u003e /fullpath/DBNAME/dumps/DBNAME_20xx0101.sql.compressed\n        20xx_02_DBNAME_20xx0201.sql.compressed -\u003e /fullpath/DBNAME/dumps/DBNAME_20xx0202.sql.compressed\n        20xx_03_DBNAME_20xx0301.sql.compressed -\u003e /fullpath/DBNAME/dumps/DBNAME_20xx0202.sql.compressed\n      weekly/\n        20xx_01_DBNAME_20xx0101.sql.compressed -\u003e /fullpath/DBNAME/dumps/DBNAME_20xx0101.sql.compressed\n        20xx_02_DBNAME_20xx0108.sql.compressed -\u003e /fullpath/DBNAME/dumps/DBNAME_20xx0108.sql.compressed\n      daily/\n        20xx_01_01_DBNAME_20xx0101.sql.compressed -\u003e /fullpath/DBNAME/dumps/DBNAME_20xx0101.sql.compressed\n        20xx_02_01_DBNAME_20xx0108.sql.compressed -\u003e /fullpath/DBNAME/dumps/DBNAME_20xx0108.sql.compressed\n\n- Indeed:\n\n    - First thing to do after after a backup is to look if a folder has more than the\n      configured backups per each type of rotation (month, week, days, snapshots)\n      and clean the oldest first.\n    - Then we will just have to prune hardlinks where linked count is stricly inferior to 2,\n      meaning that no one of the retention policies link this backup anymore. It\n      is what we can call an orphan and is willing to be pruned.\n    - Indeed, this means that **our backups are only in the dumps folder**.\n\n- How do I see that other directories contains only hard links from dump directory?\n\n    - You can see the hard links with ls in two ways. Using `ls -i` to get the\n      real inode number in first col or `ls -l` to get the hard link counters.\n::\n\n    # ls -il /var/backup/postgresql/localhost/foobar/dumps/\n    total 13332\n    14044 -rw-r----- 5 root root 1237208 22 mars  16:19 foobar_2014-03-22_16-19-34.sql\n    14049 -rw-r----- 2 root root 1237208 22 mars  16:25 foobar_2014-02-22_11-25-53.sql\n    14054 -rw-r----- 2 root root 1237208 22 mars  16:27 foobar_2014-01-22_15-27-22.sql\n    (...)\n    # ls -il /var/backup/postgresql/localhost/foobar/weekly/\n    total 1212\n    14044 -rw-r----- 5 root root 1237208 22 mars  16:19 foobar_2014_12.sql\n    ___^ inode       ^\n    _________________^ here we see the hard link counter on this file\n\n\n\nBackup types\n-------------\nPostgreSQL \u0026 MySQL specificities\n++++++++++++++++++++++++++++++++++++++++\n- We use traditionnal postgreSQL environment variables to set the host, port, password and user to set at backup\n  time\n\n- For PostgreSQL, you will certainly have to set only the BACKUP_TYPE to\n  postgresql\n- For MySQL you may have only to input the password\n\nAdd another backup type\n++++++++++++++++++++++++\nYou need to first read the implementations for **mysql** and **postgresql**, those are\nreally simple, then follow the next guide (you do not need to make the script\ncall your functions, they are introspected):\n\n    - Add a function **yourtype_set_connection_vars** to set any necessary extra global variable needed\n      at the connect phase to your service\n    - Add a function **yourtype_check_connectivity** that exit in error if the\n      connexion is not possible and die in error else (use the **die_in_error**\n      function)\n    - Add a function **yourtype_set_vars** to set any necessary extra global variable needed\n      to handle your service\n    - Add a function **yourtype_get_all_databases** that return a space separated\n      list of your database dbs.\n    - Add a function **yourtype_dump** that will dump a database to a file, or a\n      stub returning 0 as $? (call **/bin/true**) if it is not relevant for your\n      backup type.\n    - Add a function **yourtype_dumpall** even if one of them\n      is just an empty stub, the script will then introspect itself to find\n      them. Those functions must set the **LAST_BACKUP_STATUS** either to **\"\"**\n      on sucess or **\"failure\"** if the backup failed.\n    - Add what is needed to load the configuration in the default configuration\n      file in the **generate_configuration_file** method\n    - Hack the defaults and variables in **set_vars**, the same way, if\n      necessary.\n\nHooks\n---------\n- We provide a hook mechanism to let you configure custom code at each stage of\n  the backup program. For this, you just need to uncomment the relevant part in\n  your configuration file and implement whatever code you want, and even call\n  another script in another language.\n\n  - after the backup program starts: **pre_backup_hook**\n  - after the global backup(failure): **postglobalbackup_hook**\n  - after the global backup: **post_global_backup_failure_hook**\n  - after specific db backup: **post_dbbackup_hook**\n  - after specific db backup(failure): **post_db_backup_failure_hook**\n  - after the backups rotation: **post_rotate_hook**\n  - after the backups orphans cleanups: **post_cleanup_hook**\n  - at backup end: **post_backup_hook**\n\n- Think that you will have access in the environment of\n  the hook to all the variables defined and exported by the script.\n  You just have to check by reading the source what to test and how.\n\nOptions\n-----------\n- Read the script header to know what each option can do\n- You'll need to tweak at least:\n\n    - The database identifiers\n    - The backup root location (/var/backup/\u003ctype\u003e by default)\n    - Which type of backup to do (maybe only postgresql)\n    - The retention policy (there's a default one)\n\n\nBackup Rotation..\n------------------\nWe use hardlinks to achieve that but be aware that it may have filesystem limits:\n    - number of databases backed up (a lot if every possible anyway on modern filesystems (2^32 hardlinks)\n      and count something for the max like **366x2+57+12** for a year and a db.\n    - and all subdirs should be on the same mounted point than the **dumps** directory.\n\nDefault policy\n++++++++++++++\n- We keep the **24** last done dumps\n- We keep **14** days left\n- We keep 1 backup per week for the last **8** weeks\n- We keep 1 backup per month for the last **12** months\n\nPlease Note!!\n--------------\nI take no responsability for any data loss or corruption when using this script..\nThis script will not help in the event of a hard drive crash. If a\ncopy of the backup has not be stored offline or on another PC..\nYou should copy your backups offline regularly for best protection.\nHappy backing up...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcorpusops%2Fdb_smart_backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcorpusops%2Fdb_smart_backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcorpusops%2Fdb_smart_backup/lists"}