{"id":19029860,"url":"https://github.com/guardian/smartbackup","last_synced_at":"2026-05-16T18:34:28.653Z","repository":{"id":76012737,"uuid":"221791765","full_name":"guardian/smartbackup","owner":"guardian","description":"Simple utility to make backing up Postgres databases on Netapp volumes easy","archived":false,"fork":false,"pushed_at":"2020-06-18T09:58:29.000Z","size":66,"stargazers_count":1,"open_issues_count":2,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-07-18T20:17:07.837Z","etag":null,"topics":["backup","database-backup-solutions","netapp","postgres","postgres-replication","postgresql"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guardian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-14T21:41:51.000Z","updated_at":"2020-07-30T16:05:56.000Z","dependencies_parsed_at":"2023-02-26T04:45:37.731Z","dependency_job_id":null,"html_url":"https://github.com/guardian/smartbackup","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/guardian/smartbackup","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guardian%2Fsmartbackup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guardian%2Fsmartbackup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guardian%2Fsmartbackup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guardian%2Fsmartbackup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guardian","download_url":"https://codeload.github.com/guardian/smartbackup/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guardian%2Fsmartbackup/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267604735,"owners_count":24114548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-28T02:00:09.689Z","response_time":68,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","database-backup-solutions","netapp","postgres","postgres-replication","postgresql"],"created_at":"2024-11-08T21:15:36.297Z","updated_at":"2026-05-16T18:34:23.618Z","avatar_url":"https://github.com/guardian.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# smartbackup\n\nBackup app for Postgres databases using NetApp backed storage\n\n## Why? What does it do?\n\nDatabase backups are a pain. Traditionally you run pg_dump (or equivalent) to make a huge SQL file that can be replayed,\nand compress it.  It takes ages, needs loads of disk, and takes even longer to replay.\n\nIn our Multimedia system the database is over 600G and this becomes so problematic as to make the backup almost useless.\n\nFortunately, Netapp has a much better solution - snapshotting.\n\nWe can tell the Netapp to preserve the current storage blocks as a \"snapshot\" - no data is copied, only locked, so this \nis a fairly quick operation.\n\nOf course, the problem is that the database is still writing.  Under normal circumstances for this to work (i.e. for us to\nbe able to actually restart a b0rked system from the snapshot) the app in question must be \"quiesced\" i.e. prevented from\nwriting during the snapshot window.\n\nFortunately, Postgres is clever about this; it does not need to be prevented from writing but it _does_ need to create a\n\"consistent restore point\".  When an instance is brought up from the snapshot it discards data from after the \"restore point\"\nas potentially corrupt and attempts to replay as much as possible from any source of write-ahead logs available\n(see the Postgres documentation for many, many, more details on this).\n\nSo, our backup process becomes:\n\n```\n    ----------------------      ------------------      ---------------------------------------\n   | Set db restore point | -\u003e | Perform snapshot | -\u003e | Tell database that backup is complete |\n    ----------------------      ------------------      ---------------------------------------\n```\n\nThis is a simple app to manage this process\n\n## SnapCenter\n\nNetapp offer a product which can do this process for a whole range of databases and present a nice UI along with it.\nIndividual databases are supported by plugins which sit in a Java agent that needs installing on the database box.\n\nUnfortunately we have had a lot of problems:\n - the logs are difficult to read\n - the plugin installation and running has trouble with selinux\n - extra firewall holes are required for the agent\n - you can't use Ansible, Salt or any other config management platform to deploy easily, you're meant to do it through their UI\n - Postgres is a \"community supported\" plugin which uses shell script, embedded in Perl, called from Java, called from a server....\n - In a nutshell, we could not get it to work reliably.\n \nBut then, Netapp released OnTap 9.6 which brings with it a REST API.  And so a simpler way was born....\n\n\n## How does it work?\n\n1. Provide a config file - see `example-config.yaml` in the repo root for details on how this must be formatted and what\ninformation is required\n2. The config file gives the login details for Netapp appliances/SVMs and Postgres databases and a list of \"targets\",\ni.e. what needs to be backed up from the above lists\n3. Run `./smartbackup`.\n4. Each target is run through in turn, a restore point set, a snapshot made and the backup completed.\n5. - If a restore point fails, then that backup fails and we move on to the next. \n   - If the snapshot fails then the backup is still completed.\n   - If the backup complete operation fails then it is just logged.\n   \n## Where do i install it?\n\nWherever you want!  It accesses both NetApp and Postgres over network connections so it can run anywhere that connectivity\nis availabile.  However, you need to include (limited) plaintext credentials in the configuration files so it should be\nhosted somewhere secure in your environment.  There is no reason NOT to run it locally on your database server and in\nsome cases this might indeed be the best place to do it since you can use the local \"postgres\" admin privilege to avoid\nextra user accounts.  On the other hand, a secure little place or a Kubernetes cluster would work equally well.\n\nIt's assumed that the app will be run from a task scheduler like cron\n\n## Do I need to build it from source?\n\nNope! You can download the latest Master build from here: https://s3.eu-west-1.amazonaws.com/gnm-multimedia-deployables/smartbackup/latest/smartbackup.zip\n\n## How do I build it, if I want to?\n\nYou'll need Go 1.11 or higher installed to build - https://golang.org/dl/\n\nWith that installed, simply clone this repository and run `go build` in the root.  This will give you a binary\ncalled `smartbackup` in the same directory, built for your local environment.\nFor (really easy!) information about cross-compiling see https://golangcookbook.com/chapters/running/cross-compiling/\n\n### TL;DR\n\nGOOS={youros} GOARCH={yourarch} go build\n\nSee https://gist.github.com/asukakenji/f15ba7e588ac42795f421b48b8aede63 for supported values\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguardian%2Fsmartbackup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguardian%2Fsmartbackup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguardian%2Fsmartbackup/lists"}