{"id":13441995,"url":"https://github.com/wal-g/wal-g","last_synced_at":"2025-05-06T02:20:42.250Z","repository":{"id":37390788,"uuid":"95143992","full_name":"wal-g/wal-g","owner":"wal-g","description":"Archival and Restoration for databases in the Cloud","archived":false,"fork":false,"pushed_at":"2025-04-22T06:16:08.000Z","size":104438,"stargazers_count":3505,"open_issues_count":315,"forks_count":475,"subscribers_count":61,"default_branch":"master","last_synced_at":"2025-04-22T14:56:07.568Z","etag":null,"topics":["archiving","aws","aws-s3","azure","azure-storage","backup","cloud","go","golang","greenplum","mariadb","mongodb","mysql","postgres","postgresql","redis","s3","sqlserver"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wal-g.png","metadata":{"files":{"readme":"docs/README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-06-22T18:16:19.000Z","updated_at":"2025-04-21T19:27:38.000Z","dependencies_parsed_at":"2023-02-19T09:31:01.760Z","dependency_job_id":"c0c778a5-3516-4cec-97ee-15053f975251","html_url":"https://github.com/wal-g/wal-g","commit_stats":{"total_commits":1547,"total_committers":227,"mean_commits":6.814977973568282,"dds":0.8875242404654169,"last_synced_commit":"5d8f67224398676fec4d6cc74aa460429bce0262"},"previous_names":[],"tags_count":62,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wal-g%2Fwal-g","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wal-g%2Fwal-g/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wal-g%2Fwal-g/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wal-g%2Fwal-g/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wal-g","download_url":"https://codeload.github.com/wal-g/wal-g/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252607345,"owners_count":21775459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archiving","aws","aws-s3","azure","azure-storage","backup","cloud","go","golang","greenplum","mariadb","mongodb","mysql","postgres","postgresql","redis","s3","sqlserver"],"created_at":"2024-07-31T03:01:40.449Z","updated_at":"2025-05-06T02:20:42.231Z","avatar_url":"https://github.com/wal-g.png","language":"Go","funding_links":[],"categories":["HarmonyOS","Go","Tools \u0026 Utilities","Backup and Migration","azure","Compiled list","Backups","Repositories"],"sub_categories":["Windows Manager","Backup \u0026 Replication","plv8:"],"readme":"# WAL-G\n![Docker-tests-status](https://github.com/wal-g/wal-g/workflows/Docker%20tests/badge.svg)\n![Unit-tests-status](https://github.com/wal-g/wal-g/workflows/Unit%20tests/badge.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/wal-g/wal-g)](https://goreportcard.com/report/github.com/wal-g/wal-g)\n[![Documentation Status](https://readthedocs.org/projects/wal-g/badge/?version=latest)](https://wal-g.readthedocs.io/?badge=latest)\n\n[This documentation is also available at wal-g.readthedocs.io](https://wal-g.readthedocs.io)\n\nWAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).\n\nWAL-G is the successor of WAL-E with a number of key differences. WAL-G uses LZ4, LZMA, ZSTD, or Brotli compression, multiple processors, and non-exclusive base backups for Postgres. More information on the original design and implementation of WAL-G can be found on the Citus Data blog post [\"Introducing WAL-G by Citus: Faster Disaster Recovery for Postgres\"](https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-faster-restores-for-postgres/).\n\n**Table of Contents**\n- [Installation](#installation)\n- [Configuration](#configuration)\n- [Usage](#usage)\n- [Databases](#databases)\n- [Development](#development)\n    - [Installing](#installing)\n    - [Testing](#testing)\n    - [Development on windows](#development-on-windows)\n- [Troubleshooting](#troubleshooting)\n- [Authors](#authors)\n- [License](#license)\n- [Acknowledgments](#acknowledgments)\n- [Chat](#chat)\n\nInstallation\n----------\nA precompiled binary for Linux AMD 64 of the latest version of WAL-G can be obtained under the [Releases tab](https://github.com/wal-g/wal-g/releases).\n\nBinary name has the following format: `wal-g-DBNAME-OSNAME`, where `DBNAME` stands for the name of the database (for example pg, mysql), `OSNAME` stands for the name of the operating system used for building the binary.\n\nTo decompress the binary, use:\n\n```plaintext\ntar -zxvf wal-g-DBNAME-OSNAME-amd64.tar.gz\nmv wal-g-DBNAME-OSNAME-amd64 /usr/local/bin/wal-g\n```\n\nFor example, for Postgres and Ubuntu 18.04:\n```plaintext\ntar -zxvf wal-g-pg-ubuntu-18.04-amd64.tar.gz\nmv wal-g-pg-ubuntu-18.04-amd64 /usr/local/bin/wal-g\n```\n\nFor other systems, please consult the [Development](#development) section for more information.\n\nWAL-G supports bash and zsh autocompletion. Run `wal-g help completion` for more info.\n\nConfiguration\n-------------\n\nThere are two ways how you can configure WAL-G:\n\n1. Using environment variables\n\n2. Using a config file\n\n   `--config /path` flag can be used to specify the path where the config file is located.\n\n    We support every format that the [viper package](https://github.com/spf13/viper) supports: JSON, YAML, envfile and [others](https://github.com/spf13/viper#reading-config-files).\n\nEvery configuration variable mentioned in the following documentation can be specified either as an environment variable or a field in the config file.\n\n### Storage\nTo configure where WAL-G stores backups, please consult the [Storages](STORAGES.md) section.\n\n### Compression\n* `WALG_COMPRESSION_METHOD`\n\nTo configure the compression method used for backups. Possible options are: `lz4`, `lzma`, `zstd`, `brotli`. The default method is `lz4`. LZ4 is the fastest method, but the compression ratio is bad.\nLZMA is way much slower. However, it compresses backups about 6 times better than LZ4. Brotli and zstd are a good trade-off between speed and compression ratio, which is about 3 times better than LZ4.\n\n### Encryption\n\n* `YC_CSE_KMS_KEY_ID`\n\nTo configure Yandex Cloud KMS key for client-side encryption and decryption. By default, no encryption is used.\n\n* `YC_SERVICE_ACCOUNT_KEY_FILE`\n\nTo configure the name of a file containing private key of Yandex Cloud Service Account. If not set a token from the metadata service (http://169.254.169.254) will be used to make API calls to Yandex Cloud KMS.\n\n* `WALG_LIBSODIUM_KEY`\n\nTo configure encryption and decryption with libsodium. WAL-G uses an [algorithm](https://download.libsodium.org/doc/secret-key_cryptography/secretstream#algorithm) that only requires a secret key. libsodium keys are fixed-size keys of 32 bytes. For optimal cryptographic security, it is recommened to use a random 32 byte key. To generate a random key, you can something like `openssl rand -hex 32` (set `WALG_LIBSODIUM_KEY_TRANSFORM` to `hex`) or `openssl rand -base64 32` (set `WALG_LIBSODIUM_KEY_TRANSFORM` to `base64`).\n\n* `WALG_LIBSODIUM_KEY_PATH`\n\nSimilar to `WALG_LIBSODIUM_KEY`, but value is the path to the key on file system. The file content will be trimmed from whitespace characters.\n\n* `WALG_LIBSODIUM_KEY_TRANSFORM`\n\nThe transform that will be applied to the `WALG_LIBSODIUM_KEY` to get the required 32 byte key. Supported transformations are `base64`, `hex` or `none` (default).\nThe option `none` exists for backwards compatbility, the user input will be converted to 32 byte either via truncation or by zero-padding.\n\n* `WALG_GPG_KEY_ID`  (alternative form `WALE_GPG_KEY_ID`) ⚠️ **DEPRECATED**\n\nTo configure GPG key for encryption and decryption. By default, no encryption is used. Public keyring is cached in the file \"/.walg_key_cache\".\n\n* `WALG_PGP_KEY`\n\nTo configure encryption and decryption with OpenPGP standard. You can join multiline key using `\\n` symbols into one line (mostly used in case of daemontools and envdir).\nSet *private key* value when you need to execute ```wal-fetch``` or ```backup-fetch``` command.\nSet *public key* value when you need to execute ```wal-push``` or ```backup-push``` command.\nKeep in mind that the *private key* also contains the *public key*.\n\n* `WALG_PGP_KEY_PATH`\n\nSimilar to `WALG_PGP_KEY`, but value is the path to the key on file system.\n\n* `WALG_PGP_KEY_PASSPHRASE`\n\nIf your *private key* is encrypted with a *passphrase*, you should set *passphrase* for decrypt.\n\n* `WALG_ENVELOPE_PGP_KEY`\nTo configure encryption and decryption with the envelope PGP key stored in key management system.\nThis option allows you to securely manage your PGP keys by storing them in the KMS.\nIt is crucial to ensure that the key passed is encrypted using kms and encoded with *base64*.\nAlso both *private* and *publlic* parts should be presents in key because envelope key will be injected in metadata and used later in `wal/backup-fetch`.\n\nPlease note that currently, only Yandex Cloud Key Management Service (KMS) is supported for configuring.\nEnsure that you have set up and configured Yandex Cloud KMS mentioned below before attempting to use this feature.\n\n* `WALG_ENVELOPE_CACHE_EXPIRATION`\n\nThis setting controls kms response expiration. Default value is `0` to store keys permanent in memory.\nPlease note that if the system will not be able to redecrypt the key in kms after expiration, the previous response will be used.\n\n* `WALG_ENVELOPE_PGP_YC_ENDPOINT`\n\nEndpoint is an API endpoint of Yandex.Cloud against which the SDK is used. Most users won't need to explicitly set it.\n\n* `WALG_ENVELOPE_PGP_YC_CSE_KMS_KEY_ID`\n\nSimilar to `YC_CSE_KMS_KEY_ID`, but only used for envelope pgp keys.\n\n* `WALG_ENVELOPE_PGP_YC_SERVICE_ACCOUNT_KEY_FILE`\n\nSimilar to `YC_SERVICE_ACCOUNT_KEY_FILE`, but only used for envelope pgp keys.\n\n* `WALG_ENVELOPE_PGP_KEY_PATH`\n\nSimilar to `WALG_ENVELOPE_PGP_KEY`, but value is the path to the key on file system.\n\n\n### Monitoring\n\n* `WALG_STATSD_ADDRESS`\n\nTo enable metrics publishing to [statsd](https://github.com/statsd/statsd) or [statsd_exporter](https://github.com/prometheus/statsd_exporter). Metrics will be sent on a best-effort basis via UDP. The default port for statsd is `8125`.\n\n* `WALG_STATSD_EXTRA_TAGS`\n\nUse this setting to add static tags (`host`, `operation`, `database`, etc) to the metrics WAL-G publishes to statsd.\n\nIf you want to make demo for testing purposes, you can use graphite service from docker-compose file.\n\n### Profiling\n\nProfiling is useful for identifying bottlenecks within WAL-G.\n\n* `PROFILE_SAMPLING_RATIO`\n\nA float value between 0 and 1, defines likelihood of the profiler getting enabled. When set to 1, it will always run. This allows probabilistic sampling of invocations. Since WAL-G processes may get created several times per second (e.g. wal-g wal-push), we do not want to profile all of them.\n\n* `PROFILE_MODE`\n\nThe type of pprof profiler to use. Can be one of `cpu`, `mem`, `mutex`, `block`, `threadcreation`, `trace`, `goroutine`. See the [runtime/pprof docs](https://pkg.go.dev/runtime/pprof) for more information. Defaults to `cpu`.\n\n* `PROFILE_PATH`\n\nThe directory to store profiles in. Defaults to `$TMPDIR`.\n\n### Rate limiting\n* `WALG_NETWORK_RATE_LIMIT`\n\nNetwork traffic rate limit during the ```backup-push```/```backup-fetch``` operations in bytes per second.\n\n\n### Database-specific options\n**More options are available for the chosen database. See it in [Databases](#databases)**\n\nUsage\n-----\n\nWAL-G currently supports these commands for all type of databases:\n\n### ``backup-list``\n\nLists names and creation time of available backups.\n\n``--pretty``  flag prints list in a table\n\n``--json`` flag prints list in JSON format, pretty-printed if combined with ``--pretty``\n\n``--detail`` flag prints extra backup details, pretty-printed if combined with ``--pretty``, json-encoded if combined with ``--json``\n\n### ``delete``\n\nIs used to delete backups and WALs before them. By default, ``delete`` will perform a dry run. If you want to execute deletion, you have to add ``--confirm`` flag at the end of the command. Backups marked as permanent will not be deleted.\n\n``delete`` can operate in four modes: ``retain``, ``before``, ``everything`` and ``target``.\n\n``retain`` [FULL|FIND_FULL] %number% [--after %name|time%]\n\nif ``FULL`` is specified, keep ``%number%`` full backups and everything in the middle. If with ``--after`` flag is used keep\n$number$ the most recent backups and backups made after ``%name|time%`` (including).\n\n``before`` [FIND_FULL] %name%\n\nIf `FIND_FULL` is specified, WAL-G will calculate minimum backup needed to keep all deltas alive. If ``FIND_FULL`` is not specified, and call can produce orphaned deltas, the call will fail with the list.\n\n``everything`` [FORCE]\n\n``target`` [FIND_FULL] %name% | --target-user-data %data% will delete the backup specified by name or user data. Unlike other delete commands, this command does not delete any archived WALs.\n\n(Only in Postgres \u0026 MySQL) By default, if delta backup is provided as the target, WAL-G will also delete all the dependant delta backups. If `FIND_FULL` is specified, WAL-G will delete all backups with the same base backup as the target.\n\n### Examples\n\n``everything`` all backups will be deleted (if there are no permanent backups)\n\n``everything FORCE`` all backups, include permanent, will be deleted\n\n``retain 5`` will fail if 5th is delta\n\n``retain FULL 5`` will keep 5 full backups and all deltas of them\n\n``retain FIND_FULL 5`` will find necessary full for 5th and keep everything after it\n\n``retain 5 --after 2019-12-12T12:12:12`` keep 5 most recent backups and backups made after 2019-12-12 12:12:12\n\n``before base_000010000123123123`` will fail if `base_000010000123123123` is delta\n\n``before FIND_FULL base_000010000123123123`` will keep everything after base of base_000010000123123123\n\n``target base_0000000100000000000000C9`` delete the base backup and all dependant delta backups\n\n``  target --target-user-data \"{ \\\"x\\\": [3], \\\"y\\\": 4 }\"``     delete backup specified by user data\n\n``target base_0000000100000000000000C9_D_0000000100000000000000C4``    delete delta backup and all dependant delta backups\n\n``target FIND_FULL base_0000000100000000000000C9_D_0000000100000000000000C4`` delete delta backup and all delta backups with the same base backup\n\n**More commands are available for the chosen database engine. See it in [Databases](#databases)**\n\n## Storage tools\n`wal-g st` command series allows the direct interaction with the configured storage.\n[Storage tools documentation](StorageTools.md)\n\nDatabases\n-----------\n### PostgreSQL\n[Information about installing, configuration and usage](PostgreSQL.md)\n\n### MySQL/MariaDB\n[Information about installing, configuration and usage](MySQL.md)\n\n### SQLServer\n[Information about installing, configuration and usage](SQLServer.md)\n\n### Mongo [Beta]\n[Information about installing, configuration and usage](MongoDB.md)\n\n### FoundationDB [Work in progress]\n[Information about installing, configuration and usage](FoundationDB.md)\n\n### Redis [Beta]\n[Information about installing, configuration and usage](Redis.md)\n\n### Greenplum [Production ready]\n[Information about installing, configuration and usage](Greenplum.md)\n\n### ETCD [Work in progress]\n[Information about installing, configuration and usage](ETCD.md)\n\nDevelopment\n-----------\n\nThe following steps describe how to build WAL-G for PostgreSQL, but the process is the same for other databases. For example, to build WAL-G for MySQL, use the `make mysql_build` instead of `make pg_build`.\n\nOptional:\n\n- To build with brotli compressor and decompressor, set the `USE_BROTLI` environment variable.\n- To build with libsodium, set the `USE_LIBSODIUM` environment variable.\n- To build with lzo decompressor, set the `USE_LZO` environment variable.\n\n### Installing\n\n#### Ubuntu\n\n```sh\n# Install latest Go compiler\nsudo add-apt-repository ppa:longsleep/golang-backports\nsudo apt update\nsudo apt install golang-go\n\n# Install lib dependencies\nsudo apt install libbrotli-dev liblzo2-dev libsodium-dev curl cmake\n\n# Fetch project and build\n# Go 1.15 and below\ngo get github.com/wal-g/wal-g\n# Go 1.16+ - just clone repository to $GOPATH\n# if you want to save space add --depth=1 or --single-branch\ngit clone https://github.com/wal-g/wal-g $(go env GOPATH)/src/github.com/wal-g/wal-g\n\ncd $(go env GOPATH)/src/github.com/wal-g/wal-g\n\n# optional exports (see above)\nexport USE_BROTLI=1\nexport USE_LIBSODIUM=1\nexport USE_LZO=1\n\nmake deps\nmake pg_build\nmain/pg/wal-g --version\n```\n\nUsers can also install WAL-G by using `make pg_install`. Specifying the `GOBIN` environment variable before installing allows the user to specify the installation location. By default, `make pg_install` puts the compiled binary in the root directory (`/`).\n\n```sh\nexport USE_BROTLI=1\nexport USE_LIBSODIUM=1\nexport USE_LZO=1\nmake pg_clean\nmake deps\nGOBIN=/usr/local/bin make pg_install\n```\n\n#### macOS\n\n```sh\n# brew command is Homebrew for Mac OS\nbrew install cmake\n\n# Fetch project and build\n# Go 1.15 and below\ngo get github.com/wal-g/wal-g\n# Go 1.16+ - just clone repository to $GOPATH\n# if you want to save space add --depth=1 or --single-branch\ngit clone https://github.com/wal-g/wal-g $(go env GOPATH)/src/github.com/wal-g/wal-g\n\ncd $(go env GOPATH)/src/github.com/wal-g/wal-g\n\nexport USE_BROTLI=1\nexport USE_LIBSODIUM=\"true\" # since we're linking libsodium later\n./link_brotli.sh\n./link_libsodium.sh\nmake install_and_build_pg\n\n# if you need to install\nGOBIN=/usr/local/bin make pg_install\n```\n\nTo build on ARM64, set the corresponding `GOOS`/`GOARCH` environment variables:\n```\nenv GOOS=darwin GOARCH=arm64 make install_and_build_pg\n```\n\nTo build linux build on ARM MacOS:\n```\nGOOS=linux GOARCH=amd64 make mysql_build\n```\n\nThe compiled binary to run is `main/pg/wal-g`\n\n### Testing\n\nWAL-G relies heavily on unit tests. These tests do not require S3 configuration as the upload/download parts are tested using mocked objects. Unit tests can be run using\n```bash\nexport USE_BROTLI=1\nmake unittest\n```\nFor more information on testing, please consult [test](test), [testtools](testtools) and `unittest` section in [Makefile](Makefile).\n\nWAL-G will perform a round-trip compression/decompression test that generates a directory for data (e.g., data...), compressed files (e.g., compressed), and extracted files (e.g., extracted). These directories will only get cleaned up if the files in the original data directory match the files in the extracted one.\n\nTest coverage can be obtained using:\n```bash\nexport USE_BROTLI=1\nmake coverage\n```\nThis command generates `coverage.out` file and opens HTML representation of the coverage.\n\n### Development on Windows\n\n[Information about installing and usage](Windows.md)\n\n\nTroubleshooting\n---------------\n\nA good way to start troubleshooting problems is by setting one or both of these environment variables:\n\n* `WALG_LOG_LEVEL=DEVEL`\n\nPrints out the used configuration of WAL-G and detailed logs of the used command.\n\n* `S3_LOG_LEVEL=DEVEL`\n\nIf your commands seem to be stuck it could be that the S3 is not reachable, certificate problems or other S3 related issues.\nWith this environment variable set you can see the Requests and Responses from S3.\n\n\nAuthors\n-------\n\n* [Katie Li](https://github.com/katie31)\n* [Daniel Farina](https://github.com/fdr)\n\nSee also the list of [contributors](CONTRIBUTORS.md) who participated in this project.\n\nLicense\n-------\n\nThis project is licensed under the Apache License, Version 2.0, but the lzo support is licensed under GPL 3.0+. Please refer to the [LICENSE.md](../LICENSE.md) file for more details.\n\nAcknowledgments\n----------------\nWAL-G would not have happened without the support of [Citus Data](https://www.citusdata.com/)\n\nWAL-G came into existence as a result of the collaboration between a summer engineering intern at Citus, Katie Li, and Daniel Farina, the original author of WAL-E, who currently serves as a principal engineer on the Citus Cloud team. Citus Data also has an [open-source extension to Postgres](https://github.com/citusdata) that distributes database queries horizontally to deliver scale and performance.\n\nWAL-G development is supported by [Yandex Cloud](https://cloud.yandex.com)\n\nChat\n----\nWe have a [Slack group](https://postgresteam.slack.com/messages/CA25P48P2) and [Telegram chat](https://t.me/joinchat/C03q9FOwa7GgIIW5CwfjrQ) to discuss WAL-G usage and development. To join PostgreSQL slack, use [invite app](https://postgres-slack.herokuapp.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwal-g%2Fwal-g","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwal-g%2Fwal-g","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwal-g%2Fwal-g/lists"}