{"id":30873467,"url":"https://github.com/pgedge/pg_walsizer","last_synced_at":"2025-09-07T23:10:54.899Z","repository":{"id":310061977,"uuid":"1038118455","full_name":"pgEdge/pg_walsizer","owner":"pgEdge","description":"The pg_walsizer Postgres Extension","archived":false,"fork":false,"pushed_at":"2025-08-15T12:36:46.000Z","size":8,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-15T14:47:36.705Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"postgresql","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pgEdge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-14T16:44:13.000Z","updated_at":"2025-08-15T12:36:49.000Z","dependencies_parsed_at":"2025-08-15T14:47:40.553Z","dependency_job_id":"b0893f69-f8a6-477b-ad50-23e181d77ba2","html_url":"https://github.com/pgEdge/pg_walsizer","commit_stats":null,"previous_names":["pgedge/pg_walsizer"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/pgEdge/pg_walsizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgEdge%2Fpg_walsizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgEdge%2Fpg_walsizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgEdge%2Fpg_walsizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgEdge%2Fpg_walsizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pgEdge","download_url":"https://codeload.github.com/pgEdge/pg_walsizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgEdge%2Fpg_walsizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274107977,"owners_count":25223473,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-07T02:00:09.463Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-07T23:10:51.191Z","updated_at":"2025-09-07T23:10:54.892Z","avatar_url":"https://github.com/pgEdge.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The pg_walsizer Postgres Extension\n\nOne critical aspect of being a Postgres DBA is properly tuning configuration parameters based on system activity. Among these is [max_wal_size](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-MAX-WAL-SIZE), which determines the maximum amount of data Postgres can store purely within the WAL between checkpoints. Any time this limit is exceeded before [checkpoint_timeout](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-CHECKPOINT-TIMEOUT), Postgres will force a checkpoint to ensure this WAL data is written to the heap.\n\nBenchmarks universally demonstrate that forced checkpoints _dramatically_ reduce OLTP performance. This penalty can exceed an order of magnitude depending on the underlying storage technology and filesystem. Thus it pays to set `max_wal_size` properly in active systems.\n\nThe default `max_wal_size` is a mere 1GB, and most DBAs are well advised to increase it. But to what? One method is to watch the Postgres log for these entries:\n\n```\nLOG:  checkpoints are occurring too frequently (2 seconds apart)\nHINT:  Consider increasing the configuration parameter \"max_wal_size\".\n```\n\nIt's a great warning, but what value should a DBA use in this circumstance? This extension removes the guesswork from performing this task. It is designed to monitor the number of forced checkpoints which occur over a `checkpoint_timeout` interval. It then automatically increases `max_wal_size` to prevent further forced checkpoints based on this telemetry. In the end, `max_wal_size` will always be set properly to accommodate the maximum workload experienced by the instance.\n\nThis is what happens when numerous forced checkpoints cross the threshold and trigger a modification of `max_wal_size`:\n\n```\nLOG:  detected 4 forced checkpoints over 60 seconds\nLOG:  WAL request threshold (2) met, resizing max_wal_size\nLOG:  current max_wal_size is 512, should be 2560\nLOG:  received SIGHUP, reloading configuration files\nLOG:  parameter \"max_wal_size\" changed to \"2560\"\n```\n\nYes, I'm also surprised this actually works.\n\n## Installation\n\nInstalling this extension is simple:\n\n```bash\ngit clone git@github.com:pgedge/pg_walsizer.git\ncd pg_walsizer\nmake\nsudo make install\n```\n\nThen add it to `shared_preload_libraries`:\n\n```sql\nALTER SYSTEM SET shared_preload_libraries = 'pg_stat_statements, pg_walsizer';\n```\n\nOnce Postgres is restarted, pg_walsizer will launch and manage `max_wal_size` until it's removed or disabled.\n\n## Configuration\n\nThis extension currently accepts these parameters:\n\n| Parameter | Default | Min | Max | Description |\n|-----------|---------|-----|-----|-------------|\n| walsizer.enable | true |  |  | Walsizer will modify `max_wal_size` when enabled. |\n| walsizer.max | 4GB | 2MB | 2PB | Absolute maximum allowable value for `max_wal_size`. Walsizer will continue to recommend increases based on calculations, but will emit a warning rather than modify `max_wal_size`. |\n| walsizer.threshold | 2 | 1 | 1000 | Forced checkpoints below this number are ignored. This essentially ignores occasional scheduled batch jobs which may cause one or two forced checkpoints, but `max_wal_size` is properly sized otherwise. |\n\nAll parameters can only be modified by SIGHUP, so must exist in `postgresql.conf`, `postgresql.auto.conf`, or a file included by one of these.\n\n## Discussion\n\nThis extension may act as a learning exercise or skeleton for writing Postgres extensions which do the following:\n\n* Launch a background worker.\n* Properly manage a Postgres event loop using latches.\n* Consume backend system statistics.\n* Safely modify `postgresql.auto.conf`\n* Signal the Postgres postmaster.\n* Detect configuration changes.\n* Manipulate Node trees and lists.\n\nThis extension is largely based on the following libraries:\n\n* [SPI Worker test module](https://github.com/postgres/postgres/tree/master/src/test/modules/worker_spi)\n* [Checkpointer library](https://github.com/postgres/postgres/tree/master/src/backend/utils/activity/pgstat_checkpointer.c)\n\nIn addition to the following documentation:\n\n* [Postgres Background Worker Processes](https://www.postgresql.org/docs/current/bgworker.html)\n* [C-Language Functions](https://www.postgresql.org/docs/current/xfunc-c.html)\n\nAs always, it's best to have a copy of the Postgres source code available for reference.\n\n## Compatibility\n\nThis extension should be compatible with Postgres v15 or higher. This is when the backend stats system was reorganized to include `pgstat_fetch_stat_checkpointer` and related methods. Compatibility with older versions of Postgres is not a priority, but patches are always welcome.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpgedge%2Fpg_walsizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpgedge%2Fpg_walsizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpgedge%2Fpg_walsizer/lists"}