{"id":31287881,"url":"https://github.com/paudley/core_data","last_synced_at":"2026-05-08T03:10:21.826Z","repository":{"id":315939195,"uuid":"1061326051","full_name":"paudley/core_data","owner":"paudley","description":"Docker managed PostgreSQL server for local development ease","archived":false,"fork":false,"pushed_at":"2025-11-27T08:05:23.000Z","size":1392,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-29T22:37:03.048Z","etag":null,"topics":["docker","postgresql"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paudley.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-09-21T17:32:22.000Z","updated_at":"2025-11-27T08:04:06.000Z","dependencies_parsed_at":"2025-09-22T16:33:10.375Z","dependency_job_id":null,"html_url":"https://github.com/paudley/core_data","commit_stats":null,"previous_names":["paudley/core_data"],"tags_count":20,"template":false,"template_full_name":null,"purl":"pkg:github/paudley/core_data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paudley%2Fcore_data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paudley%2Fcore_data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paudley%2Fcore_data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paudley%2Fcore_data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paudley","download_url":"https://codeload.github.com/paudley/core_data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paudley%2Fcore_data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32764796,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-08T02:36:36.067Z","status":"ssl_error","status_checked_at":"2026-05-08T02:36:07.210Z","response_time":54,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","postgresql"],"created_at":"2025-09-24T11:28:59.890Z","updated_at":"2026-05-08T03:10:21.815Z","avatar_url":"https://github.com/paudley.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nSPDX-FileCopyrightText: 2025 Blackcat Informatics® Inc.\nSPDX-License-Identifier: MIT\n--\u003e\n\n# core\\_data\n\n[![CI](https://github.com/paudley/core_data/actions/workflows/ci.yml/badge.svg)](https://github.com/paudley/core_data/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n\nA reproducible PostgreSQL 17 platform delivered as code. core\\_data builds a hardened database image with spatial, vector, and graph extensions and ships a management CLI that automates backups, restores, QA cloning, and upgrades. Everything lives in version control so environments can be rebuilt consistently across laptops, CI, and production.\n\n## Why You Want This\n\n* Run the same Postgres 17 stack everywhere: laptop, CI runner, or production.\n* Ship with the heavy hitters pre-installed—PostGIS, pgvector, AGE, pg\\_cron, pgBackRest—without custom build scripts.\n* Automate the boring-but-critical tasks: backups, restores, QA clones, log analytics, and even major version upgrades via pgautoupgrade.\n* Treat your database like code with reproducible `.env` configs, templated init scripts, and a pytest smoke test that catches regressions early.\n\n## Published Docker Images\n\nPre-built images are available from GitHub Container Registry with full SLSA attestations and SBOM for supply chain security.\n\n### Quick Start with Published Image\n\n```bash\n# Pull the latest stable release\ndocker pull ghcr.io/paudley/core-data-postgres:latest\n\n# Or specify a version\ndocker pull ghcr.io/paudley/core-data-postgres:17.2-v1.0.0\n```\n\n### Available Tags\n\nImages follow a hybrid versioning strategy combining PostgreSQL version with semantic versioning:\n\n| Tag Pattern                             | Example       | Description                                    |\n| --------------------------------------- | ------------- | ---------------------------------------------- |\n| `latest`                                | `latest`      | Latest stable release                          |\n| `{PG_VERSION}-v{MAJOR}.{MINOR}.{PATCH}` | `17.2-v1.0.0` | Exact version (recommended for production)     |\n| `{PG_VERSION}-v{MAJOR}.{MINOR}`         | `17.2-v1.0`   | Latest patch for minor version                 |\n| `{PG_VERSION}-v{MAJOR}`                 | `17.2-v1`     | Latest minor for major version                 |\n| `{PG_VERSION}`                          | `17.2`        | Latest semantic version for PostgreSQL version |\n| `{PG_MAJOR}`                            | `17`          | Latest for PostgreSQL major version            |\n\n**Version Format**: `{PostgreSQL_Version}-v{Semantic_Version}`\n\n* Example: `17.2-v1.0.0` means PostgreSQL 17.2 with semantic version 1.0.0\n* See [docs/RELEASING.md](docs/RELEASING.md) for complete versioning details\n\n### Security \u0026 Verification\n\nAll published images include cryptographic attestations that prove build provenance and supply chain integrity:\n\n```bash\n# Verify image attestation (requires GitHub CLI)\ngh attestation verify oci://ghcr.io/paudley/core-data-postgres:17.2-v1.0.0 \\\n  --owner paudley\n\n# Pull and verify in one step\ndocker pull ghcr.io/paudley/core-data-postgres:17.2-v1.0.0\ngh attestation verify oci://ghcr.io/paudley/core-data-postgres:17.2-v1.0.0 \\\n  --owner paudley\n\n# Verify every referenced image from your .env (pretty console output)\n./scripts/manage.sh attestation-verify --env-file ci.env.example\n```\n\nGitHub invariably injects a repository-scoped `GITHUB_TOKEN` when workflows run. If that token cannot read `paudley/core_data`, the helper automatically retries the same verification anonymously (clearing `GH_TOKEN`/`GITHUB_TOKEN`) so public attestations still succeed without extra flags.\n\n### Published GHCR images\n\nEvery service in the stack is published with the same attested tag:\n\n| Service    | Image reference                                   |\n|------------|----------------------------------------------------|\n| PostgreSQL | `ghcr.io/paudley/core_data/postgres:\u003cstack-tag\u003e`   |\n| ValKey     | `ghcr.io/paudley/core_data/valkey:\u003cstack-tag\u003e`     |\n| PgBouncer  | `ghcr.io/paudley/core_data/pgbouncer:\u003cstack-tag\u003e`  |\n| RabbitMQ   | `ghcr.io/paudley/core_data/rabbitmq:\u003cstack-tag\u003e`   |\n| Memcached  | `ghcr.io/paudley/core_data/memcached:\u003cstack-tag\u003e`  |\n| Network Probe | `ghcr.io/paudley/core_data/network-probe:\u003cstack-tag\u003e` |\n| Network Guard | `ghcr.io/paudley/core_data/network-guard:\u003cstack-tag\u003e` |\n\nUse the optional `CORE_DATA_STACK_TAG` / `CORE_DATA_STACK_REGISTRY` environment variables (or `.env` entries) to pin a specific release across every helper; otherwise the defaults use the `latest` tag from `ghcr.io/paudley/core_data`.\n\nEvery container publishes `/README.md` and `/SECURITY.md` (copies of this repository's documentation) so you can inspect provenance and policy from inside the image with a simple `ls`.\n\nExpected verification output:\n\n```\n✓ Verification succeeded!\n\nsha256:abc123... was attested by:\nREPO              PREDICATE_TYPE                  WORKFLOW\npaudley/core_data https://slsa.dev/provenance/v1  .github/workflows/publish-docker.yml@refs/tags/17.2-v1.0.0\n```\n\n**What's Verified:**\n\n* **SLSA Provenance**: Confirms the image was built by GitHub Actions from this repository\n* **SBOM**: Software Bill of Materials listing all components and dependencies\n* **Build Reproducibility**: Links image digest to specific source code commit\n\n**Learn More:**\n\n* [Artifact Attestations Guide](https://docs.github.com/en/actions/security-guides/using-artifact-attestations-to-establish-provenance-for-builds)\n* [Release Process Documentation](docs/RELEASING.md)\n\n### Using Published Images in Docker Compose\n\nUpdate your `docker-compose.yml` to use the published image instead of building locally:\n\n```yaml\nservices:\n  postgres:\n    image: ghcr.io/paudley/core-data-postgres:17.2-v1.0.0  # Use published image\n    # Remove or comment out the 'build:' section\n    # build:\n    #   context: .\n    #   dockerfile: postgres/Dockerfile\n```\n\nOr override via `.env`:\n\n```bash\nPOSTGRES_IMAGE_NAME=ghcr.io/paudley/core-data-postgres\nPOSTGRES_IMAGE_TAG=17.2-v1.0.0\n```\n\n### Important: UID/GID Configuration for Pre-built Images\n\nPre-built images from `ghcr.io` have the `postgres` user baked in with **UID/GID 999**. Your `.env` must match:\n\n```bash\nPOSTGRES_UID=999\nPOSTGRES_GID=999\n```\n\n**Why this matters:** The `volume_prep` service uses these values to `chown` data directories. If they don't match the image's baked-in UID, you'll see permission errors like:\n\n```\nchmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted\n```\n\n**Building locally?** If you set `CORE_DATA_BUILD_IMAGE=1`, you can use your host user's UID/GID instead—the image build process will create the postgres user with your specified IDs.\n\n## Highlights\n\n* Custom Docker image with PostGIS, pgvector, Apache AGE, pgsodium, gzip, zstd, pg\\_cron, pg\\_squeeze, pgAudit, pgBadger, pgBackRest, and pgtune baked in.\n* Init scripts render configuration from templates, create application databases, and enable extensions automatically.\n* `./scripts/manage.sh` wraps lifecycle tasks: image builds, `psql`, logical dumps, pgBackRest backups/restores, QA cloning, log analysis, daily maintenance, and major upgrades via pgautoupgrade.\n* PGDATA, WAL, and pgBackRest now live on dedicated Docker named volumes for near-native Linux I/O, while `BACKUPS_HOST_PATH` (defaults to `./backups`) remains a bind mount for easy artifact exports.\n* Secrets stay in Docker secrets (`POSTGRES_PASSWORD_FILE`) and the container runs as the non-root `postgres` UID/GID at all times, keeping the least-privilege posture consistent across init and steady state.\n* TLS is enforced by default with auto-generated self-signed certificates (override with your own CA material), and a multi-stage health probe (`scripts/healthcheck.sh`) guards dependent services before they start.\n* Logging uses Docker's `local` driver with rotation and non-blocking delivery, preventing runaway JSON logs from filling the host while preserving enough history for incident response.\n* Optional profiles bundle ValKey, PgBouncer, and Memcached with sensible defaults, secrets, and CLI helpers so you can layer caches and pooling alongside PostgreSQL in one step.\n* CI smoke test (`python -m pytest -k full_workflow`) provisions a stack, exercises critical commands, and verifies upgrade safety.\n\n### Default Extension Bundle\n\ncore\\_data provisions a batteries-included extension stack in every non-template database at init time:\n\n* **Performance \u0026 Observability** — `pg_stat_statements`, `auto_explain`, `pg_buffercache`, `pg_prewarm`, `bloom`.\n* **Security \u0026 Compliance** — `pgaudit`, `pgcrypto`, `pgsodium` (Transparent Column Encryption), `\"uuid-ossp\"`.\n* **Compression** — `gzip` (gzip/gunzip in SQL), `zstd` (Zstandard compress/decompress with dictionary support in SQL).\n* **Developer Ergonomics** — `hstore`, `citext`, `pg_trgm`, `btree_gin`, `btree_gist`, `hypopg`, `intarray`, `ltree`, `tablefunc`, `unaccent`.\n* **Connectivity** — `postgres_fdw`, `dblink`.\n* **Spatial, Vector, Graph** — `postgis`, `postgis_raster`, `postgis_topology`, `vector`, `age`, `cube`, `earthdistance`.\n* **Maintenance \u0026 Testing** — `pg_cron` (kept in the `postgres` database), `pg_partman`, `pg_repack`, `pg_squeeze`, `pgstattuple`, `pgtap`.\n* **Geospatial Extras** — `postgis_tiger_geocoder`, `address_standardizer`, `address_standardizer_data_us`, `pgrouting`, `fuzzystrmatch`.\n\nThe same bundle is installed into `template1` so freshly created databases inherit the tooling automatically.\n\n#### pgsodium Key Requirement\n\npgsodium requires a 32-byte root encryption key for Transparent Column Encryption. The key is stored at `secrets/pgsodium.key` and bind-mounted read-only into the container at `/opt/core_data/secrets/pgsodium.key`. The `create-env` wizard and `bootstrap-ci` generate this key automatically. To generate one manually:\n\n```bash\nhead -c 32 /dev/urandom | od -A n -t x1 | tr -d ' \\n' \u003e secrets/pgsodium.key\nchmod 600 secrets/pgsodium.key\n```\n\n**This key is irreplaceable.** If lost, any data encrypted with pgsodium's Transparent Column Encryption becomes unrecoverable. Back it up alongside your other secret material.\n\n`pg_partman_bgw` is preloaded with a one-hour interval targeting the `postgres` database under the `postgres` superuser. Adjust `pg_partman_bgw.dbname`/`role` in `postgresql.conf.tpl` (or via `postgresql.pgtune.conf`) if you manage partitions from a different control schema.\n\nUse `./scripts/manage.sh partman-show-config` to inspect tracked parents, `partman-maintenance` to run `run_maintenance_proc()` on demand, and `partman-create-parent schema.table control_column '1 day'` to bootstrap new partition sets without hand-writing SQL.\n\nRun `./scripts/manage.sh async-queue bootstrap` when you want a lightweight background-job queue. It provisions an `asyncq.jobs` table plus helpers (`enqueue`, `dequeue`, `complete`, `fail`, `extend_lease`) that rely on `FOR UPDATE SKIP LOCKED`, `pg_notify`, and UUID leasing. Point a worker at the queue with `SELECT * FROM asyncq.dequeue('default');` in a loop and call `asyncq.complete(...)` or `asyncq.fail(...)` as you process jobs.\n\n## Quick Start\n\n1. Bootstrap environment config: `./scripts/manage.sh create-env` to walk through password creation, host UID/GID selection, and resource sizing (writes `.env` + secrets).\n2. Build and start the stack:\n   ```bash\n   ./scripts/manage.sh build-image\n   ./scripts/manage.sh up\n   ```\n3. Verify health (multi-stage probe):\n   ```bash\n   docker compose exec postgres /opt/core_data/scripts/healthcheck.sh\n   ./scripts/manage.sh psql -c 'SELECT 1;'\n   ```\n\n4. Explore the CLI: `./scripts/manage.sh help` for grouped commands (lifecycle, CI, backups, audits, extensions, cache/messaging, security).\n5. CI runners: see `CI_USAGE.md` for the pipelines-focused flow (attestations, `ci-up`/`ci-down`, pytest markers).\n\n## Project Ethos\nWe optimize for **data infrastructure as code ➜ automated admin ➜ human-friendly ➜ best-practices by default (including security and performance)**. In practice that means:\n\n1. **Data infrastructure as code.** Declarative, reproducible Postgres once, versioned forever.\n2. **Automated admin.** Every recurring task should be scriptable and CI-friendly before we worry about shell ergonomics.\n3. **Human-friendly.** CLI helpers, sensible prompts, and clear docs matter—but only after the first two goals are met.\n4. **Best-practices by default.** Security and performance guardrails (cap drops, TLS, tuned memory, backups) are enabled out of the box, with deliberate escape hatches when absolutely required.\n\nSee `docs/security_philosophy.md` for how capability hardening and related controls fit into that priority order.\n\n## Operational Defaults\n- **Resource guardrails.** Container memory, CPU, and shared memory limits come from `.env`, keeping pgtune advice and runtime constraints aligned. Adjust `POSTGRES_MEMORY_LIMIT`, `POSTGRES_CPU_LIMIT`, and `POSTGRES_SHM_SIZE` to match the host.\n- **TLS everywhere.** PostgreSQL refuses non-SSL connections from the bridge network. Provide your own certificate/key via Docker secrets or rely on the init hook to mint a self-signed pair under `${PGDATA}/tls`.\n- **Named volumes for PGDATA/WAL.** `pgdata`, `pgwal`, and `pgbackrest` volumes provide near-native I/O on Linux. Override the volume definitions if you pin WAL/data to specific devices.\n- **Non-root from the start.** A one-shot `volume_prep` helper chowns the volumes before Postgres launches so the main service and sidecars run as your host user by default (UID/GID `${POSTGRES_UID}`), keeping file ownership consistent across deployments. Supply alternative IDs only when required.\n- **Automated logical backups.** The `logical_backup` sidecar runs `pg_dump`/`pg_dumpall` on the cadence defined by `LOGICAL_BACKUP_INTERVAL_SECONDS`, writes into `${BACKUPS_HOST_PATH}/logical`, prunes according to `LOGICAL_BACKUP_RETENTION_DAYS`, and skips any databases listed in `LOGICAL_BACKUP_EXCLUDE` (defaults to `postgres`). `daily-maintenance` captures the latest run in `logical_backup_status.txt` for auditing.\n- **Composable health check.** `scripts/healthcheck.sh` verifies readiness, executes `SELECT 1`, and optionally enforces replication lag ceilings before dependents start.\n- **Rotated container logs.** Docker's `local` driver with non-blocking delivery prevents runaway JSON files while retaining compressed history for incident response.\n- **Optional service profiles.** `COMPOSE_PROFILES=valkey,pgbouncer,memcached,rabbitmq` brings the cache/pooling stack online; drop profiles from the list to opt out without editing `docker-compose.yml`.\n- **Seccomp baseline.** Shipping profiles in `seccomp/` cover each service (`postgres.json`, `logical_backup.json`, `pgbouncer.json`, `valkey.json`, `memcached.json`, plus `docker-default.json` reused for RabbitMQ). `./scripts/manage.sh seccomp-status` reports the active spec, `seccomp-verify` gates compose configs, and `docs/security_philosophy.md` outlines how to regenerate traces when you need to tighten them further.\n- **AppArmor (opt-in).** Minimal profiles live in `apparmor/core_data_minimal.profile`. Load them with `./scripts/manage.sh apparmor-load` (sudo), then set `CORE_DATA_APPARMOR_\u003cSERVICE\u003e=apparmor:core_data_minimal` in `.env` for each service you want to confine. The profile denies access to high-value host paths (`/root`, `/etc/shadow`, Docker socket) while leaving normal container paths alone.\n\n### Service Add-ons\n- **ValKey** — Requires authentication by default (`valkey_password` secret), persists to the `valkey_data` volume (`appendonly yes`), exposes `valkey-cli`/`valkey-bgsave`, and is tuned via `.env` knobs such as `VALKEY_MAXMEMORY` and `VALKEY_MAXMEMORY_POLICY`.\n- **PgBouncer** — Uses SCRAM auth backed by a dedicated superuser, renders config/userlist from templates, and offers `pgbouncer-stats` / `pgbouncer-pools` helpers. Pool sizing and admin/stat users are driven by the `PGBOUNCER_*` variables.\n- **Memcached** — Lightweight hot cache with configurable memory, connection, and thread limits (`MEMCACHED_*`). The `memcached-stats` helper pipes `stats` output for quick validation. All services stay on the internal bridge network by default.\n- **RabbitMQ** — Delivers AMQP 0.9.1 alongside the management API. Credentials and cluster identity come from `secrets/rabbitmq_default_pass` and `secrets/rabbitmq_erlang_cookie`; daily maintenance captures definitions/status snapshots, and on-demand helpers (`rabbitmq-ctl`, `rabbitmq-diagnostics`, `rabbitmq-export`, `rabbitmq-overview`) streamline broker admin.\n\n## Project Layout\n```\n\ncore\\_data/\n├── .env.example # Template for environment-specific settings (never commit real secrets)\n├── docker-compose.yml # Orchestrates PostgreSQL and supporting services\n├── scripts/ # Operator tooling (manage.sh + lib modules + maintenance workflow)\n├── postgres/ # Custom image build assets, configs, and initdb scripts\n├── backups/ # Host output directory for logical dumps and reports\n├── secrets/ # Docker secret material (passwords, pgsodium.key — never committed)\n├── README.md # This guide\n├── THIRD\\_PARTY\\_LICENSES.md # Upstream license attributions for vendored tooling\n└── AGENTS.md # Contributor quick-reference \u0026 runbooks\n\n```\nIf you override the named volumes with host bind mounts, keep those directories out of version control—they contain live cluster state and pgBackRest archives.\n\n## CI Workflow\n\nUse the dedicated CI helpers when you need to spin up the published stack inside automation:\n\n1. `./scripts/manage.sh ci-verify --env-file ci.env.example --require-attestation` confirms Docker is available, checks disk/port prerequisites, and verifies that every referenced image carries a recent attestation.\n2. `./scripts/manage.sh ci-up --env-file ci.env.example --output ./backups/ci-output.json` bootstraps secrets/network allow lists, pulls/warms the prebuilt images, and emits a JSON manifest with connection details for downstream jobs.\n3. `./scripts/manage.sh ci-down --volumes --prune-data --prune-secrets` stops containers and removes ephemeral state when the pipeline finishes.\n\n`ci.env.example` shows the minimal environment required for this workflow, and `docs/examples/ci-workflow.md` includes a ready-to-copy GitHub Actions job that wires everything together.\n\n## Management CLI\n`./scripts/manage.sh` is the operator entry point. Frequently used commands:\n\n| Command | Description |\n| --- | --- |\n| `build-image` | Build the custom PostgreSQL image defined in `postgres/Dockerfile`. |\n| `create-env` | Interactive wizard that copies `.env.example`, sizes resources, seeds secrets, and writes `.env`. |\n| `up` / `down` | Start or stop the Compose stack (volumes preserved). |\n| `psql` | Open psql inside the container (respects `PGHOST`, `PGUSER`, etc.). |\n| `dump` / `dump-sql` | Produce logical backups (custom or plain format) under `/backups`. |\n| `restore-dump` | Drop and recreate a database before restoring a `.dump.gz`. |\n| `backup [--verify]` / `stanza-create` / `restore-snapshot` | Manage pgBackRest backups \u0026 optionally restore the latest backup into a throwaway data dir for checksum verification. |\n| `daily-maintenance` | Run dumps, log capture, pgBadger analysis, and retention pruning. |\n| `provision-qa` | Differential backup + targeted restore for QA databases. |\n| `config-render` | Re-render `postgresql.conf` / `pg_hba.conf` from the templates and restart PostgreSQL (terminates active connections; required for some settings like `shared_buffers` and `max_connections`). |\n| `config-check` | Compare live `postgresql.conf` / `pg_hba.conf` against rendered templates to catch drift. |\n| `audit-roles` / `audit-security` | Generate CSV/text reports covering role hygiene, passwords, and HBA/RLS posture. |\n| `audit-extensions` | Confirm bundled extensions are present and on expected versions. |\n| `audit-autovacuum` | Flag tables with high dead tuple counts or ratios. |\n| `audit-replication` | Summarise follower lag and sync state. |\n| `audit-cron` / `audit-squeeze` | Inspect pg_cron schedules and pg_squeeze activity tables. |\n| `audit-index-bloat` | Estimate index bloat using pgstattuple (supports `--min-size-mb`). |\n| `audit-buffercache` | Snapshot shared buffer usage per relation (supports `--limit`). |\n| `audit-schema` | Snapshot schema metadata for drift detection. |\n| `snapshot-pgstat` | Capture a `pg_stat_statements` baseline (CSV output) for performance trending. |\n| `diff-pgstat --base --compare` | Diff two snapshots (CSV-in/CSV-out) to highlight hot queries. |\n| `compact --level N` | Layered bloat management: 1=autovacuum audit, 2=pg_squeeze refresh, 3=pg_repack (needs `--tables`), 4=VACUUM FULL (needs `--yes`). |\n| `exercise-extensions` | Smoke-test the core extension bundle (vector, PostGIS, AGE, citext, hstore, pgcrypto, hypopg, pg_partman, etc.). |\n| `pgtap-smoke` | Run a micro pgTap plan to confirm key extensions (including hypopg/pg_partman) are registered. |\n| `test-dataset bootstrap` | Stand up a synthetic schema with spatial, vector, routing, and graph fixtures to exercise PgBouncer/GraphQL clients. |\n| `async-queue bootstrap` | Install the lightweight async queue schema (`asyncq`) with enqueue/dequeue helpers. |\n| `partman-maintenance` | Invoke `run_maintenance_proc()` for the selected database (defaults to `POSTGRES_DB`). |\n| `partman-show-config` | Print rows from `part_config` (optionally filter by `--parent schema.table`). |\n| `partman-create-parent` | Wrap `create_parent` to bootstrap managed partitions without manual SQL. |\n| `valkey-cli` | Run `valkey-cli` inside the ValKey container with secrets wired in. |\n| `valkey-bgsave` | Trigger `BGSAVE` so the ValKey RDB is flushed to the `valkey_data` volume. |\n| `rabbitmq-ctl` | Execute `rabbitmqctl` inside the RabbitMQ container (requires the rabbitmq profile). |\n| `rabbitmq-diagnostics` | Run `rabbitmq-diagnostics` commands such as `status` or `check_running`. |\n| `rabbitmq-export` | Export broker definitions to JSON (defaults to `${BACKUPS_HOST_PATH}/rabbitmq-definitions.json`). |\n| `rabbitmq-overview` | Print the summary from `rabbitmq-diagnostics status`. |\n| `pgbouncer-stats` / `pgbouncer-pools` | Emit PgBouncer `SHOW STATS` / `SHOW POOLS` via the admin console. |\n| `memcached-stats` | Fetch `stats` output from the Memcached service. |\n| `version-status` | Compare installed Postgres/extension versions with upstream releases (CSV via `--output`). |\n| `upgrade --new-version` | Orchestrate pgautoupgrade (takes backups, validates base image, restarts). |\n\nThe CLI sources modular helpers from `scripts/lib/` so each function can be imported by tests or future automation.\n\n`daily-maintenance` now emits a richer bundle under `backups/daily/\u003cYYYYMMDD\u003e/`, including `pg_stat_statements` snapshots, `pg_buffercache` heatmaps, role/extension/autovacuum/replication CSVs, pg_cron schedules, pg_squeeze activity, and a security checklist alongside logs, dumps, pgBadger HTML, and pgaudit summaries. The workflow also records the most recent sidecar dump run in `logical_backup_status.txt`, runs `partman.run_maintenance_proc()` across each database so freshly created partitions land even if the background worker interval has not elapsed, and captures version drift in `version_status.csv` (focusing on out-of-date components). Pair those reports with `config-check` to keep the rendered configs aligned with the templates. Tune the thresholds via `DAILY_PG_STAT_LIMIT`, `DAILY_BUFFERCACHE_LIMIT`, `DAILY_DEAD_TUPLE_THRESHOLD`, `DAILY_DEAD_TUPLE_RATIO`, and `DAILY_REPLICATION_LAG_THRESHOLD` as needed.\n\nNightly cron jobs also refresh pg_squeeze targets, reset `pg_stat_statements`, and run a safe `VACUUM (ANALYZE, SKIP_LOCKED, PARALLEL 4)` so statistics stay current without blocking hot tables.\n\nSet `DAILY_EMAIL_REPORT=true` and `DAILY_REPORT_RECIPIENT=ops@example.com` in `.env` to have the maintenance job email the HTML summary via `sendmail` (if available inside the container).\n\nTo compare performance snapshots between runs, capture CSVs with `snapshot-pgstat --output /backups/pg_stat_before.csv` and `snapshot-pgstat --output /backups/pg_stat_after.csv`, then run `./scripts/manage.sh diff-pgstat --base /backups/pg_stat_before.csv --compare /backups/pg_stat_after.csv --limit 25` for a ranked delta report.\n\n### Compacting storage\n`./scripts/manage.sh compact` provides escalating space-recovery options:\n\n- **Level 1** — run the autovacuum audit (no changes, just reporting).\n- **Level 2** — rerun `core_data_admin.refresh_pg_squeeze_targets()` and emit updated `pg_squeeze` findings.\n- **Level 3** — execute `pg_repack` for specific tables (`--tables schema.table[,schema.table...]`) without heavy locks.\n- **Level 4** — run `VACUUM (FULL, ANALYZE, VERBOSE)` across all tables or a comma-delimited `--scope` (requires `--yes`). Expect exclusive locks; schedule during maintenance windows.\n\nAll runs write logs under `backups/` for auditing (`pg_repack-*.log`, `vacuum-full-*.log`).\n\n## Automation \u0026 Testing\n- **CI Workflow:** `.github/workflows/ci.yml` builds the image, runs `python -m pytest -k full_workflow`, and uploads generated backups for inspection.\n- **Smoke Test:** `tests/test_manage.py` spins up a disposable environment, exercises key CLI commands (including `daily-maintenance`, pgBackRest, and `upgrade`), and tears everything down. Run locally with `python -m pytest -k full_workflow` (Docker required).\n- **Fast Tests:** `tests/test_lightweight.py` validates offline flows like help output and the vendored tooling without needing Docker.\n- **Extension Smoke:** `./scripts/manage.sh exercise-extensions --db \u003cname\u003e` plus `pgtap-smoke` provide quick feedback that the entire core extension bundle (vector/PostGIS/AGE/hstore/citext/pgcrypto/pg_partman/etc.) is ready for use.\n- **Synthetic Fixture:** `./scripts/manage.sh test-dataset bootstrap --db core_data_testkit --owner demo_user --print-password --force` creates repeatable spatial/vector/graph/routing sample data so PgBouncer, REST, and GraphQL clients can run end-to-end queries without touching production schemas.\n- **Documentation:** `AGENTS.md` offers contributor runbooks and on-call notes.\n\n## Credits\nThank you to the maintainers and communities behind the components that make core_data possible:\n- [PostgreSQL](https://www.postgresql.org/) – the database at the heart of the platform.\n- [Docker](https://www.docker.com/) \u0026 [Docker Compose](https://docs.docker.com/compose/) – containerization and orchestration.\n- [PgBackRest](https://pgbackrest.org/) – resilient backup and restore tooling.\n- [pgBadger](https://github.com/darold/pgbadger) – PostgreSQL log analytics.\n- [pg_cron](https://github.com/citusdata/pg_cron) – database-native scheduling.\n- [pg_squeeze](https://github.com/cybertec-postgresql/pg_squeeze) – automatic bloat mitigation.\n- [pgvector](https://github.com/pgvector/pgvector) – high-dimensional vector search.\n- [PostGIS](https://postgis.net/) – spatial superpowers for PostgreSQL (including Tiger Geocoder \u0026 Address Standardizer).\n- [pgRouting](https://pgrouting.org/) – network routing \u0026 graph analysis atop PostGIS.\n- [Apache AGE](https://age.apache.org/) – graph database extension.\n- [pgaudit](https://github.com/pgaudit/pgaudit) – enhanced auditing.\n- [pg_partman](https://github.com/pgpartman/pg_partman) – automated time/ID partition management.\n- [HypoPG](https://github.com/HypoPG/hypopg) – hypothetical index exploration for query tuning.\n- [pg_repack](https://github.com/reorg/pg_repack) \u0026 [pgtap](https://github.com/theory/pgtap) – maintenance \u0026 testing extensions.\n\nTheir work powers the database-as-code experience delivered by core_data.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaudley%2Fcore_data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaudley%2Fcore_data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaudley%2Fcore_data/lists"}