{"id":13941468,"url":"https://github.com/jameshy/pgdump-aws-lambda","last_synced_at":"2026-01-01T23:24:11.497Z","repository":{"id":9193305,"uuid":"61205814","full_name":"jameshy/pgdump-aws-lambda","owner":"jameshy","description":"Lambda function for executing pg_dump and streaming the output to s3.","archived":false,"fork":false,"pushed_at":"2024-08-03T14:27:07.000Z","size":4480,"stargazers_count":227,"open_issues_count":7,"forks_count":107,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-06-29T12:17:40.547Z","etag":null,"topics":["aws-lambda","backup","nodejs","pgdump","postgres"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jameshy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-06-15T12:20:33.000Z","updated_at":"2025-05-30T06:03:57.000Z","dependencies_parsed_at":"2024-06-21T19:19:40.564Z","dependency_job_id":"df1f33b4-d615-4bb4-a056-078d09fdbae2","html_url":"https://github.com/jameshy/pgdump-aws-lambda","commit_stats":{"total_commits":133,"total_committers":16,"mean_commits":8.3125,"dds":"0.40601503759398494","last_synced_commit":"15981acf9e501128996811c8f76ccdaa8931b636"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/jameshy/pgdump-aws-lambda","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jameshy%2Fpgdump-aws-lambda","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jameshy%2Fpgdump-aws-lambda/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jameshy%2Fpgdump-aws-lambda/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jameshy%2Fpgdump-aws-lambda/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jameshy","download_url":"https://codeload.github.com/jameshy/pgdump-aws-lambda/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jameshy%2Fpgdump-aws-lambda/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266067259,"owners_count":23871324,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-lambda","backup","nodejs","pgdump","postgres"],"created_at":"2024-08-08T02:01:19.531Z","updated_at":"2026-01-01T23:24:11.452Z","avatar_url":"https://github.com/jameshy.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# pgdump-aws-lambda\n\n![ci status](https://github.com/jameshy/pgdump-aws-lambda/actions/workflows/node.js.yml/badge.svg)\n[![Coverage Status](https://coveralls.io/repos/github/jameshy/pgdump-aws-lambda/badge.svg?branch=master)](https://coveralls.io/github/jameshy/pgdump-aws-lambda?branch=master)\n\nAn AWS Lambda function that runs pg_dump and streams the output to s3.\n\nIt can be configured to run periodically using CloudWatch events.\n\n## Quick start\n\n1. Create an AWS lambda function:\n    - Author from scratch\n    - Runtime: Node.js 20.x\n    - Architecture: x86_64\n2. tab \"Code\" -\u003e \"Upload from\" -\u003e \".zip file\":\n    - Upload ([pgdump-aws-lambda.zip](https://github.com/jameshy/pgdump-aws-lambda/releases/latest))\n    - tab \"Configuration\" -\u003e \"General Configuration\" -\u003e \"Edit\"\n        - Timeout: 15 minutes\n        - Edit the role and attach the policy \"AmazonS3FullAccess\"\n    - Save\n3. Give your lambda permissions permissions to write to S3:\n    - tab \"Configuration\" -\u003e \"Permissions\"\n    - click the existing Execution role\n    - \"Add permissions\" -\u003e \"Attach policies\"\n    - select \"AmazonS3FullAccess\" and click \"Add Permissions\"\n\n4. Test\n\n    - Create new test event, e.g.:\n\n    ```json\n    {\n        \"PGDATABASE\": \"dbname\",\n        \"PGUSER\": \"postgres\",\n        \"PGPASSWORD\": \"password\",\n        \"PGHOST\": \"host\",\n        \"S3_BUCKET\": \"db-backups\",\n        \"ROOT\": \"hourly-backups\"\n    }\n    ```\n\n    - _Test_ and check the output\n\n5. Create a CloudWatch rule:\n    - Event Source: Schedule -\u003e Fixed rate of 1 hour\n    - Targets: Lambda Function (the one created in step #1)\n    - Configure input -\u003e Constant (JSON text) and paste your config (as per previous step)\n\n#### File Naming\n\nThis function will store your backup with the following s3 key:\n\ns3://${S3_BUCKET}${ROOT}/YYYY-MM-DD/YYYY-MM-DD_HH-mm-ss.backup\n\n#### AWS Firewall\n\n-   If you run the Lambda function outside a VPC, you must enable public access to your database instance, a non VPC Lambda function executes on the public internet.\n-   If you run the Lambda function inside a VPC, you must allow access from the Lambda Security Group to your database instance. Also you must either add a NAT gateway ([chargeable](https://aws.amazon.com/vpc/pricing/)) to your VPC so the Lambda can connect to S3 over the Internet, or add an [S3 VPC endpoint (free)](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) and allow traffic to the appropriate S3 prefixlist.\n\n#### Encryption\n\nYou can add an encryption key to your event, e.g.\n\n```json\n{\n    \"PGDATABASE\": \"dbname\",\n    \"PGUSER\": \"postgres\",\n    \"PGPASSWORD\": \"password\",\n    \"PGHOST\": \"host\",\n    \"S3_BUCKET\": \"db-backups\",\n    \"ROOT\": \"hourly-backups\",\n    \"ENCRYPT_KEY\": \"c0d71d7ae094bdde1ef60db8503079ce615e71644133dc22e9686dc7216de8d0\"\n}\n```\n\nThe key should be exactly 64 hex characters (32 hex bytes).\n\nWhen this key is present the function will do streaming encryption directly from pg_dump -\u003e S3.\n\nIt uses the aes-256-cbc encryption algorithm with a random IV for each backup file.\nThe IV is stored alongside the backup in a separate file with the .iv extension.\n\nYou can decrypt such a backup with the following bash command:\n\n```bash\nopenssl enc -aes-256-cbc -d \\\n-in postgres-27-12-2019@13-19-13.backup \\\n-out postgres-27-12-2019@13-19-13.unencrypted.backup \\\n-K c0d71d7ae094bdde1ef60db8503079ce615e71644133dc22e9686dc7216de8d0 \\\n-iv $(\u003c postgres-27-12-2019@13-19-13.backup.iv)\n```\n\n#### S3 Upload Part Size\n\nIf you experience lamba timeouts while uploading file parts to S3 you can try increasing the part size of each file chunk (might need to increase lambda resources). For instance on a 2GB file using the default part size of 5MB would result on ~400 parts, pushing all this parts was exceeding the 15min timeout for lambdas, by increasing the part size to 1GB the transmit time was reduced to ~3 minutes.\n\n```json\n{\n    \"S3_PART_SIZE\": 1073741824,\n}\n```\n\n#### IAM-based Postgres authentication\n\nYour context may require that you use IAM-based authentication to log into the Postgres service.\nSupport for this can be enabled my making your Cloudwatch Event look like this.\n\n```json\n{\n    \"PGDATABASE\": \"dbname\",\n    \"PGUSER\": \"postgres\",\n    \"PGHOST\": \"host\",\n    \"S3_BUCKET\": \"db-backups\",\n    \"ROOT\": \"hourly-backups\",\n    \"USE_IAM_AUTH\": true\n}\n```\n\nIf you supply `USE_IAM_AUTH` with a value of `true`, the `PGPASSWORD` var may be omitted in the CloudWatch event.\nIf you still provide it, it will be ignored.\n\n#### SecretsManager-based Postgres authentication\n\nIf you prefer to not send DB details/credentials in the event parameters, you can store such details in SecretsManager and just provide the SecretId, then the function will fetch your DB details/credentials from the secret value.\n\nNOTE: the execution role for the Lambda function must have access to GetSecretValue for the given secret.\n\nSupport for this can be enabled by setting the SECRETS_MANAGER_SECRET_ID, so your Cloudwatch Event looks like this:\n\n```json\n{\n    \"SECRETS_MANAGER_SECRET_ID\": \"my/secret/id\",\n    \"S3_BUCKET\": \"db-backups\",\n    \"ROOT\": \"hourly-backups\"\n}\n```\n\nIf you supply `SECRETS_MANAGER_SECRET_ID`, you can ommit the 'PG\\*' keys, and they will be fetched from your SecretsManager secret value instead with the following mapping:\n\n| Secret Value | PG-Key     |\n| ------------ | ---------- |\n| username     | PGUSER     |\n| password     | PGPASSWORD |\n| dbname       | PGDATABASE |\n| host         | PGHOST     |\n| port         | PGPORT     |\n\nYou can provide overrides in your event to any PG\\* keys as event parameters will take precedence over secret values.\n\n#### Multiple databases\n\nIf you'd like to export multiple databases in a single event, you can add a comma-separated list of database names to the PGDATABASE setting. The results will return in a list.\n\n```json\n{\n    \"PGDATABASE\": \"dbname1,dbname2,dbname3\",\n    \"PGUSER\": \"postgres\",\n    \"PGPASSWORD\": \"password\",\n    \"PGHOST\": \"host\",\n    \"S3_BUCKET\": \"db-backups\",\n    \"ROOT\": \"hourly-backups\"\n}\n```\n\nNOTE: The 15 minute timeout for lambda still applies.\n\n## Developer\n\n#### Bundling a new `pg_dump` binary\n\n1. Launch an EC2 instance with the Amazon Linux 2023 AMI (ami-0649bea3443ede307)\n2. Connect via SSH and:\n\n```bash\n# install packages required for building\nsudo dnf install make automake gcc gcc-c++ readline-devel zlib-devel openssl-devel libicu-devel\n# build and install postgres from source\nwget https://ftp.postgresql.org/pub/source/v16.3/postgresql-16.3.tar.gz\ntar zxf postgresql-16.3.tar.gz\ncd postgresql-16.3\n./configure --with-ssl=openssl\nmake\nsudo make install\nexit\n```\n\n#### Download the binaries\n\n```bash\nmkdir bin/postgres-16.3\nscp ec2-user@your-ec2-server:/usr/local/pgsql/bin/pg_dump ./bin/postgres-16.3/pg_dump\nscp ec2-user@your-ec2-server:/usr/local/pgsql/lib/libpq.so.5 ./bin/postgres-16.3/libpq.so.5\n```\n\n3. To use the new postgres binary pass PGDUMP_PATH in the event:\n\n```json\n{\n    \"PGDUMP_PATH\": \"bin/postgres-16.3\"\n}\n```\n\n#### Creating a new function zip\n\n`npm run makezip`\n\n#### Contributing\n\nPlease submit issues and PRs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjameshy%2Fpgdump-aws-lambda","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjameshy%2Fpgdump-aws-lambda","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjameshy%2Fpgdump-aws-lambda/lists"}