{"id":13532792,"url":"https://github.com/ankane/pgslice","last_synced_at":"2025-11-17T14:10:17.527Z","repository":{"id":7968612,"uuid":"56999385","full_name":"ankane/pgslice","owner":"ankane","description":"Postgres partitioning as easy as pie","archived":false,"fork":false,"pushed_at":"2025-03-27T23:46:51.000Z","size":268,"stargazers_count":1159,"open_issues_count":0,"forks_count":66,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-04-28T11:58:59.760Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ankane.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-24T23:40:03.000Z","updated_at":"2025-04-23T03:34:56.000Z","dependencies_parsed_at":"2023-12-26T20:36:26.403Z","dependency_job_id":"f98c1bcc-a08c-43a0-80be-3db106c7dc07","html_url":"https://github.com/ankane/pgslice","commit_stats":{"total_commits":407,"total_committers":14,"mean_commits":"29.071428571428573","dds":0.4471744471744472,"last_synced_commit":"404ea40e3fc8b1d4b06869d52b3ac8402ef44c42"},"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fpgslice","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fpgslice/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fpgslice/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fpgslice/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ankane","download_url":"https://codeload.github.com/ankane/pgslice/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251311332,"owners_count":21569008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T07:01:13.772Z","updated_at":"2025-11-17T14:10:17.521Z","avatar_url":"https://github.com/ankane.png","language":"Ruby","readme":"# pgslice\n\nPostgres partitioning as easy as pie. Works great for both new and existing tables, with zero downtime and minimal app changes. No need to install anything on your database server. Archive older data on a rolling basis to keep your database size under control.\n\n:tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)\n\n[![Build Status](https://github.com/ankane/pgslice/actions/workflows/build.yml/badge.svg)](https://github.com/ankane/pgslice/actions)\n\n## Install\n\npgslice is a command line tool. To install, run:\n\n```sh\ngem install pgslice\n```\n\nThis will give you the `pgslice` command. If installation fails, you may need to install [dependencies](#dependencies).\n\nYou can also install it with [Homebrew](#homebrew) or [Docker](#docker).\n\n## Steps\n\n1. Ensure the table you want to partition has been created. We’ll refer to this as `\u003ctable\u003e`.\n\n2. Specify your database credentials\n\n  ```sh\n  export PGSLICE_URL=postgres://localhost/myapp_development\n  ```\n\n3. Create an intermediate table\n\n  ```sh\n  pgslice prep \u003ctable\u003e \u003ccolumn\u003e \u003cperiod\u003e\n  ```\n\n  The column should be a `timestamp`, `timestamptz`, or `date` column and period can be `day`, `month`, or `year`.\n\n  This creates a partitioned table named `\u003ctable\u003e_intermediate` using range partitioning.\n\n4. Add partitions to the intermediate table\n\n  ```sh\n  pgslice add_partitions \u003ctable\u003e --intermediate --past 3 --future 3\n  ```\n\n  Use the `--past` and `--future` options to control the number of partitions.\n\n5. *Optional, for tables with data* - Fill the partitions in batches with data from the original table\n\n  ```sh\n  pgslice fill \u003ctable\u003e\n  ```\n\n  Use the `--batch-size` and `--sleep` options to control the speed (defaults to `10000` and `0` respectively)\n\n  To sync data across different databases, check out [pgsync](https://github.com/ankane/pgsync).\n\n6. Analyze tables\n\n  ```sh\n  pgslice analyze \u003ctable\u003e\n  ```\n\n7. Swap the intermediate table with the original table\n\n  ```sh\n  pgslice swap \u003ctable\u003e\n  ```\n\n  The original table is renamed `\u003ctable\u003e_retired` and the intermediate table is renamed `\u003ctable\u003e`.\n\n8. Fill the rest (rows inserted between the first fill and the swap)\n\n  ```sh\n  pgslice fill \u003ctable\u003e --swapped\n  ```\n\n9. Back up the retired table with a tool like [pg_dump](https://www.postgresql.org/docs/current/static/app-pgdump.html) and drop it\n\n  ```sql\n  pg_dump -c -Fc -t \u003ctable\u003e_retired $PGSLICE_URL \u003e \u003ctable\u003e_retired.dump\n  psql -c \"DROP TABLE \u003ctable\u003e_retired\" $PGSLICE_URL\n  ```\n\n## Sample Output\n\npgslice prints the SQL commands that were executed on the server. To print without executing, use the `--dry-run` option.\n\n```sh\npgslice prep visits created_at month\n```\n\n```sql\nBEGIN;\n\nCREATE TABLE \"public\".\"visits_intermediate\" (LIKE \"public\".\"visits\" INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING STORAGE INCLUDING COMMENTS INCLUDING STATISTICS INCLUDING GENERATED INCLUDING COMPRESSION) PARTITION BY RANGE (\"created_at\");\n\nCREATE INDEX ON \"public\".\"visits_intermediate\" USING btree (\"created_at\");\n\nCOMMENT ON TABLE \"public\".\"visits_intermediate\" is 'column:created_at,period:month,cast:date,version:3';\n\nCOMMIT;\n```\n\n```sh\npgslice add_partitions visits --intermediate --past 1 --future 1\n```\n\n```sql\nBEGIN;\n\nCREATE TABLE \"public\".\"visits_202408\" PARTITION OF \"public\".\"visits_intermediate\" FOR VALUES FROM ('2024-08-01') TO ('2024-09-01');\n\nALTER TABLE \"public\".\"visits_202408\" ADD PRIMARY KEY (\"id\");\n\nCREATE TABLE \"public\".\"visits_202409\" PARTITION OF \"public\".\"visits_intermediate\" FOR VALUES FROM ('2024-09-01') TO ('2024-10-01');\n\nALTER TABLE \"public\".\"visits_202409\" ADD PRIMARY KEY (\"id\");\n\nCREATE TABLE \"public\".\"visits_202410\" PARTITION OF \"public\".\"visits_intermediate\" FOR VALUES FROM ('2024-10-01') TO ('2024-11-01');\n\nALTER TABLE \"public\".\"visits_202410\" ADD PRIMARY KEY (\"id\");\n\nCOMMIT;\n```\n\n```sh\npgslice fill visits\n```\n\n```sql\n/* 1 of 3 */\nINSERT INTO \"public\".\"visits_intermediate\" (\"id\", \"user_id\", \"ip\", \"created_at\")\n    SELECT \"id\", \"user_id\", \"ip\", \"created_at\" FROM \"public\".\"visits\"\n    WHERE \"id\" \u003e 0 AND \"id\" \u003c= 10000 AND \"created_at\" \u003e= '2024-08-01'::date AND \"created_at\" \u003c '2024-11-01'::date\n\n/* 2 of 3 */\nINSERT INTO \"public\".\"visits_intermediate\" (\"id\", \"user_id\", \"ip\", \"created_at\")\n    SELECT \"id\", \"user_id\", \"ip\", \"created_at\" FROM \"public\".\"visits\"\n    WHERE \"id\" \u003e 10000 AND \"id\" \u003c= 20000 AND \"created_at\" \u003e= '2024-08-01'::date AND \"created_at\" \u003c '2024-11-01'::date\n\n/* 3 of 3 */\nINSERT INTO \"public\".\"visits_intermediate\" (\"id\", \"user_id\", \"ip\", \"created_at\")\n    SELECT \"id\", \"user_id\", \"ip\", \"created_at\" FROM \"public\".\"visits\"\n    WHERE \"id\" \u003e 20000 AND \"id\" \u003c= 30000 AND \"created_at\" \u003e= '2024-08-01'::date AND \"created_at\" \u003c '2024-11-01'::date\n```\n\n```sh\npgslice analyze visits\n```\n\n```sql\nANALYZE VERBOSE \"public\".\"visits_202408\";\n\nANALYZE VERBOSE \"public\".\"visits_202409\";\n\nANALYZE VERBOSE \"public\".\"visits_202410\";\n\nANALYZE VERBOSE \"public\".\"visits_intermediate\";\n```\n\n```sh\npgslice swap visits\n```\n\n```sql\nBEGIN;\n\nSET LOCAL lock_timeout = '5s';\n\nALTER TABLE \"public\".\"visits\" RENAME TO \"visits_retired\";\n\nALTER TABLE \"public\".\"visits_intermediate\" RENAME TO \"visits\";\n\nALTER SEQUENCE \"public\".\"visits_id_seq\" OWNED BY \"public\".\"visits\".\"id\";\n\nCOMMIT;\n```\n\n## Adding Partitions\n\nTo add partitions, use:\n\n```sh\npgslice add_partitions \u003ctable\u003e --future 3\n```\n\nAdd this as a cron job to create a new partition each day, month, or year.\n\n```sh\n# day\n0 0 * * * pgslice add_partitions \u003ctable\u003e --future 3 --url ...\n\n# month\n0 0 1 * * pgslice add_partitions \u003ctable\u003e --future 3 --url ...\n\n# year\n0 0 1 1 * pgslice add_partitions \u003ctable\u003e --future 3 --url ...\n```\n\nAdd a monitor to ensure partitions are being created.\n\n```sql\nSELECT 1 FROM\n    pg_catalog.pg_class c\nINNER JOIN\n    pg_catalog.pg_namespace n ON n.oid = c.relnamespace\nWHERE\n    c.relkind = 'r' AND\n    n.nspname = 'public' AND\n    c.relname = '\u003ctable\u003e_' || to_char(NOW() + INTERVAL '3 days', 'YYYYMMDD')\n    -- for months, use to_char(NOW() + INTERVAL '3 months', 'YYYYMM')\n    -- for years, use to_char(NOW() + INTERVAL '3 years', 'YYYY')\n```\n\n## Archiving Partitions\n\nBack up and drop older partitions each day, month, or year.\n\n```sh\npg_dump -c -Fc -t \u003ctable\u003e_202409 $PGSLICE_URL \u003e \u003ctable\u003e_202409.dump\npsql -c \"DROP TABLE \u003ctable\u003e_202409\" $PGSLICE_URL\n```\n\nIf you use [Amazon S3](https://aws.amazon.com/s3/) for backups, [s3cmd](https://github.com/s3tools/s3cmd) is a nice tool.\n\n```sh\ns3cmd put \u003ctable\u003e_202409.dump s3://\u003cs3-bucket\u003e/\u003ctable\u003e_202409.dump\n```\n\n## Schema Updates\n\nOnce a table is partitioned, make schema updates on the master table only (not partitions). This includes adding, removing, and modifying columns, as well as adding and removing indexes and foreign keys.\n\n## Additional Commands\n\nTo undo prep (which will delete partitions), use:\n\n```sh\npgslice unprep \u003ctable\u003e\n```\n\nTo undo swap, use:\n\n```sh\npgslice unswap \u003ctable\u003e\n```\n\n## Additional Options\n\nSet the tablespace when adding partitions\n\n```sh\npgslice add_partitions \u003ctable\u003e --tablespace fastspace\n```\n\n## App Considerations\n\nThis set up allows you to read and write with the original table name with no knowledge it’s partitioned. However, there are a few things to be aware of.\n\n### Reads\n\nWhen possible, queries should include the column you partition on to limit the number of partitions the database needs to check. For instance, if you partition on `created_at`, try to include it in queries:\n\n```sql\nSELECT * FROM\n    visits\nWHERE\n    user_id = 123 AND\n    -- for performance\n    created_at \u003e= '2024-09-01' AND created_at \u003c '2024-09-02'\n```\n\nFor this to be effective, ensure `constraint_exclusion` is set to `partition` (the default value) or `on`.\n\n```sql\nSHOW constraint_exclusion;\n```\n\n## Frameworks\n\n### Rails\n\nSpecify the primary key for partitioned models to ensure it’s returned.\n\n```ruby\nclass Visit \u003c ApplicationRecord\n  self.primary_key = \"id\"\nend\n```\n\n### Other Frameworks\n\nPlease submit a PR if additional configuration is needed.\n\n## One Off Tasks\n\nYou can also use pgslice to reduce the size of a table without partitioning by creating a new table, filling it with a subset of records, and swapping it in.\n\n```sh\npgslice prep \u003ctable\u003e --no-partition\npgslice fill \u003ctable\u003e --where \"id \u003e 1000\" # use any conditions\npgslice analyze \u003ctable\u003e\npgslice swap \u003ctable\u003e\npgslice fill \u003ctable\u003e --where \"id \u003e 1000\" --swapped\n```\n\n## Triggers\n\nTriggers aren’t copied from the original table. You can set up triggers on the intermediate table if needed.\n\n## Data Protection\n\nAlways make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained) when connecting to a database over a network you don’t fully trust. Your best option is to connect over SSH or a VPN. Another option is to use `sslmode=verify-full`. If you don’t do this, your database credentials can be compromised.\n\n## Additional Installation Methods\n\n### Homebrew\n\nWith Homebrew, you can use:\n\n```sh\nbrew install pgslice\n```\n\n### Docker\n\nGet the [Docker image](https://hub.docker.com/r/ankane/pgslice) with:\n\n```sh\ndocker pull ankane/pgslice\nalias pgslice=\"docker run --rm -e PGSLICE_URL ankane/pgslice\"\n```\n\nThis will give you the `pgslice` command.\n\n## Dependencies\n\nIf installation fails, your system may be missing Ruby or libpq.\n\nOn Mac, run:\n\n```sh\nbrew install libpq\n```\n\nOn Ubuntu, run:\n\n```sh\nsudo apt-get install ruby-dev libpq-dev build-essential\n```\n\n## Upgrading\n\nRun:\n\n```sh\ngem install pgslice\n```\n\nTo use master, run:\n\n```sh\ngem install specific_install\ngem specific_install https://github.com/ankane/pgslice.git\n```\n\n## Reference\n\n- [PostgreSQL Manual](https://www.postgresql.org/docs/current/static/ddl-partitioning.html)\n\n## Related Projects\n\nAlso check out:\n\n- [Dexter](https://github.com/ankane/dexter) - The automatic indexer for Postgres\n- [PgHero](https://github.com/ankane/pghero) - A performance dashboard for Postgres\n- [pgsync](https://github.com/ankane/pgsync) - Sync Postgres data to your local machine\n\n## History\n\nView the [changelog](https://github.com/ankane/pgslice/blob/master/CHANGELOG.md)\n\n## Contributing\n\nEveryone is encouraged to help improve this project. Here are a few ways you can help:\n\n- [Report bugs](https://github.com/ankane/pgslice/issues)\n- Fix bugs and [submit pull requests](https://github.com/ankane/pgslice/pulls)\n- Write, clarify, or fix documentation\n- Suggest or add new features\n\nTo get started with development:\n\n```sh\ngit clone https://github.com/ankane/pgslice.git\ncd pgslice\nbundle install\ncreatedb pgslice_test\nbundle exec rake test\n```\n\nTo test against different versions of Postgres with Docker, use:\n\n```sh\ndocker run -p=8000:5432 postgres:16\nTZ=Etc/UTC PGSLICE_URL=postgres://postgres@localhost:8000/postgres bundle exec rake\n```\n\nOn Mac, you must use [Docker Desktop](https://www.docker.com/products/docker-desktop/) for the port mapping to localhost to work.\n","funding_links":[],"categories":["others","Ruby","HA/Failover/Sharding","Gems","Awesome Ruby CLIs"],"sub_categories":["Zabbix","Table Partitioning","Database"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fpgslice","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fankane%2Fpgslice","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fpgslice/lists"}