{"id":19458470,"url":"https://github.com/postgrespro/pg_pathman","last_synced_at":"2025-05-15T09:05:13.459Z","repository":{"id":5250985,"uuid":"52430435","full_name":"postgrespro/pg_pathman","owner":"postgrespro","description":"Partitioning tool for PostgreSQL","archived":false,"fork":false,"pushed_at":"2024-11-15T08:25:29.000Z","size":4828,"stargazers_count":587,"open_issues_count":51,"forks_count":67,"subscribers_count":63,"default_branch":"master","last_synced_at":"2025-04-14T22:16:27.681Z","etag":null,"topics":["customscan","fdw","hash","partition-table","partitioning","pathman","postgresql","query-optimization","range","runtimeappend"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/postgrespro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-24T09:31:11.000Z","updated_at":"2025-04-14T05:59:31.000Z","dependencies_parsed_at":"2023-02-18T11:17:35.244Z","dependency_job_id":"c95be618-4632-499e-8cb2-e465d22c040f","html_url":"https://github.com/postgrespro/pg_pathman","commit_stats":{"total_commits":1479,"total_committers":28,"mean_commits":52.82142857142857,"dds":0.4401622718052738,"last_synced_commit":"73aafcfc7c088f891890b6896b7b4d62e36f6931"},"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Fpg_pathman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Fpg_pathman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Fpg_pathman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Fpg_pathman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/postgrespro","download_url":"https://codeload.github.com/postgrespro/pg_pathman/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248968918,"owners_count":21191162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["customscan","fdw","hash","partition-table","partitioning","pathman","postgresql","query-optimization","range","runtimeappend"],"created_at":"2024-11-10T17:27:14.474Z","updated_at":"2025-04-14T22:16:36.464Z","avatar_url":"https://github.com/postgrespro.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.com/postgrespro/pg_pathman.svg?branch=master)](https://travis-ci.com/postgrespro/pg_pathman)\n[![PGXN version](https://badge.fury.io/pg/pg_pathman.svg)](https://badge.fury.io/pg/pg_pathman)\n[![codecov](https://codecov.io/gh/postgrespro/pg_pathman/branch/master/graph/badge.svg)](https://codecov.io/gh/postgrespro/pg_pathman)\n[![GitHub license](https://img.shields.io/badge/license-PostgreSQL-blue.svg)](https://raw.githubusercontent.com/postgrespro/pg_pathman/master/LICENSE)\n\n### NOTE: this project is not under development anymore\n\n`pg_pathman` supports Postgres versions [11..15], but most probably it won't be ported to later releases. [Native partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html) is pretty mature now and has almost everything implemented in `pg_pathman`'; we encourage users switching to it. We are still maintaining the project (fixing bugs in supported versions), but no new development is going to happen here.\n\n# pg_pathman\n\nThe `pg_pathman` module provides optimized partitioning mechanism and functions to manage partitions.\n\nThe extension is compatible with:\n\n * PostgreSQL 12, 13;\n * PostgreSQL with core-patch: 11, 14, 15;\n * Postgres Pro Standard 11, 12, 13, 14, 15;\n * Postgres Pro Enterprise;\n\nTake a look at our Wiki [out there](https://github.com/postgrespro/pg_pathman/wiki).\n\n## Overview\n**Partitioning** means splitting one large table into smaller pieces. Each row in such table is moved to a single partition according to the partitioning key. PostgreSQL \u003c= 10 supports partitioning via table inheritance: each partition must be created as a child table with CHECK CONSTRAINT:\n\n```plpgsql\nCREATE TABLE test (id SERIAL PRIMARY KEY, title TEXT);\nCREATE TABLE test_1 (CHECK ( id \u003e= 100 AND id \u003c 200 )) INHERITS (test);\nCREATE TABLE test_2 (CHECK ( id \u003e= 200 AND id \u003c 300 )) INHERITS (test);\n```\n\nPostgreSQL 10 provides native partitioning:\n\n```plpgsql\nCREATE TABLE test(id int4, value text) PARTITION BY RANGE(id);\nCREATE TABLE test_1 PARTITION OF test FOR VALUES FROM (1) TO (10);\nCREATE TABLE test_2 PARTITION OF test FOR VALUES FROM (10) TO (20);\n```\n\nIt's not so different from the classic approach; there are implicit check constraints, and most of its limitations are still relevant.\n\nDespite the flexibility, this approach forces the planner to perform an exhaustive search and to check constraints on each partition to determine whether it should be present in the plan or not. Large amount of partitions may result in significant planning overhead.\n\nThe `pg_pathman` module features partition managing functions and optimized planning mechanism which utilizes knowledge of the partitions' structure. It stores partitioning configuration in the `pathman_config` table; each row contains a single entry for a partitioned table (relation name, partitioning column and its type). During the initialization stage the `pg_pathman` module caches some information about child partitions in the shared memory, which is used later for plan construction. Before a SELECT query is executed, `pg_pathman` traverses the condition tree in search of expressions like:\n\n```\nVARIABLE OP CONST\n```\nwhere `VARIABLE` is a partitioning key, `OP` is a comparison operator (supported operators are =, \u003c, \u003c=, \u003e, \u003e=), `CONST` is a scalar value. For example:\n\n```plpgsql\nWHERE id = 150\n```\n\nBased on the partitioning type and condition's operator, `pg_pathman` searches for the corresponding partitions and builds the plan. Currently `pg_pathman` supports two partitioning schemes:\n\n* **RANGE** - maps rows to partitions using partitioning key ranges assigned to each partition. Optimization is achieved by using the binary search algorithm;\n* **HASH** - maps rows to partitions using a generic hash function.\n\nMore interesting features are yet to come. Stay tuned!\n\n## Feature highlights\n\n * HASH and RANGE partitioning schemes;\n * Partitioning by expression and composite key;\n * Both automatic and manual [partition management](#post-creation-partition-management);\n * Support for integer, floating point, date and other types, including domains;\n * Effective query planning for partitioned tables (JOINs, subselects etc);\n * `RuntimeAppend` \u0026 `RuntimeMergeAppend` custom plan nodes to pick partitions at runtime;\n * [`PartitionFilter`](#custom-plan-nodes): an efficient drop-in replacement for INSERT triggers;\n * [`PartitionRouter`](#custom-plan-nodes) and [`PartitionOverseer`](#custom-plan-nodes) for cross-partition UPDATE queries (instead of triggers);\n * Automatic partition creation for new INSERTed data (only for RANGE partitioning);\n * Improved `COPY FROM` statement that is able to insert rows directly into partitions;\n * [User-defined callbacks](#additional-parameters) for partition creation event handling;\n * Non-blocking [concurrent table partitioning](#data-migration);\n * FDW support (foreign partitions);\n * Various [GUC](#disabling-pg_pathman) toggles and configurable settings.\n * Partial support of [`declarative partitioning`](#declarative-partitioning) (from PostgreSQL 10).\n\n## Installation guide\nTo install `pg_pathman`, execute this in the module's directory:\n\n```shell\nmake install USE_PGXS=1\n```\n\n\u003e **Important:** Don't forget to set the `PG_CONFIG` variable (`make PG_CONFIG=...`) in case you want to test `pg_pathman` on a non-default or custom build of PostgreSQL. Read more [here](https://wiki.postgresql.org/wiki/Building_and_Installing_PostgreSQL_Extension_Modules).\n\nModify the **`shared_preload_libraries`** parameter in `postgresql.conf` as following:\n\n```\nshared_preload_libraries = 'pg_pathman'\n```\n\n\u003e **Important:** `pg_pathman` may cause conflicts with some other extensions that use the same hook functions. For example, `pg_pathman` uses `ProcessUtility_hook` to handle COPY queries for partitioned tables, which means it may interfere with `pg_stat_statements` from time to time. In this case, try listing libraries in certain order: `shared_preload_libraries = 'pg_stat_statements, pg_pathman'`.\n\nIt is essential to restart the PostgreSQL instance. After that, execute the following query in psql:\n```plpgsql\nCREATE SCHEMA pathman;\nGRANT USAGE ON SCHEMA pathman TO PUBLIC;\nCREATE EXTENSION pg_pathman WITH SCHEMA pathman;\n```\n\nDone! Now it's time to setup your partitioning schemes.\n\n\u003e **Security notice**: pg_pathman is believed to be secure against\nsearch-path-based attacks mentioned in Postgres\n[documentation](https://www.postgresql.org/docs/current/sql-createextension.html). However,\nif *your* calls of pathman's functions doesn't exactly match the signature, they\nmight be vulnerable to malicious overloading. If in doubt, install pathman to clean schema where nobody except superusers have CREATE object permission to avoid problems.\n\n\u003e **Windows-specific**: pg_pathman imports several symbols (e.g. None_Receiver, InvalidObjectAddress) from PostgreSQL, which is fine by itself, but requires that those symbols are marked as `PGDLLIMPORT`. Unfortunately, some of them are not exported from vanilla PostgreSQL, which means that you have to either use Postgres Pro Standard/Enterprise (which includes all necessary patches), or patch and build your own distribution of PostgreSQL.\n\n## How to update\nIn order to update pg_pathman:\n\n1. Install the latest _stable_ release of pg_pathman.\n2. Restart your PostgreSQL cluster.\n3. Execute the following queries:\n\n```plpgsql\n/* only required for major releases, e.g. 1.4 -\u003e 1.5 */\nALTER EXTENSION pg_pathman UPDATE;\nSET pg_pathman.enable = t;\n```\n\n## Available functions\n\n### Module's version\n\n```plpgsql\npathman_version()\n```\nAlthough it's possible to get major and minor version numbers using `\\dx pg_pathman`, it doesn't show the actual [patch number](http://semver.org/). This function returns a complete version number of the loaded pg_pathman module in `MAJOR.MINOR.PATCH` format.\n\n### Partition creation\n```plpgsql\ncreate_hash_partitions(parent_relid     REGCLASS,\n                       expression       TEXT,\n                       partitions_count INTEGER,\n                       partition_data   BOOLEAN DEFAULT TRUE,\n                       partition_names  TEXT[] DEFAULT NULL,\n                       tablespaces      TEXT[] DEFAULT NULL)\n```\nPerforms HASH partitioning for `relation` by partitioning expression `expr`. The `partitions_count` parameter specifies the number of partitions to create; it cannot be changed afterwards. If `partition_data` is `true` then all the data will be automatically copied from the parent table to partitions. Note that data migration may took a while to finish and the table will be locked until transaction commits. See `partition_table_concurrently()` for a lock-free way to migrate data. Partition creation callback is invoked for each partition if set beforehand (see `set_init_callback()`).\n\n```plpgsql\ncreate_range_partitions(parent_relid    REGCLASS,\n                        expression      TEXT,\n                        start_value     ANYELEMENT,\n                        p_interval      ANYELEMENT,\n                        p_count         INTEGER DEFAULT NULL\n                        partition_data  BOOLEAN DEFAULT TRUE)\n\ncreate_range_partitions(parent_relid    REGCLASS,\n                        expression      TEXT,\n                        start_value     ANYELEMENT,\n                        p_interval      INTERVAL,\n                        p_count         INTEGER DEFAULT NULL,\n                        partition_data  BOOLEAN DEFAULT TRUE)\n\ncreate_range_partitions(parent_relid    REGCLASS,\n                        expression      TEXT,\n                        bounds          ANYARRAY,\n                        partition_names TEXT[] DEFAULT NULL,\n                        tablespaces     TEXT[] DEFAULT NULL,\n                        partition_data  BOOLEAN DEFAULT TRUE)\n```\nPerforms RANGE partitioning for `relation` by partitioning expression `expr`, `start_value` argument specifies initial value, `p_interval` sets the default range for auto created partitions or partitions created with `append_range_partition()` or `prepend_range_partition()` (if `NULL` then auto partition creation feature won't work), `p_count` is the number of premade partitions (if not set then `pg_pathman` tries to determine it based on expression's values). The `bounds` array can be built using `generate_range_bounds()`. Partition creation callback is invoked for each partition if set beforehand.\n\n```plpgsql\ngenerate_range_bounds(p_start     ANYELEMENT,\n                      p_interval  INTERVAL,\n                      p_count     INTEGER)\n\ngenerate_range_bounds(p_start     ANYELEMENT,\n                      p_interval  ANYELEMENT,\n                      p_count     INTEGER)\n```\nBuilds `bounds` array for `create_range_partitions()`.\n\n\n### Data migration\n\n```plpgsql\npartition_table_concurrently(relation   REGCLASS,\n                             batch_size INTEGER DEFAULT 1000,\n                             sleep_time FLOAT8 DEFAULT 1.0)\n```\nStarts a background worker to move data from parent table to partitions. The worker utilizes short transactions to copy small batches of data (up to 10K rows per transaction) and thus doesn't significantly interfere with user's activity. If the worker is unable to lock rows of a batch, it sleeps for `sleep_time` seconds before the next attempt and tries again up to 60 times, and quits if it's still unable to lock the batch.\n\n```plpgsql\nstop_concurrent_part_task(relation REGCLASS)\n```\nStops a background worker performing a concurrent partitioning task. Note: worker will exit after it finishes relocating a current batch.\n\n### Triggers\n\nTriggers are no longer required nor for INSERTs, neither for cross-partition UPDATEs. However, user-supplied triggers *are supported*:\n\n* Each **inserted row** results in execution of `BEFORE/AFTER INSERT` trigger functions of a *corresponding partition*.\n* Each **updated row** results in execution of `BEFORE/AFTER UPDATE` trigger functions of a *corresponding partition*.\n* Each **moved row** (cross-partition update) results in execution of `BEFORE UPDATE` + `BEFORE/AFTER DELETE` + `BEFORE/AFTER INSERT` trigger functions of *corresponding partitions*.\n\n### Post-creation partition management\n```plpgsql\nreplace_hash_partition(old_partition REGCLASS,\n                       new_partition REGCLASS,\n                       lock_parent   BOOLEAN DEFAULT TRUE)\n```\nReplaces specified partition of HASH-partitioned table with another table. The `lock_parent` parameter will prevent any INSERT/UPDATE/ALTER TABLE queries to parent table.\n\n\n```plpgsql\nsplit_range_partition(partition_relid REGCLASS,\n                      split_value     ANYELEMENT,\n                      partition_name  TEXT DEFAULT NULL,\n                      tablespace      TEXT DEFAULT NULL)\n```\nSplit RANGE `partition` in two by `split_value`. Partition creation callback is invoked for a new partition if available.\n\n```plpgsql\nmerge_range_partitions(variadic partitions REGCLASS[])\n```\nMerge several adjacent RANGE partitions. Partitions are automatically ordered by increasing bounds; all the data will be accumulated in the first partition.\n\n```plpgsql\nappend_range_partition(parent_relid   REGCLASS,\n                       partition_name TEXT DEFAULT NULL,\n                       tablespace     TEXT DEFAULT NULL)\n```\nAppend new RANGE partition with `pathman_config.range_interval` as interval.\n\n```plpgsql\nprepend_range_partition(parent_relid   REGCLASS,\n                        partition_name TEXT DEFAULT NULL,\n                        tablespace     TEXT DEFAULT NULL)\n```\nPrepend new RANGE partition with `pathman_config.range_interval` as interval.\n\n```plpgsql\nadd_range_partition(parent_relid   REGCLASS,\n                    start_value    ANYELEMENT,\n                    end_value      ANYELEMENT,\n                    partition_name TEXT DEFAULT NULL,\n                    tablespace     TEXT DEFAULT NULL)\n```\nCreate new RANGE partition for `relation` with specified range bounds. If `start_value` or `end_value` are NULL then corresponding range bound will be infinite.\n\n```plpgsql\ndrop_range_partition(partition TEXT, delete_data BOOLEAN DEFAULT TRUE)\n```\nDrop RANGE partition and all of its data if `delete_data` is true.\n\n```plpgsql\nattach_range_partition(parent_relid    REGCLASS,\n                       partition_relid REGCLASS,\n                       start_value     ANYELEMENT,\n                       end_value       ANYELEMENT)\n```\nAttach partition to the existing RANGE-partitioned relation. The attached table must have exactly the same structure as the parent table, including the dropped columns. Partition creation callback is invoked if set (see `pathman_config_params`).\n\n```plpgsql\ndetach_range_partition(partition_relid REGCLASS)\n```\nDetach partition from the existing RANGE-partitioned relation.\n\n```plpgsql\ndisable_pathman_for(parent_relid REGCLASS)\n```\nPermanently disable `pg_pathman` partitioning mechanism for the specified parent table and remove the insert trigger if it exists. All partitions and data remain unchanged.\n\n```plpgsql\ndrop_partitions(parent_relid REGCLASS,\n                delete_data  BOOLEAN DEFAULT FALSE)\n```\nDrop partitions of the `parent` table (both foreign and local relations). If `delete_data` is `false`, the data is copied to the parent table first. Default is `false`.\n\nTo remove partitioned table along with all partitions fully, use conventional\n`DROP TABLE relation CASCADE`. However, care should be taken in somewhat rare\ncase when you are running logical replication and `DROP` was executed by\nreplication apply worker, e.g. via trigger on replicated table. `pg_pathman`\nuses `pathman_ddl_trigger` event trigger to remove the record about dropped\ntable from `pathman_config`, and this trigger by default won't fire on replica,\nleading to inconsistent state when `pg_pathman` thinks that the table still\nexists, but in fact it doesn't. If this is the case, configure this trigger to\nfire on replica too:\n\n```plpgsql\nALTER EVENT TRIGGER pathman_ddl_trigger ENABLE ALWAYS;\n```\n\nPhysical replication doesn't have this problem since DDL as well as\n`pathman_config` table is replicated too; master and slave PostgreSQL instances\nare basically identical, and it is only harmful to keep this trigger in `ALWAYS`\nmode.\n\n\n### Additional parameters\n\n\n```plpgsql\nset_interval(relation REGCLASS, value ANYELEMENT)\n```\nUpdate RANGE partitioned table interval. Note that interval must not be negative and it must not be trivial, i.e. its value should be greater than zero for numeric types, at least 1 microsecond for `TIMESTAMP` and at least 1 day for `DATE`.\n\n```plpgsql\nset_enable_parent(relation REGCLASS, value BOOLEAN)\n```\nInclude/exclude parent table into/from query plan. In original PostgreSQL planner parent table is always included into query plan even if it's empty which can lead to additional overhead. You can use `disable_parent()` if you are never going to use parent table as a storage. Default value depends on the `partition_data` parameter that was specified during initial partitioning in `create_range_partitions()` function. If the `partition_data` parameter was `true` then all data have already been migrated to partitions and parent table disabled. Otherwise it is enabled.\n\n```plpgsql\nset_auto(relation REGCLASS, value BOOLEAN)\n```\nEnable/disable auto partition propagation (only for RANGE partitioning). It is enabled by default.\n\n```plpgsql\nset_init_callback(relation REGCLASS, callback REGPROC DEFAULT 0)\n```\nSet partition creation callback to be invoked for each attached or created partition (both HASH and RANGE). If callback is marked with SECURITY INVOKER, it's executed with the privileges of the user that produced a statement which has led to creation of a new partition (e.g. `INSERT INTO partitioned_table VALUES (-5)`). The callback must have the following signature: `part_init_callback(args JSONB) RETURNS VOID`. Parameter `arg` consists of several fields whose presence depends on partitioning type:\n```json\n/* RANGE-partitioned table abc (child abc_4) */\n{\n    \"parent\":           \"abc\",\n    \"parent_schema\":    \"public\",\n    \"parttype\":         \"2\",\n    \"partition\":        \"abc_4\",\n    \"partition_schema\": \"public\",\n    \"range_max\":        \"401\",\n    \"range_min\":        \"301\"\n}\n\n/* HASH-partitioned table abc (child abc_0) */\n{\n    \"parent\":           \"abc\",\n    \"parent_schema\":    \"public\",\n    \"parttype\":         \"1\",\n    \"partition\":        \"abc_0\",\n    \"partition_schema\": \"public\"\n}\n```\n\n```plpgsql\nset_set_spawn_using_bgw(relation REGCLASS, value BOOLEAN)\n```\nWhen INSERTing new data beyond the partitioning range, use SpawnPartitionsWorker to create new partitions in a separate transaction.\n\n## Views and tables\n\n#### `pathman_config` --- main config storage\n```plpgsql\nCREATE TABLE IF NOT EXISTS pathman_config (\n    partrel         REGCLASS NOT NULL PRIMARY KEY,\n    expr            TEXT NOT NULL,\n    parttype        INTEGER NOT NULL,\n    range_interval  TEXT,\n    cooked_expr     TEXT);\n```\nThis table stores a list of partitioned tables.\n\n#### `pathman_config_params` --- optional parameters\n```plpgsql\nCREATE TABLE IF NOT EXISTS pathman_config_params (\n    partrel         REGCLASS NOT NULL PRIMARY KEY,\n    enable_parent   BOOLEAN NOT NULL DEFAULT TRUE,\n    auto            BOOLEAN NOT NULL DEFAULT TRUE,\n    init_callback   TEXT DEFAULT NULL,\n    spawn_using_bgw BOOLEAN NOT NULL DEFAULT FALSE);\n```\nThis table stores optional parameters which override standard behavior.\n\n#### `pathman_concurrent_part_tasks` --- currently running partitioning workers\n```plpgsql\n-- helper SRF function\nCREATE OR REPLACE FUNCTION show_concurrent_part_tasks()\nRETURNS TABLE (\n    userid     REGROLE,\n    pid        INT,\n    dbid       OID,\n    relid      REGCLASS,\n    processed  INT,\n    status     TEXT)\nAS 'pg_pathman', 'show_concurrent_part_tasks_internal'\nLANGUAGE C STRICT;\n\nCREATE OR REPLACE VIEW pathman_concurrent_part_tasks\nAS SELECT * FROM show_concurrent_part_tasks();\n```\nThis view lists all currently running concurrent partitioning tasks.\n\n#### `pathman_partition_list` --- list of all existing partitions\n```plpgsql\n-- helper SRF function\nCREATE OR REPLACE FUNCTION show_partition_list()\nRETURNS TABLE (\n    parent     REGCLASS,\n    partition  REGCLASS,\n    parttype   INT4,\n    expr       TEXT,\n    range_min  TEXT,\n    range_max  TEXT)\nAS 'pg_pathman', 'show_partition_list_internal'\nLANGUAGE C STRICT;\n\nCREATE OR REPLACE VIEW pathman_partition_list\nAS SELECT * FROM show_partition_list();\n```\nThis view lists all existing partitions, as well as their parents and range boundaries (NULL for HASH partitions).\n\n#### `pathman_cache_stats` --- per-backend memory consumption\n```plpgsql\n-- helper SRF function\nCREATE OR REPLACE FUNCTION @extschema@.show_cache_stats()\nRETURNS TABLE (\n\tcontext     TEXT,\n\tsize        INT8,\n\tused        INT8,\n\tentries     INT8)\nAS 'pg_pathman', 'show_cache_stats_internal'\nLANGUAGE C STRICT;\n\nCREATE OR REPLACE VIEW @extschema@.pathman_cache_stats\nAS SELECT * FROM @extschema@.show_cache_stats();\n```\nShows memory consumption of various caches.\n\n## Declarative partitioning\n\nFrom PostgreSQL 10 `ATTACH PARTITION`, `DETACH PARTITION`\nand `CREATE TABLE .. PARTITION OF` commands could be used with tables\npartitioned by `pg_pathman`:\n\n```plpgsql\nCREATE TABLE child1 (LIKE partitioned_table);\n\n--- attach new partition\nALTER TABLE partitioned_table ATTACH PARTITION child1\n\tFOR VALUES FROM ('2015-05-01') TO ('2015-06-01');\n\n--- detach the partition\nALTER TABLE partitioned_table DETACH PARTITION child1;\n\n-- create a partition\nCREATE TABLE child2 PARTITION OF partitioned_table\n\tFOR VALUES IN ('2015-05-01', '2015-06-01');\n```\n\n## Custom plan nodes\n`pg_pathman` provides a couple of [custom plan nodes](https://wiki.postgresql.org/wiki/CustomScanAPI) which aim to reduce execution time, namely:\n\n- `RuntimeAppend` (overrides `Append` plan node)\n- `RuntimeMergeAppend` (overrides `MergeAppend` plan node)\n- `PartitionFilter` (drop-in replacement for INSERT triggers)\n- `PartitionOverseer` (implements cross-partition UPDATEs)\n- `PartitionRouter` (implements cross-partition UPDATEs)\n\n`PartitionFilter` acts as a *proxy node* for INSERT's child scan, which means it can redirect output tuples to the corresponding partition:\n\n```plpgsql\nEXPLAIN (COSTS OFF)\nINSERT INTO partitioned_table\nSELECT generate_series(1, 10), random();\n               QUERY PLAN\n-----------------------------------------\n Insert on partitioned_table\n   -\u003e  Custom Scan (PartitionFilter)\n         -\u003e  Subquery Scan on \"*SELECT*\"\n               -\u003e  Result\n(4 rows)\n```\n\n`PartitionOverseer` and `PartitionRouter` are another *proxy nodes* used\nin conjunction with `PartitionFilter` to enable cross-partition UPDATEs\n(i.e. when update of partitioning key requires that we move row to another\npartition). Since this node has a great deal of side effects (ordinary `UPDATE` becomes slower;\ncross-partition `UPDATE` is transformed into `DELETE + INSERT`),\nit is disabled by default.\nTo enable it, refer to the list of [GUCs](#disabling-pg_pathman) below.\n\n```plpgsql\nEXPLAIN (COSTS OFF)\nUPDATE partitioned_table\nSET value = value + 1 WHERE value = 2;\n                       QUERY PLAN\n---------------------------------------------------------\n Custom Scan (PartitionOverseer)\n   -\u003e  Update on partitioned_table_2\n         -\u003e  Custom Scan (PartitionFilter)\n               -\u003e  Custom Scan (PartitionRouter)\n                     -\u003e  Seq Scan on partitioned_table_2\n                           Filter: (value = 2)\n(6 rows)\n```\n\n`RuntimeAppend` and `RuntimeMergeAppend` have much in common: they come in handy in a case when WHERE condition takes form of:\n```\nVARIABLE OP PARAM\n```\nThis kind of expressions can no longer be optimized at planning time since the parameter's value is not known until the execution stage takes place. The problem can be solved by embedding the *WHERE condition analysis routine* into the original `Append`'s code, thus making it pick only required scans out of a whole bunch of planned partition scans. This effectively boils down to creation of a custom node capable of performing such a check.\n\n----------\n\nThere are at least several cases that demonstrate usefulness of these nodes:\n\n```plpgsql\n/* create table we're going to partition */\nCREATE TABLE partitioned_table(id INT NOT NULL, payload REAL);\n\n/* insert some data */\nINSERT INTO partitioned_table\nSELECT generate_series(1, 1000), random();\n\n/* perform partitioning */\nSELECT create_hash_partitions('partitioned_table', 'id', 100);\n\n/* create ordinary table */\nCREATE TABLE some_table AS SELECT generate_series(1, 100) AS VAL;\n```\n\n\n - **`id = (select ... limit 1)`**\n```plpgsql\nEXPLAIN (COSTS OFF, ANALYZE) SELECT * FROM partitioned_table\nWHERE id = (SELECT * FROM some_table LIMIT 1);\n                                             QUERY PLAN\n----------------------------------------------------------------------------------------------------\n Custom Scan (RuntimeAppend) (actual time=0.030..0.033 rows=1 loops=1)\n   InitPlan 1 (returns $0)\n     -\u003e  Limit (actual time=0.011..0.011 rows=1 loops=1)\n           -\u003e  Seq Scan on some_table (actual time=0.010..0.010 rows=1 loops=1)\n   -\u003e  Seq Scan on partitioned_table_70 partitioned_table (actual time=0.004..0.006 rows=1 loops=1)\n         Filter: (id = $0)\n         Rows Removed by Filter: 9\n Planning time: 1.131 ms\n Execution time: 0.075 ms\n(9 rows)\n\n/* disable RuntimeAppend node */\nSET pg_pathman.enable_runtimeappend = f;\n\nEXPLAIN (COSTS OFF, ANALYZE) SELECT * FROM partitioned_table\nWHERE id = (SELECT * FROM some_table LIMIT 1);\n                                    QUERY PLAN\n----------------------------------------------------------------------------------\n Append (actual time=0.196..0.274 rows=1 loops=1)\n   InitPlan 1 (returns $0)\n     -\u003e  Limit (actual time=0.005..0.005 rows=1 loops=1)\n           -\u003e  Seq Scan on some_table (actual time=0.003..0.003 rows=1 loops=1)\n   -\u003e  Seq Scan on partitioned_table_0 (actual time=0.014..0.014 rows=0 loops=1)\n         Filter: (id = $0)\n         Rows Removed by Filter: 6\n   -\u003e  Seq Scan on partitioned_table_1 (actual time=0.003..0.003 rows=0 loops=1)\n         Filter: (id = $0)\n         Rows Removed by Filter: 5\n         ... /* more plans follow */\n Planning time: 1.140 ms\n Execution time: 0.855 ms\n(306 rows)\n```\n\n - **`id = ANY (select ...)`**\n```plpgsql\nEXPLAIN (COSTS OFF, ANALYZE) SELECT * FROM partitioned_table\nWHERE id = any (SELECT * FROM some_table limit 4);\n                                                QUERY PLAN\n-----------------------------------------------------------------------------------------------------------\n Nested Loop (actual time=0.025..0.060 rows=4 loops=1)\n   -\u003e  Limit (actual time=0.009..0.011 rows=4 loops=1)\n         -\u003e  Seq Scan on some_table (actual time=0.008..0.010 rows=4 loops=1)\n   -\u003e  Custom Scan (RuntimeAppend) (actual time=0.002..0.004 rows=1 loops=4)\n         -\u003e  Seq Scan on partitioned_table_70 partitioned_table (actual time=0.001..0.001 rows=10 loops=1)\n         -\u003e  Seq Scan on partitioned_table_26 partitioned_table (actual time=0.002..0.003 rows=9 loops=1)\n         -\u003e  Seq Scan on partitioned_table_27 partitioned_table (actual time=0.001..0.002 rows=20 loops=1)\n         -\u003e  Seq Scan on partitioned_table_63 partitioned_table (actual time=0.001..0.002 rows=9 loops=1)\n Planning time: 0.771 ms\n Execution time: 0.101 ms\n(10 rows)\n\n/* disable RuntimeAppend node */\nSET pg_pathman.enable_runtimeappend = f;\n\nEXPLAIN (COSTS OFF, ANALYZE) SELECT * FROM partitioned_table\nWHERE id = any (SELECT * FROM some_table limit 4);\n                                       QUERY PLAN\n-----------------------------------------------------------------------------------------\n Nested Loop Semi Join (actual time=0.531..1.526 rows=4 loops=1)\n   Join Filter: (partitioned_table.id = some_table.val)\n   Rows Removed by Join Filter: 3990\n   -\u003e  Append (actual time=0.190..0.470 rows=1000 loops=1)\n         -\u003e  Seq Scan on partitioned_table (actual time=0.187..0.187 rows=0 loops=1)\n         -\u003e  Seq Scan on partitioned_table_0 (actual time=0.002..0.004 rows=6 loops=1)\n         -\u003e  Seq Scan on partitioned_table_1 (actual time=0.001..0.001 rows=5 loops=1)\n         -\u003e  Seq Scan on partitioned_table_2 (actual time=0.002..0.004 rows=14 loops=1)\n... /* 96 scans follow */\n   -\u003e  Materialize (actual time=0.000..0.000 rows=4 loops=1000)\n         -\u003e  Limit (actual time=0.005..0.006 rows=4 loops=1)\n               -\u003e  Seq Scan on some_table (actual time=0.003..0.004 rows=4 loops=1)\n Planning time: 2.169 ms\n Execution time: 2.059 ms\n(110 rows)\n```\n\n - **`NestLoop` involving a partitioned table**, which is omitted since it's occasionally shown above.\n\n----------\n\nIn case you're interested, you can read more about custom nodes at Alexander Korotkov's [blog](http://akorotkov.github.io/blog/2016/06/15/pg_pathman-runtime-append/).\n\n\n## Examples\n\n### Common tips\n- You can easily add **_partition_** column containing the names of the underlying partitions using the system attribute called **_tableoid_**:\n```plpgsql\nSELECT tableoid::regclass AS partition, * FROM partitioned_table;\n```\n\n- Though indices on a parent table aren't particularly useful (since it's supposed to be empty), they act as prototypes for indices on partitions. For each index on the parent table, `pg_pathman` will create a similar index on every partition.\n\n- All running concurrent partitioning tasks can be listed using the `pathman_concurrent_part_tasks` view:\n```plpgsql\nSELECT * FROM pathman_concurrent_part_tasks;\n userid | pid  | dbid  | relid | processed | status\n--------+------+-------+-------+-----------+---------\n dmitry | 7367 | 16384 | test  |    472000 | working\n(1 row)\n```\n\n- `pathman_partition_list` in conjunction with `drop_range_partition()` can be used to drop RANGE partitions in a more flexible way compared to good old `DROP TABLE`:\n```plpgsql\nSELECT drop_range_partition(partition, false) /* move data to parent */\nFROM pathman_partition_list\nWHERE parent = 'part_test'::regclass AND range_min::int \u003c 500;\nNOTICE:  1 rows copied from part_test_11\nNOTICE:  100 rows copied from part_test_1\nNOTICE:  100 rows copied from part_test_2\n drop_range_partition\n----------------------\n dummy_test_11\n dummy_test_1\n dummy_test_2\n(3 rows)\n```\n\n- You can turn foreign tables into partitions using the `attach_range_partition()` function. Rows that were meant to be inserted into parent will be redirected to foreign partitions (as usual, PartitionFilter will be involved), though by default it is prohibited to insert rows into partitions provided not by `postgres_fdw`. Only superuser is allowed to set `pg_pathman.insert_into_fdw` [GUC](#disabling-pg_pathman) variable.\n\n### HASH partitioning\nConsider an example of HASH partitioning. First create a table with some integer column:\n```plpgsql\nCREATE TABLE items (\n    id       SERIAL PRIMARY KEY,\n    name     TEXT,\n    code     BIGINT);\n\nINSERT INTO items (id, name, code)\nSELECT g, md5(g::text), random() * 100000\nFROM generate_series(1, 100000) as g;\n```\nNow run the `create_hash_partitions()` function with appropriate arguments:\n```plpgsql\nSELECT create_hash_partitions('items', 'id', 100);\n```\nThis will create new partitions and move the data from parent to partitions.\n\nHere's an example of the query performing filtering by partitioning key:\n```plpgsql\nSELECT * FROM items WHERE id = 1234;\n  id  |               name               | code\n------+----------------------------------+------\n 1234 | 81dc9bdb52d04dc20036dbd8313ed055 | 1855\n(1 row)\n\nEXPLAIN SELECT * FROM items WHERE id = 1234;\n                                     QUERY PLAN\n------------------------------------------------------------------------------------\n Append  (cost=0.28..8.29 rows=0 width=0)\n   -\u003e  Index Scan using items_34_pkey on items_34  (cost=0.28..8.29 rows=0 width=0)\n         Index Cond: (id = 1234)\n```\n\nNotice that the `Append` node contains only one child scan which corresponds to the WHERE clause.\n\n\u003e **Important:** pay attention to the fact that `pg_pathman` excludes the parent table from the query plan.\n\nTo access parent table use ONLY modifier:\n```plpgsql\nEXPLAIN SELECT * FROM ONLY items;\n                      QUERY PLAN\n------------------------------------------------------\n Seq Scan on items  (cost=0.00..0.00 rows=1 width=45)\n```\n### RANGE partitioning\nConsider an example of RANGE partitioning. Let's create a table containing some dummy logs:\n```plpgsql\nCREATE TABLE journal (\n    id      SERIAL,\n    dt      TIMESTAMP NOT NULL,\n    level   INTEGER,\n    msg     TEXT);\n\n-- similar index will also be created for each partition\nCREATE INDEX ON journal(dt);\n\n-- generate some data\nINSERT INTO journal (dt, level, msg)\nSELECT g, random() * 6, md5(g::text)\nFROM generate_series('2015-01-01'::date, '2015-12-31'::date, '1 minute') as g;\n```\nRun the `create_range_partitions()` function to create partitions so that each partition would contain the data for one day:\n```plpgsql\nSELECT create_range_partitions('journal', 'dt', '2015-01-01'::date, '1 day'::interval);\n```\nIt will create 365 partitions and move the data from parent to partitions.\n\nNew partitions are appended automaticaly by insert trigger, but it can be done manually with the following functions:\n```plpgsql\n-- add new partition with specified range\nSELECT add_range_partition('journal', '2016-01-01'::date, '2016-01-07'::date);\n\n-- append new partition with default range\nSELECT append_range_partition('journal');\n```\nThe first one creates a partition with specified range. The second one creates a partition with default interval and appends it to the partition list. It is also possible to attach an existing table as partition. For example, we may want to attach an archive table (or even foreign table from another server) for some outdated data:\n```plpgsql\nCREATE FOREIGN TABLE journal_archive (\n    id      INTEGER NOT NULL,\n    dt      TIMESTAMP NOT NULL,\n    level   INTEGER,\n    msg     TEXT)\nSERVER archive_server;\n\nSELECT attach_range_partition('journal', 'journal_archive', '2014-01-01'::date, '2015-01-01'::date);\n```\n\u003e **Important:** the definition of the attached table must match the one of the existing partitioned table, including the dropped columns.\n\nTo merge to adjacent partitions, use the `merge_range_partitions()` function:\n```plpgsql\nSELECT merge_range_partitions('journal_archive', 'journal_1');\n```\nTo split partition by value, use the `split_range_partition()` function:\n```plpgsql\nSELECT split_range_partition('journal_366', '2016-01-03'::date);\n```\nTo detach partition, use the `detach_range_partition()` function:\n```plpgsql\nSELECT detach_range_partition('journal_archive');\n```\n\nHere's an example of the query performing filtering by partitioning key:\n```plpgsql\nSELECT * FROM journal WHERE dt \u003e= '2015-06-01' AND dt \u003c '2015-06-03';\n   id   |         dt          | level |               msg\n--------+---------------------+-------+----------------------------------\n 217441 | 2015-06-01 00:00:00 |     2 | 15053892d993ce19f580a128f87e3dbf\n 217442 | 2015-06-01 00:01:00 |     1 | 3a7c46f18a952d62ce5418ac2056010c\n 217443 | 2015-06-01 00:02:00 |     0 | 92c8de8f82faf0b139a3d99f2792311d\n ...\n(2880 rows)\n\nEXPLAIN SELECT * FROM journal WHERE dt \u003e= '2015-06-01' AND dt \u003c '2015-06-03';\n                            QUERY PLAN\n------------------------------------------------------------------\n Append  (cost=0.00..58.80 rows=0 width=0)\n   -\u003e  Seq Scan on journal_152  (cost=0.00..29.40 rows=0 width=0)\n   -\u003e  Seq Scan on journal_153  (cost=0.00..29.40 rows=0 width=0)\n(3 rows)\n```\n\n### Disabling `pg_pathman`\nThere are several user-accessible [GUC](https://www.postgresql.org/docs/9.5/static/config-setting.html) variables designed to toggle the whole module or specific custom nodes on and off:\n\n - `pg_pathman.enable` --- disable (or enable) `pg_pathman` **completely**\n - `pg_pathman.enable_runtimeappend` --- toggle `RuntimeAppend` custom node on\\off\n - `pg_pathman.enable_runtimemergeappend` --- toggle `RuntimeMergeAppend` custom node on\\off\n - `pg_pathman.enable_partitionfilter` --- toggle `PartitionFilter` custom node on\\off (for INSERTs)\n - `pg_pathman.enable_partitionrouter` --- toggle `PartitionRouter` custom node on\\off (for cross-partition UPDATEs)\n - `pg_pathman.enable_auto_partition` --- toggle automatic partition creation on\\off (per session)\n - `pg_pathman.enable_bounds_cache` --- toggle bounds cache on\\off (faster updates of partitioning scheme)\n - `pg_pathman.insert_into_fdw` --- allow INSERTs into various FDWs `(disabled | postgres | any_fdw)`\n - `pg_pathman.override_copy` --- toggle COPY statement hooking on\\off\n\nTo **permanently** disable `pg_pathman` for some previously partitioned table, use the `disable_pathman_for()` function:\n```plpgsql\nSELECT disable_pathman_for('range_rel');\n```\nAll sections and data will remain unchanged and will be handled by the standard PostgreSQL inheritance mechanism.\n\n## Feedback\nDo not hesitate to post your issues, questions and new ideas at the [issues](https://github.com/postgrespro/pg_pathman/issues) page.\n\n## Authors\n[Ildar Musin](https://github.com/zilder)\n[Alexander Korotkov](https://github.com/akorotkov)\n[Dmitry Ivanov](https://github.com/funbringer)\n[Maksim Milyutin](https://github.com/maksm90)\n[Ildus Kurbangaliev](https://github.com/ildus)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostgrespro%2Fpg_pathman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpostgrespro%2Fpg_pathman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostgrespro%2Fpg_pathman/lists"}