{"id":13746069,"url":"https://github.com/klando/pgfincore","last_synced_at":"2025-05-09T06:31:21.067Z","repository":{"id":46123615,"uuid":"2112962","full_name":"klando/pgfincore","owner":"klando","description":"this repos is moving to https://codeberg.org/c2main/pgfincore | Mirror of http://git.postgresql.org/gitweb/?p=pgfincore.git;a=summary","archived":false,"fork":false,"pushed_at":"2024-12-15T11:26:45.000Z","size":254,"stargazers_count":104,"open_issues_count":0,"forks_count":12,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-12-15T12:20:14.290Z","etag":null,"topics":["database","postgres","postgresql","postgresql-extension"],"latest_commit_sha":null,"homepage":"http://villemain.org/projects/pgfincore","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/klando.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS"}},"created_at":"2011-07-27T13:35:54.000Z","updated_at":"2024-11-28T16:28:58.000Z","dependencies_parsed_at":"2024-01-15T00:45:59.142Z","dependency_job_id":null,"html_url":"https://github.com/klando/pgfincore","commit_stats":null,"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klando%2Fpgfincore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klando%2Fpgfincore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klando%2Fpgfincore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klando%2Fpgfincore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/klando","download_url":"https://codeload.github.com/klando/pgfincore/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253206069,"owners_count":21871158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","postgres","postgresql","postgresql-extension"],"created_at":"2024-08-03T06:00:45.576Z","updated_at":"2025-05-09T06:31:20.759Z","avatar_url":"https://github.com/klando.png","language":"C","funding_links":[],"categories":["C"],"sub_categories":[],"readme":"[![CI](https://github.com/klando/pgfincore/actions/workflows/main.yml/badge.svg?branch=master)](https://github.com/klando/pgfincore/actions/workflows/main.yml)\n\n# PgFincore\n\n--------------------------------------------------------------\n A set of functions to manage pages in memory from PostgreSQL\n--------------------------------------------------------------\n\nA set of functions to handle low-level management of relations using mincore to\nexplore cache memory.\n\n## DESCRIPTION\n\nWith PostgreSQL, each Table or Index is splitted in segments of (usually) 1GB,\nand each segment is splitted in pages in memory then in blocks for the\nfilesystem.\n\nThose functions let you know which and how many disk block from a relation are\nin the page cache of the operating system. It can provide the result as a VarBit\nand can be stored in a table. Then using this table, it is possible to restore\nthe page cache state for each block of the relation, even in another server,\nthanks to Streaming Replication.\n\nOther functions are used to set a *POSIX_FADVISE* flag on the entire relation\n(each segment). The more usefull are probably *WILLNEED* and *DONTNEED* which\npush and pop blocks of each segments of a relation from page cache,\nrespectively.\n\nEach functions are call with at least a table name or an index name (or oid)\nas a parameter and walk each segment of the relation.\n\n## DOWNLOAD\n\nYou can grab the latest code with git:\n\n    git clone git://git.postgresql.org/git/pgfincore.git\n    or\n    git://github.com/klando/pgfincore.git\n\nAnd the project is on pgfoundry : http://pgfoundry.org/projects/pgfincore\n\n## INSTALL\n\nFrom source code:\n\n    make clean\n    make\n    su\n    make install\n\nFor PostgreSQL \u003e= 9.1, log in your database and:\n\n    mydb=# CREATE EXTENSION pgfincore;\n\nFor other release, create the functions from the sql script (it should be in\nyour contrib directory):\n\n    psql mydb -f pgfincore.sql\n\nPgFincore is also shipped with Debian scripts to build your own package:\n\n    aptitude install debhelper postgresql-server-dev-all postgresql-server-dev-9.1\n    # or postgresql-server-dev-8.4|postgresql-server-dev-9.0\n    make deb\n    dpkg -i ../postgresql-9.1-pgfincore_1.1.1-1_amd64.deb\n\nPgFincore is packaged for *RPM* at http://yum.postgresql.org/\nPgFincore is packaged for *debian* at http://pgapt.debian.net/\n\n\n\n\n## EXAMPLES\n\nHere are some examples of usage. If you want more details go to Documentation_\n\n### Get current state of a relation\n\nMay be useful:\n\n    cedric=# select * from pgfincore('pgbench_accounts');\n          relpath       | segment | os_page_size | rel_os_pages | pages_mem | group_mem | os_pages_free | databit | pages_dirty | group_dirty \n    --------------------+---------+--------------+--------------+-----------+-----------+---------------+---------+-------------+-------------\n     base/11874/16447   |       0 |         4096 |       262144 |    262144 |         1 |         81016 |         |           0 |           0\n     base/11874/16447.1 |       1 |         4096 |        65726 |     65726 |         1 |         81016 |         |           0 |           0\n    (2 rows)\n    \n    Time: 31.563 ms\n\n### Load a table or an index in OS Page Buffer\n\nYou may want to try to keep a table or an index into the OS Page Cache, or\npreload a table before your well know big query is executed (reducing the query\ntime).\n\nTo do so, just execute the following query:\n\n    cedric=# select * from pgfadvise_willneed('pgbench_accounts');\n          relpath       | os_page_size | rel_os_pages | os_pages_free \n    --------------------+--------------+--------------+---------------\n     base/11874/16447   |         4096 |       262144 |        169138\n     base/11874/16447.1 |         4096 |        65726 |        103352\n    (2 rows)\n     \n    Time: 4462,936 ms\n\n  * The column *os_page_size* report that page size is 4KB.\n  * The column *rel_os_pages* is the number of pages of the specified file.\n  * The column *os_pages_free* is the number of free pages in memory (for caching).\n\n### Snapshot and Restore the OS Page Buffer state of a table or an index (or more)\n\nYou may want to restore a table or an index into the OS Page Cache as it was\nwhile you did the snapshot. For example if you have to reboot your server, then\nwhen PostgreSQL start up the first queries might be slower because neither\nPostgreSQL or the OS have pages in their respective cache about the relations\ninvolved in those first queries.\n\nExecuting a snapshot and a restore is very simple:\n\n    -- Snapshot\n    cedric=# create table pgfincore_snapshot as\n    cedric-#   select 'pgbench_accounts'::text as relname,*,now() as date_snapshot\n    cedric-#   from pgfincore('pgbench_accounts',true);\n    \n    -- Restore\n    cedric=# select * from pgfadvise_loader('pgbench_accounts', 0, true, true,\n                           (select databit from  pgfincore_snapshot\n                            where relname='pgbench_accounts' and segment = 0));\n         relpath      | os_page_size | os_pages_free | pages_loaded | pages_unloaded \n    ------------------+--------------+---------------+--------------+----------------\n     base/11874/16447 |         4096 |         80867 |       262144 |              0\n    (1 row)\n    \n    Time: 35.349 ms\n\n * The column *pages_loaded* report how many pages have been read to memory\n   (they may have already been in memoy)\n * The column *pages_unloaded* report how many pages have been removed from\n   memory (they may not have already been in memoy);\n\n## SYNOPSIS\n\n    pgsysconf(OUT os_page_size bigint, OUT os_pages_free bigint,\n              OUT os_total_pages bigint)\n      RETURNS record\n\n    pgsysconf_pretty(OUT os_page_size text, OUT os_pages_free text,\n                     OUT os_total_pages text)\n      RETURNS record\n\n    pgfadvise(IN relname regclass, IN fork text, IN action int,\n              OUT relpath text, OUT os_page_size bigint,\n              OUT rel_os_pages bigint, OUT os_pages_free bigint)\n      RETURNS setof record\n\n    pgfadvise_willneed(IN relname regclass,\n                       OUT relpath text, OUT os_page_size bigint,\n                       OUT rel_os_pages bigint, OUT os_pages_free bigint)\n      RETURNS setof record\n\n    pgfadvise_dontneed(IN relname regclass,\n                       OUT relpath text, OUT os_page_size bigint,\n                       OUT rel_os_pages bigint, OUT os_pages_free bigint)\n      RETURNS setof record\n\n    pgfadvise_normal(IN relname regclass,\n                     OUT relpath text, OUT os_page_size bigint,\n                     OUT rel_os_pages bigint, OUT os_pages_free bigint)\n      RETURNS setof record\n\n    pgfadvise_sequential(IN relname regclass,\n                         OUT relpath text, OUT os_page_size bigint,\n                         OUT rel_os_pages bigint, OUT os_pages_free bigint)\n      RETURNS setof record\n\n    pgfadvise_random(IN relname regclass,\n                     OUT relpath text, OUT os_page_size bigint,\n                     OUT rel_os_pages bigint, OUT os_pages_free bigint)\n      RETURNS setof record\n\n    pgfadvise_loader(IN relname regclass, IN fork text, IN segment int,\n                     IN load bool, IN unload bool, IN databit varbit,\n                     OUT relpath text, OUT os_page_size bigint,\n                     OUT os_pages_free bigint, OUT pages_loaded bigint,\n                     OUT pages_unloaded bigint)\n      RETURNS setof record\n\n    pgfadvise_loader(IN relname regclass, IN segment int,\n                     IN load bool, IN unload bool, IN databit varbit,\n                     OUT relpath text, OUT os_page_size bigint,\n                     OUT os_pages_free bigint, OUT pages_loaded bigint,\n                     OUT pages_unloaded bigint)\n      RETURNS setof record\n\n    pgfincore(IN relname regclass, IN fork text, IN getdatabit bool,\n              OUT relpath text, OUT segment int, OUT os_page_size bigint,\n              OUT rel_os_pages bigint, OUT pages_mem bigint,\n              OUT group_mem bigint, OUT os_pages_free bigint,\n              OUT databit varbit, OUT pages_dirty bigint,\n              OUT group_dirty bigint)\n      RETURNS setof record\n\n    pgfincore(IN relname regclass, IN getdatabit bool,\n              OUT relpath text, OUT segment int, OUT os_page_size bigint,\n              OUT rel_os_pages bigint, OUT pages_mem bigint,\n              OUT group_mem bigint, OUT os_pages_free bigint,\n              OUT databit varbit, OUT pages_dirty bigint,\n              OUT group_dirty bigint)\n     RETURNS setof record\n\n    pgfincore(IN relname regclass,\n              OUT relpath text, OUT segment int, OUT os_page_size bigint,\n              OUT rel_os_pages bigint, OUT pages_mem bigint,\n              OUT group_mem bigint, OUT os_pages_free bigint,\n              OUT databit varbit, OUT pages_dirty bigint,\n              OUT group_dirty bigint)\n      RETURNS setof record\n\n## DOCUMENTATION\n\n### pgsysconf\n\nThis function output size of OS blocks, number of free page in the OS Page Buffer.\n\n    cedric=# select * from pgsysconf();\n     os_page_size | os_pages_free | os_total_pages \n    --------------+---------------+----------------\n             4096 |         80431 |        4094174\n\n### pgsysconf_pretty\n\nThe same as above, but with pretty output.\n\n    cedric=# select * from pgsysconf_pretty();\n     os_page_size | os_pages_free | os_total_pages \n    --------------+---------------+----------------\n    4096 bytes   | 314 MB        | 16 GB\n\n### pgfadvise_WILLNEED\n\nThis function set *WILLNEED* flag on the current relation. It means that the\nOperating Sytem will try to load as much pages as possible of the relation.\nMain idea is to preload files on server startup, perhaps using cache hit/miss\nratio or most required relations/indexes.\n\n    cedric=# select * from pgfadvise_willneed('pgbench_accounts');\n          relpath       | os_page_size | rel_os_pages | os_pages_free \n    --------------------+--------------+--------------+---------------\n     base/11874/16447   |         4096 |       262144 |         80650\n     base/11874/16447.1 |         4096 |        65726 |         80650\n\n### pgfadvise_DONTNEED\n\nThis function set *DONTNEED* flag on the current relation. It means that the\nOperating System will first unload pages of the file if it need to free some\nmemory. Main idea is to unload files when they are not usefull anymore (instead\nof perhaps more interesting pages)\n\n    cedric=# select * from pgfadvise_dontneed('pgbench_accounts');\n          relpath       | os_page_size | rel_os_pages | os_pages_free\n    --------------------+--------------+--------------+---------------\n     base/11874/16447   |         4096 |       262144 |        342071\n     base/11874/16447.1 |         4096 |        65726 |        408103\n\n\n### pgfadvise_NORMAL\n\nThis function set *NORMAL* flag on the current relation.\n\n### pgfadvise_SEQUENTIAL\n\nThis function set *SEQUENTIAL* flag on the current relation.\n\n### pgfadvise_RANDOM\n\nThis function set *RANDOM* flag on the current relation.\n\n### pgfadvise_loader\n\nThis function allow to interact directly with the Page Cache.\nIt can be used to load and/or unload page from memory based on a varbit\nrepresenting the map of the pages to load/unload accordingly.\n\nWork with relation pgbench_accounts, segment 0, arbitrary varbit map:\n\n    -- Loading and Unloading\n    cedric=# select * from pgfadvise_loader('pgbench_accounts', 0, true, true, B'111000');\n         relpath      | os_page_size | os_pages_free | pages_loaded | pages_unloaded \n    ------------------+--------------+---------------+--------------+----------------\n     base/11874/16447 |         4096 |        408376 |            3 |              3\n \n    -- Loading\n    cedric=# select * from pgfadvise_loader('pgbench_accounts', 0, true, false, B'111000');\n         relpath      | os_page_size | os_pages_free | pages_loaded | pages_unloaded \n    ------------------+--------------+---------------+--------------+----------------\n     base/11874/16447 |         4096 |        408370 |            3 |              0\n \n    -- Unloading\n    cedric=# select * from pgfadvise_loader('pgbench_accounts', 0, false, true, B'111000');\n        relpath      | os_page_size | os_pages_free | pages_loaded | pages_unloaded \n    ------------------+--------------+---------------+--------------+----------------\n     base/11874/16447 |         4096 |        408370 |            0 |              3\n\n### pgfincore\n\nThis function provide information about the file system cache (page cache). \n\n    cedric=# select * from pgfincore('pgbench_accounts');\n          relpath       | segment | os_page_size | rel_os_pages | pages_mem | group_mem | os_pages_free | databit | pages_dirty | group_dirty  \n    --------------------+---------+--------------+--------------+-----------+-----------+---------------+---------+-------------+-------------\n     base/11874/16447   |       0 |         4096 |       262144 |         3 |         1 |        408444 |         |           0 |           0\n     base/11874/16447.1 |       1 |         4096 |        65726 |         0 |         0 |        408444 |         |           0 |           0\n\nFor the specified relation it returns:\n\n  * relpath : the relation path \n  * segment : the segment number analyzed \n  * os_page_size : the size of one page\n  * rel_os_pages : the total number of pages of the relation\n  * pages_mem : the total number of relation's pages in page cache.\n    (not the shared buffers from PostgreSQL but the OS cache)\n  * group_mem : the number of groups of adjacent pages_mem\n  * os_page_free : the number of free page in the OS page cache\n  * databit : the varbit map of the file, because of its size it is useless to output\n    Use pgfincore('pgbench_accounts',true) to activate it.\n  * pages_dirty : if HAVE_FINCORE constant is define and the platorm provides the relevant information, like pages_mem but for dirtied pages \n  * group_dirty : if HAVE_FINCORE constant is define and the platorm provides the relevant information, like group_mem but for dirtied pages \n\n## DEBUG\n\nYou can debug the PgFincore with the following error level: *DEBUG1* and\n*DEBUG5*.\n\nFor example:\n\n    set client_min_messages TO debug1; -- debug5 is only usefull to trace each block\n\n## REQUIREMENTS\n\n * PgFincore needs mincore() or fincore() and POSIX_FADVISE\n\n## LIMITATIONS\n\n * PgFincore has a limited mode when POSIX_FADVISE is not provided by the platform.\n\n * PgFincore needs PostgreSQL \u003e= 8.3\n\n * PgFincore does not work on windows.\n\n## SEE ALSO\n\nData Bene, PostgreSQL Expertise, Technical Support and Assistance, Trainings:\n\n  https://www.data-bene.io\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fklando%2Fpgfincore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fklando%2Fpgfincore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fklando%2Fpgfincore/lists"}