{"id":22815887,"url":"https://github.com/serpent7776/pg_worker_pool","last_synced_at":"2025-04-14T00:32:47.744Z","repository":{"id":267364802,"uuid":"899176806","full_name":"serpent7776/pg_worker_pool","owner":"serpent7776","description":"Postgres extension creating a pool of background workers","archived":false,"fork":false,"pushed_at":"2025-01-11T16:02:50.000Z","size":20,"stargazers_count":23,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-11T17:22:20.303Z","etag":null,"topics":["c","extension","postgres","postgresql","worker","worker-pool"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/serpent7776.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-05T19:02:26.000Z","updated_at":"2025-01-11T16:02:54.000Z","dependencies_parsed_at":"2024-12-09T23:30:16.348Z","dependency_job_id":"b6241768-6db2-4ddd-9212-56d3905b2c48","html_url":"https://github.com/serpent7776/pg_worker_pool","commit_stats":null,"previous_names":["serpent7776/pg_worker_pool"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serpent7776%2Fpg_worker_pool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serpent7776%2Fpg_worker_pool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serpent7776%2Fpg_worker_pool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serpent7776%2Fpg_worker_pool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/serpent7776","download_url":"https://codeload.github.com/serpent7776/pg_worker_pool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237413189,"owners_count":19306035,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c","extension","postgres","postgresql","worker","worker-pool"],"created_at":"2024-12-12T14:05:35.454Z","updated_at":"2025-02-06T03:39:58.978Z","avatar_url":"https://github.com/serpent7776.png","language":"C","readme":"# pg_worker_pool\n\nA Postgresql extension that creates a pool of named background workers.\n\nThis uses [Postgresql background workers](https://www.postgresql.org/docs/current/bgworker.html) functionality.\n\nKeep in mind that Postgresql has limited number of available background workers. This is specified by `max_worker_processes` configuration. Changing that limit requires server restart.\n\nCurrently this extension has hard-coded number of workers which is set to `8`.\nIf you want to change it, you can set `MAX_WORKERS` variable in `pg_worker_pool.c` to any other number and recompile the extension.\nIncreasing this number will let you start more workers.\nMaking this larger than `max_worker_processes` will yield no effect though. This is a Postgresql limitation.\nYou could also decrease that number, but I'm not sure why would one do that. You can control how many workers actually start by choosing appropriate worker name.\n\nBackground workers are created on-demand and exit as soon as they aren't needed.\n\n## Usage\n\nUpon creation, this extension creates `worker_pool` schema.\n\n`worker_pool.submit` can be used to add query to be executed on a specified background worker. This also starts the worker at the end of current transaction (if committed). If the current transaction is aborted, the worker is not started.\nIf you start more than one worker in the same transaction, the order in which they are started in unspecified. It might be and likely will be different than the order of submission. Do not rely on this ordering.\n\n`worker_pool.launch` can be used to start a specified worker without adding any queries to the queue.\n\n`worker_pool.jobs` stores all queries submitted to be executed.\n\nWorker name can be any string with max length of `NAMEDATALEN` (by default 64 characters).\n\nThis extension must be loaded via `shared_preload_libraries` to correctly allocate required shared memory.\n\n## How to install\n\nRegular `git clone` + `make` + `make install` should be enough.\n\nNote that `pg_config` utility needs to be in your `$PATH`.\nIt is commonly part of `postgresql-common` package on Debian/Ubuntu and `postgresql-libs` on Arch systems.\n\n```\ngit clone https://github.com/serpent7776/pg_worker_pool.git\ncd pg_worker_pool\nmake\nsudo make install\n```\n\n## Example\n\n```sql\nCREATE EXTENSION pg_worker_pool;\nCALL worker_pool.submit('foo', 'create index myindex_1 on my_big_table (id)');\nCALL worker_pool.submit('foo', 'create index myindex_2 on my_big_table (name)');\nCALL worker_pool.submit('bar', 'create index otherindex_1 on other_big_table (author)');\nCALL worker_pool.submit('bar', 'create index otherindex_2 on other_big_table (title)');\n\nSELECT * FROM worker_pool.jobs ORDER BY id DESC;\n id | worker_name |                      query_text                       | status\n----+-------------+-------------------------------------------------------+--------\n  4 | bar         | create index otherindex_2 on other_big_table (title)  | done\n  3 | bar         | create index otherindex_1 on other_big_table (author) | done\n  2 | foo         | create index myindex_2 on my_big_table (name)         | done\n  1 | foo         | create index myindex_1 on my_big_table (id)           | done\n(4 rows)\n```\n\nThis will start two background workers, `foo` and `bar`.\n`foo` will create an indices on table `my_big_table` and `bar` on table `other_big_table`.\n`foo` and `bar` will run independently of each other, but all indices submitted to the same worker will be created in order.\n\n## FAQ\n\n### How many workers should I start?\n\nIt's hard to answer that in general case, but you probably don't want to start more than the number of CPUs you have on the machine, especially for long running queries.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserpent7776%2Fpg_worker_pool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserpent7776%2Fpg_worker_pool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserpent7776%2Fpg_worker_pool/lists"}