{"id":15705310,"url":"https://github.com/brahmlower/postgres-virtual-tables","last_synced_at":"2025-03-30T15:43:24.808Z","repository":{"id":86081635,"uuid":"188176525","full_name":"brahmlower/postgres-virtual-tables","owner":"brahmlower","description":"Attempting to support dynamically defined virtual database tables in postgresql","archived":false,"fork":false,"pushed_at":"2023-05-22T22:26:06.000Z","size":50,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-05T17:17:22.573Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brahmlower.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-23T06:42:02.000Z","updated_at":"2023-08-30T02:48:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"aa091d0e-5ce4-44d6-be3f-fd213ceb1fca","html_url":"https://github.com/brahmlower/postgres-virtual-tables","commit_stats":{"total_commits":26,"total_committers":1,"mean_commits":26.0,"dds":0.0,"last_synced_commit":"dfa1c7b6eb46d2c388d11ea189ac21d3805529c8"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brahmlower%2Fpostgres-virtual-tables","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brahmlower%2Fpostgres-virtual-tables/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brahmlower%2Fpostgres-virtual-tables/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brahmlower%2Fpostgres-virtual-tables/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brahmlower","download_url":"https://codeload.github.com/brahmlower/postgres-virtual-tables/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246342713,"owners_count":20761938,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T20:15:23.494Z","updated_at":"2025-03-30T15:43:24.789Z","avatar_url":"https://github.com/brahmlower.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Postgres Virtual Tables\n\nThe idea is that users could \"create a database table\" without actually making a\nnew table in the database. This will of course not be a perfect abstraction, but\nthe hope is that it provides some additional flexability in designing and changing\nvirtual tables.\n\nTerminology is important for clear communication. Frequently used terms and their\nexacty meaning are listed here:\n\n- table - A normal postgresql table\n- column - A normal column on a postgres table\n- vtable - A virtual table conceptually defined by data in postgres tables\n- vcolumn - A column on the virtual table\n\n## Structure\n\nAs of right now, there are three tables, two of which hold the definition for\nvtables and their structures, and third contains the data within the vtables.\n\n### vtable_table\n\nThis is a pretty straightforward table- it holds a primary key ID and a name for\nthe vtable. This just serves as the root definition of a the vtables, and could be\nexpanded to track other info like vtable owner or permissions.\n\n### vtable_column\n\nThe vcolumns for a vtable are defined here, and consists of the vtable ID that\nthe column belongs to, the name of the vcolumn, and the ordering of the vcolumn.\nAt this time there is a column called 'column_type', which would be used to\ndefine the type of data being stored, however this functionality has not been\nimplemented yet.\n\n### vtable_cell\n\nLike a cell on an excel spreadsheet, records in this table represent a single\ncolumn within a row in a vtable.\n\n## Project Setup\n\nStart the test datbase, then load the schema and test data\n\n```\ndocker-compose up -d\nPGPASSWORD=test psql -U test -h localhost test -c \"\\i deploy.sql\"\nPGPASSWORD=test psql -U test -h localhost test -c \"\\i vtable_building.sql\"\n```\n\nIf you need to reset everything, you can import the revert script (or just\ndocker-compose down)\n\n```\nPGPASSWORD=test psql -U test -h localhost test -c \"\\i revert.sql\"\n```\n\n## Basic usage\n\nThis section will lightly detail how to use the tables and functions provided.\nUntil the project reaches a mature state, this will be a mix of loose notes and\nreference for myself. Boring or inconsequential command output has been ommited.\n\n### Manually create vtable access function!\n\nWe can create a function dynamically configured to show data from a specific vtable.\n\n```\ntest=# select vtable_func_creator(100);\ntest=# select * from vtable_100();\n id |     name     | height | city  |       country        | owner_id | is_public \n----+--------------+--------+-------+----------------------+----------+-----------\n 50 | Burj Khalifa |    828 | Dubai | United Arab Emirates |        0 | t\n(1 row)\n```\n\n### Views on vtable access functions!\n\nCreate a view on top of the temporary vtable access function.\n\n```\ntest=# create view buildings as select * from vtable_100();\ntest=# select * from buildings;\n id |     name     | height | city  |       country        | owner_id | is_public \n----+--------------+--------+-------+----------------------+----------+-----------\n 50 | Burj Khalifa |    828 | Dubai | United Arab Emirates |        0 | t\n(1 row)\n```\n\n### Creating functions and views the easy way\n\nManually creating functions and views is only any good for testing. There's a function that will remove any existing access function and view for a given vtable, and create new ones (useful in cases where the vtable has been updated). But in our case, we can use it to create the functions and views for the first time:\n\n```\ntest=# select * from buildings;\nERROR:  relation \"buildings\" does not exist\nLINE 1: select * from buildings;\n                      ^\ntest=# select vtable_access_rebuild(1);\nNOTICE:  view \"buildings\" does not exist, skipping\nNOTICE:  function vtable_1() does not exist, skipping\n vtable_access_rebuild \n-----------------------\n \n(1 row)\ntest=# select * from buildings;\n...\n```\n\n\n### Vtables preserve specified types!\n\nProve to ourselves that the virtual table is preserving the specified column\ntypes specified in vtable_column.\n\n```\ntest=# select column_name, column_type from vtable_column where column_name = ANY(ARRAY['city', 'owner_id', 'is_public']);\n column_name | column_type \n-------------+-------------\n city        | text\n owner_id    | integer\n is_public   | boolean\n(3 rows)\n\ntest=# select pg_typeof(city), pg_typeof(owner_id), pg_typeof(is_public) from buildings;\n pg_typeof | pg_typeof | pg_typeof \n-----------+-----------+-----------\n text      | integer   | boolean\n(1 row)\n\n```\n\n### Thoughs on future work\n\n#### Better record insertion\n\nIt'd be nice to have a cleaner way of inserting data into a vtable. Current method\nkinda sucks. Possibly make another temporary function crafted for the particular\nvirtual table. Would prevent errors like below:\n\n```\ntest=# select * from vtable_insert(100, ARRAY['Home', '15', 'Puyallup', 'United States of America', '0', 'false']);\ntest=# select * from buildings;\n id |     name     | height |   city   |         country          | owner_id | is_public \n----+--------------+--------+----------+--------------------------+----------+-----------\n 50 | Burj Khalifa |    828 | Dubai    | United Arab Emirates     |        0 | t\n 51 | Home         |     15 | Puyallup | United States of America |        0 | f\n(2 rows)\ntest=# select * from vtable_insert(100, ARRAY['Home', '15', 'Puyallup', 'United States of America', '0', 'poop']);\ntest=# select * from buildings;\nERROR:  invalid input syntax for type boolean: \"poop\"\nCONTEXT:  PL/pgSQL function pg_temp_3.vtable_100() line 3 at RETURN QUERY\n```\n\n#### Row locks, transactions, and alters oh my!\n\n**The documentation below is out of date as of 5/25/2019, and will be updated shortly\nto reflect new data access functions, views, and rebuild triggers.**\n\nAdditionally, it's possible for the vtable to be altered while another session is\nopen, at which point the existing temporary access functions will be broken. For\nexample, in Session 1 we have our view for the vtable 'buildings' set up and we\ncan query from it just fine:\n\n```\n-- Session 1\ntest=# select * from buildings;\n id |     name     | height | city  |       country        | owner_id | is_public \n----+--------------+--------+-------+----------------------+----------+-----------\n 50 | Burj Khalifa |    828 | Dubai | United Arab Emirates |        0 | t\n(1 row)\n\n```\n\nBut then in Session 2, representing a different user, we decide we're going to\nchange the shape of the vtable, and remove the 'city' column.\n\n```\n-- Session 2\ntest=# select * from vtable_column where table_id = 100 and column_name = 'city';\n id  | table_id | column_name | column_type | column_references | column_position \n-----+----------+-------------+-------------+-------------------+-----------------\n2022 |      100 | city        | text        |                   |               3\n(1 row)\ntest=# select vtable_alter_remove_column(100, 2022);\n vtable_alter_remove_column \n----------------------------\n \n(1 row)\n```\n\nNow the vtable has one less column then the temporary access function in Session 1\nis expecting, so when the user attempts to query their view again, it fails:\n\n```\n-- Session 1\ntest=# select * from buildings;\nERROR:  invalid return type\nDETAIL:  Query-specified return tuple has 7 columns but crosstab returns 6.\nCONTEXT:  PL/pgSQL function pg_temp_3.vtable_100() line 3 at RETURN QUERY\n```\n\nThis can be somewhat prevented by using row-level locks and transactions. For\nthis example we'll reset the database to its original state. Session 1 create\nthe temporary access function and view again, but will also read the vtable\ncolumns with a 'for shared', which will prevent the resulting rows from being\nmodified for the duration of the transaction.\n\n```\n-- Session 1\ntest=# begin;\ntest=# select * from vtable_column where table_id = 100 for share;\n...\n(6 rows)\ntest=# select * from buildings;\n id |     name     | height | city  |       country        | owner_id | is_public \n----+--------------+--------+-------+----------------------+----------+-----------\n 50 | Burj Khalifa |    828 | Dubai | United Arab Emirates |        0 | t\n(1 row)\n```\n\nSession 1 now has a row-level lock on the columns that define the virtual table.\nIf Session 2 comes along now and tries to modify the table, their change will be\nblocked until the Session 1 transaction is closed.\n\n```\n-- Session 2\ntest=# select vtable_alter_remove_column(100, 2022);\n```\n\nThis just blocks. I'm not sure if what the default timeout value is, but this is\nmost certainly not what you want if you're letting users modify the virtual\ntables (I mean, what else are you doing if not that?), so we should set the\n`statement_timeout` setting when opening the database session. Here we'll set it\nto 10 seconds, just as an example.\n\n```\n-- Session 2\nSET statement_timeout = 10000;\n```\n\nAs soon as the transaction on Session 1 is closed, the alter statement\ncompletes. This isn't ideal behavior though! We don't want to be locking longer\nthan we need to, and we should be able to read from the table without making any\nmanual changes when the vtable changes unexpectedly.\n\nCouple options:\n1) try to find a way to detect when the table changes, then rebuild our temporary\n    access function and acompanying view\n2) Don't use temporary functions/view. Use real ones that persist across sessions,\n    and then recreate them based on triggers set up on the vtable_column table.\n\n## Continued research\n\nReading from the vtables sucks right now. It can be sorta hacked together using\nthe `crosstab` function, but you're required to list out all the columns, which\nbreaks the autonomy here. More reading and testing is needed:\n\n- https://www.postgresql.org/docs/9.5/tablefunc.html#AEN137962\n- https://dba.stackexchange.com/questions/158181/pivot-with-2-columns-using-crosstab\n- https://stackoverflow.com/questions/12879672/dynamically-generate-columns-for-crosstab-in-postgresql\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrahmlower%2Fpostgres-virtual-tables","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrahmlower%2Fpostgres-virtual-tables","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrahmlower%2Fpostgres-virtual-tables/lists"}