{"id":13453879,"url":"https://github.com/alitrack/duckdb_fdw","last_synced_at":"2025-08-26T20:51:37.608Z","repository":{"id":45771743,"uuid":"302498474","full_name":"alitrack/duckdb_fdw","owner":"alitrack","description":"DuckDB Foreign Data Wrapper for PostgreSQL","archived":false,"fork":false,"pushed_at":"2024-12-21T14:55:11.000Z","size":1855,"stargazers_count":374,"open_issues_count":21,"forks_count":24,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-06-07T23:51:49.140Z","etag":null,"topics":["duckdb","duckdb-fdw","fdw","foreign-data-wrapper","postgresql"],"latest_commit_sha":null,"homepage":"","language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alitrack.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-09T01:06:26.000Z","updated_at":"2025-06-06T06:09:30.000Z","dependencies_parsed_at":"2024-02-04T06:27:53.716Z","dependency_job_id":"8a5be352-6bb8-41db-8c7e-730cf6ab15e2","html_url":"https://github.com/alitrack/duckdb_fdw","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/alitrack/duckdb_fdw","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alitrack%2Fduckdb_fdw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alitrack%2Fduckdb_fdw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alitrack%2Fduckdb_fdw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alitrack%2Fduckdb_fdw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alitrack","download_url":"https://codeload.github.com/alitrack/duckdb_fdw/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alitrack%2Fduckdb_fdw/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272254474,"owners_count":24901049,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-26T02:00:07.904Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","duckdb-fdw","fdw","foreign-data-wrapper","postgresql"],"created_at":"2024-07-31T08:00:48.769Z","updated_at":"2025-08-26T20:51:37.581Z","avatar_url":"https://github.com/alitrack.png","language":"PLpgSQL","funding_links":[],"categories":["Libraries Powered by DuckDB","PLpgSQL","C"],"sub_categories":["Web Clients"],"readme":"# DuckDB Foreign Data Wrapper for PostgreSQL\n\nThis is a foreign data wrapper (FDW) to connect [PostgreSQL](https://www.postgresql.org/)\nto [DuckDB](https://duckdb.org/) database file. This FDW works with PostgreSQL 9.6 ... 16 and works with exact same version of `libduckdb`.\n\n\u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/2/29/Postgresql_elephant.svg\" align=\"center\" height=\"100\" alt=\"PostgreSQL\"/\u003e\t+\t\u003cimg src=\"https://user-images.githubusercontent.com/41448637/222924178-7e622cad-fec4-49e6-b8fb-33be4447f17d.png\" align=\"center\" height=\"100\" alt=\"DuckDB\"/\u003e\n\n## Contents\n\n1. [Features](#features)\n2. [Supported platforms](#supported-platforms)\n3. [Installation](#installation)\n4. [Usage](#usage)\n5. [Functions](#functions)\n6. [Identifier case handling](#identifier-case-handling)\n7. [Generated columns](#generated-columns)\n8. [Character set handling](#character-set-handling)\n9. [Examples](#examples)\n10. [Limitations](#limitations)\n11. [Tests](#tests)\n12. [Contributing](#contributing)\n13. [Useful links](#useful-links)\n14. [License](#license)\n\n## Features\n\n### Common features\n\n- Transactions  \n- Support `TRUNCATE` by deparsing into `DELETE` statement without `WHERE` clause\n- Allow control over whether foreign servers keep connections open after transaction completion. This is controlled by `keep_connections` and defaults to on\n- Support list cached connections to foreign servers by using function `duckdb_fdw_get_connections()`\n- Support discard cached connections to foreign servers by using function `duckdb_fdw_disconnect()`, `duckdb_fdw_disconnect_all()`.\n- Support Bulk `INSERT` by using `batch_size` option\n- Support `INSERT`/`UPDATE` with generated column\n\n### Pushdowning\n\n- *not described*\n\n### Notes about pushdowning\n\n- *not described*\n\n### Notes about features\n\nAlso see [Limitations](#limitations)\n\n## Supported platforms\n\n`duckdb_fdw` was developed on macOS and tested on Linux, so it should run on any reasonably POSIX-compliant system.\n\n## Installation\n\n### Package installation\n\nThere's a `duckdb_fdw` rpm available on Pigsty's PGSQL [yum repository](https://repo.pigsty.cc/repo) for el8 and el9\n\n\n### Source installation\n\nPrerequisites:\n\n- `postgresql-server-{version}-dev`\n- `gcc`\n- `make`\n\n#### 1. Download source\n\n```BASH\ngit clone https://github.com/alitrack/duckdb_fdw\ncd duckdb_fdw\n```\n\n#### 2. Download DuckDB library\n\nFor example, we want to compile under Linux AMD64 with DuckDB v1.0.0, just download [libduckdb-linux-amd64.zip](https://github.com/duckdb/duckdb/releases/download/v1.0.0/libduckdb-linux-amd64.zip)\n\n```bash\nwget -c https://github.com/duckdb/duckdb/releases/download/v1.0.0/libduckdb-linux-amd64.zip\nunzip -d . libduckdb-linux-amd64.zip\n\n# you can also put the libduckdb.so to a directory in LD_LIBRARY_PATH, such as /usr/lib64\ncp libduckdb.so $(pg_config --libdir)\n```\n\nBeware that this libduckdb.so is build on ubuntu with higher glibc version, to use `duckdb_fdw` on el8 / el9, you have to compile `[libduckdb-src.zip`](https://github.com/duckdb/duckdb/releases/download/v1.0.0/libduckdb-src.zip) from source\n\n#### 3. Build and install duckdb_fdw\n\nAdd a directory of `pg_config` to PATH and build and install `duckdb_fdw`.\n\n```sh\nmake USE_PGXS=1\nmake install USE_PGXS=1\n```\n\nIf you want to build `duckdb_fdw` in a source tree of PostgreSQL, use\n\n```sh\nmake\nmake install\n```\n\n## Usage\n\n## CREATE SERVER options\n\n`duckdb_fdw` accepts the following options via the `CREATE SERVER` command:\n\n- **database** as *string*, **required**\n\n  DuckDB database path.\n\n- **truncatable** as *boolean*, optional, default *false*\n\n  Allows foreign tables to be truncated using the `TRUNCATE` command.\n  \n- **keep_connections** as *boolean*, optional, default *false*\n  \n  Allows to keep connections to DuckDB while there is no SQL operations between PostgreSQL and DuckDB.\n  \n- **batch_size** as *integer*, optional, default *1*\n\n  Specifies the number of rows which should be inserted in a single `INSERT` operation. This setting can be overridden for individual tables.\n  \n- **temp_directory** as *string*,  optional, default *NULL*\n  \n  Specifies the directory to which to write temp files.\n  \n## CREATE USER MAPPING options\n\nThere is no user or password conceptions in DuckDB, hence `duckdb_fdw` no need any `CREATE USER MAPPING` command.\n\nIn OS `duckdb_fdw` works as executed code with permissions of user of PostgreSQL server. Usually it is `postgres` OS user. For interacting with DuckDB database without access errors ensure this user have permissions on DuckDB file and, sometimes, directory of the file.\n\n- read permission on all directories by path to the DuckDB database file;\n- read permission on DuckDB database file;\n\n## CREATE FOREIGN TABLE options\n\n`duckdb_fdw` accepts the following table-level options via the\n`CREATE FOREIGN TABLE` command:\n\n- **table** as *string*, optional, no default\n\n  DuckDB table name. Use if not equal to name of foreign table in PostgreSQL. Also see about [identifier case handling](#identifier-case-handling).\n\n- **truncatable** as *boolean*, optional, default from the same `CREATE SERVER` option\n  \n  See `CREATE SERVER` options section for details.\n\n- **batch_size** as *integer*, optional, default from the same `CREATE SERVER` option\n\n  See `CREATE SERVER` options section for details.  \n  \n`duckdb_fdw` accepts the following column-level options via the\n`CREATE FOREIGN TABLE` command:\n\n- **column_name** as *string*, optional, no default\n\n  This option gives the column name to use for the column on the remote server. Also see about [identifier case handling](#identifier-case-handling).\n\n- **column_type** as *string*, optional, no default\n\n  Option to convert INT DuckDB column (epoch Unix Time) to be treated/visualized as TIMESTAMP in PostgreSQL.\n\n- **key** as *boolean*, optional, default *false*\n\n  Indicates a column as a part of primary key or unique key of DuckDB table.\n  \n## IMPORT FOREIGN SCHEMA options\n\n`duckdb_fdw` supports [IMPORT FOREIGN SCHEMA](https://www.postgresql.org/docs/current/sql-importforeignschema.html)\n(PostgreSQL 9.5+) and accepts no custom options for this command.\n\n## TRUNCATE support\n\n`duckdb_fdw` implements the foreign data wrapper `TRUNCATE` API, available\nfrom PostgreSQL 14.\n\nAs SQlite does not provide a `TRUNCATE` command, it is simulated with a\nsimple unqualified `DELETE` operation.\n\n`TRUNCATE ... CASCADE` support *not described*.\n\n## Functions\n\nAs well as the standard `duckdb_fdw_handler()` and `duckdb_fdw_validator()`\nfunctions, `duckdb_fdw` provides the following user-callable utility functions:\n\n- SETOF record **duckdb_fdw_get_connections**(server_name text, valid bool)\n\n- bool **duckdb_fdw_disconnect**(text)\n\n  Closes connection from PostgreSQL to DuckDB in the current session.\n\n- bool **duckdb_fdw_disconnect_all()**\n\n- **duckdb_fdw_version()**;\n\n  Returns standard \"version integer\" as `major version * 10000 + minor version * 100 + bugfix`.\n\n```\nduckdb_fdw_version\n--------------------\n              10000  \n```\n\n### DuckDB_execute\n\n```sql\nFUNCTION duckdb_execute(server name, stmt text) RETURNS void\n```\n\nThis function can be used to execute arbitrary SQL statements on the remote DuckDB server. That will only work with statements that do not return results (typically DDL statements).\n\nBe careful when using this function, since it might disturb the transaction management of duckdb_fdw. Remember that running a DDL statement in DuckDB will issue an implicit COMMIT.\nYou are best advised to use this function outside multi-statement transactions.\n\nIt is very useful to use command that duckdb_fdw does not support, for example,\n\n- add more table or view to DuckDB directly.\n  \n```sql\nSELECT duckdb_execute('duckdb_server'\n,'create or replace view iris_parquet  as select * from parquet_scan(''temp/iris.parquet'');');\n\ncreate foreign TABLE duckdb.iris_parquet(\n\"Sepal.Length\" float,  \n\"Sepal.Width\" float,\n\"Petal.Length\" float,\n\"Petal.Width\" float,  \n\"Species\" text)\n      SERVER duckdb_server OPTIONS (table 'iris_parquet');\n\n-- or an easy way\n\nIMPORT FOREIGN SCHEMA public limit to (iris_parquet) FROM SERVER  \nduckdb_server INTO duckdb;\n```\n\n- run Copy command on Foreign table\n\n```sql  \nSELECT duckdb_execute('duckdb_server'\n,'CREATE TABLE test (a INTEGER, b INTEGER, c VARCHAR(10));\n');\nSELECT duckdb_execute('duckdb_server'  \n,'COPY test FROM ''/tmp/test.csv'';');\n```\n\n## Identifier case handling\n\nPostgreSQL folds identifiers to lower case by default. DuckDB *behaviour not described*. It's important\nto be aware of potential issues with table and column names.\n\n## Generated columns\n\nDuckDB provides support for [generated columns](https://www.duckdb.org/gencol.html).\nBehaviour of `duckdb_fdw` with these columns _isn't yet described_.\n\nNote that while `duckdb_fdw` will `INSERT` or `UPDATE` the generated column value\nin DuckDB, there is nothing to stop the value being modified within DuckDB,\nand hence no guarantee that in subsequent `SELECT` operations the column will\nstill contain the expected generated value. This limitation also applies to\n`postgres_fdw`.\n\nFor more details on generated columns see:\n\n- [Generated Columns](https://www.postgresql.org/docs/current/ddl-generated-columns.html)\n- [CREATE FOREIGN TABLE](https://www.postgresql.org/docs/current/sql-createforeigntable.html)\n\n## Character set handling\n\n**Yet not described**\n\n## Examples\n\n### Install the extension\n\nOnce for a database you need, as PostgreSQL superuser.\n\n```sql\nCREATE EXTENSION duckdb_fdw;\n```\n\n### Create a foreign server with appropriate configuration:\n\nOnce for a foreign datasource you need, as PostgreSQL superuser. Please specify DuckDB database path using `database` option.\n\n```sql\nCREATE SERVER duckdb_server\nFOREIGN DATA WRAPPER duckdb_fdw\nOPTIONS (\n    database '/path/to/database'\n);\n```\n\n### Grant usage on foreign server to normal user in PostgreSQL:\n\nOnce for a normal user (non-superuser) in PostgreSQL, as PostgreSQL superuser. It is a good idea to use a superuser only where really necessary, so let's allow a normal user to use the foreign server (this is not required for the example to work, but it's secirity recomedation).\n\n```sql\nGRANT USAGE ON FOREIGN SERVER duckdb_server TO pguser;\n```\n\nWhere `pguser` is a sample user for works with foreign server (and foreign tables).\n\n### User mapping\n\nThere is no user or password conceptions in DuckDB, hence `duckdb_fdw` no need any `CREATE USER MAPPING` command. About access problems see in [CREATE USER MAPPING options](#create-user-mapping-options).\n\n### Create foreign table\n\nAll `CREATE FOREIGN TABLE` SQL commands can be executed as a normal PostgreSQL user if there were correct `GRANT USAGE ON FOREIGN SERVER`. No need PostgreSQL supersuer for secirity reasons but also works with PostgreSQL supersuer.\n\nPlease specify `table` option if DuckDB table name is different from foreign table name.\n\n```sql\n\tCREATE FOREIGN TABLE t1 (\n\t  a integer,\n\t  b text\n\t)\n\tSERVER duckdb_server\n\tOPTIONS (\n\t  table 't1_duckdb'\n\t);\n```\n\nIf you want to update tables, please add `OPTIONS (key 'true')` to a primary key or unique key like the following:\n\n```sql\n\tCREATE FOREIGN TABLE t1(\n\t  a integer OPTIONS (key 'true'),\n\t  b text\n\t)\n\tSERVER duckdb_server \n\tOPTIONS (\n\t  table 't1_duckdb'\n\t);\n```\n\nIf you need to convert INT DuckDB column (epoch Unix Time) to be treated/visualized as `TIMESTAMP` in PostgreSQL, please add `OPTIONS (column_type 'INT')` when defining FOREIGN table at PostgreSQL like the following:\n\n```sql\n\tCREATE FOREIGN TABLE t1(\n\t  a integer,\n\t  b text,\n\t  c timestamp without time zone OPTIONS (column_type 'INT')\n\t)\n\tSERVER duckdb_server\n\tOPTIONS (\n\t  table 't1_duckdb'\n\t);\n```\n\nAs above, but with aliased column names:\n\n```sql\n\tCREATE FOREIGN TABLE t1(\n\t  a integer,\n\t  b text OPTIONS (column_name 'test_id'),\n\t  c timestamp without time zone OPTIONS (column_type 'INT', column_name 'unixtime')\n\t)\n\tSERVER duckdb_server\n\tOPTIONS (\n\t  table 't1_duckdb'\n\t);\n```\n\n### Import a DuckDB database as schema to PostgreSQL:\n\n```sql\n\tIMPORT FOREIGN SCHEMA someschema\n\tFROM SERVER duckdb_server\n\tINTO public;\n```\n\nNote: `someschema` has no particular meaning and can be set to an arbitrary value.\n\n### Access foreign table\n\nFor the table from previous examples\n\n```sql\n\tSELECT * FROM t1;\n```\n\n## Limitations\n\n- `INSERT` into a partitioned table which has foreign partitions is not supported. Error `Not support partition insert` will display.\n- `TRUNCATE` in `duckdb_fdw` always delete data of both parent and child tables (no matter user inputs `TRUNCATE table CASCADE` or `TRUNCATE table RESTRICT`) if there are foreign-keys references with `ON DELETE CASCADE` clause.\n- `RETURNING` is not supported.\n\n## Tests\n\nAll tests are based on `make check`, main testing script see in [test.sh](test.sh) file. We don't profess a specific environment. You can use any POSIX-compliant system. \nTesting scripts from PosgreSQL-side is multi-versioned. Hence, you need install PostgreSQL packages in versions listed in [sql](sql) directory. \nPostgreSQL server locale for messages in tests must be *english*. About base testing mechanism see in [PostgreSQL documentation](https://www.postgresql.org/docs/current/regress-run.html).\n\nTesting directory have structure as following:\n\n```\n+---sql\n    +---11.7\n    |       filename1.sql\n    |       filename2.sql\n    | \n    +---12.12\n    |       filename1.sql\n    |       filename2.sql\n    | \n.................  \n    \\---15.0\n           filename1.sql\n           filename2.sql\n```\n\nThe test cases for each version are based on the test of corresponding version of PostgreSQL.\nYou can execute test by `test.sh` directly. \nThe version of PostgreSQL is detected automatically by `$(VERSION)` variable in Makefile.\n\n## Contributing\n\nOpening issues and pull requests on GitHub are welcome.\n\nYou don't need to squash small commits to one big in pull requests.\n\nFor pull request, please make sure these items below for testing:\n\n- Create test cases (if needed) for the latest version of PostgreSQL supported by `duckdb_fdw`.\n- Execute test cases and update expectations for the latest version of PostgreSQL\n- Test creation and execution for other PostgreSQL versions are welcome but not required.\n\n## Useful links\n\n### Source\n\n- https://github.com/pgspider/sqlite_fdw\n- https://pgxn.org/dist/duckdb_fdw/\n \nReference FDW realisation, `postgres_fdw`\n\n- https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD\n\n### General FDW Documentation\n\n- https://www.postgresql.org/docs/current/ddl-foreign-data.html\n- https://www.postgresql.org/docs/current/sql-createforeigndatawrapper.html\n- https://www.postgresql.org/docs/current/sql-createforeigntable.html\n- https://www.postgresql.org/docs/current/sql-importforeignschema.html\n- https://www.postgresql.org/docs/current/fdwhandler.html\n- https://www.postgresql.org/docs/current/postgres-fdw.html\n\n### Other FDWs\n\n- https://wiki.postgresql.org/wiki/Fdw\n- https://pgxn.org/tag/fdw/\n\n## Special thanks\n\nAuthors of https://github.com/pgspider/sqlite_fdw\n\n## License\n\n[MIT License](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falitrack%2Fduckdb_fdw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falitrack%2Fduckdb_fdw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falitrack%2Fduckdb_fdw/lists"}