{"id":14063366,"url":"https://github.com/pgspider/parquet_s3_fdw","last_synced_at":"2026-01-16T15:08:37.663Z","repository":{"id":38223680,"uuid":"343668846","full_name":"pgspider/parquet_s3_fdw","owner":"pgspider","description":"ParquetS3 Foreign Data Wrapper for PostgresSQL","archived":false,"fork":false,"pushed_at":"2025-03-18T09:45:24.000Z","size":1375,"stargazers_count":225,"open_issues_count":26,"forks_count":32,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-07-29T16:23:55.563Z","etag":null,"topics":["fdw","foreign-data-wrapper","foreign-tables","parquet","parquets3-fdw","postgresql","postgresql-extension","s3"],"latest_commit_sha":null,"homepage":"","language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pgspider.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-03-02T06:27:26.000Z","updated_at":"2025-07-24T02:57:24.000Z","dependencies_parsed_at":"2024-01-29T19:30:49.682Z","dependency_job_id":"e54b8965-de40-481b-9e46-67b7fb7fa309","html_url":"https://github.com/pgspider/parquet_s3_fdw","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/pgspider/parquet_s3_fdw","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgspider%2Fparquet_s3_fdw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgspider%2Fparquet_s3_fdw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgspider%2Fparquet_s3_fdw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgspider%2Fparquet_s3_fdw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pgspider","download_url":"https://codeload.github.com/pgspider/parquet_s3_fdw/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgspider%2Fparquet_s3_fdw/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28479406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fdw","foreign-data-wrapper","foreign-tables","parquet","parquets3-fdw","postgresql","postgresql-extension","s3"],"created_at":"2024-08-13T07:03:18.161Z","updated_at":"2026-01-16T15:08:37.636Z","avatar_url":"https://github.com/pgspider.png","language":"PLpgSQL","funding_links":[],"categories":["PLpgSQL"],"sub_categories":[],"readme":"# Parquet S3 Foreign Data Wrapper for PostgreSQL\n\nThis PostgreSQL extension is a Foreign Data Wrapper (FDW) for accessing Parquet file on local file system and [Amazon S3][2].\nThis version of parquet_s3_fdw can work for PostgreSQL 13, 14, 15, 16 and 17.\n\nApache Parquet foreign data wrapper supporting S3 access for PostgreSQL.\n\n\n## Installation\n### 1. Build requirements\n* CMake 3.29.3\n* C++11 compiler\n* libcurl-devel\n* openssl-devel\n* libuuid-devel\n* pulseaudio-libs-devel\n### 2. Install dependent libraries\n* `libarrow` and `libparquet`: Confirmed version is 16.1.0 (required).\nPlease refer to [building guide](https://github.com/apache/arrow/blob/master/docs/source/developers/cpp/building.rst).\n\n* `AWS SDK for C++ (libaws-cpp-sdk-core libaws-cpp-sdk-s3)`: Confirmed version is 1.11.335 (required).\nPlease refer to [bulding guide](https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/setup-linux.html)\n\nAttention!  \n- We recommend to build `libarrow`, `libparquet` and `AWS SDK for C++` from the source code. We failed to link if using pre-compiled binaries because gcc version is different between arrow and AWS SDK.\n\n- Build `libarrow` with the options -DARROW_WITH_SNAPPY=ON, -DARROW_WITH_ZSTD=ON to support compression with SNAPPY and ZSTD.\n\n### 3. Build and install parquet_s3_fdw\n```sh\nmake install\n```\nor in case when PostgreSQL is installed in a custom location:\n```sh\nmake install PG_CONFIG=/path/to/pg_config\n```\nIt is possible to pass additional compilation flags through either custom\n`CCFLAGS` or standard `PG_CFLAGS`, `PG_CXXFLAGS`, `PG_CPPFLAGS` variables.\n\n## Usage\n### Load extension\n```sql\nCREATE EXTENSION parquet_s3_fdw;\n```\n\n### Create server\n```sql\nCREATE SERVER parquet_s3_srv FOREIGN DATA WRAPPER parquet_s3_fdw;\n```\nIf using [MinIO][3] instead of AWS S3, please use use_minio option for create server.\n```sql\nCREATE SERVER parquet_s3_srv FOREIGN DATA WRAPPER parquet_s3_fdw OPTIONS (use_minio 'true');\n```\n\n### Create user mapping\nYou have to specify user name and password if accessing Amazon S3.\n```sql\nCREATE USER MAPPING FOR public SERVER parquet_s3_srv OPTIONS (user 's3user', password 's3password');\n```\n\n### Create foreign table\nNow you should be able to create foreign table from Parquet files. Currently `parquet_s3_fdw` supports the following column [types](https://github.com/apache/arrow/blob/master/cpp/src/arrow/type.h) (to be extended shortly):\n\n|   Arrow type |  SQL type |\n|-------------:|----------:|\n|         INT8 |      INT2 |\n|        INT16 |      INT2 |\n|        INT32 |      INT4 |\n|        INT64 |      INT8 |\n|        FLOAT |    FLOAT4 |\n|       DOUBLE |    FLOAT8 |\n|    TIMESTAMP | TIMESTAMP |\n|       DATE32 |      DATE |\n|       STRING |      TEXT |\n|       BINARY |     BYTEA |\n|         LIST |     ARRAY |\n|          MAP |     JSONB |\n\nCurrently `parquet_s3_fdw` doesn't support structs and nested lists.\n\nFollowing options are supported:\n* **filename** - space separated list of paths to Parquet files to read. You can specify the path on AWS S3 by starting with `s3://`. The mix of local path and S3 path is not supported;\n* **dirname** - path to directory having Parquet files to read;\n* **sorted** - space separated list of columns that Parquet files are presorted by; that would help postgres to avoid redundant sorting when running query with `ORDER BY` clause or in other cases when having a presorted set is beneficial (Group Aggregate, Merge Join); If column has space character in the name, the column name must be double quoted to differentiate space character among columns.\n* **files_in_order** - specifies that files specified by `filename` or returned by `files_func` are ordered according to `sorted` option and have no intersection rangewise; this allows to use `Gather Merge` node on top of parallel Multifile scan (default `false`);\n* **use_mmap** - whether memory map operations will be used instead of file read operations (default `false`);\n* **use_threads** - enables Apache Arrow's parallel columns decoding/decompression (default `false`);\n* **files_func** - user defined function that is used by parquet_s3_fdw to retrieve the list of parquet files on each query; function must take one `JSONB` argument and return text array of full paths to parquet files;\n* **files_func_arg** - argument for the function, specified by **files_func**.\n* **max_open_files** - the limit for the number of Parquet files open simultaneously.\n* **region** - the value of AWS region used to connect to (default `ap-northeast-1`).\n* **endpoint** - the address and port used to connect to (default `127.0.0.1:9000`).\n* **key** - column option, indicates a column as a part of primary key or unique key of parquet file\n* **column_name** - (string) This option, which can be specified for a column of a foreign table, gives the column name to use for the column on the parquet file. If this option is omitted, the column's name is used.\n* **compression_type** - the compression is used for table and column.\n\nForeign table may be created for a single Parquet file and for a set of files. It is also possible to specify a user defined function, which would return a list of file paths. Depending on the number of files and table options `parquet_s3_fdw` may use one of the following execution strategies:\n\n| Strategy                | Description              |\n|-------------------------|--------------------------|\n| **Single File**         | Basic single file reader\n| **Multifile**           | Reader which process Parquet files one by one in sequential manner |\n| **Multifile Merge**     | Reader which merges presorted Parquet files so that the produced result is also ordered; used when `sorted` option is specified and the query plan implies ordering (e.g. contains `ORDER BY` clause) |\n| **Caching Multifile Merge** | Same as `Multifile Merge`, but keeps the number of simultaneously open files limited; used when the number of specified Parquet files exceeds `max_open_files` |\n\nGUC variables:\n* **parquet_fdw.use_threads** - global switch that allow user to enable or disable threads (default `true`);\n* **parquet_fdw.enable_multifile** - enable Multifile reader (default `true`).\n* **parquet_fdw.enable_multifile_merge** - enable Multifile Merge reader (default `true`).\n\nExample:\n```sql\nCREATE FOREIGN TABLE userdata (\n    id           int,\n    first_name   text,\n    last_name    text\n)\nSERVER parquet_s3_srv\nOPTIONS (\n    filename 's3://bucket/dir/userdata1.parquet'\n);\n```\n\n### Access foreign table\n```sql\nSELECT * FROM userdata;\n```\n\n## Parallel queries\n`parquet_s3_fdw` also supports [parallel query execution](https://www.postgresql.org/docs/current/parallel-query.html) (not to confuse with multi-threaded decoding feature of Apache Arrow).\n\n## Row group filter\nApache Parquet has logical horizontal partitioning of the data into ['row group'](https://parquet.apache.org/docs/concepts/). `parquet_s3_fdw` can filter `row group` by condition in `WHERE` clause, this is a performance feature (not to confuse with `WHERE` clause push-down). Folowing conditions can be supported:\n- `EXPR OP CONST`\n- `CONST OP EXPR`\n- `BOOL_VAR` or `BOOL_VAR = true`\n- `!BOOL_VAR` or `BOOL_VAR = false`\n\nFor example:\n```sql\n-- only row group 1 will be read\nEXPLAIN VERBOSE\nSELECT * FROM example1 WHERE one \u003c= 1;\nDEBUG:  parquet_s3_fdw: skip rowgroup 2\n                             QUERY PLAN                             \n--------------------------------------------------------------------\n Foreign Scan on public.example1  (cost=0.00..0.03 rows=2 width=93)\n   Output: one, two, three, four, five, six, seven\n   Filter: (example1.one \u003c= 1)\n   Reader: Single File\n   Row groups: 1\n(5 rows)\n```\n\nLimitation:\n- Currently only Var (foreign table column) as expression is supported. Will be extended in future.\n- Do not support filter with `string` column in parquet file.\n\nFor schemaless mode refer [this](#schemaless-mode).\n\n## Import\n`parquet_s3_fdw` also supports [`IMPORT FOREIGN SCHEMA`](https://www.postgresql.org/docs/current/sql-importforeignschema.html) command to discover parquet files in the specified directory on filesystem and create foreign tables according to those files. It can be used as follows:\n\n```sql\nIMPORT FOREIGN SCHEMA \"/path/to/directory\"\nFROM SERVER parquet_s3_srv\nINTO public;\n```\n\nIt is important that `remote_schema` here is a path to a local filesystem directory and is double quoted.\n\nAnother way to import parquet files into foreign tables is to use `import_parquet_s3` or `import_parquet_s3_explicit`:\n\n```sql\nCREATE FUNCTION import_parquet_s3(\n    tablename   text,\n    schemaname  text,\n    servername  text,\n    userfunc    regproc,\n    args        jsonb,\n    options     jsonb)\n\nCREATE FUNCTION import_parquet_s3_explicit(\n    tablename   text,\n    schemaname  text,\n    servername  text,\n    attnames    text[],\n    atttypes    regtype[],\n    userfunc    regproc,\n    args        jsonb,\n    options     jsonb)\n```\n\nThe only difference between `import_parquet_s3` and `import_parquet_s3_explicit` is that the latter allows to specify a set of attributes (columns) to import. `attnames` and `atttypes` here are the attributes names and attributes types arrays respectively (see the example below).\n\n`userfunc` is a user-defined function. It must take a `jsonb` argument and return a text array of filesystem paths to parquet files to be imported. `args` is user-specified jsonb object that is passed to `userfunc` as its argument. A simple implementation of such function and its usage may look like this:\n\n```sql\nCREATE FUNCTION list_parquet_s3_files(args jsonb)\nRETURNS text[] AS\n$$\nBEGIN\n    RETURN array_agg(args-\u003e\u003e'dir' || '/' || filename)\n           FROM pg_ls_dir(args-\u003e\u003e'dir') AS files(filename)\n           WHERE filename ~~ '%.parquet';\nEND\n$$\nLANGUAGE plpgsql;\n\nSELECT import_parquet_s3_explicit(\n    'abc',\n    'public',\n    'parquet_srv',\n    array['one', 'three', 'six'],\n    array['int8', 'text', 'bool']::regtype[],\n    'list_parquet_files',\n    '{\"dir\": \"/path/to/directory\"}',\n    '{\"sorted\": \"one\"}'\n);\n```\n\n## Features\n- Support SELECT of parquet file on local file system or Amazon S3.\n- Support INSERT, DELETE, UPDATE (Foreign modification).\n- Support MinIO access instead of Amazon S3.\n- Allow control over whether foreign servers keep connections open after transaction completion. This is controlled by keep_connections and defaults to on.\n- Support parquet_s3_fdw function parquet_s3_fdw_get_connections() to report open foreign server connections.\n\n## Schemaless mode\n- The feature will enable user to use schemaless feature:\n  - No specific foreign foreign schema (column difinition) for each parquet file.\n  - The schemaless foreign table has only one jsonb column to represent the data from the parquet file by following rule:\n    - Jsonb Key: parquet column name.\n    - Jsonb Value: parquet column data.\n- By use schemaless mode, there are several benefits:\n  - Flexibility over data structure of parquet file: By merging all column data into one jsonb column, a schemaless foreign table can query any parquet file that has all column can be mapped with the postgres type.\n  - No pre-defined foreign table schemas (column difinition). The lack of schema means that foreign table will query all column from parquet file — including those that user do not yet use.\n\n### Schemaless mode usage\n- Schemaless mode is enabled by `schemaless` option:\n  - `schemaless` option is `true`: enable schemaless mode.\n  - `schemaless` option is `false`: disable schemaless mode (We call it `non-schemaless` mode).\n  - If `schemaless` option is not configured, default value is false.\n  - `schemaless` option is supported in `CREATE FOREIGN TABLE`, `IMPORT FOREIGN SCHEMA`, `import_parquet_s3()` and `import_parquet_s3_explicit()`.\n- Schemaless foreign table needs at least one jsonb column to represent data:\n  - If there is more than 1 jsonb column, only one column is populated, all other columns are treated with NULL value.\n  - If there is no jsonb column, all column are treated with NULL value.\n  - Example:\n    ```sql\n    CREATE FOREIGN TABLE example_schemaless (\n      id int,\n      v jsonb\n    ) OPTIONS (filename '/path/to/parquet_file', schemaless 'true');\n    SELECT * FROM example_schemaless;\n    id |                                                                v                                                                \n    ----+---------------------------------------------------------------------------------------------------------------------------------\n        | {\"one\": 1, \"six\": \"t\", \"two\": [1, 2, 3], \"five\": \"2018-01-01\", \"four\": \"2018-01-01 00:00:00\", \"seven\": 0.5, \"three\": \"foo\"}\n        | {\"one\": 2, \"six\": \"f\", \"two\": [null, 5, 6], \"five\": \"2018-01-02\", \"four\": \"2018-01-02 00:00:00\", \"seven\": null, \"three\": \"bar\"}\n    (2 rows)\n    ```\n- Create foreign table:\n  With  `IMPORT FOREIGN SCHEMA`, `import_parquet_s3()` and `import_parquet_s3_explicit()`, foreign table will create with fixed column difinition like below:\n  ```sql\n  CREATE FOREIGN TABLE example (\n    v jsonb\n  ) OPTIONS (filename '/path/to/parquet_file', schemaless 'true');\n  ```\n- Query data:\n  ```sql\n  -- non-schemaless mode\n  SELECT * FROM example;\n   one |    two     | three |        four         |    five    | six | seven \n  -----+------------+-------+---------------------+------------+-----+-------\n     1 | {1,2,3}    | foo   | 2018-01-01 00:00:00 | 2018-01-01 | t   |   0.5\n     2 | {NULL,5,6} | bar   | 2018-01-02 00:00:00 | 2018-01-02 | f   |      \n  (2 rows)\n  -- schemaless mode\n  SELECT * FROM example_schemaless;\n                                                                    v\n  ---------------------------------------------------------------------------------------------------------------------------------\n   {\"one\": 1, \"six\": \"t\", \"two\": [1, 2, 3], \"five\": \"2018-01-01\", \"four\": \"2018-01-01 00:00:00\", \"seven\": 0.5, \"three\": \"foo\"}\n   {\"one\": 2, \"six\": \"f\", \"two\": [null, 5, 6], \"five\": \"2018-01-02\", \"four\": \"2018-01-02 00:00:00\", \"seven\": null, \"three\": \"bar\"}\n  (2 rows)\n  ```\n\n- Fetch values in jsonb expression:  \n  - Use `-\u003e\u003e` jsonb arrow operator which return text type. User may cast type the jsonb expression to get corresponding data representation.  \n  - For example, `v-\u003e\u003e'col'` expression of fetch value `col` will be column name `col` in parquet file and we call it `schemaless variable` or `slvar`.  \n    ```sql\n    SELECT v-\u003e\u003e'two', sqrt((v-\u003e\u003e'one')::int) FROM example_schemaless;\n      ?column?   |        sqrt        \n    --------------+--------------------\n    [1, 2, 3]    |                  1\n    [null, 5, 6] | 1.4142135623730951\n    (2 rows)\n    ```\n\n- Some feature is different with `non-schemaless` mode\n  - Rowgroup filter support: in schemaless mode, parquet_s3_fdw can support execute row group filter with some `WHERE` condition below:\n    - `slvar::type {operator} const`. For example: `(v-\u003e\u003e'int64_col')::int8 = 100`\n    - `const {operator} slvar ::type`. For example: `100 = (v-\u003e\u003e'int64_col')::int8`\n    - `slvar::boolean is true/false`. For example: `(v-\u003e\u003e'bool_col')::boolean is false`\n    - `!(slvar::boolean)`. For example: `!(v-\u003e\u003e'bool_col')::boolean`\n    - Jsonb `exist` operator: `((v-\u003e\u003e'col')::jsonb) ? element`, `(v-\u003e'col') ? element` and `v ? 'col'`\n    - The cast function must be mapped with the parquet column type, otherwise, the filter will be skipped.\n  - To use presort column of parquet file, user must be:\n    - define column name in `sorted` option same as `non-schemaless mode`\n    - Use `slvar` instead of column name in the `ORDER BY` clause.\n    - If the sorted parquet column is not a text column, please add the explicit cast to the mapped type of this column.\n    - For example:\n      ```sql\n      CREATE FOREIGN TABLE example_sorted (v jsonb)\n      SERVER parquet_s3_srv\n      OPTIONS (filename '/path/to/example1.parquet /path/to/example2.parquet', sorted 'int64_col', schemaless 'true');\n      EXPLAIN (COSTS OFF) SELECT * FROM example_sorted ORDER BY (v-\u003e\u003e'int64_col')::int8;\n                QUERY PLAN           \n      --------------------------------\n      Foreign Scan on example_sorted\n        Reader: Multifile Merge\n        Row groups: \n          example1.parquet: 1, 2\n          example2.parquet: 1\n      (5 rows)\n      ```\n  - Support for arrow Nested List and Map: these type will be treated as nested jsonb value which can access by `-\u003e` operator.  \n  For example:\n    ```sql\n    SELECT * FROM example_schemaless;\n                                      v\n    ----------------------------------------------------------------------------\n    {\"array_col\": [19, 20], \"jsonb_col\": {\"1\": \"foo\", \"2\": \"bar\", \"3\": \"baz\"}}\n    {\"array_col\": [21, 22], \"jsonb_col\": {\"4\": \"test1\", \"5\": \"test2\"}}\n    (2 rows)\n\n    SELECT v-\u003e'array_col'-\u003e1, v-\u003e'jsonb_col'-\u003e'1' FROM example3;\n    ?column? | ?column? \n    ----------+----------\n    20       | \"foo\"\n    22       | \n    (2 rows)\n    ```\n\n  - Postgres cost for caculate `(jsonb-\u003e\u003e'col')::type` is much larger than fetch column directly in `non-schemaless` mode, The query plan of `schemaless` mode can be different with `non-schemaless` mode in some complex query.\n\n- For other feature, `schemaless` mode works same as `non-schemaless` mode.\n\n## Write-able FDW\nThe user can issue an insert, update and delete statement for the foreign table, which has set the key columns.\n\n### Key columns\n- in non-schemaless mode: The key columns can be set while creating a parquet_s3_fdw foreign table object with OPTIONS(key 'true'):\n```sql\nCREATE FOREIGN TABLE userdata (\n    id1          int OPTIONS(key 'true'),\n    id2          int OPTIONS(key 'true'),\n    first_name   text,\n    last_name    text\n) SERVER parquet_s3_srv\nOPTIONS (\n    filename 's3://bucket/dir/userdata1.parquet'\n);\n```\n- in schemaless mode The key columns can be set while creating a parquet_s3_fdw foreign table object with `key_columns` option:\n```sql\nCREATE FOREIGN TABLE userdata (\n    v JSONB\n) SERVER parquet_s3_srv\nOPTIONS (\n    filename 's3://bucket/dir/userdata1.parquet',\n    schemaless 'true',\n    key_columns 'id1 id2'\n);\n```\n- `key_columns` option can be use in IMPORT FOREIGN SCHEMA feature:\n```sql\n-- in schemaless mode\nIMPORT FOREIGN SCHEMA 's3://data/' FROM SERVER parquet_s3_srv INTO tmp_schema\nOPTIONS (sorted 'c1', schemaless 'true', key_columns 'id1 id2');\n-- corresponding CREATE FOREIGN TABLE\nCREATE FOREIGN TABLE tbl1 (\n      v jsonb\n) SERVER parquet_s3_srv\nOPTIONS (filename 's3://data/tbl1.parquet', sorted 'c1', schemaless 'true', key_columns 'id1 id2');\n\n-- in non-schemaless mode\nIMPORT FOREIGN SCHEMA 's3://data/' FROM SERVER parquet_s3_srv INTO tmp_schema\nOPTIONS (sorted 'c1', schemaless 'true', key_columns 'id1 id2');\n-- corresponding CREATE FOREIGN TABLE\nCREATE FOREIGN TABLE tbl1 (\n      id1 INT OPTIONS (key 'true'),\n      id2 INT OPTIONS (key 'true'),\n      c1  TEXT,\n      c2  FLOAT\n) SERVER parquet_s3_srv\nOPTIONS (filename 's3://data/tbl1.parquet', sorted 'c1');\n```\n### insert_file_selector option\nUser defined function signature that is used by parquet_s3_fdw to retrieve the target parquet file on INSERT query:\n```sql\nCREATE FUNCTION insert_file_selector_func(one INT8, dirname text)\nRETURNS TEXT AS\n$$\n    SELECT (dirname || '/example7.parquet')::TEXT;\n$$\nLANGUAGE SQL;\n\nCREATE FOREIGN TABLE example_func (one INT8 OPTIONS (key 'true'), two TEXT)\nSERVER parquet_s3_srv\nOPTIONS (\n    insert_file_selector 'insert_file_selector_func(one, dirname)',\n    dirname '/tmp/data_local/data/test',\n    sorted 'one');\n```\n- insert_file_selector function signature spec:\n  - Syntax: `[function name]([arg name] , [arg name] ...)`\n  - Default return type is `TEXT` (full paths to parquet file)\n  - `[arg name]`: must be foreign table column name or `dirname`\n  - args value:\n    - `dirname` arg: value of dirname option.\n    - `column` args: get from inserted slot by name.\n\n### Sorted columns:\nparquet_s3_fdw supports keeping the sorted column still sorted in the modify feature.\n\nColumn name is double quoted if it has space character.\n\n```sql\nCREATE FOREIGN TABLE example (\n    'C 1'        int,\n    c2           int\n) SERVER parquet_s3_srv\nOPTIONS (\n    filename 's3://bucket/dir/exampe.parquet' sorted '\"C 1\"');\n```\n### Parquet file schema:\nBasically, the parquet file schema is defined according to a list of column names and corresponding types, but in parquet_s3_fdw's scan, it assumes that all columns with the same name have the same type. So, in modify feature, this assumption will be use also.\n\n### Type mapping from postgres to arrow type:\n- primitive type mapping:\n  |        SQL type        |   Arrow type |\n  |-----------------------:|-------------:|\n  |                   BOOL |         BOOL |\n  |                   INT2 |        INT16 |\n  |                   INT4 |        INT32 |\n  |                   INT8 |        INT64 |\n  |                 FLOAT4 |        FLOAT |\n  |                 FLOAT8 |       DOUBLE |\n  |  TIMESTAMP/TIMESTAMPTZ |    TIMESTAMP |\n  |                   DATE |       DATE32 |\n  |                   TEXT |       STRING |\n  |                  BYTEA |       BINARY |\n- Default time precision for arrow::TIMESTAMP is microsecond an in UTC timezone.\n- LIST are created by its element type, just support primitive type for element.\n- MAP are created by its jsonb element type:\n  |  jsonb type   |   Arrow type |\n  |--------------:|-------------:|\n  |          text |       STRING |\n  |       numeric |       FLOAT8 |\n  |       boolean |         BOOL |\n  |          null |       STRING |\n  |   other types |       STRING |\n\n- In schemaless mode:\n  - The mapping for primitive jsonb type is same as MAP in non-schemaless mode.\n  - For first nested jsonb in schemaless mode:\n    |  jsonb type   |   Arrow type |\n    |--------------:|-------------:|\n    |         array |         LIST |\n    |        object |          MAP |\n  - Element type of LIST and MAP is same as MAP type in non-schemaless mode.\n\n### INSERT\n```sql\n-- non-schemaless mode\nCREATE FOREIGN TABLE example_insert (\n    c1 INT2 OPTIONS (key 'true'),\n    c2 TEXT,\n    c3 BOOLEAN\n) SERVER parquet_s3_srv OPTIONS (filename 's3://data/example_insert.parquet');\n\nINSERT INTO example_insert VALUES (1, 'text1', true), (2, DEFAULT, false), ((select 3), (select i from (values('values are fun!')) as foo (i)), true);\nINSERT 0 3\n\nSELECT * FROM example_insert;\n c1 |       c2        | c3 \n----+-----------------+----\n  1 | text1           | t\n  2 |                 | f\n  3 | values are fun! | t\n(3 rows)\n\n-- schemaless mode\nCREATE FOREIGN TABLE example_insert_schemaless (\n    v JSONB\n) SERVER parquet_s3_srv OPTIONS (filename 's3://data/example_insert.parquet', schemaless 'true', key_column 'c1');\n\nINSERT INTO example_insert_schemaless VALUES ('{\"c1\": 1, \"c2\": \"text1\", \"c3\": true}'), ('{\"c1\": 2, \"c2\": null, \"c3\": false}'), ('{\"c1\": 3, \"c2\": \"values are fun!\", \"c3\": true}');\n\nSELECT * FROM example_insert_schemaless;\n                       v                       \n-----------------------------------------------\n {\"c1\": 1, \"c2\": \"text1\", \"c3\": \"t\"}\n {\"c1\": 2, \"c2\": null, \"c3\": \"f\"}\n {\"c1\": 3, \"c2\": \"values are fun!\", \"c3\": \"t\"}\n(3 rows)\n\n```\n- Select file to insert:\n  - In case, option `insert_file_selector` exists, target file is the result of this function.\n    - If target file does not exist, create new file with the same name of target file.\n    - If target file exists, but its schema does not match with list columns of insert record, an error message will be raised.\n  - In case, option `insert_file_selector` does not exist:\n    - target file is the first file whose schema matches the inserted record (all columns of inserted record exist in  the target file).\n    - If no file that meets its schema matches the columns of insert record and `dirname` option has specified. Creating new file with name format:\n    ``` [foreign_table_name]_[date_time].parquet ```\n    - Otherwise, an error message will be raised.\n- The new file schema:\n  - In non-schemaless mode, the new file will have all columns existed in foreign table.\n  - In schemaless mode, the new file will have all column specify in jsonb value.\n  - Column information:\n    - Get from existed file list.\n    - If column does not exist in any file: create bases on [pre-defined mapping type](#type-mapping-from-postgres-to-arrow-type).\n### UPDATE/DELETE\n```sql\n-- non-schemaless mode\nCREATE FOREIGN TABLE example (\n    c1 INT2 OPTIONS (key 'true'),\n    c2 TEXT,\n    c3 BOOLEAN\n) SERVER parquet_s3_srv OPTIONS (filename 's3://data/example.parquet');\n\nSELECT * FROM example;\n c1 |       c2        | c3 \n----+-----------------+----\n  1 | text1           | t\n  2 |                 | f\n  3 | values are fun! | t\n(3 rows)\n\nUPDATE example SET c3 = false WHERE c2 = 'text1';\nUPDATE 1\n\nSELECT * FROM example;\n c1 |       c2        | c3 \n----+-----------------+----\n  1 | text1           | f\n  2 |                 | f\n  3 | values are fun! | t\n(3 rows)\n\nDELETE FROM example WHERE c1 = 2;\nDELETE 1\n\nSELECT * FROM example;\n c1 |       c2        | c3 \n----+-----------------+----\n  1 | text1           | f\n  3 | values are fun! | t\n(2 rows)\n\n-- schemaless mode\nCREATE FOREIGN TABLE example_schemaless (\n    v JSONB\n) SERVER parquet_s3_srv OPTIONS (filename 's3://data/example.parquet', schemaless 'true', key_columns 'c1');\n\nSELECT * FROM example_schemaless;\n                       v                       \n-----------------------------------------------\n {\"c1\": 1, \"c2\": \"text1\", \"c3\": \"t\"}\n {\"c1\": 2, \"c2\": null, \"c3\": \"f\"}\n {\"c1\": 3, \"c2\": \"values are fun!\", \"c3\": \"t\"}\n(3 rows)\n\nUPDATE example_schemaless SET v='{\"c3\":false}' WHERE v-\u003e\u003e'c2' = 'text1';\nUPDATE 1\n\nSELECT * FROM example_schemaless;\n                       v                       \n-----------------------------------------------\n {\"c1\": 1, \"c2\": \"text1\", \"c3\": \"f\"}\n {\"c1\": 2, \"c2\": null, \"c3\": \"f\"}\n {\"c1\": 3, \"c2\": \"values are fun!\", \"c3\": \"t\"}\n(3 rows)\n\nDELETE FROM example_schemaless WHERE (v-\u003e\u003e'c1')::int = 2;\nDELETE 1\n\nSELECT * FROM example_schemaless;\n                       v                       \n-----------------------------------------------\n {\"c1\": 1, \"c2\": \"text1\", \"c3\": \"f\"}\n {\"c1\": 3, \"c2\": \"values are fun!\", \"c3\": \"t\"}\n(2 rows)\n```\n### Compression\n- Support 03 compression options: zstd, snappy and uncompressed. Compression options can be specified at both column and table.\n- If there is no compression option\n  - In case of existed file, the original compression of parquet file will be retained.\n  - In case of new created file, the default compression is uncompressed.\n- If there is only table compression option, the compression will be applied to all columns.\n- If there are both table and column options, the compression at column level will be prioritized to use.\n- If there is column compression option:\n  - In non-schemaless mode, the compression will be applied to the specified column only. Compression in other columns will be retained.\n  - In schemaless mode, the compression will be applied to all columns of parquet file.\n\n- Example about compression for both table and column in non-schemaless mode.\n  - 'snappy' will be applied to column c1 and c3 and 'zstd' will be applied to column c2.\n```sql\n-- non-schemaless mode\nCREATE FOREIGN TABLE example_insert (\n    c1 INT2 OPTIONS (key 'true'),\n    c2 TEXT OPTIONS (compression_type 'zstd'),\n    c3 BOOLEAN\n) SERVER parquet_s3_srv OPTIONS (filename 's3://data/example_insert.parquet', compression_type 'snappy');\n```\n\n- Example about compression for both table and column in schemaless mode.\n  - 'zstd' will be applied to all columns.\n```sql\n-- schemaless mode\nCREATE FOREIGN TABLE example_insert (\n    v jsonb OPTIONS (compression_type 'zstd'),\n) SERVER parquet_s3_srv OPTIONS (filename 's3://data/example_insert.parquet', compression_type 'snappy');\n```\n\n- Example about compression for column\n  - 'zstd' is applied to column c2, the original format is retained by c1 and c3 in non-schemaless mode\n  - 'zstd' is applied to all columns in schemaless mode\n```sql\n-- non-schemaless mode\nCREATE FOREIGN TABLE example_insert (\n    c1 INT2 OPTIONS (key 'true'),\n    c2 TEXT OPTIONS (compression_type 'zstd'),\n    c3 BOOLEAN\n) SERVER parquet_s3_srv OPTIONS (filename 's3://data/example_insert.parquet');\n\n-- schemaless mode\nCREATE FOREIGN TABLE example_insert (\n    v jsonb OPTIONS (compression_type 'zstd'),\n) SERVER parquet_s3_srv OPTIONS (filename 's3://data/example_insert.parquet');\n```\n\n- Example about compression for table\n  - 'zstd' is applied to all columns\n\n```sql\n-- non-schemaless mode\nCREATE FOREIGN TABLE example_insert (\n    c1 INT2 OPTIONS (key 'true'),\n    c2 TEXT,\n    c3 BOOLEAN\n) SERVER parquet_s3_srv OPTIONS (filename 's3://data/example_insert.parquet', compression_type 'zstd');\n\n-- schemaless mode\nCREATE FOREIGN TABLE example_insert (\n    v jsonb,\n) SERVER parquet_s3_srv OPTIONS (filename 's3://data/example_insert.parquet', compression_type 'zstd');\n```\n\n## Limitations\n- Transaction is not supported.\n- Cannot create a single foreign table using parquet files on both file system and Amazon S3.\n- The 4th and 5th arguments of `import_parquet_s3_explicit()` function are meaningless in `schemaless` mode.\n  - These arguments should be defined as `NULL` value.\n  - If these arguments is not NULL value the `WARNING` below will occur:\n    ```\n    WARNING: parquet_s3_fdw: attnames and atttypes are expected to be NULL. They are meaningless for schemaless table.\n    HINT: Schemaless table imported always contain \"v\" column with \"jsonb\" type.\n    ```\n- `schemaless` mode does not support create partition table by `CREATE TABLE parent_tbl (v jsonb) PARTITION BY RANGE((v-\u003e\u003e'a')::int)`.\n- In modifying features:\n  - `parquet_s3_fdw` modifies the parquet file by creating a modifiable cache data from the target parquet file and overwriting the old one:\n    - Performance won't be good for large files.\n    - When exact same file is modifying concurrently, the result would be inconsistent.\n  - WITH CHECK OPTION, ON CONFLICT and RETURNING are not supported.\n  - `sorted` columns only supports the following types: `int2`, `int4`, `int8`, `date`, `timestamp`, `float4`, `float8`.\n  - `key` columns only supports the following types: `int2`, `int4`, `int8`, `date`, `timestamp`, `float4`, `float8` and `text`.\n  - `key` columns values must be unique, `parquet_s3_fdw` does not support checking for unique values for key columns, user must do that.\n  - `key` columns only required for UPDATE/DELETE.\n\n- Does not support compression level setting for the compression type 'zstd'. The default compression level (3) is supported.\n\n## Contributing\nOpening issues and pull requests on GitHub are welcome.\n\n## License\nCopyright (c) 2021, TOSHIBA Corporation  \nCopyright (c) 2018 - 2019, adjust GmbH\n\nPermission to use, copy, modify, and distribute this software and its documentation for any purpose, without fee, and without a written agreement is hereby granted, provided that the above copyright notice and this paragraph and the following two paragraphs appear in all copies.\n\nSee the [`LICENSE.md`][4] file for full details.\n\n[1]: https://github.com/adjust/parquet_fdw\n[2]: https://aws.amazon.com/s3/\n[3]: https://min.io/\n[4]: LICENSE.md\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpgspider%2Fparquet_s3_fdw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpgspider%2Fparquet_s3_fdw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpgspider%2Fparquet_s3_fdw/lists"}