{"id":37782609,"url":"https://github.com/pgspider/pgspider_ext","last_synced_at":"2026-01-16T15:09:24.597Z","repository":{"id":38275416,"uuid":"399727035","full_name":"pgspider/pgspider_ext","owner":"pgspider","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-20T01:14:27.000Z","size":3929,"stargazers_count":23,"open_issues_count":0,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-12-20T02:23:49.374Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pgspider.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"License","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-25T07:20:29.000Z","updated_at":"2024-12-20T01:13:08.000Z","dependencies_parsed_at":"2024-12-20T02:22:48.555Z","dependency_job_id":"b77490ea-560b-46a3-978b-6c339add468b","html_url":"https://github.com/pgspider/pgspider_ext","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/pgspider/pgspider_ext","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgspider%2Fpgspider_ext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgspider%2Fpgspider_ext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgspider%2Fpgspider_ext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgspider%2Fpgspider_ext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pgspider","download_url":"https://codeload.github.com/pgspider/pgspider_ext/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgspider%2Fpgspider_ext/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28479406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-16T15:09:24.490Z","updated_at":"2026-01-16T15:09:24.585Z","avatar_url":"https://github.com/pgspider.png","language":"PLpgSQL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PGSpider Extension\nPGSpider Extension(pgspider_ext) is an extension to construct High-Performance SQL Cluster Engine for distributed big data.\npgspider_ext enables PostgreSQL to access a number of data sources using Foreign Data Wrapper(FDW) and retrieves the distributed data source vertically.  \nCode of pgspider_ext is one of extension of PostgreSQL. We call PostgreSQL installed pgspider_ext as **\"PGSpider\"**.   \nUsage of PostgreSQL installed pgspider_ext is the same as PostgreSQL. You can use any client applications such as libpq and psql.\n\n## Features\n* Node partitioned table  \n    User can get records in multi tables on some data sources by one SQL easily.\n\tIf there are tables with similar schema in each data source, PGSpider can view them as a single virtual table.  \n    PGSpider run under Declarative Partitioning feature. Even if tables on data sources does not have a partition key, PGSpider creates a partition key automatically based on child node.  \n\tFor example, tables on data sources have 2 column(i and t).\n\tData on node1:\n\t\u003cpre\u003e\n\tSELECT * FROM t1_node1;\n\t  i | t\n\t----+---\n\t 10 | a\n\t 11 | b\n\t(2 rows)\n\t\u003c/pre\u003e\n\tData on node2:\n\t\u003cpre\u003e\n\tSELECT * FROM t1_node2;\n\t  i | t\n\t----+---\n\t 20 | c\n\t 21 | d\n\t(2 rows)\n\t\u003c/pre\u003e\n\tIf you create a partition table t1 using pgspider_ext from t1_node1 and t1_node2, the partitioned table has 3 columns (i, t and node). x is a column of partition key. You can distinguish data sources by column 'node'.  \n\tQuery on PGSpider:\n\t\u003cpre\u003e\n\t  i | t | node\n\t----+---+-------\n\t 10 | a | node1\n\t 11 | b | node1\n\t 20 | c | node2\n\t 21 | d | node2\n\t(4 rows)\n\t\u003c/pre\u003e\t\n\n\u003ccenter\u003e\u003cimg src=\"./images/structure.png\" width=70%\u003e\u003c/center\u003e\n\n* Parallel processing  \n    When PGSpider executes query, PGSpider expands partitioned table to child tables and fetches results from child nodes in parallel. \n\n* Pushdown   \n    WHERE clause and aggregation functions can be pushed down to child nodes.\n    The shippability depends on child FDW.\n\n## How to install pgspider_ext\n\nThe current version can work with PostgreSQL 13.15, 15.7, 16.3 and 17.0.\n\nDownload PostgreSQL source code.\n\u003cpre\u003e\nhttps://ftp.postgresql.org/pub/source/v17.0/postgresql-17.0.tar.gz\n\u003c/pre\u003e\n\nDecompress PostgreSQL source code. \n\u003cpre\u003e\ntar xvf postgresql-17.0.tar.gz\n\u003c/pre\u003e\n\nDownload pgspider_ext source code into \"contrib/pgspider_ext\" directory.\n\u003cpre\u003e\ngit clone XXX \n\u003c/pre\u003e\n\nBuild and install PostgreSQL and pgspider_ext.\n\u003cpre\u003e\ncd postgresql-17.0\n./configure\nmake\nsudo make install\ncd contrib/pgspider_ext\nmake \nsudo make install\n\u003c/pre\u003e\n\n## Usage\nFor example, there are 3 nodes (1 parent node and 2 child nodes).\nPGSpider runs on the parent node. And 2 child nodes are data sources.\nEach child node has PostgreSQL. They are accessed by PGSpider.  \n\nPlease install PostgreSQL on child nodes and install PostgreSQL FDW into PGSpider. \n\nInstall PostgreSQL FDW \n\u003cpre\u003e\ncd ../postgres_fdw\nmake \nsudo make install\n\u003c/pre\u003e\n\n### Start PGSpider\nYou can start PGSpider as same as PostgreSQL. \n\u003cpre\u003e\n/usr/local/pgsql\n\u003c/pre\u003e\n\nCreate database cluster and start server.\n\u003cpre\u003e\ncd /usr/local/pgsql/bin\n./initdb -D ~/pgspider_db\n./pg_ctl -D ~/pgspider_db start\n\u003c/pre\u003e\n\nConnect to PGSpider.\n\u003cpre\u003e\n./psql postgres\n\u003c/pre\u003e\n\n### Load extension\nPGSpider(Parent node)\n\u003cpre\u003e\nCREATE EXTENSION pgspider_ext;\n\u003c/pre\u003e\n\nPostgreSQL FDW\n\u003cpre\u003e\nCREATE EXTENSION postgres_fdw;\n\u003c/pre\u003e\n\n### Create server\nCreate PGSpider server.\n\u003cpre\u003e\nCREATE SERVER spdsrv FOREIGN DATA WRAPPER pgspider_ext;\n\u003c/pre\u003e\n\nCreate servers of child PostgreSQL nodes  \n\u003cpre\u003e\nCREATE SERVER pgsrv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS(host '127.0.0.1', port '5433', dbname 'postgres');\nCREATE SERVER pgsrv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS(host '127.0.0.1', port '5434', dbname 'postgres');\n\u003c/pre\u003e\n\n### Create user mapping\nCreate user mapping for PGSpider server.\nNo need to specify options.\n\u003cpre\u003e\nCREATE USER MAPPING FOR CURRENT_USER SERVER spdsrv;\n\u003c/pre\u003e\n\nUser mapping for PostgreSQL servers.\n\u003cpre\u003e\nCREATE USER MAPPING FOR CURRENT_USER SERVER pgsrv1 OPTIONS(user 'user', password 'pass');\nCREATE USER MAPPING FOR CURRENT_USER SERVER pgsrv2 OPTIONS(user 'user', password 'pass');\n\u003c/pre\u003e\n\n### Create foreign tables\nCreate foreign tables of child nodes according to data source FDW usage.\nIn this example, each PostgreSQL server has table 't1' which has 2 columns ('i' and 't').\nYou can also create them by using IMPORT FOREIGN SCHEMA if data source FDW supports it.\n\u003cpre\u003e\nCREATE FOREIGN TABLE t1_pg1_child (i int, t text) SERVER pgsrv1 OPTIONS (table_name 't1');\nCREATE FOREIGN TABLE t1_pg2_child (i int, t text) SERVER pgsrv2 OPTIONS (table_name 't1');\n\u003c/pre\u003e\n\n### Create partition table\nCreate a partition parent table and partition child tables.\nPartition child tables are corresponding to each foreign table created at the previous step.  \nYou need to declare a partition key column **at the last** in addition to columns of data source table.  \n\nA partition parent table:  \nIn this example, we define 'node' column as a partition key column.\n\u003cpre\u003e\nCREATE TABLE t1(i int, t integer, node text) PARTITION BY LIST (node);\n\u003c/pre\u003e\n\nPartition child tables:\n\u003cpre\u003e\nCREATE FOREIGN TABLE t1_pg1 PARTITION OF t1 FOR VALUES IN ('node1') SERVER spdsrv;\nCREATE FOREIGN TABLE t1_pg2 PARTITION OF t1 FOR VALUES IN ('node2') SERVER spdsrv OPTIONS (child_name 't1_child2');\n\u003c/pre\u003e\n\n't1_a' is corresponding to the foreign table 't1_pg1'.\nPGSpider searches the corresponding foreign table by name having \"[table name]_child\" by default.\nYou can specify the name by the 'child_name' option.\n\n### Access a partition table\n\u003cpre\u003e\nSELECT * FROM t1;\n  i | t | node\n----+---+-------\n 10 | a | node1\n 11 | b | node1\n 20 | c | node2\n 21 | d | node2\n(4 rows)\n\u003c/pre\u003e\n\n## Note\nIf you want to pushdown aggregate functions, you needs to execute:\n\u003cpre\u003e\nSET enable_partitionwise_aggregate TO on;\n\u003c/pre\u003e\n\nWe have confirmed that PGSpider can connect to:\n- PostgreSQL ([postgres_fdw](https://github.com/postgres/postgres))\n- MySQL ([mysql_fdw](https://github.com/pgspider/mysql_fdw))\n- DynamoDB ([dynamodb_fdw](https://github.com/pgspider/dynamodb_fdw))\n- GridDB ([griddb_fdw](https://github.com/pgspider/griddb_fdw))\n- JDBC ([jdbc_fdw](https://github.com/pgspider/jdbc_fdw))\n- ODBC ([odbc_fdw](https://github.com/pgspider/odbc_fdw))\n- Parquet S3 ([parquet_s3_fdw](https://github.com/pgspider/parquet_s3_fdw))\n- SQLite ([sqlite_fdw](https://github.com/pgspider/sqlite_fdw))\n- MongoDB ([mongo_fdw](https://github.com/pgspider/mongo_fdw))\n- InfluxDB ([influxdb_fdw](https://github.com/pgspider/influxdb_fdw))\n\nCurrently, PostgreSQL has a bug which will cause duplicate data when parallel query is executed.\nIt happens when leader node and worker nodes scan the same data.\nWe can avoid this bug by disabling leader node from collecting data by executing:\n\u003cpre\u003e\nSET parallel_leader_participation TO off;\n\u003c/pre\u003e\n\n## Test execution\n\nPGSpider Extension has several test scripts to execute test for each data source.\n| No | Test script file name | Corresponding test files | Remark |\n| ----------- | ----------- | ----------- | ----------- |\n| 1\t| test_dynamodb.sh | ported_dynamodb_fdw.sql | Test for dynamodb_fdw |\n| 2\t| test_griddb.sh | ported_griddb_fdw.sql | Test for griddb_fdw |\n| 3\t| test_jdbc.sh | ported_jdbc_fdw_griddb.sql\u003cbr\u003eported_jdbc_fdw_mysql.sql\u003cbr\u003eported_jdbc_fdw_postgres.sql| Test for jdbc_fdw |\n| 4\t| test_mongodb.sh | ported_mongodb_fdw.sql | Test for mongo_fdw |\n| 5\t| test_mysql.sh | ported_mysql_fdw.sql | Test for mysql_fdw |\n| 6\t| test_odbc.sh | ported_odbc_fdw_mysql.sql\u003cbr\u003eported_odbc_fdw_postgres.sql| Test for odbc_fdw |\n| 7\t| test_parquet_s3_fdw.sh | ported_parquet_s3_fdw_local.sql\u003cbr\u003eported_parquet_s3_fdw_server.sql | Test for parquet_s3_fdw |\n| 8\t| test_pgspider_core_fdw_parquet_s3.sh | ported_pgspider_core_fdw_import_s3.sql\u003cbr\u003eported_pgspider_core_fdw_parquet_s3_fdw.sql\u003cbr\u003eported_pgspider_core_fdw_parquet_s3_fdw2.sql | Test for pgspider_ext with parquet_s3_fdw, ported from pgspider_core_fdw |\n| 9 | test_pgspider_core_fdw_postgres_fdw.sh | ported_pgspider_core_fdw_postgres_fdw.sql | Test for pgspider_ext with postgres_fdw, ported from pgspider_core_fdw |\n| 10 | test_postgres.sh | pgspider_ext.sql\u003cbr\u003epgspider_ext2\u003cbr\u003epgspider_ext3 | Test for postgres_fdw |\n| 11 | test_sqlite.sh | ported_sqlite_fdw.sql | Test for sqlite_fdw |\n| 12 | test_influxdb.sh | ported_influxdb_fdw.sql | Test for influxdb_fdw |\n\nUser can execute test script to execute test for corresponding data source.\n\u003cpre\u003e\n./test_sqlite.sh\n\u003c/pre\u003e\n\nFor some test script, user need to update the paths inside test script or specify the environment variable before execution.\n### test_dynamodb.sh\n- Export LD_LIBRARY_PATH to the installed folder of AWS C++ SDK (which contains some libaws library).\n\u003cpre\u003e\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib64\n\u003c/pre\u003e\n- Update DYNAMODB_ENDPOINT in sql/init_data/dynamodb_fdw/dynamodb_init.sh if necessary.\n\u003cpre\u003e\nDYNAMODB_ENDPOINT=\"http://localhost:8000\"\n\u003c/pre\u003e\n- Make sure the data in sql/parameters/dynamodb_parameters.conf matches with data of data source.\n### test_griddb.sh\n- Download and build GridDB's C Client, rename the folder to griddb and put it into pgspider_ext/sql/init_data/griddb_fdw directory.\n- Export LD_LIBRARY_PATH to the bin folder of GridDB's C Client.\n\u003cpre\u003e\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/jenkins/postgres/postgresql-17.0/contrib/pgspider_ext/sql/init_data/griddb_fdw/griddb/bin\n\u003c/pre\u003e\n- Make sure the data in sql/parameters/griddb_parameters.conf matches with data of data source.\n### test_jdbc.sh\n- Update paths in sql/init_data/jdbc_fdw/jdbc_mysql_init.sh and sql/init_data/jdbc_fdw/jdbc_postgres_init.sh.\n\u003cpre\u003e\nexport PGS_SRC_DIR=\"/home/jenkins/postgres/postgresql-17.0/\"\n\nexport PGS_BIN_DIR=\"/home/jenkins/postgres/postgresql-17.0/PGS\"\nexport FDW_DIR=\"/home/jenkins/postgres/postgresql-17.0/contrib/pgspider_ext\"\n\u003c/pre\u003e\n- Update DB_DRIVERPATH, DB_DATA in sql/parameters/jdbc_griddb_parameters.conf, sql/parameters/jdbc_mysql_parameters.conf and sql/parameters/jdbc_postgres_parameters.conf.\n\u003cpre\u003e\n\\set DB_DRIVERPATH\t'\\'/home/jenkins/src/jdbc/gridstore-jdbc-5.5.0.jar\\''\n\n\\set DB_DATA\t\t'\\'/home/jenkins/postgres/postgresql-17.0/contrib/pgspider_ext/sql/init_data/jdbc_fdw/data'\n\u003c/pre\u003e\n- Make sure the data in sql/parameters/jdbc_griddb_parameters.conf, sql/parameters/jdbc_mysql_parameters.conf and sql/parameters/jdbc_postgres_parameters.conf matches with data of data source.\n### test_mongodb.sh\n- Make sure the data in sql/parameters/mongodb_parameters.conf matches with data of data source.\n### test_mysql.sh\n- Make sure the data in sql/parameters/mysql_parameters.conf matches with data of data source.\n- Open test_mysql.sh and executes the command in comment block manually.\n\u003cpre\u003e\n# --connect to mysql with root user\n# mysql -u root -p\n\n# --run below\n# CREATE DATABASE mysql_fdw_post;\n# SET GLOBAL validate_password.policy = LOW;\n# SET GLOBAL validate_password.length = 1;\n# SET GLOBAL validate_password.mixed_case_count = 0;\n# SET GLOBAL validate_password.number_count = 0;\n# SET GLOBAL validate_password.special_char_count = 0;\n# CREATE USER 'edb'@'localhost' IDENTIFIED BY 'edb';\n# GRANT ALL PRIVILEGES ON mysql_fdw_post.* TO 'edb'@'localhost';\n# GRANT SUPER ON *.* TO 'edb'@localhost;\n\n# Set time zone to default time zone of make check PST.\n# SET GLOBAL time_zone = '+00:00';\n# SET GLOBAL log_bin_trust_function_creators = 1;\n# SET GLOBAL local_infile=1;\n\u003c/pre\u003e\n### test_odbc.sh\n- Update paths in test_odbc.sh\n\u003cpre\u003e\nexport PGS_SRC_DIR=\"/home/jenkins/release_FDW/postgresql-17.0\"\nexport PGS_BIN_DIR=\"/home/jenkins/release_FDW/postgresql-17.0/PGS\"\nexport ODBC_FDW_DIR=\"/home/jenkins/release_FDW/postgresql-17.0/contrib/odbc_fdw\"\n\u003c/pre\u003e\n- Make sure the data in sql/parameters/odbc_mysql_parameters.conf, sql/parameters/odbc_postgres_parameters.conf matches with data of data source.\n### test_parquet_s3_fdw.sh\n- Make sure the data in sql/parameters/parquet_s3_local_parameters.conf, sql/parameters/parquet_s3_server_parameters.conf matches with data of data source.\n### test_pgspider_core_fdw_parquet_s3.sh\n- If do not use docker, comment out the following codes in test_pgspider_core_fdw_parquet_s3.sh\n\u003cpre\u003e\n# comment out these following code if do not use docker\ncontainer_name='minio_server'\n\nif [ ! \"$(docker ps -q -f name=^/${container_name}$)\" ]; then\n    if [ \"$(docker ps -aq -f status=exited -f status=created -f name=^/${container_name}$)\" ]; then\n        # cleanup\n        docker rm ${container_name} \n    fi\n    # run minio container\n   sudo docker run -d --name ${container_name} -it -p 9000:9000 -e \"MINIO_ACCESS_KEY=minioadmin\" -e \"MINIO_SECRET_KEY=minioadmin\" -v /tmp/data_s3:/data minio/minio server /data\nfi\n\u003c/pre\u003e\n### test_pgspider_core_fdw_postgres_fdw.sh\n- Update path in sql/init_data/pgspider_core_fdw/ported_postgres_setup.sh.\n\u003cpre\u003e\nPOSTGRES_HOME=/home/jenkins/postgres/postgresql-17.0/PGS\n\u003c/pre\u003e\n### test_influxdb.sh\n- Make sure the data in sql/parameters/influxdb_parameters.conf matches with data of data source.\n\n## Limitations\n- PGSpider Extension does not support INSERT, UPDATE, DELETE.\n\n## Contributing\nOpening issues and pull requests on GitHub are welcome.\n\n## License\nCopyright and license information can be found in the\nfile [`License`][1] .\n\n[1]: License","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpgspider%2Fpgspider_ext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpgspider%2Fpgspider_ext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpgspider%2Fpgspider_ext/lists"}