{"id":7839316,"url":"https://github.com/ildus/clickhouse_fdw","last_synced_at":"2025-04-06T01:08:56.048Z","repository":{"id":36764081,"uuid":"184786765","full_name":"ildus/clickhouse_fdw","owner":"ildus","description":"ClickHouse FDW for PostgreSQL","archived":false,"fork":false,"pushed_at":"2024-04-23T21:25:16.000Z","size":1381,"stargazers_count":266,"open_issues_count":37,"forks_count":56,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-04-02T05:42:11.935Z","etag":null,"topics":["binary","clickhouse","fdw","http","postgresql","pushdown"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ildus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-03T16:22:42.000Z","updated_at":"2025-03-06T10:40:24.000Z","dependencies_parsed_at":"2024-04-23T22:35:31.345Z","dependency_job_id":"eb14273a-1835-44b8-8ead-692739e6c292","html_url":"https://github.com/ildus/clickhouse_fdw","commit_stats":null,"previous_names":["adjust/clickhouse_fdw"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ildus%2Fclickhouse_fdw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ildus%2Fclickhouse_fdw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ildus%2Fclickhouse_fdw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ildus%2Fclickhouse_fdw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ildus","download_url":"https://codeload.github.com/ildus/clickhouse_fdw/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247419860,"owners_count":20936012,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary","clickhouse","fdw","http","postgresql","pushdown"],"created_at":"2024-04-13T00:11:35.246Z","updated_at":"2025-04-06T01:08:56.006Z","avatar_url":"https://github.com/ildus.png","language":"C++","readme":"`clickhouse_fdw` - ClickHouse Foreign Data Wrapper for PostgreSQL\n=================================================================\n\nThe `clickhouse_fdw` is open-source. It is a Foreign Data Wrapper (FDW) for `ClickHouse` column oriented database.\n\nSupported PostgreSQL Versions\n------------------------------\n\nPostgreSQL 11-14\n\nInstallation\n----------------\n\n###### Prerequisite\n\nThe `clickhouse_fdw` uses an HTTP interface provided by ClickHouse. `libcurl` and\n`uuid` libraries should be installed in the system. Make sure that ClickHouse\nuses UTC timezone.\n\n###### Installing `clickhouse_fdw`\n\n```\ngit clone git@github.com:ildus/clickhouse_fdw.git\ncd clickhouse_fdw\nmkdir build \u0026\u0026 cd build\ncmake ..\nmake \u0026\u0026 make install\n```\n\n###### Centos 7 Notes\n\nYou should be using modern compiler, available in [devtoolset-7](https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/). Once installed, activate it with the command:\n\n```sh\nsource scl_source enable devtoolset-7\n```\n\nMinimal libcurl compatible with clickhouse-fdw is 7.43.0. It is not available in the official Centos repo. You can download recent RPMs from [here](https://cbs.centos.org/koji/buildinfo?buildID=1408)\n\nYou can upgrade libcurl with command:\n\n```sh\nsudo rpm -Uvh  *curl*rpm --nodeps\nsudo yum install libmetalink\n```\n\n`uuid` can be installed with:\n\n```sh\nyum install libuuid-devel\n```\n\nUsage\n-----\n\nYou need to set up the sample database and tables in the ClickHouse database.\nHere we create a sample database name test_database and two sample tables\ntax_bills_nyc and tax_bills:\n\n    CREATE DATABASE test_database;\n    USE test_database;\n    CREATE TABLE tax_bills_nyc\n    (\n        bbl Int64,\n        owner_name String,\n        address String,\n        tax_class String,\n        tax_rate String,\n        emv Float64,\n        tbea Float64,\n        bav Float64,\n        tba String,\n        property_tax String,\n        condonumber String,\n        condo String,\n        insertion_date DateTime MATERIALIZED now()\n    )\n    ENGINE = MergeTree PARTITION BY tax_class ORDER BY (owner_name);\n\n    CREATE TABLE tax_bills\n    (\n        bbl Int64,\n        owner_name String\n    )\n    ENGINE = MergeTree\n    PARTITION BY bbl\n    ORDER BY bbl;\n\nDownload the sample data from the taxbills.nyc website and put the data in the table:\n\n    curl -X GET 'http://taxbills.nyc/tax_bills_june15_bbls.csv' | \\\n\t\tclickhouse-client --input_format_allow_errors_num=10 \\\n\t\t--query=\"INSERT INTO test_database.tax_bills_nyc FORMAT CSV\"\n\nNow the data is ready in the ClickHouse, the next step is to set up the PostgreSQL side.\nFirst we need to create a FDW extension and a ClickHouse foreign server:\n\n    CREATE EXTENSION clickhouse_fdw;\n    CREATE SERVER clickhouse_svr FOREIGN DATA WRAPPER clickhouse_fdw OPTIONS(dbname 'test_database');\n\nBy default the server will use `http` protocol. But we could use binary protocol:\n\n    CREATE SERVER clickhouse_svr FOREIGN DATA WRAPPER clickhouse_fdw\n\t\tOPTIONS(dbname 'test_database', driver 'binary');\n\nAvailable parameters:\n\n\t* dbname\n\t* host\n\t* port\n\t* driver\n\nNow create user mapping and foreign tables:\n\n    CREATE USER MAPPING FOR CURRENT_USER SERVER clickhouse_svr OPTIONS (user 'default', password '');\n    IMPORT FOREIGN SCHEMA \"test_database\" FROM SERVER clickhouse_svr INTO public;\n\n\tSELECT bbl,tbea,bav,insertion_date FROM tax_bills_nyc LIMIT 5;\n        bbl     | tbea  |  bav   |   insertion_date\n\t------------+-------+--------+---------------------\n     1001200009 | 72190 | 648900 | 2019-08-03 11:04:38\n     4000620001 |  8961 |  80550 | 2019-08-03 11:04:38\n     4157860094 | 13317 | 119700 | 2019-08-03 11:04:38\n     4123850237 |    50 |    450 | 2019-08-03 11:04:38\n     4103150163 |  2053 |  18450 | 2019-08-03 11:04:38\n\n    INSERT INTO tax_bills SELECT bbl, tbea from tax_bills_nyc LIMIT 100;\n\n    EXPLAIN VERBOSE SELECT bbl,tbea,bav,insertion_date FROM tax_bills_nyc LIMIT 5;\n                                         QUERY PLAN\n    --------------------------------------------------------------------------------------------\n    Limit  (cost=0.00..0.00 rows=1 width=32)\n    Output: bbl, tbea, bav, insertion_date\n     -\u003e  Foreign Scan on public.tax_bills_nyc  (cost=0.00..0.00 rows=0 width=32)\n             Output: bbl, tbea, bav, insertion_date\n             Remote SQL: SELECT bbl, tbea, bav, insertion_date FROM test_database.tax_bills_nyc\n    (5 rows)\n\nAggregatingMergeTree\n--------------------\n\nFor columns where you need `Merge` prefix in aggregations just add `AggregateFunction` option with aggregation function name. Example:\n\n```\nALTER TABLE table ALTER COLUMN b OPTIONS (SET AggregateFunction 'count');\n```\n\nOr use `IMPORT SCHEMA` for automatic definitions.\n\n[1]: https://www.postgresql.org/\n[2]: http://www.clickhouse.com\n[3]: https://github.com/ildus/clickhouse_fdw/issues/new\n","funding_links":[],"categories":["Language bindings"],"sub_categories":["Other sdk/libraries"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fildus%2Fclickhouse_fdw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fildus%2Fclickhouse_fdw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fildus%2Fclickhouse_fdw/lists"}