{"id":26544952,"url":"https://github.com/quackscience/duckdb-extension-clickhouse-native","last_synced_at":"2025-03-22T04:15:08.236Z","repository":{"id":267049353,"uuid":"900111398","full_name":"quackscience/duckdb-extension-clickhouse-native","owner":"quackscience","description":"Experimental ClickHouse Native Client and Native file reader Extension for DuckDB chsql","archived":false,"fork":false,"pushed_at":"2025-02-19T11:54:33.000Z","size":118,"stargazers_count":6,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-19T12:35:29.603Z","etag":null,"topics":["clickhouse","clickhouse-client","clickhouse-native","duckdb","duckdb-extension","reader"],"latest_commit_sha":null,"homepage":"https://duckdb.org/community_extensions/extensions/chsql_native.html","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/quackscience.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-07T22:21:44.000Z","updated_at":"2025-01-13T01:24:17.000Z","dependencies_parsed_at":"2024-12-08T00:22:54.834Z","dependency_job_id":"3f44a6f5-549c-4579-92cf-037340611c55","html_url":"https://github.com/quackscience/duckdb-extension-clickhouse-native","commit_stats":null,"previous_names":["quackmagic/duckdb-extension-clickhouse-native","quackscience/duckdb-extension-clickhouse-native"],"tags_count":1,"template":false,"template_full_name":"duckdb/extension-template-rs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Fduckdb-extension-clickhouse-native","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Fduckdb-extension-clickhouse-native/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Fduckdb-extension-clickhouse-native/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Fduckdb-extension-clickhouse-native/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/quackscience","download_url":"https://codeload.github.com/quackscience/duckdb-extension-clickhouse-native/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244902911,"owners_count":20529115,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","clickhouse-client","clickhouse-native","duckdb","duckdb-extension","reader"],"created_at":"2025-03-22T04:15:07.781Z","updated_at":"2025-03-22T04:15:08.229Z","avatar_url":"https://github.com/quackscience.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"https://community-extensions.duckdb.org/extensions/chsql.html\" target=\"_blank\"\u003e\n\u003cimg src=\"https://github.com/user-attachments/assets/9003897d-db6f-4a79-9443-9b72766b511b\" width=200\u003e\n\u003c/a\u003e\n\n# DuckDB Clickhouse Native Extension for [chsql](https://github.com/quackscience/duckdb-extension-clickhouse-sql)\nExperimental ClickHouse Native Client and Native file reader for DuckDB chsql\n\n\n### 📦 Installation\n```sql\nINSTALL chsql_native FROM community;\nLOAD chsql_native;\n```\n\n## 🤖 Native Client\nThe extension provides an experimental clickhouse native client: `clickhouse_scan`\n### 🏁 Settings\n```bash\n# Local Setup, Insecure\nexport CLICKHOUSE_URL=\"tcp://localhost:9000\"\n# Remote Setup, Secure\nexport CLICKHOUSE_URL=\"tcp://user:pass@remote:9440/?secure=true\u0026skip_verify=true\"\n```\n### ✏️ Usage\n```sql\nD SELECT * FROM clickhouse_scan(\"SELECT version(), 'hello', 123\");\n┌────────────┬─────────┬────────┐\n│ version()  │ 'hello' │  123   │\n│  varchar   │ varchar │ uint32 │\n├────────────┼─────────┼────────┤\n│ 24.10.2.80 │ hello   │    123 │\n└────────────┴─────────┴────────┘\n```\n\n## 🤖 Native Reader\nThe extension provides an experimental clickhouse native file reader: `clickhouse_native`\n\n### 🏁 Input\nGenerate some native files with `clickhouse-local` or `clickhouse-server`\n\n```sql\n--- simple w/ one row, two columns\nSELECT version(), number FROM numbers(1) INTO OUTFILE '/tmp/numbers.clickhouse' FORMAT Native;\n--- simple w/ one column, 100000 rows\nSELECT number FROM numbers(100000) INTO OUTFILE '/tmp/100000.clickhouse' FORMAT Native;\n--- complex w/ multiple types\nSELECT * FROM system.functions LIMIT 10 INTO OUTFILE '/tmp/functions.clickhouse' FORMAT Native;\n```\n\n### ✏️ Usage\nRead ClickHouse Native files with DuckDB. Reads are full-scans at this time.\n\n```sql\nD SELECT * FROM clickhouse_native('/tmp/numbers.clickhouse');\n┌──────────────┬─────────┐\n│  version()   │ number  │\n│   varchar    │  int32  │\n├──────────────┼─────────┤\n│ 24.12.1.1273 │ 0       │\n└──────────────┴─────────┘\n```\n```sql\nD SELECT count(*), max(number) FROM clickhouse_native('/tmp/100000.clickhouse');\n┌──────────────┬─────────────┐\n│ count_star() │ max(number) │\n│    int64     │    int32    │\n├──────────────┼─────────────┤\n│       100000 │       99999 │\n└──────────────┴─────────────┘\n```\n```sql\nD SELECT * FROM clickhouse_native('/tmp/functions.clickhouse') WHERE alias_to != '' LIMIT 10;\n┌────────────────────┬──────────────┬──────────────────┬──────────────────────┬──────────────┬─────────┬───┬─────────┬───────────┬────────────────┬──────────┬────────────┐\n│        name        │ is_aggregate │ case_insensitive │       alias_to       │ create_query │ origin  │ … │ syntax  │ arguments │ returned_value │ examples │ categories │\n│      varchar       │    int32     │      int32       │       varchar        │   varchar    │ varchar │   │ varchar │  varchar  │    varchar     │ varchar  │  varchar   │\n├────────────────────┼──────────────┼──────────────────┼──────────────────────┼──────────────┼─────────┼───┼─────────┼───────────┼────────────────┼──────────┼────────────┤\n│ connection_id      │            0 │                1 │ connectionID         │              │ System  │ … │         │           │                │          │            │\n│ rand32             │            0 │                0 │ rand                 │              │ System  │ … │         │           │                │          │            │\n│ INET6_ATON         │            0 │                1 │ IPv6StringToNum      │              │ System  │ … │         │           │                │          │            │\n│ INET_ATON          │            0 │                1 │ IPv4StringToNum      │              │ System  │ … │         │           │                │          │            │\n│ truncate           │            0 │                1 │ trunc                │              │ System  │ … │         │           │                │          │            │\n│ ceiling            │            0 │                1 │ ceil                 │              │ System  │ … │         │           │                │          │            │\n│ replace            │            0 │                1 │ replaceAll           │              │ System  │ … │         │           │                │          │            │\n│ from_utc_timestamp │            0 │                1 │ fromUTCTimestamp     │              │ System  │ … │         │           │                │          │            │\n│ mapFromString      │            0 │                0 │ extractKeyValuePairs │              │ System  │ … │         │           │                │          │            │\n│ str_to_map         │            0 │                1 │ extractKeyValuePairs │              │ System  │ … │         │           │                │          │            │\n├────────────────────┴──────────────┴──────────────────┴──────────────────────┴──────────────┴─────────┴───┴─────────┴───────────┴────────────────┴──────────┴────────────┤\n│ 10 rows                                                                                                                                           12 columns (11 shown) │\n└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘\n```\n\n#### Notes\n\n\u003e The reader is a clear room implementation of the ClickHouse Native file format using no code or libraries from ClickHouse Inc. As such it is potentially incomplete, imperfect and might not be compatible with all files. USE AT YOUR OWN RISK!\n\n### 🐎 Performance\nSimple CLI _cold start_ count() test using `duckdb` vs. `clickhouse-local` and 1M rows\n#### DuckDB\n```sql\n# time duckdb -c \"LOAD chsql_native; SELECT count(*) FROM clickhouse_native('/tmp/1M.clickhouse');\"\n┌──────────────┐\n│ count_star() │\n│    int64     │\n├──────────────┤\n│      1000000 │\n└──────────────┘\n\nreal\t0m0.095s\nuser\t0m0.077s\nsys\t0m0.029s\n```\n#### clickhouse-local\n```sql\n# time clickhouse local \"SELECT count(*) FROM '/tmp/1M.clickhouse'\";\n1000000\n\nreal\t0m0.141s\nuser\t0m0.086s\nsys\t0m0.043s\n```\n\n\u003cbr\u003e\n\n### ⛑️ Extension Status\n- [x] Basic Fomat Reading\n  - [x] Column Extraction\n  - [x] Blocks Parser \u0026 Iterator\n  - [x] Type Mapping WIP\n    - [x] Strings\n    - [x] Integers\n    - [x] Enums\n    - [ ] ??? as String\n  - [ ] Compression support\n- [x] Basic Native Client\n  - [x] clickhouse-rs binding\n  - [x] TLS Support\n  - [x] Type Mapping WIP\n    - [x] Strings\n    - [x] Integers\n    - [ ] Everything Else\n\n\u003cbr\u003e\n\n\n### ⚙️ Dev Build\nYou can easily modify the code and build a local extension for testing and development.\n\n#### Requirements\n- Rust\n\n1) Clone and Compile the extension on your system\n\n```bash\ncd /usr/src\ngit clone --recurse-submodules https://github.com/quackscience/duckdb-extension-clickhouse-native\ncd duckdb-extension-clickhouse-native\nmake configure \u0026\u0026 make\n```\n\n2) Download and Run DuckDB with -unsigned\n```\nwget https://github.com/duckdb/duckdb/releases/download/v1.1.3/duckdb_cli-linux-amd64.zip \u0026\u0026 unzip duckdb_cli-linux-amd64.zip\n./duckdb -unsigned\n```\n\n3) Load your local extension build\n```sql\nD LOAD '/usr/src/duckdb-extension-clickhouse-native/build/debug/clickhouse_native.duckdb_extension';\n```\n\n\n----\n\n###### Disclaimer\n\u003e DuckDB ® is a trademark of DuckDB Foundation.\n\u003e ClickHouse® is a trademark of ClickHouse Inc. All trademarks, service marks, and logos mentioned or depicted are the property of their respective owners. The use of any third-party trademarks, brand names, product names, and company names is purely informative or intended as parody and does not imply endorsement, affiliation, or association with the respective owners.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquackscience%2Fduckdb-extension-clickhouse-native","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquackscience%2Fduckdb-extension-clickhouse-native","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquackscience%2Fduckdb-extension-clickhouse-native/lists"}