{"id":20673732,"url":"https://github.com/cpg314/polarhouse","last_synced_at":"2025-09-27T02:30:54.918Z","repository":{"id":218395580,"uuid":"746278857","full_name":"cpg314/polarhouse","owner":"cpg314","description":"Interoperability between Polars and Clickhouse","archived":false,"fork":false,"pushed_at":"2025-02-26T22:27:51.000Z","size":90,"stargazers_count":9,"open_issues_count":3,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-08T18:54:20.725Z","etag":null,"topics":["apache-arrow","clickhouse","polars","rust"],"latest_commit_sha":null,"homepage":"https://c.pgdm.ch/code","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cpg314.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-21T15:51:21.000Z","updated_at":"2025-03-13T16:06:14.000Z","dependencies_parsed_at":"2024-01-21T18:28:23.864Z","dependency_job_id":"68db50e0-e5cb-4cfe-b1bd-aa63d736cb2a","html_url":"https://github.com/cpg314/polarhouse","commit_stats":null,"previous_names":["cpg314/polarhouse"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/cpg314/polarhouse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fpolarhouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fpolarhouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fpolarhouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fpolarhouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cpg314","download_url":"https://codeload.github.com/cpg314/polarhouse/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fpolarhouse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277171430,"owners_count":25773212,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-27T02:00:08.978Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-arrow","clickhouse","polars","rust"],"created_at":"2024-11-16T20:42:23.289Z","updated_at":"2025-09-27T02:30:54.902Z","avatar_url":"https://github.com/cpg314.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"**Polarhouse** connects together\n\n- [Polars](https://pola.rs/) Dataframes (backed by the [Apache Arrow](https://arrow.apache.org/) columnar format)\n- the [Clickhouse](https://clickhouse.com/) columnar database.\n\nMore specifically, it allows:\n\n- inserting Polars Dataframes into Clickhouse tables (and creating these if necessary).\n- and vice-versa retrieving Clickhouse query results as Polars Dataframes.\n\nPolarhouse uses the native TCP Clickhouse protocol via the [`klickhouse`](https://github.com/Protryon/klickhouse) crate. It maps the Polars and Clickhouse types, and builds Polars `Series` (resp. Clickhouse columns) after transforming the data if necessary.\n\n```\nPolars\n┌──────────┬─────────┬──────┬───────────────────────────┐\n│ name     ┆ is_rich ┆ age  ┆ address                   │\n│ ---      ┆ ---     ┆ ---  ┆ ---                       │\n│ str      ┆ u8      ┆ i32  ┆ struct[2]                 │\n╞══════════╪═════════╪══════╪═══════════════════════════╡\n│ Batman   ┆ 1       ┆ 30   ┆ {{\"Chicago\",\"IL\"},\"USA\"}  │\n│ Superman ┆ null    ┆ null ┆ {{\"New York\",\"NY\"},\"USA\"} │\n└──────────┴─────────┴──────┴───────────────────────────┘\nClickhouse\n┌─name─────┬─is_rich─┬──age─┬─address.city.city─┬─address.city.state─┬─address.country─┐\n│ Batman   │ true    │   30 │ Chicago           │ IL                 │ USA             │\n│ Superman │ null    │ null │ New York          │ NY                 │ USA             │\n└──────────┴─────────┴──────┴───────────────────┴────────────────────┴─────────────────┘\n```\n\nThis is not yet published on `crates.io`, as it depends on a fork of the `klickhouse` crate exposing the `Block::read` method (see [this PR](https://github.com/Protryon/klickhouse/pull/80)).\n\n## Polars to Clickhouse\n\n### Rust\n\n```rust\nlet ch = klickhouse::Client::connect(\"localhost:9000\", Default::default()).await?;\n\nlet df: DataFrame = ...\n\n// Deduce table schema from the dataframe\nlet table = polarhouse::ClickhouseTable::from_polars_schema(table_name, df.schema(), [])?;\n\n// Create Clickhouse table corresponding to the Dataframe (optional)\ntable.create(\u0026ch, TableCreateOptions { primary_keys: \u0026[\"name\"] , ..Default::default() }).await?;\n\n// Insert dataframe contents into table\ntable.insert_df(df, \u0026ch).await?;\n```\n\n## Clickhouse to Polars\n\n### Rust\n\n```rust\nlet ch = klickhouse::Client::connect(\"localhost:9000\", Default::default()).await?;\n\n// Retrieve Clickhouse query results as a Dataframe.\nlet df: DataFrame = polarhouse::get_df_query(\n    klickhouse::SelectBuilder::new(table_name).select(\"*\"),\n    Default::default(),\n    \u0026ch,\n).await?;\n```\n\n### Python\n\n```python\nfrom polarhouse import Client\nclient = await Client.connect(\"localhost:9000\", caching=True)\ndf = await self.client.get_df_query(\"SELECT * from superheros\")\n\n```\n\n## Status\n\n\u003cp style=\"background:rgba(255,181,77,0.16);padding:0.75em;\"\u003e\nThis is for now only a proof of concept.\n\u003c/p\u003e\n\n## Alternative solutions\n\n- Use the `Arrow`, `ArrowStream` or `Parquet` [Clickhouse input/output formats](https://clickhouse.com/docs/en/interfaces/formats), which can be read and written from Polars.\n- Write an [Arrow Database Connectivity](https://arrow.apache.org/docs/format/ADBC.html) driver for Clickhouse, and use [Polars' ADBC support](https://docs.pola.rs/user-guide/io/database/).\n- Clickhouse to Polars: [ConnectorX](https://github.com/sfu-db/connector-x) (uses the [MySQL interface](https://clickhouse.com/docs/en/interfaces/mysql)).\n\n## Tests\n\n```\n$ docker run --network host --rm --name clickhouse clickhouse/clickhouse-server:latest\n$ cargo nextest run -r --nocapture\n```\n\n## Supported types\n\n- [x] Integers\n- [x] Floating points\n- [x] Strings\n- [x] Booleans\n- [x] Categorical (Polars) / Low cardinality (Clickhouse)\n- [x] Structs (Polars), which get flattened into Clickhouse, with fields names separated by `.`\n- [x] Nullables\n- [x] Lists (Polars) / Arrays (Clickhouse)\n- [x] UUIDs (mapped to Strings in Polars)\n- [ ] Arrays (Polars)\n- [ ] Tuples\n- [ ] DateTime\n- [ ] Time\n- [ ] Duration\n- [ ] ...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpg314%2Fpolarhouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcpg314%2Fpolarhouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpg314%2Fpolarhouse/lists"}