{"id":13696163,"url":"https://github.com/ClickHouse/spark-clickhouse-connector","last_synced_at":"2025-05-03T14:30:59.790Z","repository":{"id":36966217,"uuid":"369723273","full_name":"ClickHouse/spark-clickhouse-connector","owner":"ClickHouse","description":"Spark ClickHouse Connector build on DataSourceV2 API","archived":false,"fork":false,"pushed_at":"2025-02-01T12:24:53.000Z","size":2519,"stargazers_count":196,"open_issues_count":47,"forks_count":71,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-04-12T14:56:39.493Z","etag":null,"topics":["arrow","clickhouse","datasourcev2","grpc","http","spark"],"latest_commit_sha":null,"homepage":"https://clickhouse.com/docs/en/integrations/apache-spark","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ClickHouse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-05-22T05:26:16.000Z","updated_at":"2025-03-18T07:06:04.000Z","dependencies_parsed_at":"2024-02-21T06:25:32.023Z","dependency_job_id":"afb494f4-1f14-4bfb-8ed6-07f1ad5c9417","html_url":"https://github.com/ClickHouse/spark-clickhouse-connector","commit_stats":null,"previous_names":["clickhouse/spark-clickhouse-connector","housepower/spark-clickhouse-connector"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fspark-clickhouse-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fspark-clickhouse-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fspark-clickhouse-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fspark-clickhouse-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ClickHouse","download_url":"https://codeload.github.com/ClickHouse/spark-clickhouse-connector/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252203166,"owners_count":21710901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arrow","clickhouse","datasourcev2","grpc","http","spark"],"created_at":"2024-08-02T18:00:36.809Z","updated_at":"2025-05-03T14:30:57.187Z","avatar_url":"https://github.com/ClickHouse.png","language":"Scala","funding_links":[],"categories":["大数据","Language bindings"],"sub_categories":["Scala"],"readme":"Spark ClickHouse Connector\n===\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.clickhouse.spark/clickhouse-spark-runtime-3.5_2.12/badge.svg)](https://search.maven.org/search?q=clickhouse-spark-runtime)\n[![License](https://img.shields.io/github/license/clickhouse/spark-clickhouse-connector)](https://github.com/ClickHouse/spark-clickhouse-connector/blob/main/LICENSE)\n\nBuild on Apache Spark DataSourceV2 API.\n\n![Overview](docs/imgs/scc_overview.drawio.png)\n\n## Usage\n\nSee the [documentation](https://clickhouse.com/docs/en/integrations/apache-spark) for how to use this connector.\n\n## Requirements\n\n- Java 8 or 17\n- Scala 2.12 or 2.13\n- Apache Spark 3.3 or 3.4 or 3.5\n\nNotes:\n1. As of 0.5.0, this connector switches from ClickHouse raw gRPC Client to \n   [ClickHouse Official Java Client](https://github.com/ClickHouse/clickhouse-jdbc), which brings HTTP protocol support,\n   extends the range of supported versions of ClickHouse Server.\n2. Since 0.6.0, HTTP becomes the default protocol.\n3. Since 0.7.0, gRPC is deprecated and not recommended, it may be removed in the future.\n4. Since 0.8.0, gRPC is removed.\n5. Since 0.8.0, the project groupId is rename from `com.github.housepower` to `com.clickhouse.spark`, and\n   the class `xenon.clickhouse.ClickHouseCatalog` is renamed to `com.clickhouse.spark.ClickHouseCatalog`\n\n## Compatible Matrix\n\n| Version | Compatible Spark Versions | ClickHouse JDBC version |\n|---------|---------------------------|-------------------------|\n| main    | Spark 3.3, 3.4, 3.5       | 0.6.3                   |\n| 0.8.0   | Spark 3.3, 3.4, 3.5       | 0.6.3                   |\n| 0.7.3   | Spark 3.3, 3.4            | 0.4.6                   |\n| 0.6.0   | Spark 3.3                 | 0.3.2-patch11           |\n| 0.5.0   | Spark 3.2, 3.3            | 0.3.2-patch11           |\n| 0.4.0   | Spark 3.2, 3.3            | Not depend on           |\n| 0.3.0   | Spark 3.2, 3.3            | Not depend on           |\n| 0.2.1   | Spark 3.2                 | Not depend on           |\n| 0.1.2   | Spark 3.2                 | Not depend on           |\n\n## Build\n\nBuild w/o test\n\n`./gradlew clean build -x test`\n\n## Test\n\nThe project leverage [Testcontainers](https://www.testcontainers.org/) and [Docker Compose](https://docs.docker.com/compose/)\nto do integration tests, you should install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/)\nbefore running test, and check more details on [Testcontainers document](https://www.testcontainers.org/) if you'd\nlike to run test with remote Docker daemon.\n\nRun all test\n\n`./gradlew clean test`\n\nRun all test w/ Spark 3.3 and Scala 2.13\n\n`./gradlew clean test -Dspark_binary_version=3.3 -Dscala_binary_version=2.13`\n\nRun single test\n\n`./gradlew test --tests=ConvertDistToLocalWriteSuite`\n\nTest against custom ClickHouse image\n\n`CLICKHOUSE_IMAGE=custom-org/clickhouse-server:custom-tag ./gradlew test`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FClickHouse%2Fspark-clickhouse-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FClickHouse%2Fspark-clickhouse-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FClickHouse%2Fspark-clickhouse-connector/lists"}