{"id":22096992,"url":"https://github.com/streamnative/pulsar-io-lakehouse","last_synced_at":"2025-08-25T21:05:30.691Z","repository":{"id":37101430,"uuid":"485802605","full_name":"streamnative/pulsar-io-lakehouse","owner":"streamnative","description":"pulsar lakehouse connector","archived":false,"fork":false,"pushed_at":"2025-03-24T04:40:45.000Z","size":1298,"stargazers_count":33,"open_issues_count":26,"forks_count":23,"subscribers_count":24,"default_branch":"master","last_synced_at":"2025-07-25T01:59:51.737Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamnative.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-04-26T13:39:54.000Z","updated_at":"2025-06-19T19:36:45.000Z","dependencies_parsed_at":"2023-11-06T02:28:45.783Z","dependency_job_id":"be2aa5f1-417f-4c66-800f-3b662c8bac09","html_url":"https://github.com/streamnative/pulsar-io-lakehouse","commit_stats":null,"previous_names":[],"tags_count":340,"template":false,"template_full_name":"streamnative/pulsar-io-template","purl":"pkg:github/streamnative/pulsar-io-lakehouse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-io-lakehouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-io-lakehouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-io-lakehouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-io-lakehouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamnative","download_url":"https://codeload.github.com/streamnative/pulsar-io-lakehouse/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-io-lakehouse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272131794,"owners_count":24878986,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-25T02:00:12.092Z","response_time":1107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T04:13:44.680Z","updated_at":"2025-08-25T21:05:30.663Z","avatar_url":"https://github.com/streamnative.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Pulsar IO :: Lakehouse Connector\n\nThe Lakehouse connector is a Pulsar IO connector for synchronizing data between Lakehouse (Delta Lake, Iceberg and Hudi) and Pulsar. It contains two types of connectors:\n\n***Lakehouse source connector***\nCurrently support `DeltaLake`\n\nThis source connector can capture data changes from delta lake through [DSR](https://github.com/delta-io/connectors/wiki/Delta-Standalone-Reader) and writes data to Pulsar topics.\n\n***Lakehouse sink connector***\nCurrently support `DeltaLake`, `Hudi` and `Iceberg`.\n\nThis sink connector can consume pulsar topic data and write into Lakehouse and users can use other big-data engines to process the delta lake table data further.\n\n\nCurrently, Lakehouse connector versions (x.y.z) are based on Pulsar versions (x.y.z).\n\n| Delta connector version | Pulsar version                                       | Doc |\n| :--------------- |:-----------------------------------------------------| :------------------------------|\n2.9.x| [2.9.2](https://github.com/apache/pulsar/tree/v2.9.2)| - [Lakehouse source connector](docs/lakehouse-source.md)\u003cbr\u003e\u003cbr\u003e- [Lakehouse sink connector](docs/lakehouse-sink.md)\n\n\nLakehouse Demos\n| Lakehouse | Demo                                      |\n| :--------------- |:-----------------------------------------------------|\n| Delta Lake| [Delta Lake Source and Sink Demo](docs/delta-lake-demo.md)  |\n\n\n## Project layout\n\nBelow are the sub folders and files of this project and their corresponding descriptions.\n\n  ```bash\n    ├── conf // stores configuration examples.\n    ├── docs // stores user guides.\n    ├── src // stores source codes.\n    │   ├── checkstyle // stores checkstyle configuration files.\n    │   ├── license // stores license headers. You can use `mvn license:format` to format the project with the stored license header.\n    │   │   └── ALv2\n    │   ├── main // stores all main source files.\n    │   │   └── java\n    │   ├── spotbugs // stores spotbugs configuration files.\n    │   └── test // stores all related tests.\n    │ \n  ```\n\n## Build delta connector\n\nRequirements:\n* Java [JDK 11](https://adoptium.net/?variant=openjdk11) or [JDK 8](https://adoptium.net/?variant=openjdk8)\n* Maven 3.6.1+\n\nCompile and install without cloud dependency:\n\n```bash\n$ mvn clean install -DskipTests\n```\n\nCompile and install with cloud dependency (Including `aws`, `gcs` and `azure`):\n\n```bash\n$ mvn clean install -P cloud -DskipTests\n```\n\nRun Unit Tests:\n\n```bash\n$ mvn test\n```\n\nRun Individual Unit Test:\n\n```bash\n$ mvn test -Dtest=unit-test-name (e.g: ParquetReaderTest)\n```\n\n## License\n\nLicensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fpulsar-io-lakehouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamnative%2Fpulsar-io-lakehouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fpulsar-io-lakehouse/lists"}