{"id":22951622,"url":"https://github.com/unexist/showcase-hadoop-cdc-quarkus","last_synced_at":"2026-05-01T09:31:58.985Z","repository":{"id":80479642,"uuid":"575724279","full_name":"unexist/showcase-hadoop-cdc-quarkus","owner":"unexist","description":"Showcase for Hadoop with CDC on Quarkus [MIRROR]","archived":false,"fork":false,"pushed_at":"2024-03-28T10:40:58.000Z","size":26192,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-01T22:49:31.348Z","etag":null,"topics":["cdc","debezium","hadoop","iceberg","quarkus"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/unexist.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-08T06:41:32.000Z","updated_at":"2022-12-08T07:28:09.000Z","dependencies_parsed_at":"2023-12-29T17:29:43.949Z","dependency_job_id":"2aa213b7-7cb0-4ea5-99df-9ba743f45e50","html_url":"https://github.com/unexist/showcase-hadoop-cdc-quarkus","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/unexist/showcase-hadoop-cdc-quarkus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unexist%2Fshowcase-hadoop-cdc-quarkus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unexist%2Fshowcase-hadoop-cdc-quarkus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unexist%2Fshowcase-hadoop-cdc-quarkus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unexist%2Fshowcase-hadoop-cdc-quarkus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/unexist","download_url":"https://codeload.github.com/unexist/showcase-hadoop-cdc-quarkus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unexist%2Fshowcase-hadoop-cdc-quarkus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32492142,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","debezium","hadoop","iceberg","quarkus"],"created_at":"2024-12-14T15:17:07.253Z","updated_at":"2026-05-01T09:31:58.968Z","avatar_url":"https://github.com/unexist.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"= Showcase for Hadoop with CDC on Quarkus\n\nThis project holds a showcase for Hadoop with CDC on Quarkus.\n\n== Make targets\n\nFollowing make targets exist in the subfolder **podman**:\n\n=== Container and co\n\n- **pd-machine-create** - Create a suitable Qemu machine\n- **pd-pod-create** - Create the pod with port mappings\n- **pd-pod-rm** - Remove pod\n- **pd-pod-recreate** - Remove and recreate the pod\n- **pd-build** - Build all images\n- **pd-init** - Create machine, pod and build all images\n- **pd-start** - Start all container\n\nFollowing make targets are available **here**:\n\n=== Tools\n\n- **todo** - Create todo entry via curl\n- **list** - List todo entries via curl\n- **kat-listen** - Listen for Kafka messages\n- **kat-send** - Send Kafka message\n- **psql** - Use psql CLI to connect to Postgres\n\n=== Hive\n\n- **beeline** - Start beeline CLI and connect to Hive\n- **beeline-hive-select** - Select data from **hive_todos**\n- **beeline-debezium-select** - Select data from **debezium_todos**\n- **beeline-spark-select** - Select data from **spark_messages** and **spark_todos**\n\n=== Spark\n\n- **spark-shell** - Start Spark shell and connect to Spark\n- **spark-beeline** - Start Spark Beeline and connect to Spark\n\n- **data-init** - Init all Hive data\n- **copy** - Copy the Scala jar into the Hadoop container\n\n=== Browser\n\n- **open-namenode** - Open the namenode in a browser\n- **open-datanode** - Open the datanode in a browser\n- **open-spark-master** - Open the Spark master in a browser\n- **open-spark-slave** - Open the Spark slave in a browser\n- **open-spark-shell** - Open the Spark shell in a browser\n- **open-resourcemanager** - Open the Resoucemanager in a browser\n- **open-debezium** - Open the Debezium in a browser\n- **open-app** - Open the Quarkus Dev tools in a browser\n\n== How to use\n\n=== Initial setup\n\n. Create podman machine: `make -C podman pd-machine-create`\n. Start podman machine: `make -C podman pd-machine-start`\n. Create pod: `make -C podman pd-pod-create`\n. Build all containers: `make -C podman pd-build`\n. Start all containers: `make -C podman pd-start`\n\n=== Start everything on host\n\n. Init Hive tables: `make init`\n. Compile scala jar: `make scala`\n. SSH into Hadoop pod: `make ssh`\n\n=== Run everything on ssh\n\n. Copy scala jar to Hadoop: `make copy`\n. Init Spark tables: `make init`\n. Run scala jar: `make run`\n. Create todo via curl: `make todo`\n\n== Problems\n\n=== Dockerfile\n\nApparently, the datanodes use components like C-libraries which I couldn't get NOT to dump core\ndump with Alpine/Musl.\n\n=== Podman\n\nStarting with Podman 4.4.1 they dropped the default privileges for chroot, which led to following\nproblems on connection:\n\n```\nssh: Connection closed by 127.0.0.1 port 22\nsshd: chroot(\"/run/sshd\"): Operation not permitted [preauth]\n```\n\n- https://github.com/containers/podman/releases/tag/v4.4.1\n- https://forum.gitlab.com/t/failure-on-ssh-push-pull/82244\n\n=== Scala\n\n```text\njava.lang.NoSuchMethodError: 'scala.collection.immutable.ArraySeq scala.runtime.ScalaRunTime$.wrapRefArray(java.lang.Object[])'\n```\n\n```text\nCaused by: java.lang.ClassNotFoundException: scala.$less$colon$less\n```\n\nMake sure the Scala version of the jars/dependencies match the Scala version of Spark.\n\nThis can be easily checked with:\n\n```shell\nmvn dependency:tree\n```\n\n== Hive\n\nThe jdbc connection string for either anonymous or *hduser* for Hive is following:\n\n[source,txt]\n----\njdbc:hive2://localhost:10000/default\n----\n\nAnd adding the external Debezium table works:\n\n[source,sql]\n----\nadd jar /home/hduser/hive/lib/iceberg-hive-runtime-1.1.0.jar;\ncreate external table debezium stored by 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' location 'hdfs://localhost:9000/warehouse/debeziumevents/debeziumcdc_showcase_public_todos' TBLPROPERTIES ('iceberg.catalog'='location_based_table')\"\n----\n\n== Spark\n\nSpark executors use submitted values for JAVA_HOME:\n\nhttps://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_Set_Java_Home_Howto.md\n\n== Hadoop\n\nHadoop includes a Jakarta enabled version of Jetty, but still in v3.3.6 many of the servlets\nimplement interfaces of `javax.servlets.*` and this doesn't work in a Jakarta project.\n\n[source,text]\n----\njava.lang.RuntimeException: java.lang.NoSuchMethodError: 'void org.eclipse.jetty.servlet.ServletHolder.\u003cinit\u003e(javax.servlet.Servlet)'\n----\n\n== Links\n\n=== Hadoop\n\n- https://medium.com/analytics-vidhya/hadoop-single-node-cluster-on-docker-e88c3d09a256\n- https://github.com/rancavil/hadoop-single-node-cluster\n- https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html\n- https://www.edureka.co/blog/install-hadoop-single-node-hadoop-cluster\n- https://stackoverflow.com/questions/41266403/how-to-access-hadoop-web-ui-in-linux\n- https://www.digitalocean.com/community/tutorials/how-to-install-hadoop-in-stand-alone-mode-on-ubuntu-20-04\n- https://www.ibm.com/docs/el/db2-big-sql/5.0?topic=applications-impersonation-in-big-sql\n\n==== MapReduce\n\n- https://m-mansur-ashraf.blogspot.com/2013/02/testing-mapreduce-with-mrunit.html\n- https://cwiki.apache.org/confluence/display/MRUNIT/MRUnit+Tutorial\n\n==== Config defaults\n\n- https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml\n- https://hadoop.apache.org/docs/r2.7.1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml\n\n=== Debezium\n\n- https://hub.docker.com/r/debezium/server\n- https://debezium.io/documentation/reference/stable/operations/debezium-server.html\n- https://github.com/memiiso/debezium-server-iceberg\n- https://debezium.io/blog/2021/10/20/using-debezium-create-data-lake-with-apache-iceberg/\n- https://iceberg.apache.org/\n- https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#FsURIvsHTTP_URL\n- https://stackoverflow.com/questions/59978213/debezium-could-not-access-file-decoderbufs-using-postgres-11-with-default-plug\n- https://debezium.io/documentation/reference/stable/development/engine.html#database-history-properties\n\n=== Iceberg\n\n- https://iceberg.apache.org/docs/latest/hive/#create-external-table-overlaying-an-existing-iceberg-table\n- https://iceberg.apache.org/releases/#downloads\n- https://iceberg.apache.org/docs/latest/java-api-quickstart/\n- https://tabular.io/blog/java-api-part-3/\n\n=== Spark\n\n- https://www.dremio.com/blog/introduction-to-apache-iceberg-using-spark/\n- https://spark.apache.org/docs/latest/sql-getting-started.html\n- https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html\n- https://sparkbyexamples.com/apache-hive/how-to-connect-spark-to-remote-hive/\n- https://codait.github.io/spark-bench/\n- https://sparkbyexamples.com/spark/spark-split-dataframe-column-into-multiple-columns/\n- https://www.adaltas.com/en/2019/04/18/spark-streaming-data-pipelines-with-structured-streaming/\n\n=== Scala\n\n- https://davidb.github.io/scala-maven-plugin/usage.html\n- https://www.alibabacloud.com/help/en/e-mapreduce/latest/use-spark-to-write-data-to-an-iceberg-table-in-streaming-mode\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funexist%2Fshowcase-hadoop-cdc-quarkus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funexist%2Fshowcase-hadoop-cdc-quarkus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funexist%2Fshowcase-hadoop-cdc-quarkus/lists"}