{"id":27645070,"url":"https://github.com/justkode/spark-on-k8s-example","last_synced_at":"2026-05-12T07:40:09.263Z","repository":{"id":235719761,"uuid":"618511464","full_name":"JustKode/spark-on-k8s-example","owner":"JustKode","description":"Running Spark On Kubernetes Example","archived":false,"fork":false,"pushed_at":"2023-03-27T17:05:06.000Z","size":3,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-24T00:59:44.235Z","etag":null,"topics":["kubernetes","pyspark","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JustKode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-03-24T16:17:02.000Z","updated_at":"2023-03-27T14:43:05.000Z","dependencies_parsed_at":"2024-04-24T12:14:34.814Z","dependency_job_id":null,"html_url":"https://github.com/JustKode/spark-on-k8s-example","commit_stats":null,"previous_names":["justkode/spark-on-k8s-example"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustKode%2Fspark-on-k8s-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustKode%2Fspark-on-k8s-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustKode%2Fspark-on-k8s-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustKode%2Fspark-on-k8s-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JustKode","download_url":"https://codeload.github.com/JustKode/spark-on-k8s-example/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250540965,"owners_count":21447427,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kubernetes","pyspark","spark"],"created_at":"2025-04-24T00:59:46.750Z","updated_at":"2026-05-12T07:40:09.215Z","avatar_url":"https://github.com/JustKode.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Spark On Kubernetes Example\nSpark On Kubernetes Example. Need to set up the programs below.\n\n- Docker\n- Minikube\n- kubectl\n- Spark 3.0 \u003e=\n\n### Pyspark Image build \u0026 push\nBefore you run spark application on kubernetes, you need to build a spark image that can run in kubernetes cluster.\n\nFor this purpose, you can use `./bin/docker-image-tool.sh`. Options are described below.\n\n- `-f file`: (Optional) Dockerfile to build for JVM based Jobs. By default builds the Dockerfile shipped with Spark. For Java 17, use `-f kubernetes/dockerfiles/spark/Dockerfile.java17`\n- `-p file`: (Optional) Dockerfile to build for PySpark Jobs. Builds Python dependencies and ships with Spark. Skips building PySpark docker image if not specified.\n- `-R file`: (Optional) Dockerfile to build for SparkR Jobs. Builds R dependencies and ships with Spark. Skips building SparkR docker image if not specified.\n- `-r repo`: Repository address.\n- `-t tag`: Tag to apply to the built image, or to identify the image to be pushed.\n\nFirst, you need to build base image of spark\n\n```bash\n$ cd $SPARK_DIR  # spark directory\n$ eval $(minikube docker-env)\n$ ./bin/docker-image-tool.sh -r k8s -t 1.0 -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build\n```\n\nSecond, build application image from base image of spark\n\n```bash\n$ cd \u003cthis repository path\u003e/python\n$ eval $(minikube docker-env)\n$ docker build -t pyspark-on-k8s:1.0 .\n$ minikube image load pyspark-on-k8s:1.0\n```\n\nYou can see all builded docker image by `docker image ls`\n\n```bash\n$ docker image ls\nREPOSITORY                           TAG       IMAGE ID       CREATED         SIZE\npyspark-on-k8s                       1.0       00a4af077a09   About a minute ago   938MB\nk8s/spark-py                         1.0       985cf805549a   13 days ago     938MB\nk8s/spark                            1.0       bd8ba88688d4   13 days ago     601MB\n...\n```\n\n### How To Run Spark Application\nBefore running spark application, you need to make **k8s service account and k8s clusterrolebinding.** Because Spark on k8s works in a way that the **driver pod calls the executor pod**, the driver pod must be authorized to edit the pod via clusterrolebinding.\n\n```bash\n$ kubectl create serviceaccount spark\n$ kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default\n```\n\nBy Spark Submit, you can **run spark application on k8s cluster.**\n\n```bash\n$ kubectl proxy\nStarting to serve on 127.0.0.1:8001\n\n# In another terminal\n$ kubectl create namespace spark-job\n\n# rdd_example\n$ ./bin/spark-submit \\\n    --master k8s://http://127.0.0.1:8001 \\\n    --deploy-mode cluster \\\n    --name rdd-example \\\n    --class org.apache.spark.examples.SparkPi \\\n    --conf spark.kubernetes.container.image=pyspark-on-k8s:1.0 \\\n    --conf spark.kubernetes.driver.pod.name=rdd-example-pod \\\n    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \\\n    --verbose \\\n    \"local:///python/rdd_example.py\"\n\n$ kubectl logs rdd-example-pod  # log check\n\n# dataframe_example\n$ ./bin/spark-submit \\\n    --master k8s://http://127.0.0.1:8001 \\\n    --deploy-mode cluster \\\n    --name dataframe-example \\\n    --class org.apache.spark.examples.SparkPi \\\n    --conf spark.kubernetes.container.image=pyspark-on-k8s:1.0 \\\n    --conf spark.kubernetes.driver.pod.name=dataframe-example-pod \\\n    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \\\n    --verbose \\\n    \"local:///python/dataframe_example.py\"\n\n$ kubectl logs dataframe-example-pod  # log check\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustkode%2Fspark-on-k8s-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjustkode%2Fspark-on-k8s-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustkode%2Fspark-on-k8s-example/lists"}