{"id":15208883,"url":"https://github.com/okdp/spark-images","last_synced_at":"2026-02-05T19:01:09.468Z","repository":{"id":231493246,"uuid":"777625019","full_name":"OKDP/spark-images","owner":"OKDP","description":"Collection of Apache Spark docker images for OKDP","archived":false,"fork":false,"pushed_at":"2026-02-04T18:10:16.000Z","size":147,"stargazers_count":2,"open_issues_count":6,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-02-05T05:26:13.949Z","etag":null,"topics":["apache-spark","big-data","docker","k8s-spark","kubernetes","spark-kubernetes","spark-python","spark-r","spark-sql"],"latest_commit_sha":null,"homepage":"https://okdp.io","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OKDP.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-03-26T07:55:47.000Z","updated_at":"2026-01-22T16:43:13.000Z","dependencies_parsed_at":"2024-06-10T16:25:03.385Z","dependency_job_id":"8e8e49a7-17c2-4674-81da-ae3177425bfb","html_url":"https://github.com/OKDP/spark-images","commit_stats":{"total_commits":22,"total_committers":3,"mean_commits":7.333333333333333,"dds":"0.13636363636363635","last_synced_commit":"fbd0388c847ee081946678cdc617b3283330aced"},"previous_names":["okdp/spark-images"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/OKDP/spark-images","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fspark-images","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fspark-images/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fspark-images/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fspark-images/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OKDP","download_url":"https://codeload.github.com/OKDP/spark-images/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fspark-images/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29130094,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-05T18:55:47.139Z","status":"ssl_error","status_checked_at":"2026-02-05T18:55:04.010Z","response_time":65,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","big-data","docker","k8s-spark","kubernetes","spark-kubernetes","spark-python","spark-r","spark-sql"],"created_at":"2024-09-28T07:03:03.073Z","updated_at":"2026-02-05T19:01:09.442Z","avatar_url":"https://github.com/OKDP.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![ci](https://github.com/okdp/spark-images/actions/workflows/ci.yml/badge.svg)](https://github.com/okdp/spark-images/actions/workflows/ci.yml)\n[![Release](https://img.shields.io/github/v/release/okdp/spark-images)](https://github.com/okdp/spark-images/releases/latest)\n[![License Apache2](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0)\n\n\nCollection of [Apache Spark](https://spark.apache.org/) docker images for [OKDP Platform](https://okdp.io/).\n\nCurrently, the images are built from the [Apache Spark project distribution](https://archive.apache.org/dist/spark) and the requirement may evolve to produce them from the [source code](https://github.com/apache/spark).\n\nThe image relashionship is described by the following diagram:\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"docs/images/spark-images.drawio.svg\"\u003e\n\u003c/p\u003e\n\n\n\n\n| Image | Description |\n|:---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `JRE` | The JRE LTS base image supported by Apache Spark depending on the version. This includes Java 11/17/21. Please, check the [reference versions](.build/reference-versions.yml) or [Apache Spark website](https://spark.apache.org/docs/latest/) for more information. |\n| `spark-base` | The Apache Spark base image with official spark binaries (scala/java) and without OKDP extensions. |\n| `spark` | The Apache Spark image with official spark binaries (scala/java) and OKDP extensions. | \n| `spark-py` | The Apache Spark image with official spark binaries (scala/java), OKDP extensions and python support. | \n| `spark-r` | The Apache Spark image with official spark binaries (scala/java), OKDP extensions and R support. | \n\n# Tagging\n\nThe project builds the images with a long format tags. Each tag combines multiple compatible versions combinations.\n\nThere are multiple tags levels and the format to use depends on your convenience in term of stability and reproducibility.\n\nThe images are pushed to [quay.io/okdp](https://quay.io/organization/okdp) repository with the following [tags](.build/images.yml):\n\n| Images | Tags |\n|:--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| spark-base, spark | spark-\u003cSPARK_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e\u003c/br\u003e\u003c/br\u003espark-\u003cSPARK_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e-\u003cBUILD_DATE\u003e\u003c/br\u003e\u003c/br\u003espark-\u003cSPARK_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e-\u003cRELEASE_VERSION\u003e\u003c/br\u003e\u003c/br\u003espark-\u003cSPARK_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e-\u003cBUILD_DATE\u003e-\u003cRELEASE_VERSION\u003e |\n| spark-py | spark-\u003cSPARK_VERSION\u003e-python-\u003cPYTHON_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e\u003c/br\u003e\u003c/br\u003espark-\u003cSPARK_VERSION\u003e-python-\u003cPYTHON_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e-\u003cBUILD_DATE\u003e\u003c/br\u003e\u003c/br\u003espark-\u003cSPARK_VERSION\u003e-python-\u003cPYTHON_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e-\u003cRELEASE_VERSION\u003e\u003c/br\u003e\u003c/br\u003espark-\u003cSPARK_VERSION\u003e-python-\u003cPYTHON_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e-\u003cBUILD_DATE\u003e-\u003cRELEASE_VERSION\u003e |\n| spark-r | spark-\u003cSPARK_VERSION\u003e-r-\u003cR_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e\u003c/br\u003e\u003c/br\u003e spark-\u003cSPARK_VERSION\u003e-r-\u003cR_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e-\u003cBUILD_DATE\u003e\u003c/br\u003e\u003c/br\u003espark-\u003cSPARK_VERSION\u003e-r-\u003cR_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e-\u003cRELEASE_VERSION\u003e\u003c/br\u003e\u003c/br\u003espark-\u003cSPARK_VERSION\u003e-r-\u003cR_VERSION\u003e-scala-\u003cSCALA_VERSION\u003e-java-\u003cJAVA_VERSION\u003e-\u003cBUILD_DATE\u003e-\u003cRELEASE_VERSION\u003e |\n\n\u003e [!NOTE]\n\u003e 1. `\u003cRELEASE_VERSION\u003e` corresponds to the Github [release version](https://github.com/okdp/spark-images/releases) or [git tag](https://github.com/okdp/spark-images/tags) without the leading `v`.\n\u003e Ex.: 1.0.0\n\u003e \n\u003e 2. `\u003cBUILD_DATE\u003e` corresponds to the images build date with the `YYYY-MM-DD` format. The latest release tag is rebuilt every week to ensure the OS image is up to date against the latest security updates.\n\u003e \n\u003e You may need to switch to the latest release version if your are using the long form tag image with a `\u003cRELEASE_VERSION\u003e`. Please, check the [changelog](https://github.com/okdp/spark-images/releases) to see the notable impacts.\n\u003e\n\u003e An example of `py-spark` image with a long form tag including `spark/java/scala/python` compatible versions and a `\u003cBUILD_DATE\u003e` with a `\u003cRELEASE_VERSION\u003e` is: \n\u003e \n\u003e `quay.io/okdp/spark-py:spark-3.5.1-python-3.11-scala-2.13-java-17-2024-04-04-1.0.0`.\n\u003e\n\u003e The corresponding changelog is [releases/tag/v1.0.0](https://github.com/okdp/spark-images/releases/tag/v1.0.0).\n\u003e\n\u003e 3. You can also use the latest tag without `\u003cBUILD_DATE\u003e` and `\u003cRELEASE_VERSION\u003e` which is always up to date with the latest security updates. \n\u003e \n\u003e An example of `py-spark` image with the latest tag is: `quay.io/okdp/spark-py:spark-3.5.1-python-3.11-scala-2.13-java-17`\n\u003e\n\n# Patching and Dependency Management System\n\nThis project automatically applies security fixes and dependency updates to Spark source code during builds using a patch and pombump system.\n\n**Key Features:**\n- āœ… **Source code patches** for critical security fixes\n- āœ… **Automated dependency updates** via pombump\n- āœ… **Version-specific configurations** \n- āœ… **Build optimization** and compatibility\n\n## How It Works\n\n### Configuration-Based Processing\n\nThe system uses `.build/pre-build-patch-pombump.yml` to determine which Spark versions should receive patches and/or dependency updates:\n\n```yaml\ncontrols:\n - spark_version: \"3.4.1\"\n python_version: \"3.11\"\n java_version: \"17\"\n hadoop_version: \"3.3.6\"\n patch_files: [] # No source patches needed, but pombump will run\n```\n\n### Processing Logic\n\n**If a Spark version is present in the configuration file:**\n\n1. **Source Download**: The system downloads the Spark source code\n2. **Patch Application**: Applies any source code patches (if `patch_files` is not empty)\n3. **Dependency Updates**: Runs pombump to update Maven dependencies to secure versions\n4. **Build Context**: Uses the patched/updated source for Docker builds\n\n**If a Spark version is not in the configuration:**\n- Uses original Spark distribution without modifications\n\n### Pombump Dependency Management\n\nFor versions in the configuration, pombump automatically updates dependencies to secure versions:\n\n```yaml\n# From pombump-properties.yaml\n- property: log4j.version\n value: \"2.25.0\" # Updates to secure Log4j version\n- property: fasterxml.jackson.version \n value: \"2.14.2\" # Updates Jackson for security\n```\n\nThis ensures all builds use the latest secure dependency versions, even without source code changes.\n\nšŸ“– **[Read the full patching documentation ā†’](PATCH-POMBUMP.md)**\n\n**Quick Reference:**\n- Patch configuration: [`.build/pre-build-patch-pombump.yml`](.build/pre-build-patch-pombump.yml)\n- Patch files: [`spark-base/spark-X.Y/`](spark-base/)\n- Application logic: [`.github/actions/patch-pombump/`](.github/actions/patch-pombump/)\n\n# Alternatives\n\n- [Official images](https://github.com/apache/spark-docker)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokdp%2Fspark-images","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fokdp%2Fspark-images","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokdp%2Fspark-images/lists"}