{"id":20046438,"url":"https://github.com/okdp/jupyterlab-docker","last_synced_at":"2025-05-05T09:31:40.895Z","repository":{"id":221418544,"uuid":"753008820","full_name":"OKDP/jupyterlab-docker","owner":"OKDP","description":"okdp jupyterlab docker images","archived":false,"fork":false,"pushed_at":"2024-05-17T14:35:01.000Z","size":2540,"stargazers_count":0,"open_issues_count":0,"forks_count":3,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-05-18T09:31:23.688Z","etag":null,"topics":["datascience","docker","jupyter","jupyter-notebook","jupyterhub","jupyterlab","k8s-spark","python","spark","spark-kubernetes","spark-python"],"latest_commit_sha":null,"homepage":"https://okdp.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OKDP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-05T09:40:28.000Z","updated_at":"2024-05-17T14:35:04.000Z","dependencies_parsed_at":"2024-02-07T21:40:04.737Z","dependency_job_id":"b1e22ea6-98a3-4c55-8a19-64822ef5eac7","html_url":"https://github.com/OKDP/jupyterlab-docker","commit_stats":null,"previous_names":["okdp/jupyterlab-docker"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fjupyterlab-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fjupyterlab-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fjupyterlab-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OKDP%2Fjupyterlab-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OKDP","download_url":"https://codeload.github.com/OKDP/jupyterlab-docker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224437683,"owners_count":17311041,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datascience","docker","jupyter","jupyter-notebook","jupyterhub","jupyterlab","k8s-spark","python","spark","spark-kubernetes","spark-python"],"created_at":"2024-11-13T11:23:53.086Z","updated_at":"2024-11-13T11:23:53.640Z","avatar_url":"https://github.com/OKDP.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OKDP Jupyter Images\n\n[![Build, test, tag, and push jupyter images](https://github.com/OKDP/jupyterlab-docker/actions/workflows/main.yml/badge.svg)](https://github.com/OKDP/jupyterlab-docker/actions/workflows/main.yml)\n\nOKDP jupyter docker images based on [jupyter docker-stacks](https://github.com/jupyter/docker-stacks) source dockerfiles. It includes (read only copy) [jupyter docker-stacks](https://github.com/jupyter/docker-stacks) repository as a [git-subtree](https://www.atlassian.com/git/tutorials/git-subtree) sub project.\n\nThe project leverages the features provided by [jupyter docker-stacks](https://github.com/jupyter/docker-stacks):\n- Build from the original [source docker files](docker-stacks/images)\n- Customize the images by using docker ```build-arg``` build arguments\n- Run the original [tests](docker-stacks/tests) at every pipeline trigger\n\nThe project provides an up to date jupyter lab images especially for pyspark.\n\n# Images build workflow\n## Build/Test\n\nThe [ci](.github/workflows/ci.yml) build pipeline contains 6 main reusable workflows:\n\n1. [build-base-images-template](.github/workflows/build-base-images-template.yml): docker-stacks-foundation, base-notebook, minimal-notebook, scipy-notebook\n2. [build-datascience-images-template](.github/workflows/build-datascience-images-template.yml): r-notebook, julia-notebook, tensorflow-notebook, pytorch-notebook\n3. [build-spark-images-template](.github/workflows/build-spark-images-template.yml): pyspark-notebook, all-spark-notebook\n4. [publish](.github/workflows/publish.yml): push the built images to the container registry (main branch only)\n5. [auto-rerun](.github/workflows/auto-rerun.yml): partially re-run jobs in case of failures (github runner issues/main branch only)\n6. [ci](.github/workflows/ci.yml): run ci pipeline at every contribution\n\n![build pipeline](doc/_images/build-pipeline.png)\n\nThe build is based on the [version compatibility matrix](.build/.versions.yml).\n\nThe [build-matrix](.build/.versions.yml#L42) section defines the components versions to build. It behaves like a filter of the parent [compatibility-matrix](.build/.versions.yml#L5) section to limit the versions combintations to build. The build process ensures only the compatible versions are built:\n\nFor example, the following build-matrix:\n\n```yaml\nbuild-matrix:\n  python_version: ['3.9', '3.10', '3.11']\n  spark_version: [3.2.4, 3.3.4, 3.4.2, 3.5.0]\n  java_version: [11, 17]\n  scala_version: [2.12]\n```\n\nWill build the following versions combinations in regards to [compatibility-matrix](.build/.versions.yml#5) section:\n- spark3.3.4-python3.10-java17-scala2.12\n- spark3.5.0-python3.11-java17-scala2.12\n- spark3.4.2-python3.11-java17-scala2.12\n- spark3.2.4-python3.9-java11-scala2.12\n\nBy default, if no filter is specified:\n\n```yaml\nbuild-matrix:\n```\n\nAll compatible versions combinations are built.\n\nFinally, all the images are tested against the original [tests](docker-stacks/tests) at every pipeline trigger\n\n## Publishing\n\nDevelopment images with tags ```-\u003cGIT-BRANCH\u003e-latest``` suffix (ex.: spark3.2.4-python3.9-java11-scala2.12-\u003cGIT-BRANCH\u003e-latest) are produced at every pipeline run regardless of the git branch (main or not).\n\nThe [official images](#tagging) are publiqhed to the [okdp quay.io registry](https://quay.io/organization/okdp):\n\n1. At every release, and,\n2. Periodically, every monday at 05H00 GMT\n\n\n## Tagging\n\nThe project builds the images with a long format tags. Each tag combines multiple compatible versions combinations.\n\nThere are multiple tags levels and the format to use is depending on your convenience in term of stability and reproducibility.\n\nHere are some examples:\n\n### scipy-notebook:\n- python-3.11-2024-02-06\n- python-3.11.7-2024-02-06\n- python-3.11.7-hub-4.0.2-lab-4.1.0\n- python-3.11.7-hub-4.0.2-lab-4.1.0-2024-02-06\n\n### datascience-notebook:\n- python-3.9-2024-02-06\n- python-3.9.18-2024-02-06\n- python-3.9.18-hub-4.0.2-lab-4.1.0\n- python-3.9.18-hub-4.0.2-lab-4.1.0-2024-02-06\n- python-3.9.18-r-4.3.2-julia-1.10.0-2024-02-06\n- python-3.9.18-r-4.3.2-julia-1.10.0-hub-4.0.2-lab-4.1.0\n- python-3.9.18-r-4.3.2-julia-1.10.0-hub-4.0.2-lab-4.1.0-2024-02-06\n\n### pyspark-notebook:\n- spark-3.5.0-python-3.11-java-17-scala-2.12\n- spark-3.5.0-python-3.11-java-17-scala-2.12-2024-02-06\n- spark-3.5.0-python-3.11.7-java-17.0.9-scala-2.12.18-hub-4.0.2-lab-4.1.0\n- spark-3.5.0-python-3.11.7-java-17.0.9-scala-2.12.18-hub-4.0.2-lab-4.1.0-2024-02-06\n- spark-3.5.0-python-3.11.7-r-4.3.2-java-17.0.9-scala-2.12.18-hub-4.0.2-lab-4.1.0\n- spark-3.5.0-python-3.11.7-r-4.3.2-java-17.0.9-scala-2.12.18-hub-4.0.2-lab-4.1.0-2024-02-06\n\nPlease, check the [okdp quay.io container registry](https://quay.io/organization/okdp) for more images and tags.\n\n# Running github actions\n## Official registry (quai.io) credentials\n\nCreate the following [secrets and configuration variables](https://docs.github.com/en/actions/learn-github-actions/variables#creating-configuration-variables-for-a-repository) when running with your own github account or organization:\n\n| Variable               | Type                    | Default  | Description                                 |\n| -----------------------|-------------------------| ---------| ------------------------------------------- |\n| `REGISTRY`             | Configuration variable  | quay.io  | Container registry                          |\n| `REGISTRY_USERNAME`    | Secret variable         |          | Container registry username                 |\n| `REGISTRY_ROBOT_TOKEN` | Secret variable         |          | Container registry password or access token `(Scopes: write:packages/delete:packages)` |\n\n## Running locally with act\n\n[Act](https://github.com/nektos/act) can be used to build and test locally.\n\nHere is an example command:\n\n```shell\n$ act  --container-architecture linux/amd64  \\\n       -W .github/workflows/ci.yml \\\n       --env ACT_SKIP_TESTS=\u003ctrue|false\u003e \\\n       --secret GITHUB_TOKEN=\u003cGITHUB_TOKEN\u003e \\\n       --rm\n```\n\nset the option ```--container-architecture linux/amd64``` if you are running locally with Apple's M1/M2 chips.\n\nFor more information:\n\n```shell\n$ act  --help\n```\n\n# OKDP custom extensions\n\n1. [Tagging extension](.build/python/okdp/extension/tagging) is based on the original [jupyter docker-stacks](docker-stacks/tagging) source files \n2. [Patchs](.build/python/okdp/patch/README.md) patchs the original [jupyter docker-stacks](docker-stacks/tests) in order to run the tests\n3. [Version compatibility matrix](.build/python/okdp/extension/matrix) to generate all the compatible versions combintations for pyspark\n4. [Unit tests](.build/python/tests) in order to test okdp extension at every pipeline run","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokdp%2Fjupyterlab-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fokdp%2Fjupyterlab-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokdp%2Fjupyterlab-docker/lists"}