{"id":22423855,"url":"https://github.com/statcan/aaw-kubeflow-containers","last_synced_at":"2025-08-01T07:32:35.016Z","repository":{"id":37020625,"uuid":"253309273","full_name":"StatCan/aaw-kubeflow-containers","owner":"StatCan","description":"Containers built to be used with Kubeflow for Data Science","archived":false,"fork":false,"pushed_at":"2024-10-29T18:06:32.000Z","size":2617,"stargazers_count":24,"open_issues_count":32,"forks_count":21,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-10-29T18:41:29.365Z","etag":null,"topics":["aaw","containers","daaas","kubeflow","kubernetes"],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StatCan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-05T18:55:33.000Z","updated_at":"2024-10-28T17:57:51.000Z","dependencies_parsed_at":"2023-09-21T22:51:48.714Z","dependency_job_id":"3e33e5e0-8af1-4fe7-aacb-89004afa3901","html_url":"https://github.com/StatCan/aaw-kubeflow-containers","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StatCan%2Faaw-kubeflow-containers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StatCan%2Faaw-kubeflow-containers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StatCan%2Faaw-kubeflow-containers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StatCan%2Faaw-kubeflow-containers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StatCan","download_url":"https://codeload.github.com/StatCan/aaw-kubeflow-containers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228348339,"owners_count":17905899,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aaw","containers","daaas","kubeflow","kubernetes"],"created_at":"2024-12-05T18:13:16.976Z","updated_at":"2025-08-01T07:32:35.004Z","avatar_url":"https://github.com/StatCan.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aaw-kubeflow-containers \u003c!-- omit in toc --\u003e\n\nContainer images to be used with kubeflow on the AAW platform for Data Science \u0026 other workloads.\n\n## Table of Contents \u003c!-- omit in toc --\u003e\n\n- [Introduction](#introduction)\n- [List of maintained images in this github repository](#list-of-maintained-images-in-this-github-repository)\n- [Using](#using)\n  - [Building and Tagging Docker Images](#building-and-tagging-docker-images)\n  - [Pulling and Pushing Docker Images](#pulling-and-pushing-docker-images)\n  - [Testing images](#testing-images)\n    - [Running and Connecting to Images Locally/Interactively](#running-and-connecting-to-images-locallyinteractively)\n    - [Automated Testing](#automated-testing)\n- [General Development Workflow](#general-development-workflow)\n  - [Running an AAW Container Locally](#running-an-aaw-container-locally)\n  - [Testing Locally](#testing-locally)\n  - [Testing On-Platform](#testing-on-platform)\n  - [Overview of images](#overview-of-images)\n  - [Adding new software](#adding-new-software)\n  - [Adding new Images](#adding-new-images)\n  - [Adding new Stages](#adding-new-stages)\n  - [Modifying and Testing CI](#modifying-and-testing-ci)\n- [Other Development Notes](#other-development-notes)\n  - [Github CI](#github-ci)\n  - [The `latest` and `v1` tags for the master branch](#the-latest-and-v1-tags-for-the-master-branch)\n  - [Set User File Permissions](#set-user-file-permissions)\n  - [Troubleshooting](#troubleshooting)\n- [Structure](#structure)\n\n## Introduction\n\nOur Container images are based on the community driven [jupyter/docker-stacks](https://github.com/jupyter/docker-stacks).\nWe chose those images because they are continuously updated and install the most common utilities.\nThis enables us to focus only on the additional toolsets that we require to enable our data scientists.\nThese customized images are maintained by the AAW team and are the default images available on the kubeflow UI.\n\nThis is different from the [aaw-contrib-containers](https://github.com/StatCan/aaw-contrib-containers) as those images are built by AAW user-base.\nThese are often created when a user's workload is more specific and our generic images are not suitable for them.\nThose images can be used via the `custom-image` feature in kubeflow and do not populate the default images drop-down.\nAdditionally, the AAW team is not responsible for maintaining those images.\n\n## List of maintained images in this github repository\n\n| Image Name                         | Notes                                                                                                                                           | Extra Installations  |\n| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |\n| jupyterlab-cpu                     | The base experience. A jupyterlab notebook with various installations.                                                                          | VsCode, R, Julia     |\n| jupyterlab-tensorflow              | For users looking to leverage a GPU machine. Comes installed with tensorflow                                                                    | tensorflow-gpu       |\n| remote-desktop                     | For users looking to have a desktop-like experience.                                                                                            | Open M++, QGIS       |\n| rstudio                            | For users looking to have a rstudio tuned experience.                                                                                           |                      |\n| sas                                | Similar to our jupyterlab-cpu image, except with SAS. This is only available to Statistics Canada employees as that is what our license allows. |                      |\n\n## Using\n\n### Building and Tagging Docker Images\n\nUse `make build/IMAGENAME` to build a `Dockerfile`.\nThis by default generates images with:\n\n- `repo=k8scc01covidacr.azurecr.io`\n- `tag=BRANCH_NAME`\n  For example: `k8scc01covidacr.azurecr.io/IMAGENAME:BRANCH_NAME`.\n\n`make build` also accepts arguments for REPO and TAG to override these behaviours.\nFor example, `make build/jupyterlab-cpu REPO=myrepo TAG=notLatest`.\n\n`make post-build/IMAGENAME` is meant for anything that is commonly done after building an image, but currently only adds common tags.\nIt adds tags of SHA, SHORT_SHA, and BRANCH_NAME to the given image,\nand accepts a `SOURCE_FULL_IMAGE_NAME` argument if you're trying to tag an existing image that has a non-typical name.\nFor example:\n\n- `make post-build/IMAGENAME` will apply SHA, SHORT_SHA, and BRANCH_NAME tags to `k8scc01covidacr.azurecr.io/IMAGENAME:BRANCH_NAME` (eg: using the default REPO and TAG names)\n- `make post-build/IMAGENAME SOURCE_FULL_IMAGE_NAME=oldRepo/oldImage:oldTag REPO=newRepo` will make the following new aliases for `oldRepo/oldImage:oldTag REPO=newRepo`:\n  - `newRepo/IMAGENAME:SHA`\n  - `newRepo/IMAGENAME:SHORT_SHA`\n  - `newRepo/IMAGENAME:BRANCH_NAME`\n\n### Pulling and Pushing Docker Images\n\n`make pull/IMAGENAME` and `make push/IMAGENAME` work similarly to `make build/IMAGENAME`.\nThey either push a local image to the acr, or pull an exisitng one from acr to local. \nThe `REPO` and `TAG` arguments are available to override their default values.\n\n**Note:** To use `make pull` or `make push`,\nyou must first log in to ACR (`az acr login -n k8scc01covidacr`)\n\n**Note:** `make push` by default does `docker push --all-tags` in order to push the SHA, SHORT_SHA, etc., tags.\n\n### Testing images\n\n#### Running and Connecting to Images Locally/Interactively\n\nTo test an image interactively, use `make dev/IMAGENAME`.\nThis calls `docker run` on a built image,\nautomatically forwarding ports to your local machine and providing a link to connect to.\nOnce the docker container is running, it will serve a localhost url to connect to the notebook.\n\n#### Automated Testing\n\nAutomated tests are included for the generated Docker images using `pytest`.\nThis testing suite is modified from the [docker-stacks](https://github.com/jupyter/docker-stacks) test suite.\nImage testing is invoked through `make test/IMAGENAME` (with optional `REPO` and `TAG` arguments like `make build`).\n\nTesting of a given image consists of general and image-specific tests:\n\n```\n└── tests\n    ├── general                             # General tests applied to all images\n    │   └── some_general_test.py\n    ├── jupyterlab-cpu                      # Test applied to a specific image\n    │   └── some_jupyterlab-cpu-specific_test.py\n    └── jupyterlab-tensorflow\n```\n\nWhere `tests/general` tests are applied to all images,\nand `tests/IMAGENAME` are applied only to a specific image.\nPytest will start the image locally and then run the provided tests to determine if Jupyterlab is running, python packages are working properly, etc.\nTests are formatted using typical pytest formats (python files with `def test_SOMETHING()` functions).\n`conftest.py` defines some standard scaffolding for image management, etc.\n\n---\n\n## General Development Workflow\n\n### Running an AAW Container Locally\n\n1. Clone the repository with `git clone https://github.com/StatCan/aaw-kubeflow-containers`.\n2. Run `make install-python-dev-venv` to build a development Python virtual environment.\n3. Build your image using `make build/IMAGENAME DIRECTORY=STAGENAME`, e.g. run `make build/jupyterlab-tensorflow DIRECTORY=cmd`.\n4. Test your image using automated tests through `make test/IMAGENAME`, e.g. run `make test/jupyterlab-tensorflow`.\n5. Veiw your images with `docker images`. You should see a table printed in the console with your images. For example you may see:\n\n```\nusername@hostname:~$ docker images\nREPOSITORY                                         TAG            IMAGE ID       CREATED          SIZE\nk8scc01covidacr.azurecr.io/jupyterlab-tensorflow   v1             13f8dc0e4f7a   26 minutes ago   14.6GB\nk8scc01covidacr.azurecr.io/jupyterlab-pytorch      v1             2b9acb795079   19 hours ago     15.5GB\njupyter/datascience-notebook                       9ed3b8de5de1   9a0c8d86de1a   5 weeks ago      4.25GB\n```\n\n7. Run your image with `make dev/IMAGENAME`,\n   e.g. `make dev/jupyterlab-tensorflow`.\n8. Open [http://localhost:8888](http://localhost:8888) or `\u003cip-address-of-server\u003e:8888`.\n\n### Testing Locally\n\n1. Clone the repo\n2. (optional) `make pull/IMAGENAME TAG=SOMEEXISTINGTAG` to pull an existing version of the image you are working on\n   (this could be useful as a build cache to reduce development time below)\n3. Edit an image via the [image stages](/images) that are used to create it.\n4. Build your edited stages and any dependencies using `make build/IMAGENAME DIRECTORY=STAGENAME`\n5. Test your image:\n   - using automated tests through `make test/IMAGENAME`\n   - manually by `docker run -it -p 8888:8888 REPO/IMAGENAME:TAG`,\n     then opening it in [http://localhost:8888](http://localhost:8888)\n\n### Testing On-Platform\n\nGitHub Actions CI is enabled to do building, scanning, automated testing, pushing of our images to ACR.\n\nBuild, test, and scan CI triggers on:\n\n- any push to master\n- any push to an open PR\n  This allows for easy scanning and automated testing for images.\n\nPushing images to ACR triggers on:\n\n- any push to master\n- any push to an open PR **that also has the `auto-deploy` label**\n\nPushes to the `master` branch will push to the k8scc01covidacr.azurecr.io ACR and these are accessible from both the dev and prod cluster.\nYou can access these images using any of the following:\n\n- k8scc01covidacr.azurecr.io/IMAGENAME:SHA\n- k8scc01covidacr.azurecr.io/IMAGENAME:SHORT_SHA\n- k8scc01covidacr.azurecr.io/IMAGENAME:latest\n- k8scc01covidacr.azurecr.io/IMAGENAME:v1\n\nPushes to an open PR **that also has the `auto-deploy` label on the PR** are accessible only from the dev cluster\nYou can access these images using any of the following:\n\n- k8scc01covidacrdev.azurecr.io/IMAGENAME:SHA\n- k8scc01covidacrdev.azurecr.io/IMAGENAME:SHORT_SHA\n- k8scc01covidacrdev.azurecr.io/IMAGENAME:BRANCHNAME\n- k8scc01covidacrdev.azurecr.io/IMAGENAME:dev (for convenience in testing)\n\nImages pushed to the dev acr are only available to the DEV cluster, attempting to use them in prod will fail.\n\nNOTE: ACR has an image retention policy of 7 days. However it is currently disabled.\n\n### Overview of images\n\nEach directory in the images folder makes up one stage of the build process.\nThey each contain the Dockerfile that directs the build, and all related files.\n\nThe relationship between the stages and the final product is as shown below.\n![The flowchart showing the stages and their order](./docs/images/image-stages.png)\n\nAll output images should meet the following criteria:\n\n### Adding new software\n\nSoftware needs to be added by modifying the relevant image stage,\nthen following the normal build instructions starting with the Generate Dockerfiles step.\n\nBe selective with software installation as image sizes are already quite big (16Gb plus),\nand increasing that size would negatively impact the time it takes up for a workspace server to come up\n(as well as first time image pulls to a node).\nIn such cases it may be more relevant to make an image under\n[aaw-contrib-containers](https://github.com/StatCan/aaw-contrib-containers) as mentioned earlier.\n\n### Adding new Images\n\n1. Identify what stages are needed, \nand if the image will use any existing images and workflows\n2. For each new stage to be added, \nfollow the process in the [Adding new Stages](#adding-new-stages) section\n\n### Adding new Stages\n\n1. Identify where the new stage will be placed in the build order\n2. Create a new subdirectory in the `/images/` directory for the stage\n3. Add a new job to the `./github/workflows/docker.yaml` for the new stage.\nSee below for a description of all the fields.\n4. If this stage was inserted between two existing stages,\nupdate the parent values of any children of this stage\n5. If this stage creates an image that will be deployed to users.\nA job must be added to test the image in `./github/workflows/docker.yaml`,\nand the image name must be added to the matrix in `./github/workflows/docker-nightly.yaml`\nSee below for a description of all the fields\n6. Update the documentation for the new stage.\nThis is generally updating `images-stages.png` and `image-stages.drawio` in the `docs/images` folder using draw.io.\n\n\nyaml to create an image\n```yaml\n  stage-name:                                                         # The name of the stage, will be shown in the CICD workflow\n    needs: [vars, parent]                                             # All stages need vars, any stages with a parent must link their direct parent\n    uses: ./.github/workflows/docker-steps.yaml\n    with:\n      image: \"stage-name\"                                             # The name of the current stage/image\n      directory: \"directory-name\"                                     # The name of the directory in the /images/ folder. /images/base would be \"base\"\n      base-image: \"quay.io/jupyter/datascience-notebook:2024-06-17\"   # used if the stage is built from an upsteam image. Omit if stage has a local parent\n      parent-image: \"parent\"                                          # The name of the parent stage/image. Omit if stage uses an upsteam image\n      parent-image-is-diff: \"${{ needs.parent.outputs.is-diff }}\"     # Checks if the parent image had changes. Omit if stage uses an upsteam image\n      buildkit: 1                                                     # Configures the buildkit version of docker. Can be ommited if using the default 1. \n                                                                      # Can be set to 0 for building large images, like remote-desktop\n      # The following values are static between differnt stages\n      registry-name: \"${{ needs.vars.outputs.REGISTRY_NAME }}\"\n      branch-name: \"${{ needs.vars.outputs.branch-name }}\"\n    secrets:\n      REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}\n      REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}\n```\n\nyaml to create a test\n```yaml\n  imagename-test:                                       # The name of the test job, usually  imagename-test\n    needs: [vars, imagename]                            # Must contain vars and the image that will be tested\n    uses: ./.github/workflows/docker-pull-test.yaml\n    with:\n      image: \"imagename\"                                # The name of the image that will be tested\n      # The following values are static between differnt tests\n      registry-name: \"${{ needs.vars.outputs.REGISTRY_NAME }}\"\n      branch-name: \"${{ needs.vars.outputs.branch-name }}\"\n    secrets:\n      REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}\n      REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}\n      CVE_ALLOWLIST: ${{ secrets.CVE_ALLOWLIST}}\n```\n\n### Modifying and Testing CI\n\nIf making changes to CI that cannot be done on a branch (eg: changes to issue_comment triggers), you can:\n\n1. fork the 'kubeflow-containers' repo\n2. Modify the CI with\n\n- REGISTRY: (your own dockerhub repo, eg: \"j-smith\" (no need for the full url))\n- Change\n  ```\n  - uses: azure/docker-login@v1\n    with:\n      login-server: ${{ env.REGISTRY_NAME }}.azurecr.io\n      username: ${{ secrets.REGISTRY_USERNAME }}\n      password: ${{ secrets.REGISTRY_PASSWORD }}\n  ```\n  to\n  ```\n  - uses: docker/login-action@v1\n    with:\n      username: ${{ secrets.REGISTRY_USERNAME }}\n      password: ${{ secrets.REGISTRY_PASSWORD }}\n  ```\n\n3. In your forked repo, define secrets for REGISTRY_USERNAME and REGISTRY_PASSWORD with your dockerhub credentials (you should use an API token, not your actual dockerhub password)\n\n---\n\n## Other Development Notes\n\n### Github CI\n\nThe Github workflow is set up to build the images and their dependant stages.\nSee below for a flowchart of this build.\n\nThe main workflow is `docker.yaml`,\nit controls the stage build order, and what triggers the CI.\n(Pushes to master, pushes to an open pull-request, and nightly builds)\n\nThe building of a stage is controled by `docker-steps.yaml`.\nIt checks if there are changes to the stage or dependant stages.\nBuilds a new image if there are changes, \nor pulls a copy of the existing image if not.\nTesting will be performed if this is the final stage in the build of an image.\n\n![A flowchart of the Github CI workflow](./docs/images/Workflows.png)\n\n### The `latest` and `v1` tags for the master branch\n\nThese tags are intended to be `long-lived` in that they will not change.\nSubsequent pushes will clobber the previous `IMAGENAME:latest` image.\nThis means that `IMAGENAME:latest` will be updated automatically as changes are made,\nso updates to the tag are not needed.\n\nThe `v1` tag is intended for when we encounter a breaking change,\nbut still want to support the features of that current image.\n\nThe `v2` tag is currently used by [zone-kubeflow-containers](https://github.com/StatCan/zone-kubeflow-containers)\n\n**Note**:\nThe `latest` tag is shared with [zone-kubeflow-containers](https://github.com/StatCan/zone-kubeflow-containers),\nSo isn't reliable\n\n### Set User File Permissions\n\nThe Dockerfiles in this repo are intended to construct compute environments for a non-root user **jovyan** to ensure the end user has the least privileges required for their task,\nbut installation of some of the software needed by the user must be done as the **root** user.\nThis means that installation of anything that should be user editable (eg: `pip` and `conda` installs, additional files in `/home/$NB_USER`, etc.) will by default be owned by **root** and not modifiable by **jovyan**.\n**Therefore we must change the permissions of these files to allow the user specific access for modification.**\nFor example, most pip install/conda install commands occur as the root user and result in new files in the $CONDA_DIR directory that will be owned by **root** and cause issues if user **jovyan** tried to update or uninstall these packages\n(as they by default will not have permission to change/remove these files).\n\nTo fix this issue, end any `RUN` command that edits any user-editable files with:\n\n```\nfix-permissions $CONDA_DIR \u0026\u0026 \\\nfix-permissions /home/$NB_USER\n```\n\nThis fix edits the permissions of files in these locations to allow user access.\nNote that if these are not applied **in the same layer as when the new files were added** it will result in a duplication of data in the layer because the act of changing permissions on a file from a previous layer requires a copy of that file into the current layer.\nSo something like:\n\n```\nRUN add_1GB_file_with_wrong_permissions_to_NB_USER.sh \u0026\u0026 \\\n\tfix-permissions /home/$NB_USER\n```\n\nwould add a single layer of about 1GB, whereas\n\n```\nRUN add_1GB_file_with_wrong_permissions_to_NB_USER.sh\n\nRUN fix-permissions /home/$NB_USER\n```\n\nwould add two layers, each about 1GB (2GB total).\n\n### Troubleshooting\n\nIf running using a VM and RStudio image was built successfully but is not opening correctly on localhost (error 5000 page),\nchange your CPU allocation in your Linux VM settings to \u003e= 3.\nYou can also use your VM's system monitor to examine if all CPUs are 100% being used as your container is running.\nIf so, increase CPU allocation.\nThis was tested on Linux Ubuntu 20.04 virtual machine.\n\n## Structure\n\n```\n.\n├── .github/workflow                        # Github CI. Controls the stage build order\n│\n├── Makefile                                # Controls the interactions with docker commands\n│\n├── make_helpers                            # Scripts used by makefile\n│   ├── get_branch_name.sh\n│   ├── get-nvidia-stuff.sh\n│   └── post-build-hook.sh\n│\n├── images                                  # Dockerfile and required resources for stage builds\n│   ├── base                                # Common stage for most images\n│   ├── cmd                                 # Common stage for finalizing most images\n│   ├── platform\n│   ├── remote-desktop\n│   ├── rstudio\n│   ├── sas\n│   └── tensorflow\n│\n├── docs                                    # files/images used in documentation (ex. Readme's)\n│\n└── tests\n    ├── general/                            # General tests applied to all images\n    ├── jupyterlab-cpu/                     # Test applied to a specific image\n    ├── jupyterlab-pytorch/\n    ├── jupyterlab-tensorflow/\n    └── README.md\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstatcan%2Faaw-kubeflow-containers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstatcan%2Faaw-kubeflow-containers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstatcan%2Faaw-kubeflow-containers/lists"}