{"id":20687620,"url":"https://github.com/susumuota/kaggleenv","last_synced_at":"2025-11-09T23:03:26.376Z","repository":{"id":60550460,"uuid":"353775682","full_name":"susumuota/kaggleenv","owner":"susumuota","description":"GCP + Kaggle Docker + VSCode","archived":false,"fork":false,"pushed_at":"2022-02-28T04:31:30.000Z","size":160,"stargazers_count":15,"open_issues_count":1,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-11-09T23:03:07.269Z","etag":null,"topics":["docker","docker-compose","google-cloud","google-cloud-platform","jupyter","jupyter-notebook","jupyterlab","kaggle","python","python3","visual-studio-code","vscode"],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/susumuota.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-01T17:24:50.000Z","updated_at":"2024-09-09T17:02:32.000Z","dependencies_parsed_at":"2022-10-01T07:20:16.714Z","dependency_job_id":null,"html_url":"https://github.com/susumuota/kaggleenv","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/susumuota/kaggleenv","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/susumuota%2Fkaggleenv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/susumuota%2Fkaggleenv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/susumuota%2Fkaggleenv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/susumuota%2Fkaggleenv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/susumuota","download_url":"https://codeload.github.com/susumuota/kaggleenv/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/susumuota%2Fkaggleenv/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":283593385,"owners_count":26861541,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-09T02:00:05.828Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","google-cloud","google-cloud-platform","jupyter","jupyter-notebook","jupyterlab","kaggle","python","python3","visual-studio-code","vscode"],"created_at":"2024-11-16T22:57:49.066Z","updated_at":"2025-11-09T23:03:26.362Z","avatar_url":"https://github.com/susumuota.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GCP (or local machine) + Kaggle Docker + VSCode\n\n![vscode_jupyter](https://user-images.githubusercontent.com/1632335/113431667-0d1b8c80-9417-11eb-8183-e89084670f39.png)\n\nThis document describes how to setup [Kaggle Python docker image](https://github.com/Kaggle/docker-python) environment on [Google Cloud Platform (GCP)](https://cloud.google.com/) or your local machine by [Docker](https://www.docker.com/) and how to setup [Visual Studio Code (VSCode)](https://code.visualstudio.com/) to connect the environment.\n\nA primally information source comes from [Kaggle's docker-python repository](https://github.com/Kaggle/docker-python). Also, there is a [guide](https://medium.com/kaggleteam/how-to-get-started-with-data-science-in-containers-6ed48cb08266), but unfortunately it's a bit obsoleted guide written in 2016.\n\n\u003e **_Note:_** This method may take 20-30 minutes and over 18.5GB disks for data downloads.\n\n\u003e **_Note:_** If you do not use VSCode, no need to read this document. See [here](https://www.kaggle.com/product-feedback/159602).\n\nAll files in this document are available on [my repository](https://github.com/susumuota/kaggleenv).\n\nThere are 2 options, GCP or local machine. If you are going to setup the environment on your local machine, skip to `[Option 2] Setup the environment on your local machine` section.\n\n## [Option 1] Setup the environment on GCP\n\nOn GCP, [\"Vertex AI Workbench\"](https://cloud.google.com/vertex-ai/docs/workbench) would be easier than [\"Compute Engine\"](https://cloud.google.com/compute/docs/) (GCE) to setup [Kaggle Python docker image](https://github.com/Kaggle/docker-python).\n\n### Create a Vertex AI Workbench\n\n- Access https://console.cloud.google.com/vertex-ai/workbench\n- Select a project e.g. `kaggle-myproject-1` (You must [create a project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project) beforehand)\n- Click `USER-MANAGED NOTEBOOK`\n- Click `NEW NOTEBOOK`\n- Choose `Customize...`\n- Instance name: e.g. `kaggle-test-1`\n- Environment: `Kaggle Python [BETA]` (This option will automatically prepare [Kaggle Python docker image](https://github.com/Kaggle/docker-python) at startup the VM instance)\n- GPU type: e.g. `NVIDIA Tesla T4` (You must [increase GPU quota](https://cloud.google.com/compute/quotas#requesting_additional_quota) beforehand)\n  - Mark the checkbox `Install NVIDIA GPU driver automatically for me`\n\n![gcp_notebook_1](https://user-images.githubusercontent.com/1632335/115653028-636e5200-a369-11eb-9bda-8c34036591f4.png)\n\n- Open `Networking` section\n  - Mark the radio button `Networks in this project`\n  - Clear the checkbox `Allow proxy access when it's available` (This option will avoid to load unnecessary proxy Docker container)\n- Click `CREATE`\n\n![gcp_notebook_2](https://user-images.githubusercontent.com/1632335/115653237-c7911600-a369-11eb-88b1-db382a1f997e.png)\n\n- Wait for around 20-30 minutes to start up the VM instance. I guess it's because of `docker pull`. If you choose GPU type: `None`, it takes a few minutes. Check the console logs at [here](https://console.cloud.google.com/logs/).\n\n### Connect to the VM instance\n\n- [Install Cloud SDK](https://cloud.google.com/sdk/docs/quickstart). If you are using macOS and Homebrew, `brew install --cask google-cloud-sdk` may be convenient.\n\nAfter that, `gcloud` command should be available on your terminal.\n\n- SSH to the VM instance with port forwarding\n\n```\n% gcloud compute --project \"kaggle-myproject-1\" ssh --zone \"us-west1-b\" \"kaggle-test-1\" -- -L 8080:localhost:8080\n```\n\n\u003e **_Note:_** You must wait to start up the VM instance. Check the console logs at [here](https://console.cloud.google.com/logs/).\n\n\u003e **_Note:_** I recommend to limit source IP ranges for SSH and RDP port. See [here](https://cloud.google.com/vpc/docs/using-firewalls#creating_firewall_rules).\n\n- Open web browser and try to access `http://localhost:8080`\n\n\u003e **_Note:_** There is no `token=...`.\n\nIf you do not use VSCode, that's all. You do not have to do anything below.\n\n### Stop pre-installed Docker container\n\nIf you use VSCode to connect GCP Notebook, you must tweak Docker container. At the moment, VSCode can only access to remote Jupyter servers with `token` option enabled. But pre-installed Docker container disables `token` option by `c.NotebookApp.token = ''`. You must stop pre-installed Docker container and run a new Docker container with `token` option enabled instead.\n\n- Stop pre-installed Docker container\n\nStop pre-installed Docker container and turn off the startup option. See details [here](https://docs.docker.com/config/containers/start-containers-automatically/).\n\n```\n% docker ps -a\n% docker inspect -f \"{{.Name}} {{.HostConfig.RestartPolicy.Name}}\" $(docker ps -aq)\n% docker update --restart no payload-container\n% docker inspect -f \"{{.Name}} {{.HostConfig.RestartPolicy.Name}}\" $(docker ps -aq)\n% docker stop payload-container\n% docker ps -a\n```\n\n- Install `docker-compose`\n\n`docker-compose` will be convenient to run containers, even on a single container. See details [here](https://docs.docker.com/compose/install/).\n\n```\n% sudo curl -L \"https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)\" -o /usr/local/bin/docker-compose\n% sudo chmod +x /usr/local/bin/docker-compose\n```\n\nSkip to `Run Docker container` section.\n\n## [Option 2] Setup the environment on your local machine\n\nIf you setup the environment on your local machine, [install and setup Docker](https://docs.docker.com/get-docker/).\n\nAfter that, `docker` and `docker-compose` commands should be available on your terminal.\n\n```sh\n% docker -v\nDocker version 20.10.8, build 3967b7d\n% docker-compose -v\ndocker-compose version 1.29.2, build 5becea4c\n```\n\n## Run Docker container (both GCP and local machine)\n\nI prepared a [sample repository](https://github.com/susumuota/kaggleenv) of the `Dockerfile`, etc. If you do not care about details, execute these commands and skip to `Open Notebook by web browser` section.\n\n```\n% git clone https://github.com/susumuota/kaggleenv.git\n% cd kaggleenv\n% docker-compose build\n% docker-compose up -d\n% docker-compose logs\n# Find and copy http://localhost:8080/?token=...\n```\n\nOtherwise, follow the instructions below.\n\n### Create `Dockerfile`\n\nCreate a directory (e.g. `kaggleenv`) and go there. If you clone the sample repository, just `cd kaggleenv`.\n\nCreate `Dockerfile` like the following. See details [here](https://docs.docker.com/engine/reference/builder/#format). If you use CPU instead of GPU, edit `FROM` lines.\n\n```Dockerfile\n# for CPU\n# FROM gcr.io/kaggle-images/python:v109\n# for GPU\nFROM gcr.io/kaggle-gpu-images/python:v109\n\n# apply patch to enable token and change notebook directory to /kaggle/working\n# see jupyter_notebook_config.py.patch\nCOPY jupyter_notebook_config.py.patch /opt/jupyter/.jupyter/\nRUN (cd /opt/jupyter/.jupyter/ \u0026\u0026 patch \u003c jupyter_notebook_config.py.patch)\n\n# add extra modules here\n# RUN pip install -U pip\n```\n\nYou can specify a tag (e.g. `v109` or `latest`) to keep using the same environment. You can find tags from [GCR page](https://gcr.io/kaggle-images/python).\n\n### Create `jupyter_notebook_config.py.patch`\n\nThis Docker image will run Jupyter Lab with startup script `/run_jupyter.sh` and config `/opt/jupyter/.jupyter/jupyter_notebook_config.py`. It needs to be tweaked like the following.\n\n- Enable token (so that VSCode can connect properly)\n- Change notebook directory to `/kaggle/working`\n\nCreate `jupyter_notebook_config.py.patch` like the following.\n\n```patch\n--- jupyter_notebook_config.py.orig\t2021-12-19 07:04:25.000000000 +0000\n+++ jupyter_notebook_config.py\t2022-01-29 18:19:29.016821460 +0000\n@@ -11 +11 @@\n-c.ServerApp.token = \"\"\n+# c.ServerApp.token = \"\"\n@@ -17 +17,2 @@\n-c.ServerApp.notebook_dir = \"/home/jupyter\"\n+# c.ServerApp.notebook_dir = \"/home/jupyter\"\n+c.ServerApp.notebook_dir = \"/kaggle/working\"\n```\n\n\u003e **_Note:_** This patch may not work in the future version of [Kaggle Python docker image](https://github.com/Kaggle/docker-python). In that case, create a new patch with `diff -u original new \u003e patch`. At least I confirmed this patch work on `v109` tag.\n\n### Create `docker-compose.yml`\n\nCreate `docker-compose.yml` like the following. See details [here](https://docs.docker.com/compose/). This setting mounts current directory on your local machine to `/kaggle/working` on the container. If you use CPU instead of GPU, comment out `runtime: nvidia`.\n\n```yaml\nversion: \"3\"\nservices:\n  jupyter:\n    build: .\n    volumes:\n      - $PWD:/kaggle/working\n    working_dir: /kaggle/working\n    ports:\n      - \"8080:8080\"\n    hostname: localhost\n    restart: always\n    # for GPU\n    runtime: nvidia\n```\n\n### Create `.dockerignore`\n\nCreate `.dockerignore` like the following. See details [here](https://docs.docker.com/engine/reference/builder/#dockerignore-file). This setting specifies subdirectories and files that should be ignored when building Docker images. You will **mount** the current directory, so you do not need to **include** subdirectories and files into image. Especially, `input` directory should be ignored because it may include large files so that build process may take long time.\n\n```\nREADME.md\ninput\noutput\n.git\n.gitignore\n.vscode\n.ipynb_checkpoints\n```\n\n### Run `docker-compose build`\n\nRun `docker-compose build` to build the Docker image. See details [here](https://docs.docker.com/compose/reference/build/).\n\n\u003e **_Note:_** This process may take 20-30 minutes and over 18.5GB disks for data downloads on your local machine.\n\n```sh\n% docker-compose build\n```\n\nConfirm the image by `docker images`.\n\n```sh\n% docker images\nREPOSITORY            TAG       IMAGE ID       CREATED          SIZE\nkaggleenv_jupyter   latest    ............   28 minutes ago   18.5GB\n```\n\n### Run `docker-compose up -d`\n\nRun `docker-compose up -d` to start Docker container in the background. In addition, the container will automatically run at startup VM instance or local machine. See details [here](https://docs.docker.com/compose/reference/up/) and [here](https://docs.docker.com/config/containers/start-containers-automatically/).\n\n```sh\n% docker-compose up -d\n% docker ps -a\n% docker inspect -f \"{{.Name}} {{.HostConfig.RestartPolicy.Name}}\" $(docker ps -aq)\n```\n\nFind the Notebook URL on the log and copy it.\n\n```\n% docker-compose logs\n\nhttp://localhost:8080/?token=...\n```\n\n### Open Notebook by web browser\n\n- Open web browser and type the Notebook URL (`http://localhost:8080/?token=...`).\n- Create a `Python 3` Notebook.\n- Create code cells and execute `!pwd`, `!ls` and `!pip list` to confirm Python environment.\n\n![jupyter_kaggle](https://user-images.githubusercontent.com/1632335/113484058-5afcc700-94e1-11eb-9f2e-a6fd01a0121a.png)\n\n### Setup Kaggle API\n\n[Setup Kaggle API credentials](https://github.com/Kaggle/kaggle-api#api-credentials).\n\nAfter that, `~/.kaggle/kaggle.json` file should be on your local machine.\n\n- Copy `~/.kaggle/kaggle.json` to current directory **on your local machine** (so that it can be accessed from the container at `/kaggle/working/kaggle.json`)\n\n```sh\n% cp -p ~/.kaggle/kaggle.json .\n```\n\n- Create a code cell on the Notebook and confirm `/kaggle/working/kaggle.json` on the container.\n\n```sh\n!ls -l /kaggle/working/kaggle.json\n-rw------- 1 root root 65 Mar 22 07:59 /kaggle/working/kaggle.json\n```\n\n- Copy it to `~/.kaggle` directory on the container.\n\n```sh\n!cp -p /kaggle/working/kaggle.json ~/.kaggle/\n```\n\n- Remove `kaggle.json` on the current directory **on your local machine**.\n\n```sh\n% rm -i kaggle.json\n```\n\n- Try `kaggle` command on the Notebook.\n\n```sh\n!kaggle competitions list\n```\n\n### Shutdown the Vertex AI Workbench (GCP)\n\nAfter you finished your work, stop the VM instance.\n\n- Access https://console.cloud.google.com/vertex-ai/workbench/list/instances\n- Check the VM instance on the list\n- Click `STOP` or `DELETE`\n\nIf you `DELETE` the VM instance, you will not be charged anything (as far as I know).\n\nHowever, if you `STOP` the VM instance, you will be charged for resources (e.g. persistent disk) until you `DELETE` it. You should `DELETE` if you do not use it for a long time (though you must setup the environment again). See details [here](https://cloud.google.com/compute/docs/instances/stop-start-instance#billing).\n\n### Run `docker-compose down` (local machine)\n\nAfter you finished your work, run `docker-compose down` to stop Docker container. See details [here](https://docs.docker.com/compose/reference/down/).\n\n```sh\n% docker-compose down\n```\n\n## Setup VSCode to open remote Notebooks\n\nIf you are using [Visual Studio Code (VSCode)](https://code.visualstudio.com/), you can setup VSCode to connect to the remote Notebook.\n\n### [Optional] Install the latest Notebook extension\n\nThere is a revamped version of Notebook extension. See details [here](https://devblogs.microsoft.com/python/notebooks-are-getting-revamped/). I recommend installing it because this new version can handle custom extensions (e.g. key bindings) properly inside code cells, etc.\n\n![vscode_jupyter](https://user-images.githubusercontent.com/1632335/113431667-0d1b8c80-9417-11eb-8183-e89084670f39.png)\n\n### Connect to the remote Notebook\n\nConnect to the remote Notebook. See details [here](https://code.visualstudio.com/docs/python/jupyter-support#_connect-to-a-remote-jupyter-server).\n\n- Open `Command Palette...`\n- Type `Jupyter: Specify local or remote Jupyter server for connections`\n\n![vscode_palette](https://user-images.githubusercontent.com/1632335/113466765-3bca4f00-9479-11eb-914e-7d90ac073daf.png)\n\n- Choose `Existing: Specify the URI of an existing server`\n\n![vscode_existing](https://user-images.githubusercontent.com/1632335/113467276-01fb4780-947d-11eb-93f6-a4f5a974d323.png)\n\n- Specify the Notebook URL (`http://localhost:8080/?token=...`)\n\n\u003e **_Note:_** `token` must be specified.\n\n![vscode_uri](https://user-images.githubusercontent.com/1632335/113467238-c2ccf680-947c-11eb-9388-1ecd2297eb6b.png)\n\n- Press `Reload` button\n\n![vscode_reload](https://user-images.githubusercontent.com/1632335/113467629-31ab4f00-947f-11eb-9062-1bbc5566ab86.png)\n\n- Open `Command Palette...`\n- Type `Jupyter: Create New Blank Notebook`\n\n![vscode_create](https://user-images.githubusercontent.com/1632335/113467560-9f0ab000-947e-11eb-865e-62beeed43f12.png)\n\n- Create code cells and execute `!pwd`, `!ls` and `!pip list` to confirm Python environment.\n\n![vscode_new_notebook](https://user-images.githubusercontent.com/1632335/113467525-75518900-947e-11eb-86e1-e9e79d84e610.png)\n\n## Increase Docker resources (local machine)\n\nSometimes containers need much resources (e.g. memory or disk). You can increase the amount of resources from Docker preferences.\n\n- Click Docker icon\n- Choose `Preferences...`\n- Click `Resources`\n- Click `ADVANCED`\n- Increase `Memory`, e.g. `8GB`\n- Increase `Disk image size`, e.g. `128GB`\n- Click `Apply \u0026 Restart`\n\n![docker_preferences](https://user-images.githubusercontent.com/1632335/137115613-88386ee7-807e-4252-920f-73ef04d9a18a.png)\n\n## Maintain Docker containers, images and cache\n\nBasically `docker-compose up -d` and `docker-compose down` work well, but sometimes you may need to use these commands to maintain Docker containers, images and cache.\n\n- How to remove containers. See details [here](https://docs.docker.com/engine/reference/commandline/rm/).\n\n```sh\n% docker ps -a  # confirm container ids to remove\n% docker rm CONTAINER  # remove container by id\n% docker rm $(docker ps --filter status=exited -q)  # remove all containers that have exited\n```\n\n- How to remove images. See details [here](https://docs.docker.com/engine/reference/commandline/rmi/).\n\n```sh\n% docker images  # confirm image ids to remove\n% docker rmi IMAGE  # remove image by id\n```\n\n- How to remove cache. See details [here](https://docs.docker.com/engine/reference/commandline/builder_prune/) and [here](https://docs.docker.com/engine/reference/commandline/volume_prune/).\n\n```sh\n% docker system df  # confirm how much disk used by cache\n% docker builder prune\n% docker volume prune\n```\n\n## TODO\n\n- Workflow to submit local Notebook to Kaggle\n\n## Links\n\n- https://github.com/Kaggle/docker-python\n- https://medium.com/kaggleteam/how-to-get-started-with-data-science-in-containers-6ed48cb08266\n- https://github.com/susumuota/kaggleenv\n- https://cloud.google.com/vertex-ai/docs/workbench\n- https://cloud.google.com/sdk/docs/quickstart\n- https://code.visualstudio.com/docs/python/jupyter-support#_connect-to-a-remote-jupyter-server\n- https://devblogs.microsoft.com/python/notebooks-are-getting-revamped/\n- https://www.kaggle.com/product-feedback/159602\n- https://amalog.hateblo.jp/entry/data-analysis-docker  (Japanese)\n\n## Author\n\nSusumu OTA\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsusumuota%2Fkaggleenv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsusumuota%2Fkaggleenv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsusumuota%2Fkaggleenv/lists"}