{"id":13579527,"url":"https://github.com/yu-iskw/google-cloud-deep-learning-kit","last_synced_at":"2025-04-05T21:31:52.843Z","repository":{"id":145587202,"uuid":"102819123","full_name":"yu-iskw/google-cloud-deep-learning-kit","owner":"yu-iskw","description":"Create a GPU instance on GCP with Jupyter + Keras(Tensorflow) + Nvidia Docker","archived":true,"fork":false,"pushed_at":"2018-03-25T16:06:34.000Z","size":86,"stargazers_count":39,"open_issues_count":5,"forks_count":13,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-05T17:49:59.653Z","etag":null,"topics":["deep-learning","deep-neural-networks","docker","google-cloud","gpu","jupyter","keras","nvidia-docker","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yu-iskw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-09-08T04:54:22.000Z","updated_at":"2023-03-22T21:34:51.000Z","dependencies_parsed_at":"2024-01-16T21:05:52.709Z","dependency_job_id":"e50ad703-9048-4e9b-a063-9f2da45a4d18","html_url":"https://github.com/yu-iskw/google-cloud-deep-learning-kit","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yu-iskw%2Fgoogle-cloud-deep-learning-kit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yu-iskw%2Fgoogle-cloud-deep-learning-kit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yu-iskw%2Fgoogle-cloud-deep-learning-kit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yu-iskw%2Fgoogle-cloud-deep-learning-kit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yu-iskw","download_url":"https://codeload.github.com/yu-iskw/google-cloud-deep-learning-kit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247405987,"owners_count":20933802,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","deep-neural-networks","docker","google-cloud","gpu","jupyter","keras","nvidia-docker","tensorflow"],"created_at":"2024-08-01T15:01:40.206Z","updated_at":"2025-04-05T21:31:52.546Z","avatar_url":"https://github.com/yu-iskw.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"# Google Cloud Deep Learning Kit\n\nThis repository aims to create a GPU instance with Jupyter, Tensorflow and Keras on google cloud platform in an instant.\n\n![Overview](./docs/overview.png)\n\n## Requirements\n\n- make\n- Google Cloud SDK\n- Docker\n\n## Life Cycle to Use this Kit\n\n1. Create an instance with `make create-instance` or `make create-instance-cpu`\n2. Run jupyter on the instance with `make run-jupyter`\n  - It may takes 5 minutes or so.\n3. Install python libraries with `make pip-install`\n  - Put libraries you want to install in `./requirements.txt`\n4. Upload files to the instance with `make upload-files`.\n5. Make ssh tunnel to the instance with `make ssh-tunnel`\n6. Access jupyter via your web browser\n  - Default: `http://localhost:18888`\n7. Download outputs with `make download-outputs`\n8. Delete the instance with `make delete-instance`\n\n## Installed python libraries in the docker image\n\nThe pre-built docker image on Docker Hub is created with the following anaconda environment YAML.\nDon't worry, you can additionally install other python library with `make pip-install`, if you want.\n\n[Anaconda Environment YAML](./docker/environment-gpu.yml)\n\n## Commands Reference\n\n### Create a GCP instance with GPUs\nIt may take 5 minutes or so to finish to execute the startup script to install the require environment in your instance.\n\n```\nmake create-instance \\\n  INSTANCE_NAME=\"test-gpu-instance\" \\\n  GCP_PROJECT_ID=xxx-xxx-xxx\n```\n\nThe command has some options as following:\n- `INSTANCE_NAME`: GCP instance name\n- `GCP_PROJECT_ID`: GCP project ID\n- `MACHINE_TYPE`: GCP instance machine type\n- `ACCELERATOR`: Attached accelerator\n- `BOOT_DISK_SIZE`: boot disk size\n- `GCP_INSTANCE_SCOPES`: scopes\n\nPlease change `ACCELERATOR` to what you want, when you would like to modify the accelerator of a GCP instance\nThe default value of `ACCELERATOR` is `type=nvidia-tesla-k80,count=1`.\n\n### Create a GCP instance without GPUs\nWhen you would like to make a GCP instance without GPU, I would recommend you to execute `make create-instance-cpu`.\n\n```\nmake create-instance-cpu \\\n  INSTANCE_NAME=\"test-cpu-instance\" \\\n  GCP_PROJECT_ID=xxx-xxx-xxx \\\n  GCP_ZONE=us-central1-f \\\n  MACHINE_TYPE=n1-standard-32\n```\nThe command has some options as following:\n- `INSTANCE_NAME`: GCP instance name\n- `GCP_PROJECT_ID`: GCP project ID\n- `MACHINE_TYPE`: GCP instance machine type\n- `BOOT_DISK_SIZE`: boot disk size\n- `GCP_INSTANCE_SCOPES`: scopes\n\n### Run Jupyter as a docker container\n\n#### Run Jupyter as a docker container on a GPU instance\nWe can run jupyter on the docker container with the command.\nWhen a container for jupyter is running, it will restart the container.\n\n```\nmake run-jupyter \\\n  INSTANCE_NAME=\"test-gpu-instance\" \\\n  GCP_PROJECT_ID=xxx-xxx-xxx \\\n  DOCKER_IMAGE_GPU=tf-1.4-gpu\n```\n\nWe support some types of docker image for GPU.\n\n- `tf-1.4-gpu`: Tensorflow 1.4 for GPU\n\n#### Run Jupyter as a docker container on a CPU instance\nWhen you launch a GCP instance with `make create-instance-cpu`, you must use the command in order to run a container for jupyter.\nBecause the docker image with GPUs is totally different from that without GPUs.\n\n```\nmake run-jupyter-cpu \\\n  INSTANCE_NAME=\"test-gpu-instance\" \\\n  GCP_PROJECT_ID=xxx-xxx-xxx \\\n  DOCKER_IMAGE_CPU=tf-1.4-cpu\n```\n\nWe support some types of docker image for GPU.\n\n- `tf-1.4-cpu`: Tensorflow 1.4 for CPU\n\n### SSH tunnel\nIn order to access the jupyter which you launched, you have to have a SSH tunnel.\nWhen you got it, you can access `http://localhost:18888` via a web browser on your local machine.\nWhen you don't set any value with `make ssh-tunnel`, the port is the default value.\nThe default of `JUPYTER_PORT` is `18888`.\n\n```\nmake ssh-tunnel \\\n  INSTANCE_NAME=\"test-gpu-instance\" \\\n  GCP_PROJECT_ID=xxx-xxx-xxx \\\n  JUPYTER_PORT=18888\n```\n\n### Delete the instance you created\n\nThe command is used for deleting the GCP instance you created.\nPlease don't forget to do that after you finish your works!\n\n```\nmake delete-instance \\\n  INSTANCE_NAME=\"test-gpu-instance\" \\\n  GCP_PROJECT_ID=xxx-xxx-xxx\n```\n\n### Install python libraries\n\nIf you want to install python libraries other than pre-installed ones, please add them to `./requirements.txt` and then execute the below command:\nWhen you restart a container to run jupyter, we must re-execute the commahd.\n\n```\nmake pip-install \\\n  INSTANCE_NAME=\"test-gpu-instance\" \\\n  GCP_PROJECT_ID=xxx-xxx-xxx\n```\n\n### Upload your files\nThis command allows us to upload files on a local machine to the instance.\nThose files will be set at `/src` on the GCP machine and the docker container will mount it at `/src`.\n\n```\nmake upload-files \\\n  INSTANCE_NAME=\"test-gpu-instance\" \\\n  GCP_PROJECT_ID=xxx-xxx-xxx \\\n  FROM=/path/to/your/files\n```\n\n### Download ouputs\nThis command allows us to download files under `/src/outputs` in the instance.\nWhen you save trained models and some predicted results under `/src/outputs`, you can download them with it.\n\n```\nmake download-outputs \\\n  INSTANCE_NAME=\"test-gpu-instance\" \\\n  GCP_PROJECT_ID=xxx-xxx-xxx \\\n  TO=/path/to/your/destination\n```\n\n## Jupyter extensions\nJupyter in this docker image has extensions as default.\nIf you would like to turn off some extensions, please visit `http://localhost:18888/nbextensions`.\n\n## Links\n- [yuiskw/google\\-cloud\\-deep\\-learning\\-kit \\- Docker Hub](https://hub.docker.com/r/yuiskw/google-cloud-deep-learning-kit/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyu-iskw%2Fgoogle-cloud-deep-learning-kit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyu-iskw%2Fgoogle-cloud-deep-learning-kit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyu-iskw%2Fgoogle-cloud-deep-learning-kit/lists"}