https://github.com/grimmer0125/colab-network-exp

note: 1. specify ssh private key to git pull: https://superuser.com/a/912281. 2. using "!pip install PACKAGE_NAME" to install packages on colab
https://github.com/grimmer0125/colab-network-exp

colab

Last synced: 7 days ago
JSON representation

note: 1. specify ssh private key to git pull: https://superuser.com/a/912281. 2. using "!pip install PACKAGE_NAME" to install packages on colab

• Host: GitHub
• URL: https://github.com/grimmer0125/colab-network-exp
• Owner: grimmer0125
• Created: 2018-10-13T08:16:56.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2018-12-20T14:21:29.000Z (about 6 years ago)
• Last Synced: 2024-12-22T00:23:36.445Z (2 months ago)
• Topics: colab
• Homepage:
• Size: 88.9 KB
• Stars: 1
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## colab note

ref: https://medium.com/deep-learning-turkey/google-colab-free-gpu-tutorial-e113627b9f5d

### mount google drive in colab VM and download public git repo code

p.s. this mounting may be accessible by other session but is has expiration time

```

from google.colab import drive

drive.mount('/content/drive/') 

cd /content/drive/My\ Drive/Colab\ Notebooks/

!git clone https://github.com/grimmer0125/network-exp

cd network-exp

```

### upload data and trained weight via google drive web UI

Alternative way: 

```

from google.colab import files

uploaded = files.upload() 

```

But you need to move the files to the mounted google drive folder, otherwise the data is only accessable in the VM's lifetime. 

### enable GPU

```

Edit > Notebook settings or Runtime>Change runtime type and # select GPU as Hardware accelerator.

```

### start training

```

run -i classifier_from_little_data_script_3 

```

or

```

import fine_tune_model from classifier_from_little_data_script_3

fine_tune_model ()

```

### How to clone your git private repository from GitLab or Bitbucket

#### Step1: Generate a ssh key pair locally

Follow the guide: https://docs.gitlab.com/ee/ssh/. Assume `KEY_FILE_NAME` is the private key file name.

#### step2: setup your public ssl key on GitLab or bitbucket

- GitLab: Settings > Repository section by expanding the Deploy Key

- Bitbucket: Settings->Access keys

#### Step3: Prepare to git clone on colab 

**3-1: setup ssh hosts**

create the folder 

```

cd /root

mkdir .ssh

```

For Gitlab:

```

! ssh-keyscan gitlab.com >> /root/.ssh/known_hosts 

```

For Bicbucket:

`! ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts`

Use `!ssh -T [email protected]` to test if it is ok or not.

**3-2: Upload the ssh private key to colab**

Either use Google drive to upload ssh private key file to the mounted colab folder or using `uploaded = files.upload()` on colab.

Then `!chmod 600 KEY_FILE_NAME`

#### Step4: Use Git command to download the code on colab

`!GIT_SSH_COMMAND="ssh -i KEY_FILE_NAME -F /dev/null" git clone [email protected]:USER_NAME/REPO_NAME.git`

#### Step5: Use Git command to update code on colab

cd /content/drive/My\ Drive/Colab\ Notebooks/YOUR_GIT_PROJECT

!GIT_SSH_COMMAND="ssh -i ../id_ed25519 -F /dev/null" git fetch --all

!GIT_SSH_COMMAND="ssh -i ../id_ed25519 -F /dev/null" git reset --hard origin/develop