Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/liuyuweitarek/downgrade-upgrade-colab-python
Creating a Specific Version of Python and PyTorch Environment on Colab.
https://github.com/liuyuweitarek/downgrade-upgrade-colab-python
customized-project google-colab python pytorch tensorflow
Last synced: about 5 hours ago
JSON representation
Creating a Specific Version of Python and PyTorch Environment on Colab.
- Host: GitHub
- URL: https://github.com/liuyuweitarek/downgrade-upgrade-colab-python
- Owner: liuyuweitarek
- License: mit
- Created: 2024-08-25T16:55:20.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2024-08-31T10:53:43.000Z (27 days ago)
- Last Synced: 2024-09-22T06:02:14.063Z (5 days ago)
- Topics: customized-project, google-colab, python, pytorch, tensorflow
- Language: Jupyter Notebook
- Homepage:
- Size: 43.9 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Creating a Specific Version of Python and PyTorch Environment on Colab
This document will guide free users in creating a specific version of Python, PyTorch and Tensorflow execution environment on Google Colab.
Based on the steps provided here, you can create any Python(version >= 3.7) environment in Google Colab.
**Noted that this method has limitations.**
With this method, cell cannot directly execute Python code from the environment you create. You need to activate the created environment at the start of each cell execution, which is equivalent to running your command in the terminal.
i.e. Cell will be like below to execute your code:
```
%%shell
eval "$(conda shell.bash hook)"
conda activate myenv
python main.py
```
## Prerequisite
### 1. Check CUDA versions
Please refer to [this table](https://www.tensorflow.org/install/source?hl=zh-tw#gpu) to find the Python Version you're using and the corresponding supported CUDA Version.
P.S. No matter you are using PyTorch or TensorFlow.
### 2. Check your package version
#### For Pytorch
Check the table below, and look for whether the package is in this [package source list](https://download.pytorch.org/whl/torch/).
For example:
- Python 3.7(cp37)
- PyTorch-1.7.1
- CUDA Version 10.1(cu101)
- and google colab is linux based system(linux_x86_64).
I then should found `torch-1.7.1+cu101-cp37-cp37m-linux_x86_64.whl` in the list, which cound be installed in command like:
```bash
$ python -m pip install torch==1.7.1+cu101 --extra-index-url https://download.pytorch.org/whl --no-cache-dir
```
Torch Version
Available CUDA Version
Python Version
2.0.1
cu117, cu118
cp38, cp39, cp310, cp311
2.0.0
cu117, cu118
cp38, cp39, cp310, cp311
1.13.1
cu116, cu117
cp37, cp38, cp39, cp310
1.13.0
cu116, cu117
cp37, cp38, cp39, cp310
1.12.1
cu113, cu116
cp37, cp38, cp39, cp310
1.12.0
cu113, cu116
cp37, cp38, cp39, cp310
1.11.0
cu113, cu115
cp37, cp38, cp39, cp310
1.10.2
cu102, cu111, cu113
cp36, cp37, cp38, cp39
1.10.1
cu102, cu111, cu113
cp36, cp37, cp38, cp39
1.10.0
cu102, cu111, cu113
cp36, cp37, cp38, cp39
1.9.1
cu102, cu111
cp36, cp37, cp38, cp39
1.9.0
cu102, cu111
cp36, cp37, cp38, cp39
1.8.1
cu101, cu102, cu111
cp36, cp37, cp38, cp39
1.8.0
cu101, cu111
cp36, cp37, cp38, cp39
1.7.1
cu101, cu110
cp36, cp37, cp38, cp39
1.7.0
cu101, cu110
cp36, cp37, cp38
1.6.0
cu101
cp36, cp37, cp38
1.5.1
cu92, cu101
cp35, cp36, cp37, cp38
1.5.0
cu92, cu101
cp35, cp36, cp37, cp38
1.4.0
cu92
cp35, cp36, cp37, cp38
1.3.1
cu92
cp35, cp36, cp37
1.3.0
cu92
cp35, cp36, cp37
1.2.0
cu92
cp35, cp36, cp37
#### For Tensorflow
Check out [the version table](https://www.tensorflow.org/install/source?hl=zh-tw#gpu).
For example,
- Python 3.7
- Tensorflow 2.1.0
At `requirements.txt`,
```
tensorflow-gpu==2.1.0
protobuf==3.20.1
```
### 3. Install cudnn + CUDA
Based on the `CUDA Version` you determined above, find the cudnn package to install:
For example, I chose CUDA 10.1 above, so after checkouting the table below, I download `cudnn-10.1-linux-x64-v7.6.5.32.tgz`.
I chose `v7.6.5.32` because many references use this version without any issues. However, in principle, you just need to select one that matches the version compatibility listed below.
Below are the download links for CUDA versions <= 11.
> _If you are using CUDA version >= 12, you might consider downloading it directly from the official website._
|Cudnn Version|Sources|
| --- | --- |
|v5 - v8.8.0| https://developer.download.nvidia.com/compute/redist/cudnn |
|v8.4.0 - v9.3.0| https://developer.download.nvidia.com/compute/cudnn/redist/cudnn/linux-x86_64 |
cuDNNCUDAcuDNN v8.4.0 (April 1st, 2022)CUDA 11.xcuDNN v8.4.0 (April 1st, 2022)CUDA 10.2cuDNN v8.3.3 (March 18th, 2022)CUDA 11.5cuDNN v8.3.3 (March 18th, 2022)CUDA 10.2cuDNN v8.3.2 (January 10th, 2022)CUDA 11.5cuDNN v8.3.2 (January 10th, 2022)CUDA 10.2cuDNN v8.3.1 (November 22nd, 2021)CUDA 11.5cuDNN v8.3.1 (November 22nd, 2021)CUDA 10.2cuDNN v8.3.0 (November 3rd, 2021)CUDA 11.5cuDNN v8.3.0 (November 3rd, 2021)CUDA 10.2cuDNN v8.2.4 (September 2nd, 2021)CUDA 11.4cuDNN v8.2.4 (September 2nd, 2021)CUDA 10.2cuDNN v8.2.2 (July 6th, 2021)CUDA 11.4cuDNN v8.2.2 (July 6th, 2021)CUDA 10.2cuDNN v8.2.1 (June 7th, 2021)CUDA 11.xcuDNN v8.2.1 (June 7th, 2021)CUDA 10.2cuDNN v8.2.0 (April 23rd, 2021)CUDA 11.xcuDNN v8.2.0 (April 23rd, 2021)CUDA 10.2cuDNN v8.1.1 (Feburary 26th, 2021)CUDA 11.0,11.1 and 11.2cuDNN v8.1.1 (Feburary 26th, 2021)CUDA 10.2cuDNN v8.1.0 (January 26th, 2021)CUDA 11.0,11.1 and 11.2cuDNN v8.1.0 (January 26th, 2021)CUDA 10.2cuDNN v8.0.5 (November 9th, 2020)CUDA 11.1cuDNN v8.0.5 (November 9th, 2020)CUDA 11.0cuDNN v8.0.5 (November 9th, 2020)CUDA 10.2cuDNN v8.0.5 (November 9th, 2020)CUDA 10.1cuDNN v8.0.4 (September 28th, 2020)CUDA 11.1cuDNN v8.0.4 (September 28th, 2020)CUDA 11.0cuDNN v8.0.4 (September 28th, 2020)CUDA 10.2cuDNN v8.0.4 (September 28th, 2020)CUDA 10.1cuDNN v8.0.3 (August 26th, 2020)CUDA 11.0cuDNN v8.0.3 (August 26th, 2020)CUDA 10.2cuDNN v8.0.3 (August 26th, 2020)CUDA 10.1cuDNN v8.0.2 (July 24th, 2020)CUDA 11.0cuDNN v8.0.2 (July 24th, 2020)CUDA 10.2cuDNN v8.0.2 (July 24th, 2020)CUDA 10.1cuDNN v8.0.1 RC2 (June 26th, 2020)CUDA 11.0cuDNN v8.0.1 RC2 (June 26th, 2020)CUDA 10.2cuDNN v7.6.5 (November 18th, 2019)CUDA 10.2cuDNN v7.6.5 (November 5th, 2019)CUDA 10.1cuDNN v7.6.5 (November 5th, 2019)CUDA 10.0cuDNN v7.6.5 (November 5th, 2019)CUDA 9.2cuDNN v7.6.5 (November 5th, 2019)CUDA 9.0cuDNN v7.6.4 (September 27, 2019)CUDA 10.1cuDNN v7.6.4 (September 27, 2019)CUDA 10.0cuDNN v7.6.4 (September 27, 2019)CUDA 9.2cuDNN v7.6.4 (September 27, 2019)CUDA 9.0cuDNN v7.6.3 (August 23, 2019)CUDA 10.1cuDNN v7.6.3 (August 23, 2019)CUDA 10.0cuDNN v7.6.3 (August 23, 2019)CUDA 9.2cuDNN v7.6.3 (August 23, 2019)CUDA 9.0cuDNN v7.6.2 (July 22, 2019)CUDA 10.1cuDNN v7.6.2 (July 22, 2019)CUDA 10.0cuDNN v7.6.2 (July 22, 2019)CUDA 9.2cuDNN v7.6.2 (July 22, 2019)CUDA 9.0cuDNN v7.6.1 (June 24, 2019)CUDA 10.1cuDNN v7.6.1 (June 24, 2019)CUDA 10.0cuDNN v7.6.1 (June 24, 2019)CUDA 9.2cuDNN v7.6.1 (June 24, 2019)CUDA 9.0cuDNN v7.6.0 (May 20, 2019)CUDA 10.1cuDNN v7.6.0 (May 20, 2019)CUDA 10.0cuDNN v7.6.0 (May 20, 2019)CUDA 9.2cuDNN v7.6.0 (May 20, 2019)CUDA 9.0cuDNN v7.5.1 (April 22, 2019)CUDA 10.1cuDNN v7.5.1 (April 22, 2019)CUDA 10.0cuDNN v7.5.1 (April 22, 2019)CUDA 9.2cuDNN v7.5.1 (April 22, 2019)CUDA 9.0cuDNN v7.5.0 (Feb 25, 2019)CUDA 10.1cuDNN v7.5.0 (Feb 21, 2019)CUDA 10.0cuDNN v7.5.0 (Feb 21, 2019)CUDA 9.2cuDNN v7.5.0 (Feb 21, 2019)CUDA 9.0cuDNN v7.4.2 (Dec 14, 2018)CUDA 10.0cuDNN v7.4.2 (Dec 14, 2018)CUDA 9.2cuDNN v7.4.2 (Dec 14, 2018)CUDA 9.0cuDNN v7.4.1 (Nov 8, 2018)CUDA 10.0cuDNN v7.4.1 (Nov 8, 2018)CUDA 9.2cuDNN v7.4.1 (Nov 8, 2018)CUDA 9.0cuDNN v7.3.1 (Sept 28, 2018)CUDA 10.0cuDNN v7.3.1 (Sept 28, 2018)CUDA 9.2cuDNN v7.3.1 (Sept 28, 2018)CUDA 9.0cuDNN v7.3.0 (Sept 19, 2018)CUDA 10.0cuDNN v7.3.0 (Sept 19, 2018)CUDA 9.0cuDNN v7.2.1 (August 7, 2018)CUDA 9.2cuDNN v7.1.4 (May 16, 2018)CUDA 9.2cuDNN v7.1.4 (May 16, 2018)CUDA 9.0cuDNN v7.1.4 (May 16, 2018)CUDA 8.0cuDNN v7.1.3 (April 17, 2018)CUDA 9.1cuDNN v7.1.3 (April 17, 2018)CUDA 9.0cuDNN v7.1.3 (April 17, 2018)CUDA 8.0cuDNN v7.1.2 (Mar 21, 2018)CUDA 9.1 & 9.2cuDNN v7.1.2 (Mar 21, 2018)CUDA 9.0cuDNN v7.0.5 (Dec 11, 2017)CUDA 9.1cuDNN v7.0.5 (Dec 5, 2017)CUDA 9.0cuDNN v7.0.5 (Dec 5, 2017)CUDA 8.0cuDNN v7.0.4 (Nov 13, 2017)CUDA 9.0cuDNN v6.0 (April 27, 2017)CUDA 8.0cuDNN v6.0 (April 27, 2017)CUDA 7.5cuDNN v5.1 (Jan 20, 2017)CUDA 8.0cuDNN v5.1 (Jan 20, 2017)CUDA 7.5cuDNN v5 (May 27, 2016)CUDA 8.0cuDNN v5 (May 12, 2016)CUDA 7.5cuDNN v4 (Feb 10, 2016)CUDA 7.0 and later.cuDNN v3 (September 8, 2015)CUDA 7.0 and later.cuDNN v2 (March 17,2015)CUDA 6.5 and later.cuDNN v1 (cuDNN 6.5 R1)
## Get started
As proof as concept, I Use Miniconda create a Python 3.7 environment, which can use both PyTorch 1.7.1 and tensorflow 2.1.0 version here.
1. Clone the project
```bash
git clone https://github.com/liuyuweitarek/downgrade-upgrade-colab-python.git custom_env_colab
```
If have a better approach, feel free to share it with me or submit a PR. Thank you!
2. Install `cudnn-10.1-linux-x64-v7.6.5.32.tgz` from [NVIDIA website](https://developer.download.nvidia.com/compute/redist/cudnn/v7.6.5/cudnn-10.1-linux-x64-v7.6.5.32.tgz). Here is the [backup](https://drive.google.com/file/d/1R54KoKk16CcpFu3tR80dEyC_aB8UOyXc/view?usp=sharing). Place in the same folder as `custom_env_colab`.
Further,
3. Place the project in your Google Drive.
4. Follow the instructions in the `Creating Specific Version of Python and PyTorch Environment on Colab.ipynb` notebook.
## Q&A and Notes
1. How to deal with Colab time limit?
This document does not overcome the time limit issue for free users, who will need to wait for a specific period before using it again. Therefore, please **make good use of Checkpoint to save and continue training progress**.
2. Please use the version of Miniconda installer which is higher than the Python version you wish to use to create the virtual environment.
Furthermore,