https://github.com/bethgelab/docker
Information and scripts to run and develop the Bethge Lab Docker containers
https://github.com/bethgelab/docker
Last synced: 7 months ago
JSON representation
Information and scripts to run and develop the Bethge Lab Docker containers
- Host: GitHub
- URL: https://github.com/bethgelab/docker
- Owner: bethgelab
- License: mit
- Created: 2016-02-05T13:32:43.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2018-03-10T09:40:21.000Z (over 7 years ago)
- Last Synced: 2024-08-10T14:13:38.758Z (10 months ago)
- Language: Makefile
- Homepage:
- Size: 35.2 KB
- Stars: 20
- Watchers: 18
- Forks: 6
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Outdated
THIS REPOSITORY AND THE DOCKER IMAGES LISTED HERE ARE **OUTDATED**. OUR NEW DEEPLEARNING DOCKER IMAGES AND SCRIPTS CAN BE FOUND AT https://github.com/bethgelab/docker-deeplearning.
# Docker Toolchain of the Bethge Lab
This repository includes utilities to build and run the Docker images of the [Bethge Lab](http://bethgelab.org/). The toolchain is composed of four different images (more details below):
* The [__xserver__](https://github.com/bethgelab/docker-xserver) image adds user-authentication and Xserver capabilities to a base Ubuntu-image. In particular, this images fixes a file permission problem: new files created from within the container are owned by root and thus conflict with user permissions.
* The [__jupyter-notebook__](https://github.com/bethgelab/docker-jupyter-notebook) image is a fork of the official [jupyter/notebook image](https://hub.docker.com/r/jupyter/notebook/) but is based on xserver.
* The [__jupyter-scipyserver__](https://github.com/bethgelab/docker-jupyter-scipyserver) image is based on jupyter-notebook and adds many python packages needed for scientific computing such as Numpy and Scipy (both compiled against OpenBlas), Theano, Lasagne, Pandas, Seaborn and more.
* The [__jupyter-deeplearning__](https://github.com/bethgelab/docker-jupyter-deeplearning) image is based on jupyter-scipyserver (including **Lasagne**) but adds some libraries such as **Caffe**, **Keras**, **Tensorflow**, Scikit-image, Joblib and others.
* The [__jupyter-torch__](https://github.com/bethgelab/docker-jupyter-torch) image is based on jupyter-deeplearning and adds Torch including FBLuaLib and loadcaffe.
All images come with different (or no) CUDA-libraries installed. Currently we support the following configurations:
* `ubuntu-14.04`: plain __Ubuntu 14.04__
* `cuda6.5`: Ubuntu 14.04 + __Cuda 6.5__
* `cuda7.0-cudnn2`: Ubuntu 14.04 + __Cuda 7.0 + CuDNN v2__
* `cuda7.0-cudnn3`: Ubuntu 14.04 + __Cuda 7.0 + CuDNN v3__
* `cuda7.0-cudnn4`: Ubuntu 14.04 + __Cuda 7.0 + CuDNN v4__
* `cuda7.5-cudnn5`: Ubuntu 14.04 + __Cuda 7.5 + CuDNN v5__
* `cuda8.0-cudnn5`: Ubuntu 14.04 + __Cuda 8.0 + CuDNN v5__
All images are readily available from [Docker Hub](https://hub.docker.com/u/bethgelab/). To pull for example the image *jupyter-deeplearning* with Cuda 8.0 and CuDNN v5 you would do
docker pull bethgelab/jupyter-deeplearning-x:cuda8.0-cudnn5
Available tags are *ubuntu-14.04*, *cuda6.5*, *cuda7.0-cudnn2*, *cuda7.0-cudnn3*, *cuda7.0-cudnn4*, *cuda7.5-cudnn5*, *cuda8.0-cudnn5*.
### AGMB Docker wrapper
To make the employment of the containers as painless as possible we have wrapped all important flags in the script ```nv-agmb-docker``` (see root directory of repo), which wraps ```nvidia-docker``` (https://github.com/NVIDIA/nvidia-docker). To run a container, first pull the image from Docker Hub (important - otherwise the CUDA version cannot be detected) before running the command
GPU=0 ./nv-agmb-docker run -d bethgelab/jupyter-deeplearning-x:cuda8.0-cudnn5
or equivalently for any other image or tag. This command has to be run in the folder in which the agmb-docker script was placed. The script takes care of setting up the NVIDIA host driver environment inside the Docker container, adds the current user, mounts his home-directory in which it finally starts the jupyter notebook. Some properties are specific to users within the AG Bethge lab, but as an external user one can override all settings. As the most stripped-down version, use
GPU=0 ./nv-agmb-docker run -e GROUPS=sudo -e USER_HOME=$HOME -d bethgelab/jupyter-deeplearning-x:cuda8.0-cudnn5
Note that all the usual docker flags can be given. In addition, some environmental variables have a special meaning
* ```USER``` -- The username that is added to the container
* ```USER_HOME``` -- The user's home directory location (default ```/gpfs01/bethge/home/$USER_NAME```)
* ```USER_ID``` -- The user ID for the new user
* ```USER_GROUPS``` -- The groups to which the user is added (default: sudo,bethgelab:1011,cin:1019); the first group will act as the primary group
* ```USER_ENCRYPTED_PASSWORD``` -- your user password (encrypted). To generate it: ```perl -e 'print crypt('"PASSWORD"', "aa"),"\n"' ```
GPUs are exported through a list of comma-separated IDs using the environment variable ```GPU```.
The numbering is the same as reported by ```nvidia-smi``` or when running CUDA code with ```CUDA_DEVICE_ORDER=PCI_BUS_ID```, it is however **different** from the default CUDA ordering.
#### Note
The above applies to the tags from CUDA 8.0 onwards. For older images (i.e. CUDA 7.5 and before) replace `nv-agmb-docker` by `agmb-docker`.
## xserver: LDAP, Xserver & OpenBLAS
This image is a modification and extension of a Dockerfile by [Alexander Ecker](https://github.com/aecker/docker). It enables the following features:
1. Using LDAP user within a Docker container (more precisely: emulates it by using a local user with the same uid).
2. Runs an X server.
3. SSH daemon, i.e. allows `ssh -X` to run GUI within the Docker container.
4. Installs OpenBLAS.
Note that one should not override the `CMD` in this image. If you need to execute additional programs when starting the container, add them to `/usr/local/bin/startup` as follows:
`RUN echo "./mycmd" >> /usr/local/bin/startup`
## jupyter-notebook
This image is a fork of the official [jupyter/notebook image](https://hub.docker.com/r/jupyter/notebook/) with some modifications to allow a shift of the base image from plain Ubuntu to our CUDA-enhanced ldap-xserver images. The Jupyter Notebook runs as the User and listens to port 8888. At runtime the container will initialize (and display) a port-forwarding between host and container, the choice can be overriden by setting the forward manually, e.g.
GPU=0 ./agmb-docker run -p 534:8888 -d bethgelab/jupyter-deeplearning:cuda7.0-cudnn3
The notebook can then be reached by
http://localhost:534
By default the notebook will start from user home.
## jupyter-scipyserver
This image is based on jupyter-notebook and adds the following packages to both Python 2.7 and Python 3.4:
* Numpy (compiled against OpenBLAS)
* Scipy (compiled against OpenBLAS)
* pandas
* scikit-learn
* matplotlib
* seaborn
* h5py
* yt
* sympy
* patsy
* ggplot
* statsmodels
* Theano (from master)
* Lasagne (from master)
* Bokeh
* mock
* pytest
## jupyter-deeplearning
This image is based on jupyter-scipyserver and adds **Caffe 0.14** (binaries by NVIDIA). In addition, the following packages are installed for Python 2.7:
* scikit-image
* h5py
* leveldb
* networkx
* joblib
* bloscpack
* keras
* **TensorFlow**
## jupyter-torch
This image is based on jupyter-deeplearning and adds **Torch** including FBLuaLib and loadcaffe.
## Trouble-Shooting
If you run into an "[[: not found" error upon executing the agmb-docker script, then you are invoking dash and not bash. To invoke bash explicitely, do
GPU=0 bash agmb-docker run -d bethgelab/jupyter-deeplearning:cuda7.0-cudnn3
# Issues and Contributing
* Please let us know by [filing a new issue](https://github.com/bethgelab/docker/issues/new)
* You can contribute by opening a [pull request](https://help.github.com/articles/using-pull-requests/)