https://github.com/ucphhpc/jupyter_service
A docker swarm of Jupyter Notebooks
https://github.com/ucphhpc/jupyter_service
docker-swarm-cluster jupyterhub-deployment jupyterhub-swarmspawner-deployment
Last synced: 5 months ago
JSON representation
A docker swarm of Jupyter Notebooks
- Host: GitHub
- URL: https://github.com/ucphhpc/jupyter_service
- Owner: ucphhpc
- Created: 2017-10-31T09:10:36.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2025-09-11T12:34:28.000Z (10 months ago)
- Last Synced: 2025-09-11T15:46:46.559Z (10 months ago)
- Topics: docker-swarm-cluster, jupyterhub-deployment, jupyterhub-swarmspawner-deployment
- Language: Python
- Homepage:
- Size: 271 KB
- Stars: 5
- Watchers: 1
- Forks: 2
- Open Issues: 3
-
Metadata Files:
- Readme: README.rst
Awesome Lists containing this project
README
===============
jupyter_service
===============
A jupyter notebook docker swarm setup that consists of a customised JupyterHub service that spawns individual notebooks for
individual users.
- `UCPHHPC Jupyterhub `_
------------
Architecture
------------
The stack is made of a 2 layered docker swarm stack, i.e. any external
request is received by the JupyterHub proxy service, which redirects requests to the JupyterHub service where an authenticated user is allowed to start a notebook.
The authentication is defined by `Authenticators `_ where Jupyterhub allows for a custom
authenticator to be selected based on the local requirements.
Hence how a user should be authenticated before they are able to launch notebooks via the JupyterHub web interface.
The authenticator itself is selected by defining the ``authenticator_class`` variable as shown in
the ``example/non_ssl/basic_jupyterhub_config.py`` configuration file.
Beyond authentication, JupyterHub also allows for a custom `Spawner `_
scheme to be overloaded.
The default ``spawner_class`` in the ``example/non_ssl/basic_jupyterhub_config.py`` configuration file
is defined with the `jhub-swarmspawner `_ which enables the deployment of
jupyter notebooks on a `Docker Swarm Cluster `_
cluster whenever a user requests a new notebook.
-------------
Prerequisites
-------------
As a prerequisite the JupyterHub service is able to launch separate notebook services,
JupyterHub needs access to the hosts docker daemon process. This access can
be gained in a number of ways, one of which is to mount the ``/var/run/docker
.sock`` file inside the JupyterHub service as a volume and then ensuring that
the user that executes the ``deploy`` command is part of the ``docker`` system
group. This is the default approach as defined in the docker-compose.yml file.
Another approach would be to expose the docker daemon remotely on port 2376
with TLS verification as explained @ `Docker Docs `_ under "Daemon
socket option".
In addition it requires that the JupyterHub service is deployed on a swarm manager node.
See `Create a swarm `_.
Hence the restriction set in the docker-compose file that the JupyterHub service is restricted to a manager node.
Upon launching the ``example/basic_docker-compose.yml`` stack also provides an `docker-image-updater `_ service.
This service provides a continuously monitor whether new versions of the specified notebook image is available,
and if so pulls it to every swarm node and prunes previous versions when no other running notebook depends on that particular version.
-------------
Configuration
-------------
Before the stack can be deployed, the ``docker-compose.yml``, ``jupyterhub_config.py``, and ``.env`` files needs to be prepared. This can be achived by using the ``make init`` target to generate the nessesary environment for your deployment.
By default, the Makefile will select the non-SSL configuration in the ``example/non_ssl`` directory, and the `defaults.env` environment file will be produced and linked by the `.env` file.
This non-SSL version is configured to publish the JupyterHub service on port 8080. Furthermore, the example JupyterHub configuration file examples in
`example/*` uses an **insecure default password ``dummy`` that should not be used in a production environment**. This can be changed by altering the
following option in the default ``example/non_ssl/basic_jupyterhub_config.py`` file before it is copied into the ``hub/jupyterhub`` directory before calling the ``make init`` target.::
# Authenticator
c.JupyterHub.authenticator_class = "jhubauthenticators.DummyAuthenticator"
c.DummyAuthenticator.password = "dummy"
Once the selected ``jupyterhub_config.py`` file has been properly prepared, the configuration can be prepared by calling::
make init
An alternative to this is to setup the ``docker-compose.yml`` and ``hub/jupyterhub/jupyterhub_config.py`` files yourself before launching the stack.
Finally, ``make clean`` can be called if you want to remove the various configuration and environment files that have been prepared by the ``init`` call.
---------------------
Launching the Service
---------------------
Once the Configuration has been completed, the service can be launched by executing the following command inside the repository directory::
make daemon
To verify that the stack is now deployed and the services are being spawned
do::
docker stack ls
docker service ls
The ``stack`` command should return that the jupyter-service stacks is running with 3 services, i.e. the JupyterHub, the proxy, and the image-updater services.