Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/skai-x/elastic-jupyter-operator
Cloud-native way to provide elastic Jupyter Notebooks on Kubernetes. Run remote kernels, natively.
https://github.com/skai-x/elastic-jupyter-operator
cloud-native jupyter jupyter-kernels jupyter-notebook kubernetes
Last synced: about 2 months ago
JSON representation
Cloud-native way to provide elastic Jupyter Notebooks on Kubernetes. Run remote kernels, natively.
- Host: GitHub
- URL: https://github.com/skai-x/elastic-jupyter-operator
- Owner: skai-x
- License: apache-2.0
- Created: 2021-02-26T08:18:14.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2022-03-24T06:27:01.000Z (over 2 years ago)
- Last Synced: 2024-05-02T01:52:06.387Z (5 months ago)
- Topics: cloud-native, jupyter, jupyter-kernels, jupyter-notebook, kubernetes
- Language: Go
- Homepage:
- Size: 1.23 MB
- Stars: 186
- Watchers: 12
- Forks: 27
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# elastic-jupyter-operator
Elastic Jupyter Notebooks on Kubernetes
## Motivation
Jupyter is a free, open-source, interactive web tool known as a computational notebook, which researchers can use to combine software code, computational output, explanatory text, and multimedia resources in a single document.
For data scientists and machine learning engineers, Jupyter has emerged as a de facto standard. At the same time, there has been growing criticism that the way notebooks are being used leads to low resource utilization.
GPU and other hardware resources will be bound to the specified notebooks even if the data scientists do not need them currently. This project proposes some Kubernetes CRDs to solve these problems.
## Introduction
elastic-jupyter-operator provides elastic Jupyter notebook services with these features:
- Provide users the out-of-box Jupyter notebooks on Kubernetes.
- Autoscale Jupyter kernels when the kernels are not used within the given time frame to increase the resource utilization.
- Customize the kernel configuration in runtime without restarting the notebook.
Figure 1. elastic-jupyter-operator
Figure 2. Other Jupyter on Kubernetes solutions
## Deploy
```bash
kubectl apply -f ./hack/enterprise_gateway/prepare.yaml
make deploy
```
## Quickstart
You can follow the [quickstart](./docs/quick-start.md) to create the notebook server and kernel in Kubernetes like this:
```bash
NAME READY STATUS RESTARTS AGE
jovyan-fd191444-b08c-4668-ba4e-3748a54a0ac1-5789574d66-tb5cm 1/1 Running 0 146m
jupytergateway-sample-858bbc8d5c-xds44 1/1 Running 0 3h46m
jupyternotebook-sample-5bf7d9d9fb-pdv9b 1/1 Running 10 77d
```
There are three pods running in the demo:
- `jupyternotebook-sample-5bf7d9d9fb-pdv9b` is the notebook server
- `jupytergateway-sample-858bbc8d5c-xds44` is the jupyter gateway to support remote kernels
- `jovyan-fd191444-b08c-4668-ba4e-3748a54a0ac1-5789574d66-tb5cm` is the remote kernel
The kernel will be deleted if the notebook does not use it in 10 mins. And it will be recreated if there is any new run in the notebook.
## Community
Please join [![Discord][discord-badge]][discord-url]
[discord-badge]: https://img.shields.io/discord/913359799058587658?logo=Discord&style=flat-square
[discord-url]: https://discord.gg/NJsd4guhPM
## Design
Please refer to [design doc](docs/design.md)
## API Documentation
Please refer to [API doc](docs/api/generated.asciidoc)
## Special Thanks
- [jupyter/enterprise_gateway](https://github.com/jupyter/enterprise_gateway) which implements the logic to run kernels remotely