{"id":28547851,"url":"https://github.com/rs-python/rs-demo","last_synced_at":"2026-04-23T10:01:45.361Z","repository":{"id":265455470,"uuid":"727596852","full_name":"RS-PYTHON/rs-demo","owner":"RS-PYTHON","description":"Collection of Jupyter notebooks showcasing the latest features of the system","archived":false,"fork":false,"pushed_at":"2026-01-26T10:25:25.000Z","size":8688,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":0,"default_branch":"develop","last_synced_at":"2026-01-26T23:48:12.596Z","etag":null,"topics":["jupyter-notebook","software"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RS-PYTHON.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-12-05T07:24:30.000Z","updated_at":"2026-01-26T10:25:27.000Z","dependencies_parsed_at":"2025-02-11T14:27:50.442Z","dependency_job_id":"c70ce8af-3654-45be-81b0-710e3b0e4bf0","html_url":"https://github.com/RS-PYTHON/rs-demo","commit_stats":null,"previous_names":["rs-python/rs-demo"],"tags_count":30,"template":false,"template_full_name":null,"purl":"pkg:github/RS-PYTHON/rs-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RS-PYTHON%2Frs-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RS-PYTHON%2Frs-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RS-PYTHON%2Frs-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RS-PYTHON%2Frs-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RS-PYTHON","download_url":"https://codeload.github.com/RS-PYTHON/rs-demo/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RS-PYTHON%2Frs-demo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28914943,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T12:13:43.263Z","status":"ssl_error","status_checked_at":"2026-01-30T12:13:22.389Z","response_time":66,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jupyter-notebook","software"],"created_at":"2025-06-10T01:07:54.178Z","updated_at":"2026-04-23T10:01:45.348Z","avatar_url":"https://github.com/RS-PYTHON.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Running Modes\n\nIn this page, we will see how to run the Jupyter notebooks on cluster, local and hybrid mode.\n\n### Quick links\n\n  * On cluster mode:\n\n      * JupyterHub: \u003chttps://processing.ops.rs-python.eu/jupyter\u003e\n      * RS-Server website (Swagger/OpenAPI): \u003chttps://rspy.ops.rs-python.eu/docs\u003e\n      * Create an API key: \u003chttps://apikeymanager.rspy.ops.rs-python.eu/docs#/Manage%20API%20keys/create_api_key_auth_api_key_new_get\u003e\n      * Prefect dashboard (orchestrator): \u003chttps://processing.ops.rs-python.eu/\u003e\n      * Grafana (logs, traces, metrics): \u003chttps://monitoring.ops.rs-python.eu/\u003e\n\n  * On hybrid mode:\n\n      * RS-Server website (Swagger/OpenAPI): \u003chttps://rspy.ops.rs-python.eu/docs\u003e\n      * Create an API key: \u003chttps://apikeymanager.rspy.ops.rs-python.eu/docs#/Manage%20API%20keys/create_api_key_auth_api_key_new_get\u003e\n      * Prefect dashboard (orchestrator): \u003chttp://localhost:4200\u003e\n\n  * On local mode:\n\n      * RS-Server website (Swagger/OpenAPI):\n\n          * \u003chttp://localhost:8000/docs\u003e (frontend, only for visualization, not functional)\n          * \u003chttp://localhost:8001/docs\u003e (auxip)\n          * \u003chttp://localhost:8002/docs\u003e (cadip)\n          * \u003chttp://localhost:8003/catalog/api.html\u003e (catalog)\n          * \u003chttp://localhost:8005/docs\u003e (prip)\n\n      * Prefect dashboard (orchestrator): \u003chttp://localhost:4200\u003e\n      * SeaweedFS s3 bucket: \u003chttp://localhost:9101/object-store/buckets\u003e with:\n\n          * Username: `seaweedfs`\n          * Password: `Strong#Pass#1234`\n\n## Prefect and Dask\n\n### EOPF/DPR\n\n![Prefect and Dask EOPF](./doc/images/prefect-dask-eopf.drawio.png \"Prefect and Dask EOPF\")\n\nWhen calling EOPF (DPR) with Prefect and Dask:\n\n  1. The **client** (Jupyter notebook, Prefect dashboard or terminal) runs a **Prefect flow**\n  (implemented as a Python function) on the **Prefect workers** on the Kubernetes cluster.\n  1. The flow calls the **tasks** (implemented as Python functions) on the **Dask workers**\n  on the Kubernetes cluster.\n  1. The tasks call the **EOPF Python functions** that are installed as a Python package (wheel) on the Dask pods.\n\n### Staging\n\n![Prefect and Dask staging](./doc/images/prefect-dask-staging.drawio.png \"Prefect and Dask staging\")\n\nWhen calling the staging with Prefect and Dask:\n\n  1. The **client** (Jupyter notebook, Prefect dashboard or terminal) runs a **Prefect flow**\n  (implemented as a Python function) on the **Prefect workers** on the Kubernetes cluster.\n  1. The flow makes **HTTP requests** to the **rs-server-staging** web service on the Kubernetes cluster.\n  1. The service calls the **tasks** (implemented as Python functions) on the **Dask workers**\n  on the Kubernetes cluster.\n  1. The tasks call the **rs-server-staging Python functions** that are installed as a Python package (wheel)\n  on the Dask pods.\n\n## Run on cluster mode\n\nOn cluster mode, we run the Jupyter notebooks from our JupyterHub session deployed on the cluster. They connect to the services deployed on the RS-Server website (=cluster). Authentication is required for this mode.\n\n### Prerequisites\n\n* You have access to JupyterHub: \u003chttps://processing.ops.rs-python.eu/jupyter\u003e\n* You have access to the RS-Server website: \u003chttps://rspy.ops.rs-python.eu/docs\u003e\n* You have generated an API key from the RS-Server website.\n\n### Initialize the Prefect blocks\n\nBefore the first use, you need to initialize the Prefect block that contains the\nenvironment variables shared for all users between Jupyter, Prefect and Dask.\n\nRun this command line to get the existing values, if any:\n```python\n# From terminal\nprefect block inspect \"secret/env-vars\"\n\n# Or from python\nimport json\nfrom prefect.blocks.system import Secret\nprint(json.dumps(Secret.load(\"env-vars\", _sync=True).get(), indent=2))\n```\n\nRun this Python code from any Jupyter notebook to write the new values:\n\n```python\nimport os\nfrom prefect.blocks.system import Secret\n\nvalue = {\n    # S3 bucket name and subfolder.\n    # NOTE: the \"share-bucket\" block will be created automatically from these\n    # variables. So if you change these variables, please also remove the\n    # \"share-bucket\" block and it will be recreated.\n    \"PREFECT_BUCKET_NAME\": \"rs-dev-cluster-temp\",\n    \"PREFECT_BUCKET_FOLDER\": \"prefect-share\",\n\n    # Token that was used to setup the Dask clusters.\n    # See: https://gateway.dask.org/authentication.html#using-jupyterhub-s-authentication\n    \"JUPYTERHUB_API_TOKEN\": \"\u003cyour-token-value\u003e\",\n\n    # Needed to run the performance indicator prefect flow\n    # The values for the following fields should be taken from rs-infra-core inventory,\n    # file rs-infra-core/inventory/sample/host_vars/setup/apps.yml.\n    # There is a section named rs_performance_indicator. The values for the fields\n    # are set at the cluster deployment. These values should be also used here\n    # Here is the aforementioned section:\n    # rs_performance_indicator:\n    #  database:\n    #    host: postgresql-cluster-rw.database.svc.cluster.local\n    #    name: performance\n    #    password: test\n    #    username: test\n    #    secret: pi-database-password\n    \"POSTGRES_HOST\": \"\u003ccluster_postgres_host\u003e\", # default: \"postgresql-cluster-rw.database.svc.cluster.local\",\n    \"POSTGRES_USER\": \"\u003cpi_postgres_user\u003e\",\n    \"POSTGRES_PASSWORD\": \"\u003cpi_postgres_password\u003e\",\n    \"POSTGRES_PORT\": \"\u003ccluster_postgres_port\u003e\", # normally, 5432\n    \"POSTGRES_PI_DB\": \"performance\",\n    # osam url, internal to the cluster\n    \"RSPY_HOST_OSAM\": \"http://rs-server-osam.processing.svc.cluster.local:8080\",\n}\n\n# Jupyter env vars to pass to Prefect and Dask\nfor env in [\n    \"RSPY_UAC_CHECK_URL\",\n    \"RSPY_WEBSITE\",\n    \"TEMPO_ENDPOINT\",\n    \"DASK_GATEWAY_PUBLIC\",\n    \"DASK_GATEWAY_ADDRESS\",\n]:\n    value[env] = os.environ[env]\n\n# Save Prefect block\nawait Secret(value=value).save(\"env-vars\", overwrite=True)\n```\n\nFrom a bash Terminal in Jupyter, check your values with:\n```bash\n# View all configured blocks\nprefect block ls\n\n# Displays details about the configured blocks\nprefect block inspect secret/auth\n```\n\n### Run the demos on cluster mode\n\n* Open a JupyterHub session.\n* Open a terminal, check that `rs-client-libraries` is installed by running:\n\n    ```shell\n    pip show rs-client-libraries # should show the name, version, ...\n    ```\n\n* On the left, in the file explorer, go to the demos or tutorial folder and double-click a notebook to open it:\n\n![Jupyter Notebook](./doc/images/jupyter.png \"Jupyter Notebook\")\n\n### Install a new `rs-client-libraries` version\n\n#### Option 1: update the JupyterHub image (affects everyone)\n\n1. Verify in the CI/CD that the last `rs-client-libraries` modifications were merged into the `develop` branch: \u003chttps://github.com/RS-PYTHON/rs-client-libraries/actions/workflows/publish-binaries.yml\u003e\n\n1. Ask the `rs-infrastructure` administrator to run a new CI/CD workflow to publish this `rs-client-libraries` version into a new JupyterHub image.\n\n#### Option 2: from a wheel package (affects only you)\n\n1. In the CI/CD, click on the last `rs-client-libraries` branch workflow that you want to use: \u003chttps://github.com/RS-PYTHON/rs-client-libraries/actions/workflows/publish-binaries.yml\u003e, go to the `Artifacts` section and download the `.whl` package file.\n\n1. Or alternatively, build the wheel yourself from your local `rs-client-libraries` project by running: `poetry build --format wheel`\n\n1. Upload the `.whl` package file to your JupyterHub session, open a Terminal and run:\n\n    ```shell\n    # Uninstall the old version. Note: this fails if we do it for the first time because\n    # we try to uninstall the root installation of the library, but this this OK.\n    pip install pip-autoremove\n    pip pip-autoremove -y rs-client-libraries 2\u003e/dev/null\n\n    # You may have conflicts between dependencies installed for the root user\n    # and the current user. You can uninstall all current user dependencies with:\n    # for dep in $(pip freeze | cut -d \"@\" -f1); do pip uninstall -y $dep 2\u003e/dev/null; done\n\n    # The old rs-client-libraries version is still installed for the root user.\n    # This is the version you (=current user) use by default.\n    pip show rs-client-libraries | grep Location # should display: /opt/conda/lib/python3.x/site-packages\n\n    # Install the new version for the current user.\n    pip install rs_client_libraries-\u003cversion\u003e-py3-none-any.whl\n    opentelemetry-bootstrap -a install\n    ```\n\n## Run on local mode\n\nOn local mode, docker-compose and Docker images are used to run services and libraries locally (not on a cluster). There is no authentication for this mode.\n\n### Prerequisites\n\n  * You have Docker installed on your system, see: \u003chttps://docs.docker.com/engine/install/\u003e\n  * You have access to the RSPY project on GitHub: \u003chttps://github.com/RS-PYTHON\u003e\n  * You have created a personal access token (PAT) on GitHub: \u003chttps://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens\u003e\n\n    * This access token is used to retrieve the rs-server product on the github repository.\n    * You may want to create a classic PAT with the ```read:packages``` permissions.\n\n  * You have checked out this git project:\n\n    ```shell\n    git clone https://github.com/RS-PYTHON/rs-demo.git\n\n    # Get last version\n    cd rs-demo\n    git checkout develop\n    ```\n\n### Run the demos on local mode\n\nTo pull the latest Docker images, run:\n\n```shell\n# Login into the project ghcr.io (GitHub Container Registry)\n# Username: your GitHub login\n# Password: your personal access token (PAT) created above\ndocker login https://ghcr.io/v2/rs-python\n\n# From the local-mode directory, pull the images\ncd ./local-mode\ndocker compose pull\n```\n\nThen to run the demos:\n\n```shell\n# Still from the local-mode directory, if you're not there yet\ncd ./local-mode\n\n# Run all services.\n# Note: in case of port conflicts, you can kill all your running docker containers with:\n# docker rm -f $(docker ps -aq)\ndocker compose down -v; docker compose up # -d for detached\n\n# Note: we always need to call 'down' before 'up' or we'll have errors\n# when the stac database will initialize a second time.\n```\n\nNear the end of the logs you will see some Jupyter information e.g:\n```\njupyter | To access the server, open this file in a browser:\njupyter |     ...\njupyter | Or copy and paste one of these URLs:\njupyter |     ...\njupyter |     http://127.0.0.1:8888/lab?token=612cb124335d9ab80a5a6414631a7df186b2401234050001\n```\n\nOpen (ctrl-click) the ```http://127.0.0.1:8888/lab?token=...``` link to open the Jupyter web client (=Jupyter Notebook) in your browser.\n\n__Note__: the token is auto-generated by Jupyter and changes everytime you relaunch the containers. So after relaunching, your old Jupyter web session won't be available anymore.\n\nTo show the Jupyter logs from another terminal, run:\n\n```shell\ndocker compose logs jupyter\n```\n\nOn the left, in the file explorer, go to the demos or tutorial folder and double-click a notebook to open it:\n\n![Jupyter Notebook](./doc/images/jupyter.png \"Jupyter Notebook\")\n\n```shell\n# When you're done, shutdown all services and volumes (-v)\n# with Ctrl-C (if not in detached mode i.e. -d) then:\ndocker compose down -v\n\n# You can use this to remove all docker volumes\n# (use with care if you have other docker containers)\ndocker volume prune\n```\n\n### How does it work\n\nThe [docker-compose.yml](local-mode/docker-compose.yml) file uses Docker images to run all the necessary container services for the demos :\n\n  * The latest rs-server images available:\n    * Built from the CI/CD: \u003chttps://github.com/RS-PYTHON/rs-server/actions/workflows/publish-binaries.yml\u003e\n    * Available in the ghcr.io: \u003chttps://github.com/orgs/RS-PYTHON/packages\u003e\n  * The AUXIP, CADIP ... station mockups:\n    * Built from the CI/CD: \u003chttps://github.com/RS-PYTHON/rs-testmeans/actions/workflows/publish-docker.yml\u003e\n    * Also available in the ghcr.io\n  * STAC PostgreSQL database\n  * SeaweedFS S3 bucket server\n  * Jupyter server\n\nThese containers are run locally (not on a cluster). The Jupyter notebooks accessed from \u003chttp://127.0.0.1:8888\u003e are run from the containerized Jupyter server, not from your local environment. This Jupyter environment contains all the Python modules required to call the rs-server HTTP endpoints.\n\n### How to run your local rs-server code in this environment\n\nIt can be helpful to use your last rs-server code version to debug it or to test modifications without pushing them and rebuilding the Docker image. Follow these steps:\n\n1. Go to the ```local-mode``` directory and run:\n\n    ```shell\n    cp 'docker-compose.yml' 'docker-compose-debug.yml'\n    ```\n\n1. If your local `rs-server` github repository is under `/my/local/rs-server`, modify the `docker-compose-debug.yml` file to mount your local `rs-server` services:\n\n    ```yaml\n    # e.g.\n    rs-server-adgs:\n      # ...\n      volumes:\n        - /my/local/rs-server/services/common/rs_server_common:/usr/local/lib/python3.x/site-packages/rs_server_common\n        - /my/local/rs-server/services/adgs/rs_server_adgs:/usr/local/lib/python3.x/site-packages/rs_server_adgs\n        - /my/local/rs-server/services/adgs/config:/usr/local/lib/python3.x/site-packages/config\n        # - and any other useful files ...\n    ```\n\n1. Run the demo with:\n\n    ```shell\n    # Still from the local-mode directory, if you're not there yet\n    cd ./local-mode\n\n    # Run all services\n    docker compose down -v; docker compose -f docker-compose-debug.yml up # -d for detached\n    ```\n\n## Run on hybrid mode\n\nOn hybrid mode, we run the Jupyter notebooks locally, but they connect to the services deployed on the RS-Server website (=cluster). Authentication is required for this mode.\n\n### Prerequisites\n\n* You have access to the RS-Server website: \u003chttps://rspy.ops.rs-python.eu/docs\u003e\n* You have generated an API key from the RS-Server website.\n* Python is installed on your system.\n\nYou also need the rs-client-libraries project:\n\n  * If you have its source code, install it with:\n\n    ```shell\n    cd /path/to/rs-client-libraries\n    pip install poetry\n    poetry install --with dev,demo\n    poetry run opentelemetry-bootstrap -a install\n    ```\n\n  * Or if you only have its ```.whl``` package, install it with:\n\n    ```shell\n    pip install rs_client_libraries-*.whl\n    # then install jupyter lab\n    pip install jupyterlab\n    ```\n\n### Run the demos on hybrid mode\n\nFrom your terminal in the rs-demo, run:\n\n```shell\n# NOTE: at CS France premises, use this to deactivate the proxy which causes random errors\nunset no_proxy ftp_proxy https_proxy http_proxy\n\n# To use your local rs-client-libraries source code\ncd /path/to/rs-client-libraries\n# git checkout develop \u0026\u0026 git pull # maybe take the latest default branch\npoetry run /path/to/rs-demo/hybrid-mode/start-jupyterlab.sh\n\n# Or if you have installed it from rs_client_libraries-*.whl,\n# just run\n/path/to/rs-demo/hybrid-mode/start-jupyterlab.sh\n```\n\nThe Jupyter web client (=Jupyter Notebook) opens in a new tab of your browser.\n\n*WARNING*: the cluster is shut down from 18h30 to 8h00 each night and on the weekends.\n\n### How to check your Python interpreter used in notebooks\n\nIn a notebook cell, run:\n```python\nimport sys\nprint(sys.executable)\n```\n\nIf you use the rs-client-libraries poetry environment, it should show something like:\n```shell\n${HOME}/.cache/pypoetry/virtualenvs/rs-client-libraries-xxxxxxxx-py3.x/bin/python\n```\n\n## Licensing\n\nThe code in this project is licensed under Apache License 2.0.\n\n---\n\n![](https://raw.githubusercontent.com/RS-PYTHON/.github/refs/heads/main/profile/banner_logo.jpg)\n\nThis project is funded by the EU and ESA.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frs-python%2Frs-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frs-python%2Frs-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frs-python%2Frs-demo/lists"}