{"id":18304045,"url":"https://github.com/googleclouddataproc/jupyterhub-dataprocspawner","last_synced_at":"2025-04-05T15:31:06.392Z","repository":{"id":39717473,"uuid":"201310047","full_name":"GoogleCloudDataproc/jupyterhub-dataprocspawner","owner":"GoogleCloudDataproc","description":null,"archived":false,"fork":false,"pushed_at":"2022-05-27T01:04:56.000Z","size":410,"stargazers_count":14,"open_issues_count":7,"forks_count":11,"subscribers_count":30,"default_branch":"main","last_synced_at":"2025-03-21T06:33:26.010Z","etag":null,"topics":["google-cloud-dataproc"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudDataproc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-08T17:53:48.000Z","updated_at":"2023-01-20T17:08:50.000Z","dependencies_parsed_at":"2022-09-20T08:51:08.677Z","dependency_job_id":null,"html_url":"https://github.com/GoogleCloudDataproc/jupyterhub-dataprocspawner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fjupyterhub-dataprocspawner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fjupyterhub-dataprocspawner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fjupyterhub-dataprocspawner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fjupyterhub-dataprocspawner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudDataproc","download_url":"https://codeload.github.com/GoogleCloudDataproc/jupyterhub-dataprocspawner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247358531,"owners_count":20926236,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google-cloud-dataproc"],"created_at":"2024-11-05T15:27:39.118Z","updated_at":"2025-04-05T15:31:05.113Z","avatar_url":"https://github.com/GoogleCloudDataproc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DataprocSpawner\n\nDataprocSpawner enables [JupyterHub][jupyterhub] to spawn single-user [jupyter_notebooks][Jupyter notebooks] that run on [Dataproc][dataproc] clusters. This provides users with ephemeral clusters for data science without the pain of managing them.\n\n- [Product Documentation][dataproc]\n- DISCLAIMER: DataprocSpawner only supports zonal DNS names. If your project uses global DNS names, click [this][dns] for instructions on how to migrate.\n\n[jupyterhub]: https://jupyterhub.readthedocs.io/en/stable/\n[jupyter_notebooks]: https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/what_is_jupyter.html\n[dataproc]: https://cloud.google.com/dataproc\n[dns]: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal\n\nSupported Python Versions: Python \u003e= 3.6\n\n## Before you begin\n\nIn order to use this library, you first need to go through the following steps:\n\n1. [Select or create a Cloud Platform project][create_project]\n2. [Enable billing for your project][enable_billing]\n3. [Enable the Google Cloud Dataproc API][enable_api]\n4. [Setup Authentication][authentication]\n\n[create_project]: https://console.cloud.google.com/project\n[enable_billing]: https://cloud.google.com/billing/docs/how-to/modify-project#enable_billing_for_a_project\n[enable_api]: https://cloud.google.com/dataproc\n[authentication]: https://cloud.google.com/docs/authentication/getting-started#auth-cloud-implicit-python\n\n## Installation example\n\n### Locally\n\nTo try is locally for development purposes. From the root folder:\n\n```sh\nchmod +x deploy_local_example.sh\n./deploy_local_example.sh \u003cYOU_PROJECT_ID\u003e \u003cYOUR_GCS_CONFIG_LOCATIONS\u003e \u003cYOUR_AUTHENTICATED_EMAIL\u003e\n```\n\nThe script will start a local container image and authenticate it using your local credentials.\n\nNote: Although you can try the Dataproc Spawner image locally, you might run into networking communication problems.\n\n### Google Compute Engine\n\nTo try it out in the Cloud, the quickest way is to to use a test Compute Engine instance. The following takes you through the process.\n\n1. Set your working project\n\n    ```bash\n    PROJECT_ID=\u003cYOUR_PROJECT_ID\u003e\n    VM_NAME=vm-spawner\n    ```\n\n1. Run the example script which:\n\n    a. Creates a Dockerfile\n    b. Creates a jupyter_config.py example file that uses a dummy authenticator.\n    c. Deploy a Docker image of the JupyterHub spawner in [Google Container Registry][gcr]\n    d. Create a container-based Compute Engine\n    e. Returns the IP of the instance that runs JupyterHub.\n\n    ```bash\n    bash deploy_gce_example.sh ${PROJECT_ID} ${VM_NAME}\n    ```\n\n1. After the script finishes, you should see an IP displayed. You can use that IP to access your setup at `\u003cIP\u003e:8000`. You might have to wait for a few minutes until the container is deployed on the instance.\n\n[gcr]: https://cloud.google.com/container-registry/\n\n## Troubleshooting\n\nTo troubleshoot\n\n1. ssh into the VM:\n\n    ```bash\n    gcloud compute ssh ${VM_NAME}\n    ```\n\n1. From the VM console, install some useful tools:\n\n    ```bash\n    apt-get update\n    apt-get install vim\n    ```\n\n1. From the VM console, you can:\n\n    - List the running containers with `docker ps`\n    - Display container logs `docker logs -f \u003cCONTAINER_ID\u003e`\n    - Execute code in the container `docker exec -it \u003cCONTAINER_ID\u003e /bin/bash`\n    - Restart the container for changes to take effect  `docker restart \u003cCONTAINER_ID\u003e`\n\n## Notes\n\n- DataprocSpawner defaults to port 12345, the port can be set within `jupyterhub_config.py`. More info in JupyterHub's [jupyterhub_documentation].\n\n    c.Spawner.port = {port number}\n\n- The region default is ``us-central1`` for Dataproc clusters. The zone default is ``us-central1-a``. Using ``global`` is currently unsupported. To change region, pick a region and zone from this [list][locations_list] and include the following lines in ``jupyterhub_config.py``:\n\n    .. code-block:: console\n\n    c.DataprocSpawner.region = '{region}'\n    c.DataprocSpawner.zone = '{zone that is within the chosen region}'\n\n[jupyterhub_documentation]: https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.port\n[locations_list]: https://cloud.google.com/compute/docs/regions-zones/\n\n## Next\n\n- For an example on how to run the Dataproc spawner in production, refer to the [ai-notebook-extended Github repository](https://github.com/GoogleCloudPlatform/ai-notebooks-extended).\n- For a Google-supported version of the Dataproc Spawner, refer to the official [Dataproc Hub documentation](https://cloud.google.com/dataproc/docs/tutorials/dataproc-hub-admins).\n\n## Disclaimer\n\n[This is not an official Google product.](https://opensource.google.com/docs/releasing/publishing/#disclaimer)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleclouddataproc%2Fjupyterhub-dataprocspawner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogleclouddataproc%2Fjupyterhub-dataprocspawner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleclouddataproc%2Fjupyterhub-dataprocspawner/lists"}