{"id":19245337,"url":"https://github.com/dimajix/docker-jupyter-spark","last_synced_at":"2025-07-27T23:35:28.249Z","repository":{"id":149663534,"uuid":"92422035","full_name":"dimajix/docker-jupyter-spark","owner":"dimajix","description":"Docker image for Jupyter notebooks with PySpark","archived":false,"fork":false,"pushed_at":"2018-08-03T18:50:12.000Z","size":12,"stargazers_count":27,"open_issues_count":1,"forks_count":25,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-21T10:53:50.704Z","etag":null,"topics":["docker","hadoop","jupyter","pyspark","python","spark"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dimajix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-25T16:20:12.000Z","updated_at":"2024-09-13T17:32:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"bc34da29-38ec-41be-b5ce-c998607b0abf","html_url":"https://github.com/dimajix/docker-jupyter-spark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dimajix/docker-jupyter-spark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimajix%2Fdocker-jupyter-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimajix%2Fdocker-jupyter-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimajix%2Fdocker-jupyter-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimajix%2Fdocker-jupyter-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dimajix","download_url":"https://codeload.github.com/dimajix/docker-jupyter-spark/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimajix%2Fdocker-jupyter-spark/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267441462,"owners_count":24087772,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-27T02:00:11.917Z","response_time":82,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","hadoop","jupyter","pyspark","python","spark"],"created_at":"2024-11-09T17:27:30.079Z","updated_at":"2025-07-27T23:35:28.228Z","avatar_url":"https://github.com/dimajix.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Jupyter Spark Docker Container\n\nThis Docker image contains a Jupyter notebook with a PySpark kernel. Per default, the kernel runs in Spark 'local'\nmode, which does not require any cluster. But the Docker image also supports setting up a Spark standalone cluster\nwhich can be accessed from the Notebook.\n\nThe easiest way to run the Jupyter notebook is to run:\n\n    docker run -p 8888:8888 dimajix/jupyter-spark\n    \nThen when the container is running, point your webbrowser to `http://localhost:8888`, where the Jupyter notebook \nserver is running\n    \nYou can also specify your AWS credentials for accessing data inside S3 via environment variables\n    \n    docker run \\\n        -p 8888:8888 \\\n        -e AWS_ACCESS_KEY_ID=your_aws_key \\\n        -e AWS_SECRET_ACCESS_KEY=your_aws_secret \\\n        dimajix/jupyter-spark\n    \n\n# Configuration\n\nThere are some configuration options which can be changed by setting environment variables for your Docker container.\nDetails to all the options are listed below.\n\n## Jupyter Configuration\n\nThere are only two Jupyter specific configuration properties:\n\n    JUPYTER_PORT=8888\n    JUPYTER_DIR=/mnt/notebooks\n\n## Spark Kernel Configuration\n\nThe Jupyter Notebook contains a special PySpark kernel, which also has some configuration options related to Spark\nitself. Unfortunately you cannot change these settings on a per-notebook basis, but at least you can change these\nsettings per Docker container.\n\n    SPARK_MASTER=local[*]\n    SPARK_DRIVER_MEMORY=2G\n    SPARK_EXECUTOR_MEMORY=4G\n    SPARK_EXECUTOR_CORES=4\n    SPARK_NUM_EXECUTORS=1\n\n\n## S3 properties\n\nSince many users want to access data stored on AWS S3, it is also possible to specify AWS credentials and general\nsettings.\n\n    S3_PROXY_HOST=\n    S3_PROXY_PORT=\n    S3_PROXY_USE_HTTPS=false\n    S3_ENDPOINT=s3.amazonaws.com\n    S3_ENDPOINT_HTTP_PORT=80\n    S3_ENDPOINT_HTTPS_PORT=443\n\n    AWS_ACCESS_KEY_ID=\n    AWS_SECRET_ACCESS_KEY=\n\n\n# Spark Cluster Configuration\n\nAside from the Jupyter kernel / driver side there are some more Spark related configuration properties, which are used\nto setup and connect to the Spark cluster. Note that all worker nodes also require the same Python installation as on\nthe notebook server, so essentially the only deployment mode currently supported is Spark Standalone cluster using the\nsame Docker image for both the Spark master and all Spark worker nodes.\n\nThe following settings configure Spark master and all workers.\n\n    SPARK_MASTER_HOST=spark-master\n    SPARK_MASTER_PORT=7077\n\n    SPARK_WEBUI_PORT=9090\n    SPARK_WORKER_CORES=4\n    SPARK_WORKER_MEMORY=8G\n    SPARK_LOCAL_DIRS=/tmp/spark-local\n    SPARK_WORKER_DIR=/tmp/spark-worker\n\n## Hadoop Properties\n\nIt is possible to access Hadoop resources (in HDFS) from Spark. \n\n    HDFS_NAMENODE_HOSTNAME=hadoop-namenode\n    HDFS_NAMENODE_PORT=8020\n    HDFS_DEFAULT_FS=${HDFS_DEFAULT_FS=hdfs://$HDFS_NAMENODE_HOSTNAME:$HDFS_NAMENODE_PORT}\n    HDFS_REPLICATION_FACTOR=2\n\n## Running a Spark Standalone Cluster\n\nThe container already contains all components for running a Spark standalone cluster. This can be achieved by using the\ntwo commands\n* master\n* slave\n\nThe docker-compose file contains an example of a complete Spark standalone cluster with a Jupyter Notebook as the\nfrontend.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimajix%2Fdocker-jupyter-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdimajix%2Fdocker-jupyter-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimajix%2Fdocker-jupyter-spark/lists"}