{"id":13749987,"url":"https://github.com/sequenceiq/docker-spark","last_synced_at":"2025-05-09T13:31:42.990Z","repository":{"id":18540897,"uuid":"21741862","full_name":"sequenceiq/docker-spark","owner":"sequenceiq","description":null,"archived":false,"fork":false,"pushed_at":"2021-03-11T10:00:45.000Z","size":55,"stargazers_count":765,"open_issues_count":28,"forks_count":282,"subscribers_count":65,"default_branch":"master","last_synced_at":"2024-11-16T01:32:57.685Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sequenceiq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-07-11T15:45:18.000Z","updated_at":"2024-08-30T12:17:26.000Z","dependencies_parsed_at":"2022-08-19T17:01:06.534Z","dependency_job_id":null,"html_url":"https://github.com/sequenceiq/docker-spark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sequenceiq%2Fdocker-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sequenceiq%2Fdocker-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sequenceiq%2Fdocker-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sequenceiq%2Fdocker-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sequenceiq","download_url":"https://codeload.github.com/sequenceiq/docker-spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253258274,"owners_count":21879619,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T07:01:21.181Z","updated_at":"2025-05-09T13:31:42.736Z","avatar_url":"https://github.com/sequenceiq.png","language":"Shell","funding_links":[],"categories":["Shell","Resources"],"sub_categories":["Docker Images"],"readme":"Apache Spark on Docker\n==========\n\n[![DockerPulls](https://img.shields.io/docker/pulls/sequenceiq/spark.svg)](https://registry.hub.docker.com/u/sequenceiq/spark/)\n[![DockerStars](https://img.shields.io/docker/stars/sequenceiq/spark.svg)](https://registry.hub.docker.com/u/sequenceiq/spark/)\n\n\nThis repository contains a Docker file to build a Docker image with Apache Spark. This Docker image depends on our previous [Hadoop Docker](https://github.com/sequenceiq/hadoop-docker) image, available at the SequenceIQ [GitHub](https://github.com/sequenceiq) page.\nThe base Hadoop Docker image is also available as an official [Docker image](https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/).\n\n##Pull the image from Docker Repository\n```\ndocker pull sequenceiq/spark:1.6.0\n```\n\n## Building the image\n```\ndocker build --rm -t sequenceiq/spark:1.6.0 .\n```\n\n## Running the image\n\n* if using boot2docker make sure your VM has more than 2GB memory\n* in your /etc/hosts file add $(boot2docker ip) as host 'sandbox' to make it easier to access your sandbox UI\n* open yarn UI ports when running container\n```\ndocker run -it -p 8088:8088 -p 8042:8042 -p 4040:4040 -h sandbox sequenceiq/spark:1.6.0 bash\n```\nor\n```\ndocker run -d -h sandbox sequenceiq/spark:1.6.0\n```\n\n## Versions\n```\nHadoop 2.6.0 and Apache Spark v1.6.0 on Centos\n```\n\n## Testing\n\nThere are two deploy modes that can be used to launch Spark applications on YARN.\n\n### YARN-client mode\n\nIn yarn-client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.\n\n```\n# run the spark shell\nspark-shell \\\n--master yarn-client \\\n--driver-memory 1g \\\n--executor-memory 1g \\\n--executor-cores 1\n\n# execute the the following command which should return 1000\nscala\u003e sc.parallelize(1 to 1000).count()\n```\n### YARN-cluster mode\n\nIn yarn-cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application.\n\nEstimating Pi (yarn-cluster mode):\n\n```\n# execute the the following command which should write the \"Pi is roughly 3.1418\" into the logs\n# note you must specify --files argument in cluster mode to enable metrics\nspark-submit \\\n--class org.apache.spark.examples.SparkPi \\\n--files $SPARK_HOME/conf/metrics.properties \\\n--master yarn-cluster \\\n--driver-memory 1g \\\n--executor-memory 1g \\\n--executor-cores 1 \\\n$SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar\n```\n\nEstimating Pi (yarn-client mode):\n\n```\n# execute the the following command which should print the \"Pi is roughly 3.1418\" to the screen\nspark-submit \\\n--class org.apache.spark.examples.SparkPi \\\n--master yarn-client \\\n--driver-memory 1g \\\n--executor-memory 1g \\\n--executor-cores 1 \\\n$SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar\n```\n\n### Submitting from the outside of the container\nTo use Spark from outside of the container it is necessary to set the YARN_CONF_DIR environment variable to directory with a configuration appropriate for the docker. The repository contains such configuration in the yarn-remote-client directory.\n\n```\nexport YARN_CONF_DIR=\"`pwd`/yarn-remote-client\"\n```\n\nDocker's HDFS can be accessed only by root. When submitting Spark applications from outside of the cluster, and from a user different than root, it is necessary to configure the HADOOP_USER_NAME variable so that root user is used.\n\n```\nexport HADOOP_USER_NAME=root\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsequenceiq%2Fdocker-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsequenceiq%2Fdocker-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsequenceiq%2Fdocker-spark/lists"}