{"id":15728953,"url":"https://github.com/flaviostutz/spark-scala-hdfs-docker-example","last_synced_at":"2026-05-07T01:10:31.605Z","repository":{"id":84180861,"uuid":"254903918","full_name":"flaviostutz/spark-scala-hdfs-docker-example","owner":"flaviostutz","description":"Spark with Scala reading/writing files to HDFS with automatic additions of new Spark workers using Docker \"scale\"","archived":false,"fork":false,"pushed_at":"2020-04-25T00:07:11.000Z","size":501,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-06T07:29:41.033Z","etag":null,"topics":["datanode","docker","example","hdfs","namenodes","scala","scale","spark","spark-workers"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flaviostutz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-11T16:02:59.000Z","updated_at":"2020-04-25T00:07:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"19262853-dffe-4794-91f1-96555f38c89a","html_url":"https://github.com/flaviostutz/spark-scala-hdfs-docker-example","commit_stats":{"total_commits":12,"total_committers":1,"mean_commits":12.0,"dds":0.0,"last_synced_commit":"c65540ccbd29c0fc42a7409c640a3820bf1f2ee0"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flaviostutz%2Fspark-scala-hdfs-docker-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flaviostutz%2Fspark-scala-hdfs-docker-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flaviostutz%2Fspark-scala-hdfs-docker-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flaviostutz%2Fspark-scala-hdfs-docker-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flaviostutz","download_url":"https://codeload.github.com/flaviostutz/spark-scala-hdfs-docker-example/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246405382,"owners_count":20771774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datanode","docker","example","hdfs","namenodes","scala","scale","spark","spark-workers"],"created_at":"2024-10-03T23:08:10.156Z","updated_at":"2026-05-07T01:10:31.576Z","avatar_url":"https://github.com/flaviostutz.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spark-scala-hdfs-docker-example\n\nSpark with Scala reading/writing files to HDFS with automatic additions of new Spark workers using Docker \"scale\"\n\nDefaults:\n\n* Workspace dir is at /app\n\n* Jar file is at /app/app.jar.\n\n* Main class is app.Main\n\n\nIf you want an example of Spark, Scala, remote Spark Master, but inside Jupyter Notebook environment, check https://github.com/flaviostutz/spark-scala-jupyter/example\n\n## Usage\n\n* Create docker-compose.yml\n\n```yml\nversion: '3.3'\nservices:\n\n  #SPARK EXAMPLE APPLICATION\n  spark-example:\n    build: .\n    image: flaviostutz/spark-sample\n    environment:\n      - SPARK_MASTER_NAME=spark-master\n    networks:\n      - hdfs\n\n  #SPARK SERVICES\n  spark-master:\n    image: bde2020/spark-master:2.4.5-hadoop2.7\n    ports:\n      - \"8080:8080\"\n      - \"7077:7077\"\n    environment:\n      - INIT_DAEMON_STEP=setup_spark\n      - SPARK_PUBLIC_DNS=localhost\n    networks:\n      - spark\n\n  spark-worker:\n    image: bde2020/spark-worker:2.4.5-hadoop2.7\n    environment:\n      - \"SPARK_MASTER=spark://spark-master:7077\"\n      - SPARK_PUBLIC_DNS=localhost\n    networks:\n      - spark\n\n  #HDFS SERVICES\n  namenode1:\n    image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8\n    ports:\n      - \"50070:50070\"\n      - \"9870:9870\"\n    environment:\n      - CLUSTER_NAME=example1\n      - INIT_DAEMON_STEP=setup_hdfs\n      - HDFS_CONF_dfs_webhdfs_enabled=true\n      - HDFS_CONF_dfs_permissions_enabled=false\n    #   - CORE_CONF_hadoop_http_staticuser_user=root\n      - CORE_CONF_hadoop_proxyuser_hue_hosts=*\n      - CORE_CONF_hadoop_proxyuser_hue_groups=*\n    volumes:\n      - ./volumes/namenode1:/hadoop/dfs/name\n    networks:\n      - hdfs\n\n  datanode1:\n    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8\n    environment:\n      - CORE_CONF_fs_defaultFS=hdfs://namenode1:8020\n      - HDFS_CONF_dfs_webhdfs_enabled=true\n    ports:\n      - \"50075:50075\"\n      - \"9864:9864\"\n    volumes:\n      - ./volumes/datanode1:/hadoop/dfs/data\n    networks:\n      - hdfs\n```\n\n* Copy example application contents from this repo\n\n* Run ```docker-compose up --build -d```\n\n* Run ```docker-compose logs -f spark-example```\n\n  * Check Scala Spark running Hello World application\n\n* Open http://localhost:8080 and look for \"Completed Applications\"\n\n* Access namenode1 admin at http://localhost:9870/\n  * Access filebrowser from menu Utilities-\u003eBrowser the filesystem\n\n* Access datanode1 admin at http://localhost:9864/\n\n* For scaling datanodes/namenodes, copy datanode/namenode service and map a new volume to them\n\n* For scaling Spark workers, use docker-compose scale spark-worker=5, for example\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflaviostutz%2Fspark-scala-hdfs-docker-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflaviostutz%2Fspark-scala-hdfs-docker-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflaviostutz%2Fspark-scala-hdfs-docker-example/lists"}