{"id":15411054,"url":"https://github.com/oskardudycz/sparkwithscalaanddocker","last_synced_at":"2025-04-04T05:25:41.431Z","repository":{"id":79310910,"uuid":"109485825","full_name":"oskardudycz/SparkWithScalaAndDocker","owner":"oskardudycz","description":"Example showing how to run Spark with Scala and Docker","archived":false,"fork":false,"pushed_at":"2020-01-02T10:19:53.000Z","size":675,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-09T16:44:02.271Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oskardudycz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-11-04T10:30:05.000Z","updated_at":"2020-01-02T10:19:55.000Z","dependencies_parsed_at":"2023-05-25T17:15:15.227Z","dependency_job_id":null,"html_url":"https://github.com/oskardudycz/SparkWithScalaAndDocker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oskardudycz%2FSparkWithScalaAndDocker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oskardudycz%2FSparkWithScalaAndDocker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oskardudycz%2FSparkWithScalaAndDocker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oskardudycz%2FSparkWithScalaAndDocker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oskardudycz","download_url":"https://codeload.github.com/oskardudycz/SparkWithScalaAndDocker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247125973,"owners_count":20887816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T16:47:43.186Z","updated_at":"2025-04-04T05:25:41.410Z","avatar_url":"https://github.com/oskardudycz.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark With Scala And Docker\r\n\r\nProject shows how to create basic configuration of Spark workstation with Docker. It's not needed to setup on local environment Java, Scala, Spark, Hadoop on any other tool than Docker. As an example is show how to do simplest lines count in Spark.\r\n\r\nCode is placed in [/src/main/scala/](https://github.com/oskardudycz/SparkWithScalaAndDocker/blob/master/src/src/main/scala/FileLinesCount.scala)\r\n\r\n```scala\r\npackage sparkWithScalaAndDocker\r\n\r\nimport org.apache.spark.{SparkContext, SparkConf}\r\n\r\nobject FileLinesCount {\r\n  def main(args: Array[String]) {\r\n    val conf = new SparkConf().setAppName(\"SparkWithScalaAndDocker Application\")\r\n    val sc = new SparkContext(conf)\r\n\r\n    val fileName = args(0)\r\n    val lines = sc.textFile(fileName).cache\r\n\r\n    val c = lines.count\r\n    println(s\"There are $c lines in $fileName\")\r\n  }\r\n}\r\n```\r\nInstall [Docker](https://www.docker.com/get-docker)\r\n\r\nOpen cmd/shell:\r\n1. Run `init` to start docker (see details in [init.bat](https://github.com/oskardudycz/SparkWithScalaAndDocker/blob/master/init.bat)).\r\n2. Run `build` to build project (see details in [build.bat](https://github.com/oskardudycz/SparkWithScalaAndDocker/blob/master/build.bat)).\r\n3. Run `run` to run project in Spark (see details in [run.bat](https://github.com/oskardudycz/SparkWithScalaAndDocker/blob/master/run.bat))\r\n\r\n\r\nI found an issue or I have a change request\r\n--------------------------------\r\nFeel free to create an issue on GitHub. Contributions, pull requests are more than welcome!\r\n\r\n**Spark With Scala And Docker** is Copyright \u0026copy; 2017-2020 [Oskar Dudycz](http://oskar-dudycz.pl) and other contributors under the [MIT license](LICENSE).\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foskardudycz%2Fsparkwithscalaanddocker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foskardudycz%2Fsparkwithscalaanddocker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foskardudycz%2Fsparkwithscalaanddocker/lists"}