{"id":26659049,"url":"https://github.com/denisecase/spark-challenge","last_synced_at":"2025-04-11T14:10:15.686Z","repository":{"id":36251875,"uuid":"222745772","full_name":"denisecase/spark-challenge","owner":"denisecase","description":"A chance to practice skills with Spark, Java, and Maven to perform a simple word count process","archived":false,"fork":false,"pushed_at":"2024-10-03T18:45:56.000Z","size":11,"stargazers_count":0,"open_issues_count":1,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-25T10:17:35.035Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/denisecase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-19T16:57:00.000Z","updated_at":"2022-02-04T21:58:17.000Z","dependencies_parsed_at":"2022-08-08T13:47:15.386Z","dependency_job_id":null,"html_url":"https://github.com/denisecase/spark-challenge","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fspark-challenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fspark-challenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fspark-challenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fspark-challenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/denisecase","download_url":"https://codeload.github.com/denisecase/spark-challenge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248413898,"owners_count":21099377,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-25T10:17:40.376Z","updated_at":"2025-04-11T14:10:15.650Z","avatar_url":"https://github.com/denisecase.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spark-challenge\n\n## Links\n\n- [Repo](https://github.com/denisecase/spark-challenge)\n\n## Prerequisites\n\n- VS Code\n- VS Code Extension: Maven for Java\n- VS Code Extension: Java Extension Pack\n\n## Instructions 1 - Start New Maven Project\n\n1. Create a new project using Maven (you do not need to fork this repo).\n1. There should be a folder on your laptop where you keep your git projects. Go to this folder, e.g. 44517.\n1. Open this parent folder in VS Code.\n1. Click the VS Code Extensions icon. Verify you have the two required extensions. If not, install them.\n1. Click the VS Code Explorer icon. From the menu, select:\n1. View / Command Palette / Maven: Create Maven Project / archetype-quickstart-jdk8 / most recent version.\n1. When the folder window opens, click your parent folder up at the top, click \"Select Destination Folder\".\n\n## Instructions 2 - Interactive Mode\n\n```Bash\ngroupId: edu.nwmissouri.yourname\nartifactId: spark-challenge\nversion: HIT ENTER\npackage: HIT ENTER\nY: HIT ENTER\n```\n\nYou will now have a spark-challenge project folder. Exit VS Code.\n\n## Instructions 3 - Code the project\n\nChange directory into your new spark-challenge folder. Right-click and open your spark-challenge folder in VS Code.\n\nAdd a basic README.md. Use it to store your notes and commands.\n\nWhen VS Code asks: \"A build file was modified. Do you want to synchronize the Java classpath/configuration?\" Answer \"Always\" to allow VS Code to generate these artifacts automatically.\n\n## Instructions 4 - Add to POM.xml\n\nCopy the POM.xml from this repo to yours. Use CTRL-F to search for \"isl\" for \"Intelligent Systems Lab\".\nChange each occurance to match yourname in your groupId instead.\n\n## Prepare the Code\n\n```PowerShell\nmvn clean\nmvn compile\nmvn assembly:single\n```\n\n## Execute\n\n```Bash\njava -cp target/spark-challenge-1.0.0-jar-with-dependencies.jar edu.nwmissouri.isl.App \"data.txt\"\n```\n\n## Challenges - Getting Started\n\n1. Where does program execution begin?\n1. Update the main method to check that exactly one argument is provided.\n1. If not, output a message and call System.exit(0);\n1. If so, call a new method named process that uses the first item in the arg array.\n1. Add the following imports to App.java.\n\n```Java\nimport org.apache.spark.SparkConf;\nimport org.apache.spark.api.java.JavaPairRDD;\nimport org.apache.spark.api.java.JavaRDD;\nimport org.apache.spark.api.java.JavaSparkContext;\nimport scala.Tuple2;\nimport java.util.Arrays;\nimport java.nio.file.FileSystems;\nimport java.nio.file.Path;\nimport java.util.Comparator;\n```\n\n## Challenges - create a process method\n\nCreate a new private static void process method that takes one argment, a String containting the fileName (provided in the args).\n\n```Java\n\n    // define a spark configuration\n    // setMaster to local and setAppName to Challenge\n    SparkConf sparkConf = new SparkConf()...\n\n    // define a new JavaSparkContext, pass in the spark configuration object\n    JavaSparkContext sparkContext = ...\n\n    // use sparkContext.textFile() to read data into RDD given the fileName provided\n    JavaRDD\u003cString\u003e inputFile = sparkContext.textFile(fileName);\n\n    // use Java API to flatMap text into an RDD of strings (words)\n    // In 2.0, FlatMapFunction.call() returns an Iterator rather than Iterable.\n    // replace what must this be with the correct variable name\n    JavaRDD\u003cString\u003e wordsFromFile = inputFile.flatMap( whatMustThisBe -\u003e Arrays.asList(line.split(\" \")).iterator());\n\n    // use wordsFromFile and mapToPair to create a new Tuple2 for each word\n    // the tuple is in the form (word, 1)\n    // Replace each T below to create a JavaPairRDD with the associated types\n    JavaPairRDD\u003cT, T\u003e countData = wordsFromFile.mapToPair(word -\u003e new Tuple2(word, 1))\n        .reduceByKey((x, y) -\u003e (int) x + (int) y);\n\n    // create new JavaPairRDD that reverses the tuple to (n, word)\n    // use countData.mapToPair and provide a simple function that\n    // takes each pair p and outputs a new Tuple2 with p._2 first, then p._1.\n    // then call .sortByKey() to sort in reverse order,\n    // pass in Comparator.reverseOrder() to the sortByKey method\n    JavaPairRDD\u003cInteger, String\u003e output = ...\n\n    // save results to a folder (RDDs are complex) - provide a simple string value.\n    String outputFolder = ...\n\n    // get the path to your outputFolder\n    Path path = FileSystems.getDefault().getPath(outputFolder);\n\n    // use FileUtils to delete them quietly\n    // It does not work.\n    // google and find the import statement required. Add it to this file.\n    FileUtils.deleteQuietly(path.toFile());\n\n    // use call saveAsTextFile on your output JavaPairRDD to save your results.\n    // Pass in the variable holding your simple string\n    add code here\n\n    // close your spark context\n    add code here\n```\n\n## Challenges - Creating data\n\n1. What must your data file be named to make the execution command work?\n1. Create this file. Use data from one of these links.\n1. Use Maven to clean compile and build your fat jar file.\n1. Use java to run your App by including the fat jar in your classpath.\n\n## Possible Sources\n\n- \u003chttp://shakespeare.mit.edu/romeo_juliet/full.html\u003e\n- \u003chttp://shakespeare.mit.edu/midsummer/full.html\u003e\n- \u003chttp://shakespeare.mit.edu/index.html\u003e - any full play on one page from here\n\n## Finalize\n\n1. Create a cloud repo named \"spark-challenge\". Do NOT add a README in the cloud.\n1. Create a git repo in your local spark-challenge folder.\n1. Add and commit all files locally.\n1. Push your code to your cloud repo.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenisecase%2Fspark-challenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdenisecase%2Fspark-challenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenisecase%2Fspark-challenge/lists"}