{"id":18759276,"url":"https://github.com/mariussoutier/spark-intro","last_synced_at":"2025-12-02T19:30:15.349Z","repository":{"id":29509278,"uuid":"33047522","full_name":"mariussoutier/spark-intro","owner":"mariussoutier","description":"Companion code to \"Intro to Apache Spark\" talk","archived":false,"fork":false,"pushed_at":"2015-03-29T11:01:29.000Z","size":95,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-29T03:53:45.618Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mariussoutier.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-28T19:41:09.000Z","updated_at":"2015-03-29T11:01:29.000Z","dependencies_parsed_at":"2022-09-02T13:51:13.592Z","dependency_job_id":null,"html_url":"https://github.com/mariussoutier/spark-intro","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariussoutier%2Fspark-intro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariussoutier%2Fspark-intro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariussoutier%2Fspark-intro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariussoutier%2Fspark-intro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mariussoutier","download_url":"https://codeload.github.com/mariussoutier/spark-intro/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239652954,"owners_count":19675004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T17:51:06.339Z","updated_at":"2025-12-02T19:30:15.301Z","avatar_url":"https://github.com/mariussoutier.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Intro to Spark\n\nInto to exploring data and extracting useful information using [Apache Spark)(http://spark.apache.org).\nThe code was presented at the Scala User Group Cologne, slides are [here](http://www.slideshare.net/marius-soutier/spark-intro-scala-ug).\n\nThere are several jobs provided that you can run for each topic. Build the JAR using `sbt \";clean;assembly\"`.\nRun it on Spark using `spark-submit --master \u003cmaster\u003e --class \u003cf.q.ClassName\u003e target/scala-2.10/spark-demo-assembly-1.0.jar`.\n\nFor interactive exploration, you can use `spark-shell`. When you run it, it already provides a SparkContext called `sc`.\n\nIf you are using an IDE like IntelliJ or Eclipse, you should try out their worksheets or Scala console.\nWhen you are starting a SparkContext, pass the master explicitly, e.g.:\n`val sc = new SparkContext(master = \"\u003cmaster\u003e\", appName = \"Demo\")`.\n\n*Hint*: You can easily start any job or shell by using the Spark local mode, e.g. `spark-shell --master local[*]`.\n\n## Google Web Graph\n\n* Download `web-Google.txt.gz` from https://snap.stanford.edu/data/web-Google.html and put it in src/main/resources.\n* Check out the `GoogleWebGraph.scala` job.\n\n\n## GitHub Commits\n\n1. Obtain an OAuth key on GitHub (Settings \u003e Applications \u003e Personal access tokens \u003e Generate)\n2. Execute `download_github.sh \u003ctoken\u003e`\n3. Execute `ProcessGitHubData.scala`\n-\u003e the files should be in `src/main/resources/github`, one commit per line\n\nNow play around with `GitHubSql.scala`.\n\n\n## Audioscrobbler\n\n* Download Audioscrobbler profile data from http://www-etud.iro.umontreal.ca/~bergstrj/audioscrobbler_data.html and put it in src/main/resources.\n* Check out the `Audioscrobbler.scala` job.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmariussoutier%2Fspark-intro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmariussoutier%2Fspark-intro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmariussoutier%2Fspark-intro/lists"}