{"id":18607352,"url":"https://github.com/addono/id2221-final-project","last_synced_at":"2025-11-02T15:30:25.536Z","repository":{"id":82277121,"uuid":"216060639","full_name":"Addono/id2221-final-project","owner":"Addono","description":null,"archived":false,"fork":false,"pushed_at":"2021-05-07T11:42:26.000Z","size":37,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-12-27T00:27:26.355Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Addono.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-18T16:03:00.000Z","updated_at":"2021-05-07T11:42:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"23e16ee9-da6c-4103-98f0-b3a867c1bb32","html_url":"https://github.com/Addono/id2221-final-project","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Addono%2Fid2221-final-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Addono%2Fid2221-final-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Addono%2Fid2221-final-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Addono%2Fid2221-final-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Addono","download_url":"https://codeload.github.com/Addono/id2221-final-project/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239395876,"owners_count":19631336,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T02:29:10.225Z","updated_at":"2025-11-02T15:30:25.502Z","avatar_url":"https://github.com/Addono.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ID2221 - Final Project\n\nBuilds a directional graph from all public events (pushing public commits, creating repositories, ...). Each event is an edge which connects a user with a repository.\n\n## Table of Contents\n+ [About](#about)\n+ [Getting Started](#getting_started)\n+ [Usage](#usage)\n+ [Resources](#resources)\n\n## About \u003ca name = \"about\"\u003e\u003c/a\u003e\nThis project leverages the [Github Archive](https://gharchive.org) dataset. \n\nAt the moment of writing this it is 789 GiB of compressed data:\n```bash\n$ gsutil du -sh gs://data.gharchive.org\n789.57 GiB   gs://data.gharchive.org\n```\n\n## Getting Started \u003ca name = \"getting_started\"\u003e\u003c/a\u003e\n\u003c!--These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See [deployment](#deployment) for notes on how to deploy the project on a live system.--\u003e\n\n### Prerequisites\n\n* [scala](https://scala-lang.org/download/)\n* [sbt](https://www.scala-sbt.org/download.html)\n* [Google Cloud SDK](https://cloud.google.com/sdk/) (optional, alternatively use the [web console](https://console.cloud.google.com))\n\n\n### Installing\nClone the repository and build the application:\n```bash\nsbt assembly\n```\n\n## Usage \u003ca name = \"usage\"\u003e\u003c/a\u003e\n\n[Create a DataProc cluster](https://cloud.google.com/dataproc/docs/guides/create-cluster) if you don't have one running yet.\n\nLaunch the project on a DataProc cluster, make sure to update the name of the cluster, the clusters region (if not set to global), the input selector and output location.\n \n**Note: The input selector should not match files created before 1th of January 2015, so for example `201*-01-01-10` is illegal.**\n\nThe `GraphBuilder` can be used to construct the graphs without running any community detection algorithms.\n```bash\ngcloud dataproc jobs submit spark --jars target/scala-2.11/github-graphframe-builder-assembly-0.2.jar --cluster gh-archive-dataproc --region europe-west1 --class GraphBuilder -- \"2015-01-01-*\" gs://gh-grahpframes/2015-01-01\n```\n\nThe `LabelPropagationRunner` both constructs the graph and then runs the label propagation algorithm on this graph. It's first argument is the location where the output files can be written to, all preceding arguments are input selector. Each of which will be scheduled as seperate batches.\n```bash\ngcloud dataproc jobs submit spark --jars target/scala-2.11/github-graphframe-builder-assembly-0.2.jar --cluster gh-archive-dataproc --region europe-west1  --class `LabelPropagationRunner` -- gs://gh-graphframes \"2015-01-01-*\"\n```\n\n### Cleanup\nTo prevent unnecessary costs, make sure to destroy all resources which you aren't using anymore.\n\nFirst, the most expensive thing to keep running is probably going to be the DataProc cluster. Destroy it by running:\n```bash\ngcloud dataproc clusters delete gh-archive-dataproc\n``` \n\nAlso, along the way we have stored some files into Cloud Storage, e.g. the JAR we assembled or parquet files as job artifacts (see the second runtime argument):\n```bash\n# Individual files\ngsutil rm gs://\u003cbucket-name\u003e/myfilename.txt\n\n# Directories\ngsutil rm -r gs://\u003cbucket-name\u003e/\n```\n\n## Resources \u003ca name = \"resources\"\u003e\u003c/a\u003e\n* [Write and run Spark Scala jobs on Cloud Dataproc](https://cloud.google.com/dataproc/docs/tutorials/spark-scala)\n* [Github Archive](https://www.gharchive.org)\n* [GraphFrames Documentation](https://graphframes.github.io/graphframes/docs/_site/index.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faddono%2Fid2221-final-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faddono%2Fid2221-final-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faddono%2Fid2221-final-project/lists"}