{"id":15145750,"url":"https://github.com/soundcloud/spark-pagerank","last_synced_at":"2025-10-19T16:32:16.680Z","repository":{"id":29455258,"uuid":"32991590","full_name":"soundcloud/spark-pagerank","owner":"soundcloud","description":"PageRank in Spark","archived":false,"fork":false,"pushed_at":"2023-04-09T21:09:44.000Z","size":113,"stargazers_count":74,"open_issues_count":8,"forks_count":14,"subscribers_count":136,"default_branch":"master","last_synced_at":"2025-01-29T21:23:46.440Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://soundcloud.github.io/spark-pagerank","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soundcloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-27T14:38:49.000Z","updated_at":"2024-08-16T00:55:56.000Z","dependencies_parsed_at":"2022-08-31T23:51:28.291Z","dependency_job_id":null,"html_url":"https://github.com/soundcloud/spark-pagerank","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fspark-pagerank","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fspark-pagerank/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fspark-pagerank/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fspark-pagerank/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soundcloud","download_url":"https://codeload.github.com/soundcloud/spark-pagerank/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237172115,"owners_count":19266614,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-26T11:42:30.471Z","updated_at":"2025-10-19T16:32:11.379Z","avatar_url":"https://github.com/soundcloud.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![buildstatus](https://travis-ci.org/soundcloud/spark-pagerank.svg?branch=master)](https://travis-ci.org/soundcloud/spark-pagerank)\n\n# PageRank in Spark\n\nThis is an implementation of PageRank in Spark, using Spark's standard RDD API.\n\n## Features\n\n- Fast iterations\n- Parameterised \"teleport\" probability\n- Weighted edges (out-edge weights must be normalized)\n- Supports \"dangling\" vertices (no out-edges from a node)\n- Supports arbitrary (e.g. non-uniform) priors (as vertex values)\n- Various stopping criteria:\n  - Number of iterations threshold\n  - Convergence threshold (requires additional computation after each iteration)\n- Utilities for building, preparing, and validating input graphs (incl. out-edge normalization)\n\n## Usage\n\nInclude it as a dependency in your sbt project:\n`\"com.soundcloud\" %% \"spark-pagerank\" % \u003cversion\u003e`\n\n### As A Library\n\nYou can use PageRank in Spark as a library and call it from within your own drivers. You will want to do this when you have some data preparation to do that does not conform with the built-in driver data interfaces.\n\nMore examples of usage as a library can be found in the source of the built-in drivers (see below).\n\n### As Drivers\n\nWe include several built-in drivers that operate on plain-text TSV input of labeled edges as a starting point. You will prepare the graph and run PageRank in the following sequence of drivers. Use `--help` to see the arguments and usage of each driver.\n\n1. `com.soundcloud.spark.pagerank.GraphBuilderApp`: Builds a PageRank graph from (non-normalized) weighted edges in TSV format (source, destination, weight), saving the resulting graph (edges and vertices) in Parquet files in preparation for next steps.\n1. `com.soundcloud.spark.pagerank.PageRankApp`: Runs PageRank on the graph produced using the functions in `PageRankGraph` or by using the `GraphBuilderApp`.\n1. `com.soundcloud.spark.pagerank.ConvergenceCheckApp`: Compares two PageRank vectors and lets the user determine if there is convergence by outputting the sum of the component-wise difference of the vectors. Note that this is an optional tool that is mostly used for debugging. If the user is concerned with iterating until convergence, the user can specify the convergence threshold at runtime to PageRank.\n\n## Performance\n\nWe run this library on one of our behavior graphs which consists of approximately 700M vertices and 15B edges. Using the following Spark configuration, and in-memory persistence of edge and vertex RDDs, we obtain iteration times on the order of 3-5 minutes each.\n\nConfiguration example:\n\n - Spark 2.1.1\n - YARN\n - Dynamic allocation: no\n - Number of executors: 256\n - Number of executor cores: 4\n - Executor memory: 28G\n\n### Performance Tuning\n\n- Persist the edges and vertices of the graph in memory and disk (as spill): `StorageLevel.MEMORY_AND_DISK`\n- Enable Kryo serialization: `KryoSerialization.useKryo`\n\n## Publishing and Releasing\n\nTo publish an artifact to the Sonatype/Maven central repository (a snapshot or release), you need to have a Sonatype account, PGP keys and sbt plugins set up. Please follow the [sbt guideline](http://www.scala-sbt.org/release/docs/Using-Sonatype.html) for a complete guide to getting started. Once this is done, you can use the [sbt-release](https://github.com/sbt/sbt-release) plugin via the `Makefile` to publish snapshots and perform releases.\n\n### Publishing Snapshots\n\nAt any point in the development lifecycle, a snapshot can be published to the central repository.\n\n```\nmake publish\n```\n\n### Performing a Release\n\nOnce development of a version is complete, the artifact should be released to the central repository. This is a two stage process with the artifact first entering a staging repository, followed by a manual promotion process.\n\n```\nmake release\n```\n\nAfter a release to the staging repository, the [staging-to-release promotion process](http://central.sonatype.org/pages/releasing-the-deployment.html) must be followed manually before the artifact is available in the central repository.\n\n## Versioning\n\nThis library aims to adhere to [Semantic Versioning 2.0.0](http://semver.org/spec/v2.0.0.html). Violations of this scheme should be reported as bugs. Specifically, if a minor or patch version is released that breaks backward compatibility, that version should be immediately removed and/or a new version should be immediately released that restores compatibility. Breaking changes to the public API will only be introduced with new major versions.\n\n## Contributing\n\nWe welcome contributions by [pull requests](https://github.com/soundcloud/spark-pagerank/pulls) and bug reports or feature requests as [issues](https://github.com/soundcloud/spark-pagerank/issues).\n\n## Authors\n\n* [Josh Devins](https://github.com/joshdevins)\n\n### Contributors\n\n* [Max Jakob](https://github.com/maxjakob)\n\n## License\n\nCopyright (c) 2017 [SoundCloud Ltd.](http://soundcloud.com)\n\nSee the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoundcloud%2Fspark-pagerank","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoundcloud%2Fspark-pagerank","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoundcloud%2Fspark-pagerank/lists"}