{"id":16906085,"url":"https://github.com/timkam/wasp-cloud-course","last_synced_at":"2026-05-10T00:36:15.410Z","repository":{"id":77453216,"uuid":"189022150","full_name":"TimKam/wasp-cloud-course","owner":"TimKam","description":"Matrix Computation with Apache Spark in the Cloud","archived":false,"fork":false,"pushed_at":"2019-06-05T19:27:37.000Z","size":426,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-25T15:24:30.446Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TimKam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-28T12:16:08.000Z","updated_at":"2019-06-05T19:27:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"c6aacbf1-4cb9-43a9-bd96-420ae6adeea6","html_url":"https://github.com/TimKam/wasp-cloud-course","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimKam%2Fwasp-cloud-course","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimKam%2Fwasp-cloud-course/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimKam%2Fwasp-cloud-course/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimKam%2Fwasp-cloud-course/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TimKam","download_url":"https://codeload.github.com/TimKam/wasp-cloud-course/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244647956,"owners_count":20487170,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T18:40:57.119Z","updated_at":"2026-05-10T00:36:15.365Z","avatar_url":"https://github.com/TimKam.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Matrix Computation with Apache Spark in the Cloud\n\n*Christopher Blöcker, Timotheus Kampik, Tobias Sundqvist*\n\nThis repository ([https://github.com/TimKam/wasp-cloud-course](https://github.com/TimKam/wasp-cloud-course)) contains documentation and example code for running matrix computations with Apache Spark in the cloud.\nThe content was produced as part of the WASP Software Engineering and Cloud Computing Course 2019.\n\n## Problem\nThe problem to solve is scaling *matrix computations* with Apache Spark. Instead of just running them on a local machine, the computations are to be executed across several cloud instances to speed up execution time. To demonstrate the performance advantage of the cloud-based solution, any of the matrices that are available at [https://sparse.tamu.edu/](https://sparse.tamu.edu/) and any of Spark's [linear algebra methods](https://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html#module-pyspark.ml.linalg) can be picked. However, the choice of data set and methods should allow for a fruitful analysis.\n\n## Choice of Methods\nTo assess the performance of an Apache Spark cluster with different numbers of worker nodes for a linear algebra use case, we chose the following methods:\n\n1. **Singular value decomposition** (SVD): [pyspark.mllib.html#pyspark.mllib.linalg.distributed.RowMatrix.computeSVD](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.linalg.distributed.RowMatrix.computeSVD);\n\n2. **Multiply**: [pyspark.mllib.linalg.distributed.RowMatrix.multiply](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.linalg.distributed.RowMatrix.multiply)\n\nWe picked the two methods, because we expect them to be commonly used in real-life scenarios.\n\n## Choice of Data Sets\nWe selected two approximately medium-sized data sets by sorting the available data sets at [https://sparse.tamu.edu/](https://sparse.tamu.edu/) by number of non-zero entries.\nThe exact data sets we selected are:\n\n* [https://sparse.tamu.edu/PARSEC/Si87H76](https://sparse.tamu.edu/PARSEC/Si87H76);\n\n* [https://sparse.tamu.edu/DIMACS10/delaunay_n18](https://sparse.tamu.edu/DIMACS10/delaunay_n18).\n\n## Setup and Execution\nThe Docker setup is based on the [elyra/spark-py](https://hub.docker.com/r/elyra/spark-py) image and executes the analysis code you find at [https://github.com/TimKam/wasp-cloud-course/blob/master/analysis.py](https://github.com/TimKam/wasp-cloud-course/blob/master/analysis.py).\n## Resources\nWe created a pool with 4 core nodes and gave the spark executors 8 GB memory.\n\n## Analysis\n\nWe ran both operations on the aforementioned data sets on 1, 2, 3, and 4 (and 5 for data set 2) workers in Google Cloud.\nBelow are our results, averaged over 10 runs each, for the data set at [https://sparse.tamu.edu/PARSEC/Si87H76](https://sparse.tamu.edu/PARSEC/Si87H76):\n\n|              | 1 Worker              | 2 Workers              | 3 Workers              | 4 Workers              |\n| ------------ | --------------------- | ---------------------  | ---------------------  | ---------------------  |\n| **Multiply** |  0.7329313039779664 s | 0.5070641040802002 s   |   0.8104101181030273 s |   1.015389585494995 s  | \n| **SVD**      |  8.090139007568359 s  | 5.993102312088013 s    |   6.0258043050765995 s |   6.112043023109436 s  | \n\n![Multiply](multiply.png)\n\n![SVD](svd.png)\n\nWith the data set at [https://sparse.tamu.edu/DIMACS10/delaunay_n18](https://sparse.tamu.edu/DIMACS10/delaunay_n18), we got the following results:\n\n\n|              | 1 Worker              | 2 Workers              | 3 Workers              | 4 Workers              | 5 Workers              |\n| ------------ | --------------------- | ---------------------  | ---------------------  | ---------------------  | ---------------------  |\n| **Multiply** |  2.4694278955459597 s | 1.6311752080917359 s   |     2.05913450717926 s | 1.8936052083969117 s   |   1.8766150951385498 s | \n| **SVD**      |  67.83026208877564 s  | 40.50567228794098 s    |   39.880996799468996 s |   41.2277055978775 s   |   40.89326241016388 s  |\n\n![Multiply](multiply_2.png)\n\n![SVD](svd_2.png)\n\nAs can be seen, both operations are fastest with 2 workers and loose speed with more workers, with the exception of data set 2, for which we receive a minor speed improvement (possibly within the margin of error) for 3 workers.\nWe tried a third method (principal component analysis [pyspark.mllib.linalg.distributed.RowMatrix.computePrincipalComponents](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.linalg.distributed.RowMatrix.computePrincipalComponents)) that would require too much memory when executed on the selected data set. We assume in most cases, memory is the bottleneck (Von Neumann bottleneck).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimkam%2Fwasp-cloud-course","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimkam%2Fwasp-cloud-course","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimkam%2Fwasp-cloud-course/lists"}