{"id":16391168,"url":"https://github.com/queirozfcom/hadoop-spark-ml-comparison","last_synced_at":"2026-06-13T00:30:21.712Z","repository":{"id":36481897,"uuid":"40787467","full_name":"queirozfcom/hadoop-spark-ml-comparison","owner":"queirozfcom","description":"Code used for university coursework aimed at comparing Hadoop and Spark functionality, with Machine Learning (ML) tasks in mind","archived":false,"fork":false,"pushed_at":"2015-11-03T05:34:39.000Z","size":10940,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-03T17:34:24.839Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/queirozfcom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-15T21:56:41.000Z","updated_at":"2021-10-08T18:15:31.000Z","dependencies_parsed_at":"2022-08-30T15:31:01.415Z","dependency_job_id":null,"html_url":"https://github.com/queirozfcom/hadoop-spark-ml-comparison","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queirozfcom%2Fhadoop-spark-ml-comparison","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queirozfcom%2Fhadoop-spark-ml-comparison/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queirozfcom%2Fhadoop-spark-ml-comparison/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queirozfcom%2Fhadoop-spark-ml-comparison/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/queirozfcom","download_url":"https://codeload.github.com/queirozfcom/hadoop-spark-ml-comparison/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240231976,"owners_count":19768923,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T04:45:14.910Z","updated_at":"2026-06-13T00:30:21.628Z","avatar_url":"https://github.com/queirozfcom.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hadoop X Spark - Comparing Performance for Workloads \nCode used for university coursework aimed at comparing Hadoop and Spark functionality, with Machine Learning (ML) tasks in mind\n\nThis project consists of 2 (plus one discarded task) tasks we ran in order to compare the performance of Hadoop MapReduce and Spark, as well as Mahout (running on top of Hadoop MapReduce) and Spark's MLLib machine learning library.\n\n\u003e All code used for the experiments are in this repository!\n\n## The experiments\n\nAll experiments were executed on AWS infrastructure - more specifically, AWS ElasticMapReduce, on 1, 2, 4, 8 and 16 m3.xlarge nodes.\n\n**Wordcount**\n\nWe ran a standard wordcount experiment over a large dataset using Hadoop and Spark, the results being as follows:\n\n![results1](http://i.imgur.com/qvy6czI.png)\n\n**Distributed KMeans**\n\nWe ran the Distributed KMeans algorithms on Mahout (on top of Hadoop MapReduce) and on Spark's MLLib:\n\n![results2](http://i.imgur.com/HwTGUVh.png)\n\n**Naïve Distributed KMeans**\n\nWe had also intended to include a naïve implementation of Distributed KMeans (as can be seen under [naive_kmeans](https://github.com/queirozfcom/hadoop_spark_ml_comparison/tree/master/naive_kmeans)) for Hadoop MapReduce and Spark. While the Spark implementation went OK, the Hadoop version did not finish after a long wait so we decided against inluding it in the results.\n\n## Findings\n\n- For non-iterative tasks Spark starts off better than Hadoop (w.r.t. execution time) but Hadoop catches up with Spark\n\n- For iterative tasks Spark performs much better than Hadoop.\n\n- For all tasks, enabling Spark's `dynamicAllocation` led to massive performance gains.\n\n\nSpecial thanks to [Julian McAuley](http://cseweb.ucsd.edu/~jmcauley/) for letting us use the dataset prepared by him.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqueirozfcom%2Fhadoop-spark-ml-comparison","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqueirozfcom%2Fhadoop-spark-ml-comparison","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqueirozfcom%2Fhadoop-spark-ml-comparison/lists"}