{"id":21614570,"url":"https://github.com/halaway/big-data-ml","last_synced_at":"2026-05-15T12:03:37.158Z","repository":{"id":211027350,"uuid":"728017455","full_name":"halaway/big-data-ML","owner":"halaway","description":null,"archived":false,"fork":false,"pushed_at":"2023-12-06T05:04:18.000Z","size":178,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-24T20:41:32.259Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/halaway.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-06T03:44:19.000Z","updated_at":"2023-12-06T03:45:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"bb2a7059-0cab-4c3c-bc40-2e561fcafb53","html_url":"https://github.com/halaway/big-data-ML","commit_stats":null,"previous_names":["halaway/big-data-ml"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halaway%2Fbig-data-ML","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halaway%2Fbig-data-ML/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halaway%2Fbig-data-ML/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halaway%2Fbig-data-ML/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/halaway","download_url":"https://codeload.github.com/halaway/big-data-ML/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244265999,"owners_count":20425825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-24T22:08:26.456Z","updated_at":"2026-05-15T12:03:32.125Z","avatar_url":"https://github.com/halaway.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Machine Learning Models for Cancer Diagnosis: Integrating Spark\n### Introduction\nLeveraging distributed systems for tackling machine learning problems, typically increases query optimization and model performance when accounting for faster execution times and parallel data processing.\n\nFor this project, we are given a CSV file consisting of 569 rows of individual samples defining some diagnosis of either \"Benign” or ”Malignant” and twenty measured clinical variables. We then use Spark along with two machine learning models, ie. LinearSVC and Random Forest Classifiers, for predicting the positive target variable of our data set, or the ”Malignant” diagnosis.\n\n| ID       | Diagnosis  | Radius_mean   | Texture_mean  | perimeter_mean | area_mean | smoothness_mean | compactness_mean |\n| -------- |:------:| --------:| ---------------------- | -------- | ---------| --------| ---------|\n|842302 | M  | 17.99 | 10.38 |122.8 | 1001 | 0.1184| 0.2776| \n| 842517 | M | 20.57 | 21.77  |  132.9 | 1326 | 0.08474 | 0.07864 | \n\n\n\n### Integrating Spark With LinearSVC and RandomForest\nThis project is essentially contained within the ML Class, where different methods\nare used for instantiating, fitting, and plotting the model along with their performance\nmetrics. \n\n### Linear SVC Performance Metrics\n\n\u003cdiv style=\"display: flex; align-items: center;\"\u003e\n    \u003cdiv style=\"flex: 1;\"\u003e\n        \u003cimg align='left' src = \"https://github.com/halaway/big-data-ML/assets/31904474/33ebd981-cecf-4906-9091-e4d748f86f75\" width=35% height=35% \u003e \n    \u003c/div\u003e\n\u003c/div\u003e\n   \n\u003cp align='center'\u003e\n    \n| Precision       | Recall  | Accuracy   | F1-Score  | \n| -------- |:------:| --------:| ---------------------- |\n|0.999|   0.914    |\t   0.965\t  |  0.955     |  \n\u003c/p\u003e\n\n\nTesting the model's predicted values against the true values created the following metrics \nas measures of the model's predictive ability. \n\u003cbr clear=\"left\"/\u003e\n\n\n### Random Forest Performance Metrics\n\u003cimg align='left' src= \"https://github.com/halaway/big-data-ML/assets/31904474/4fd81ba1-9b36-4251-a38b-df103063720e\" width=35% height=35% \u003e \n\u003cimg align = 'center' src=\"https://github.com/halaway/big-data-ML/assets/31904474/2503670a-4817-45c0-b8c9-df5e045dc22a\" width=35% height=35%\u003e\n\n| Precision       | Recall  | Accuracy   | F1-Score  | \n| -------- |:------:| --------:| ---------------------- |\n|0.941|   0.914  |\t  0.942\t  |0.928 | \n\n\u003cbr clear=\"left\"/\u003e\n\n\n\n\n# General Use\nThe main file contains a few lines of code that create an ML Class depending on \nthe model type: LinearSVC or RandomForest.\n\nA class is created like: \n  - support_svc = ML('file/path/project3.csv')\n  - support_svc.feature_selection()\n  - ...\n### Note\nDealing with Support Vector Machines doesn't usually allow for plotting ROC curves \nsince they don't often predict probabilities(I think )\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhalaway%2Fbig-data-ml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhalaway%2Fbig-data-ml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhalaway%2Fbig-data-ml/lists"}