https://github.com/halaway/big-data-ml
https://github.com/halaway/big-data-ml
Last synced: about 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/halaway/big-data-ml
- Owner: halaway
- Created: 2023-12-06T03:44:19.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-12-06T05:04:18.000Z (over 2 years ago)
- Last Synced: 2025-01-24T20:41:32.259Z (over 1 year ago)
- Language: Python
- Size: 174 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# Machine Learning Models for Cancer Diagnosis: Integrating Spark
### Introduction
Leveraging distributed systems for tackling machine learning problems, typically increases query optimization and model performance when accounting for faster execution times and parallel data processing.
For this project, we are given a CSV file consisting of 569 rows of individual samples defining some diagnosis of either "Benign” or ”Malignant” and twenty measured clinical variables. We then use Spark along with two machine learning models, ie. LinearSVC and Random Forest Classifiers, for predicting the positive target variable of our data set, or the ”Malignant” diagnosis.
| ID | Diagnosis | Radius_mean | Texture_mean | perimeter_mean | area_mean | smoothness_mean | compactness_mean |
| -------- |:------:| --------:| ---------------------- | -------- | ---------| --------| ---------|
|842302 | M | 17.99 | 10.38 |122.8 | 1001 | 0.1184| 0.2776|
| 842517 | M | 20.57 | 21.77 | 132.9 | 1326 | 0.08474 | 0.07864 |
### Integrating Spark With LinearSVC and RandomForest
This project is essentially contained within the ML Class, where different methods
are used for instantiating, fitting, and plotting the model along with their performance
metrics.
### Linear SVC Performance Metrics
| Precision | Recall | Accuracy | F1-Score |
| -------- |:------:| --------:| ---------------------- |
|0.999| 0.914 | 0.965 | 0.955 |
Testing the model's predicted values against the true values created the following metrics
as measures of the model's predictive ability.
### Random Forest Performance Metrics
| Precision | Recall | Accuracy | F1-Score |
| -------- |:------:| --------:| ---------------------- |
|0.941| 0.914 | 0.942 |0.928 |
# General Use
The main file contains a few lines of code that create an ML Class depending on
the model type: LinearSVC or RandomForest.
A class is created like:
- support_svc = ML('file/path/project3.csv')
- support_svc.feature_selection()
- ...
### Note
Dealing with Support Vector Machines doesn't usually allow for plotting ROC curves
since they don't often predict probabilities(I think )