https://github.com/leonism/breast-cancer

How to predict a Breast Cancer patient through Machine Learning modeling with Python, using Pandas, Numpy and SciKit-Learn Libraries
https://github.com/leonism/breast-cancer

data-science machine-learning machinelearning python

Last synced: 2 months ago
JSON representation

How to predict a Breast Cancer patient through Machine Learning modeling with Python, using Pandas, Numpy and SciKit-Learn Libraries

Host: GitHub
URL: https://github.com/leonism/breast-cancer
Owner: leonism
Created: 2020-06-24T01:13:03.000Z (about 6 years ago)
Default Branch: master
Last Pushed: 2024-08-16T04:04:49.000Z (almost 2 years ago)
Last Synced: 2025-03-28T23:46:50.264Z (over 1 year ago)
Topics: data-science, machine-learning, machinelearning, python
Language: Jupyter Notebook
Homepage: https://leonism.github.io/Breast-Cancer/
Size: 6.27 MB
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# Predicting Breast Cancer Using Machine Learning

![header-main.jpg](images/predicting-breast-cancer-using-machine-learning.png)

---

Imagine harnessing the power of machine learning to predict one of the most prevalent and life-threatening diseases: breast cancer. As data science enthusiasts, we often seek new challenges to expand our skills and dive into unexplored territories. This journey not only enhances our technical prowess but also broadens our understanding of diverse fields.

This article invites you to venture beyond the realms of `digital marketing` and `media investment` into the captivating world of `healthcare`. Did you know that cancer is the second leading cause of death globally, accounting for approximately 9.6 million deaths in 2018, according to the [WHO](https://www.who.int/news-room/fact-sheets/detail/cancer#:~:text=Cancer%20is%20the%20second%20leading,-%20and%20middle-income%20countries.). This staggering statistic underscores the urgent need for innovative solutions in early detection and treatment.

Join me as we explore how machine learning can be a game-changer in predicting breast cancer symptoms. We'll utilize a comprehensive dataset from [UCI](https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+%28original%29), generously provided by academicians, to build our predictive model.

To bring this vision to life, we'll employ powerful Python libraries like [Pandas](https://pandas.pydata.org/), [Seaborn](https://seaborn.pydata.org/), and [Scikit-learn](https://scikit-learn.org/). These tools will help us explore, clean, and visualize data, ultimately leading to a robust machine learning model. Ready to embark on this exciting adventure? Let's break it down into manageable steps:

1. **Loading Libraries**
2. **Data Exploration**
3. **Data Visualization**
4. **One Hot Encoding**
5. **Feature Generation**
6. **Data Splitting**
7. **Machine Learning Modeling**
8. **Data Prediction**

Dive in and discover how you can leverage machine learning to make a meaningful impact in the fight against breast cancer.

---

# 1. Load Libraries

Much like any other data exploratory process in Pandas or Python, the initial phase involves loading the essential libraries into our working Jupyter Notebook environment. These libraries are the backbone of our data analysis and machine learning endeavors, providing us with the tools needed to manipulate, visualize, and model our data. Whether you're using `Jupyter Notebook`, `Google Colab`, or `Kaggle`, the process remains largely the same. These platforms offer robust environments that support Python and its libraries, making them ideal for data science projects.

For this tutorial, I'll stick to my faithful Jupyter Notebook environment, known for its versatility and user-friendly interface. Jupyter Notebook allows for an interactive data analysis experience, where code, visualizations, and explanatory text can coexist seamlessly. This setup will enable us to document our process comprehensively and adjust our code on the fly as we delve into the breast cancer dataset. While you're free to use any Integrated Development Environment (IDE) you prefer, Jupyter Notebook's integration with libraries like `Pandas`, `Seaborn`, and `Scikit-learn` makes it an excellent choice for this step-by-step guide.

```python
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import seaborn as sns # visualization library
```

## 1.1 Load Dataset

Start by creating a directory on your computer. Although I'm using a MacOS environment, the instructions provided here are applicable across different platforms. For the purpose of this walkthrough, let's name the directory `Project`. This will serve as our main working directory. Navigate into the `Project` folder, as this will be our base for organizing and executing the steps outlined in this tutorial. The next step is to download the breast cancer dataset from the [UCI](https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+%28original%29) site, which we'll use for our machine learning model.

Within the `Project` directory, create a new folder named `data` and copy the downloaded CSV file into this `data` directory. This organization ensures that all relevant files are neatly stored and easily accessible throughout the tutorial. By structuring our project this way, we facilitate a smooth workflow and maintain order as we progress. Now that everything is set up, we can load the dataset into our Jupyter Notebook. This step allows us to examine, manipulate, and observe the data, laying the groundwork for our machine learning exploration.

```python
df = pd.read_csv('data/breast_cancer_data.csv')
```

---

## 1.2 Dataset Size

Once we have completed the initial setup, we can proceed to analyze our dataset further. A common starting point in any data analysis project is to understand the size of the dataset. You might be wondering, just how large is our dataset? This question is easily answered using the `.shape` method in Pandas.

By applying the `.shape` method to our dataset, we can quickly obtain the number of rows and columns. This method returns a tuple representing the dimensions of the dataset, giving us an immediate sense of its scale. Understanding the size of our dataset is crucial as it informs us about the volume of data we will be working with and helps in planning subsequent data processing and analysis steps.

```python
df.shape
```

(699, 12)

### Rows & Columns

We can see we have the following information at hand:

- rows `699`
- columns `12`

## 1.3 Data Types

It's always a good idea to get cozy with our dataset, not just by looking at its size, but by understanding what it's really made of. Think of it like getting to know a new friend—you wouldn't just ask them how tall they are, right? You'd want to know their quirks, their traits, what makes them tick. The same goes for our data. Knowing the types of data in each column helps us groove through the feature generation phase with ease.

So, let's kick back and take a deeper dive. By checking out the data types of each column, we get the full picture: the numbers, the categories, the text. This insight is like the smooth rhythm of a jazz tune, guiding us to apply the right transformations and manipulations. When we're in sync with our data, everything just flows better, leading to more accurate and reliable models. To get this vibe going, we'll use the `.dtypes` attribute in Pandas. It’s our backstage pass to the inner workings of the dataset, giving us a clear overview of the structure and content. Let's get jazzy with our data and see what it’s composed of!

```python
# We need to observe the data types of each columns
df.dtypes
```

patient_id int64
clump_thickness float64
cell_size_uniformity float64
cell_shape_uniformity int64
marginal_adhesion int64
single_ep_cell_size int64
bare_nuclei object
bland_chromatin float64
normal_nucleoli float64
mitoses int64
class object
doctor_name object
dtype: object

### 1.3.1 The Data Legend

Let's lay down the smooth beats of our dataset. Here's the lowdown on the columns we have, as described by the source:

- `Patient ID`: id number
- `Clump Thickness`: 1–10
- `Uniformity of Cell Size`: 1–10
- `Uniformity of Cell Shape`: 1–10
- `Marginal Adhesion`: 1–10
- `Single Epithelial Cell Size`: 1–10
- `Bare Nuclei`: 1–10
- `Bland Chromatin`: 1–10
- `Normal Nucleoli`: 1–10
- `Mitoses`: 1–10
- `Class`: malignant or benign
- `Doctor name`: 4 different doctors

So, what’s the vibe here? The `Patient ID` is our unique identifier, ensuring each record stands out. The `Class` column is the headline act, telling us whether the tumor is `malignant` (cancerous) or `benign` (not cancerous). The rest of the columns? They're numeric medical descriptions of the tumor, except for `Doctor name`, which adds a categorical twist.

> Keep this in mind—if our goal is to predict whether a tumor is cancerous based on the other features, we’ll need to perform some one-hot encoding on the categorical data and clean up the numerical data. Just like tuning an instrument before a jam session, prepping our data ensures everything flows smoothly in our analysis.

### 1.3.2 First & Last Rows

Now that we've got the lay of the land, let's dive in and see what the top five records in our dataset look like. This peek at the first few rows will give us a quick feel for the data and help us spot any obvious issues or patterns right off the bat.

To do this, we'll use the `.head()` method in Pandas, which will show us the first five rows. It's like getting a sneak preview of the opening act before the main event. This simple step is crucial for ensuring we're on the right track and that our data is ready to roll.

```python
df.head()
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bare_nuclei
bland_chromatin
normal_nucleoli
mitoses
class
doctor_name

0
1000025
5.0
1.0
1
1
2
1
3.0
1.0
1
benign
Dr. Doe

1
1002945
5.0
4.0
4
5
7
10
3.0
2.0
1
benign
Dr. Smith

2
1015425
3.0
1.0
1
1
2
2
3.0
1.0
1
benign
Dr. Lee

3
1016277
6.0
8.0
8
1
3
4
3.0
7.0
1
benign
Dr. Smith

4
1017023
4.0
1.0
1
3
2
1
3.0
1.0
1
benign
Dr. Wong

Additionally, checking the last few records with the `.tail()` method will give us a complete sense of the dataset's structure. This combination of the first and last rows provides a balanced overview, ensuring no surprises lurk at the end. Let's groove through the data and see what stories the top and bottom rows tell us!

```python
df.tail()
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bare_nuclei
bland_chromatin
normal_nucleoli
mitoses
class
doctor_name

694
776715
3.0
1.0
1
1
3
2
1.0
1.0
1
benign
Dr. Lee

695
841769
2.0
1.0
1
1
2
1
1.0
1.0
1
benign
Dr. Smith

696
888820
5.0
10.0
10
3
7
3
8.0
10.0
2
malignant
Dr. Lee

697
897471
4.0
8.0
6
4
3
4
10.0
6.0
1
malignant
Dr. Lee

698
897471
4.0
8.0
8
5
4
5
10.0
4.0
1
malignant
Dr. Wong

# 2. Descriptive Statistics

In Descriptive statistics, we are describing our data with the help of various representative methods using `charts`, `graphs`, `tables`, `excel files`, etc. In descriptive statistics, we describe our data in some manner and present it in a meaningful way so that it can be easily understood. Most of the time it is performed on small data sets and this analysis helps us a lot to predict some future trends based on the current findings. Some measures that are used to describe a data set are measures of central tendency and measures of variability or dispersion.

## 2.1 Numerical Analysis

Let's jazz up our dataset with some sweet statistical insights! With the `.describe()` method, we're about to dive deep into the numerical nitty-gritty. This little trick gives us the lowdown on key stats like `count`, `mean`, and `standard deviation`, shedding light on the distribution and central tendencies of our numeric data.

So, why does this matter? Well, getting cozy with these numbers gives us a clearer picture of what we're working with. It's like fine-tuning our instruments before a performance—it ensures our analysis hits all the right notes. With these stats in hand, we can groove through our dataset with confidence, uncovering hidden patterns and trends along the way. Let's crank up the volume and see what the numbers have to say! 🎶

```python
df.describe()
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bland_chromatin
normal_nucleoli
mitoses

count
6.990000e+02
698.000000
698.000000
699.000000
699.000000
699.000000
695.000000
698.000000
699.000000

mean
1.071704e+06
4.416905
3.137536
3.207439
2.793991
3.216023
3.447482
2.868195
1.589413

std
6.170957e+05
2.817673
3.052575
2.971913
2.843163
2.214300
2.441191
3.055647
1.715078

min
6.163400e+04
1.000000
1.000000
1.000000
1.000000
1.000000
1.000000
1.000000
1.000000

25%
8.706885e+05
2.000000
1.000000
1.000000
1.000000
2.000000
2.000000
1.000000
1.000000

50%
1.171710e+06
4.000000
1.000000
1.000000
1.000000
2.000000
3.000000
1.000000
1.000000

75%
1.238298e+06
6.000000
5.000000
5.000000
3.500000
4.000000
5.000000
4.000000
1.000000

max
1.345435e+07
10.000000
10.000000
10.000000
10.000000
10.000000
10.000000
10.000000
10.000000

## 2.2 Categorical Analysis

Just like tuning into a different frequency, let's shift our focus to the categorical side of the spectrum. With the `.describe(include=['O'])` method, we're about to unravel the mysteries of our categorical data. While the output might be a bit more concise compared to its numerical counterpart, it still packs a punch.

By honing in on the categorical variables—those with a data type of `object`—we gain valuable insights into their distribution and uniqueness. It's like flipping through the pages of a well-worn record collection, each category offering its own distinct vibe.

So, why bother? Well, understanding the landscape of our categorical data sets the stage for deeper analysis. Just like a DJ crafting the perfect mix, these insights help us blend and remix our data with precision. With the `.describe(include=['O'])` method in hand, we're ready to spin some categorical magic and uncover the stories hidden within our dataset. Let's dive in and see what melodies await! 🎵

```python
df.describe(include=['O'])
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

.dataframe thead th {
text-align: right;
}

bare_nuclei
class
doctor_name

count
697
699
699

unique
11
2
4

top
1
benign
Dr. Doe

freq
401
458
185

---

# 3. Data Reshaping

Time to remix our data and give it a fresh new vibe! With the code snippet you've got in hand, we're about to shake things up and reshape our dataset like never before. By grooving to the beat of `df.groupby(by=['doctor_name', 'class']).count()`, we're taking our data on a whole new journey.

Picture this: we're gathering our data around the DJ booth, grouping it by the soothing sounds of `doctor_name` and the electrifying beats of `class`. Then, we crank up the volume with the aggregation function, counting up the hits in each group. It's like taking our dataset to a cool underground club, where every combination of doctor and class brings its own unique vibe.

Why does this matter? Well, reshaping our data in this way allows us to uncover fresh insights and patterns that might have been hidden before. It's like remixing a classic track—same ingredients, but with a whole new flavor. So, grab your data and let's hit the dance floor, because we're about to reshape it into something truly groovy! 🎧💃

```python
# This aggreates the data by its column names, then we pass the aggregation function (size = count)
df.groupby(by =['doctor_name', 'class']).count()
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bare_nuclei
bland_chromatin
normal_nucleoli
mitoses

doctor_name
class

Dr. Doe
benign
127
127
126
127
127
127
126
126
127
127

malignant
58
58
58
58
58
58
58
57
58
58

Dr. Lee
benign
121
121
121
121
121
121
121
119
121
121

malignant
60
60
60
60
60
60
60
60
60
60

Dr. Smith
benign
102
102
102
102
102
102
102
102
102
102

malignant
74
73
74
74
74
74
73
74
74
74

Dr. Wong
benign
108
108
108
108
108
108
108
108
107
108

malignant
49
49
49
49
49
49
49
49
49
49

```python
df.groupby(by =['class', 'doctor_name']).count()
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bare_nuclei
bland_chromatin
normal_nucleoli
mitoses

class
doctor_name

benign
Dr. Doe
127
127
126
127
127
127
126
126
127
127

Dr. Lee
121
121
121
121
121
121
121
119
121
121

Dr. Smith
102
102
102
102
102
102
102
102
102
102

Dr. Wong
108
108
108
108
108
108
108
108
107
108

malignant
Dr. Doe
58
58
58
58
58
58
58
57
58
58

Dr. Lee
60
60
60
60
60
60
60
60
60
60

Dr. Smith
74
73
74
74
74
74
73
74
74
74

Dr. Wong
49
49
49
49
49
49
49
49
49
49

```python
df.groupby(by =['bare_nuclei', 'class']).count()
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bland_chromatin
normal_nucleoli
mitoses
doctor_name

bare_nuclei
class

1
benign
386
386
386
386
386
386
383
385
386
386

malignant
15
15
15
15
15
15
15
15
15
15

10
benign
3
3
2
3
3
3
3
3
3
3

malignant
128
128
128
128
128
128
128
128
128
128

2
benign
21
21
21
21
21
21
21
21
21
21

malignant
9
9
9
9
9
9
9
9
9
9

3
benign
14
14
14
14
14
14
14
14
14
14

malignant
14
13
14
14
14
14
14
14
14
14

4
benign
6
6
6
6
6
6
6
6
6
6

malignant
13
13
13
13
13
13
13
13
13
13

5
benign
10
10
10
10
10
10
10
10
10
10

malignant
20
20
20
20
20
20
20
20
20
20

6
malignant
4
4
4
4
4
4
4
4
4
4

7
benign
1
1
1
1
1
1
1
1
1
1

malignant
7
7
7
7
7
7
6
7
7
7

8
benign
2
2
2
2
2
2
2
2
2
2

malignant
19
19
19
19
19
19
19
19
19
19

9
malignant
9
9
9
9
9
9
9
9
9
9

?
benign
14
14
14
14
14
14
14
14
14
14

malignant
2
2
2
2
2
2
2
2
2
2

# 4. Data Cleaning

Alright, time to chill and tidy up our dataset! Now that we've wrapped up the early analysis phase, it's onto the next groove: cleaning up our data. Picture this: your data rolls in with all sorts of shapes and sizes, like records in a crate waiting to be sorted. But the real magic happens when we polish it up, turning it into the complete and comprehensive masterpiece we need.

Sure, it's like sifting through a crate of vinyl, each record with its own scratches and dust. But trust me, the best jams come from the cleanest cuts. By whipping our dataset into shape, we're setting the stage for some serious feature engineering and analysis down the line. So, grab your data mop and broom, because we're about to sweep away the dust and uncover the smooth grooves beneath. Let's get cleaning! 🎶✨

## 4.1 Missing Records

Among one of the easiet way to identify whether or not your dataset has any missing data in them, would be to check them using the `.isna()` method and combine them with the `.sum()` function. It would in return, give you information on how many rows gone missing in your current dataset. Usually Pandas, would assing them with the value of `NaN`, but it can always be just a blank value in the record cell.

```python
df.isna().sum()
```

patient_id 0
clump_thickness 1
cell_size_uniformity 1
cell_shape_uniformity 0
marginal_adhesion 0
single_ep_cell_size 0
bare_nuclei 2
bland_chromatin 4
normal_nucleoli 1
mitoses 0
class 0
doctor_name 0
dtype: int64

Good to know that the `patient_id` has `0` missing values, but as you may notice, others columns much like `clump_thickness`, `cell_size_uniformity`, `bare_nuclei`, `bland_chromatin` and `normal_nucleoli`, and to put them in total, there are 9 missing rows in the dataset.

## 4.2 How To Deal With?

The real question isn't just about spotting the missing records and summing them up. The real jazz starts when you decide how to handle them before moving forward on your data wrangling journey. In this particular case, we've got a small amount of data with missing values—just `9 rows` out of `699`. That's a mere `0.012`, or less than 1% of the total dataset. With such a small fraction, I'm thinking we drop them like they're hot, using the `.dropna` method. And while we're at it, let’s break down the cool attributes that groove along with the `.dropna` method.

- `Axis`: Decides if you're dropping rows or columns. `0` means rows, while `1` goes for columns.
- `How`: Two vibes here—_any_ or _all_. If you choose `all`, it drops rows or columns that are completely empty. Opt for `any`, and it drops those with even a single missing value.
- `Inplace`: This one's crucial. If you set `inplace=True`, changes will happen right on the DataFrame you're working with. If it's `False` (which is the default), the original DataFrame stays untouched, and a new one is returned.

So, let's clean up those missing beats and keep the data flowing smoothly!

```python
df.dropna(axis=0, how='any', inplace=True)
```

```python
df
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

.dataframe thead th {
text-align: right;
}

690 rows × 12 columns

As you can see now, the rows number have been decreased, from `699` to `690`, down with `9` records, but left us with clean dataset with no empty cells in them. Let move on to check them!

```python
df.isnull().values.any()
```

False

## 4.3 Rechecking

Now that we've got our dataset shining bright with no empty records, we might still be wondering if there’s another way to double-check for any sneaky missing values. Good news, data groovers! There's a slick method called `.isnull` that performs a boolean check, giving you a smooth true or false response to your inquiry. It's like having a jazz soloist confirming every note is in place. So, let's slide into it and make sure our dataset is as clean as a crisp vinyl record. Let’s do this! 🎷✨

## 4.4 Validating

So, our dataset’s looking sharp, but let’s not stop there. If you're curious whether there are still any hidden empty cells lurking around, there's a cool cat method called `.isnull` that’s perfect for the job. This boolean checker will let us know with a simple true or false if any values are missing. It’s like having an extra pair of ears in the studio, ensuring every beat is perfect. Let’s give it a spin and make sure everything's in tip-top shape! 🎶🔍

```python
df.isnull()
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bare_nuclei
bland_chromatin
normal_nucleoli
mitoses
class
doctor_name

0
False
False
False
False
False
False
False
False
False
False
False
False

1
False
False
False
False
False
False
False
False
False
False
False
False

2
False
False
False
False
False
False
False
False
False
False
False
False

3
False
False
False
False
False
False
False
False
False
False
False
False

4
False
False
False
False
False
False
False
False
False
False
False
False

...
...
...
...
...
...
...
...
...
...
...
...
...

694
False
False
False
False
False
False
False
False
False
False
False
False

695
False
False
False
False
False
False
False
False
False
False
False
False

696
False
False
False
False
False
False
False
False
False
False
False
False

697
False
False
False
False
False
False
False
False
False
False
False
False

698
False
False
False
False
False
False
False
False
False
False
False
False

690 rows × 12 columns

So far, so groovy! From our dataset checks, everything's coming back with `False` values, and that's music to our ears. It means one thing: our dataset is spotless and ready to jam. With our data all tuned up, it's time to move on to the next leg of our journey. So, let's keep the rhythm going and dive into the next adventure. Onward to data greatness! 🎷✨

## 4.5 Duplicate Records

Alright, let's jazz things up and hunt for those duplicate records! First, we need to investigate whether our dataset is hiding any duplicate grooves in the cell records. By getting on top of this early, we can dodge potential hurdles that might throw our analysis offbeat and introduce unwanted bias.

To kick off this detective work, we’ll use the `.nunique` method. This little gem will give us some solid pointers to detect any anomalies lurking in our dataset. We’ll start by grooving through the columns that are supposed to have unique identifiers—those special `object` datatype columns. In our case, it’s the `patient_id` column. So, let’s spin that record and see if we have any duplicates in the mix! 🎷🔍

```python
df.nunique()
```

patient_id 637
clump_thickness 10
cell_size_uniformity 10
cell_shape_uniformity 10
marginal_adhesion 10
single_ep_cell_size 10
bare_nuclei 11
bland_chromatin 10
normal_nucleoli 10
mitoses 9
class 2
doctor_name 4
dtype: int64

```python
df.info()
```

Index: 690 entries, 0 to 698
Data columns (total 12 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 patient_id 690 non-null int64
1 clump_thickness 690 non-null float64
2 cell_size_uniformity 690 non-null float64
3 cell_shape_uniformity 690 non-null int64
4 marginal_adhesion 690 non-null int64
5 single_ep_cell_size 690 non-null int64
6 bare_nuclei 690 non-null object
7 bland_chromatin 690 non-null float64
8 normal_nucleoli 690 non-null float64
9 mitoses 690 non-null int64
10 class 690 non-null object
11 doctor_name 690 non-null object
dtypes: float64(4), int64(5), object(3)
memory usage: 70.1+ KB

We know for sure that our dataset is jamming with `690` rows and `12` columns (down from the previous `699`). But hold on a second—when we dig into the groove, we find that `patient_id` only has `637` records. Something's offbeat here, especially since `patient_id` should be our unique identifier. We should be seeing a solid `690` records, not just `637`.

Time to put on our detective hats and investigate this mystery. There’s gotta be some duplication in the `patient_id` column messing with our flow. Let’s dive deep, spin those records backwards, and uncover where the duplicates are hiding. This dataset is about to get a clean remix! 🎷🔍

## 4.6 Duplicate Patients

Alright, buckle up, because we're about to dive into the mystery of the duplicate `patient_id` records. Picture this: you're flipping through your dataset like a detective, and suddenly, you stumble upon some suspicious duplicates. But fear not, because we've got just the solution to unravel this enigma.

We're borrowing a slick move from the data science playbook, courtesy of the wizards over at Stack Overflow. This little trick is like shining a spotlight on the shadows, revealing all the duplicate items lurking in the shadows of our dataset. With this solution in hand, we'll shine a light on those repeat offenders and get to the bottom of this duplication dilemma. So, get ready to crack the case and uncover the truth behind those duplicate patients! 🕵️‍♂️🔍

- borrow from https://stackoverflow.com/questions/14657241/how-do-i-get-a-list-of-all-the-duplicate-items-using-pandas-in-python

```python
df[df.patient_id.duplicated(keep=False)].sort_values("patient_id")
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bare_nuclei
bland_chromatin
normal_nucleoli
mitoses
class
doctor_name

267
320675
3.0
3.0
5
2
3
10
7.0
1.0
1
malignant
Dr. Wong

272
320675
3.0
3.0
5
2
3
10
7.0
1.0
1
malignant
Dr. Smith

575
385103
5.0
1.0
2
1
2
1
3.0
1.0
1
benign
Dr. Smith

269
385103
1.0
1.0
1
1
2
1
3.0
1.0
1
benign
Dr. Doe

271
411453
5.0
1.0
1
1
2
1
3.0
1.0
1
benign
Dr. Wong

...
...
...
...
...
...
...
...
...
...
...
...
...

560
1321942
5.0
1.0
1
1
2
1
3.0
1.0
1
benign
Dr. Doe

660
1339781
1.0
1.0
1
1
2
1
2.0
1.0
1
benign
Dr. Lee

661
1339781
4.0
1.0
1
1
2
1
3.0
1.0
1
benign
Dr. Smith

672
1354840
2.0
1.0
1
1
2
1
3.0
1.0
1
benign
Dr. Wong

673
1354840
5.0
3.0
2
1
3
1
1.0
1.0
1
benign
Dr. Lee

98 rows × 12 columns

Alright, check this out: we've got a little situation on our hands. It seems like we've got `98` patient IDs making multiple appearances in our dataset. Some are showing up twice, while others are pulling off the triple play. Now, wouldn't it be sweet if we could get the lowdown on exactly how many times each patient ID is making a cameo?

Well, guess what? We're about to dive into the nitty-gritty and unravel this mystery. Picture this: we're peeling back the layers of duplication, analyzing each instance to tally up the total count. It's like detective work for data scientists—sleuthing through the numbers to uncover the truth. So, grab your magnifying glass and let's crack this case wide open. We're diving deep into the world of duplications, ready to count 'em up and bring clarity to our dataset! 🕵️‍♂️🔍

## 4.7 Duplicate Records

Let's analyze how many times a single `patient_id` value, was being recorded more than once, in the next table.

- borrow from https://stackoverflow.com/questions/38309729/count-unique-values-with-pandas-per-groups

```python
df.patient_id.value_counts()
```

patient_id
1182404 6
1276091 5
1198641 3
897471 2
411453 2
..
1231706 1
1232225 1
1236043 1
1241232 1
809912 1
Name: count, Length: 637, dtype: int64

Surpisingly, some are getting recorded more than twice, some are even getting recorded 6 times. Let's move on to the next steps on how to deal with them.

```python
df.drop_duplicates(subset="patient_id", keep='first', inplace = True)
df # let's print them.
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

637 rows × 12 columns

Great, the above code just left us with one clean and no duplicated rows of data. Now the records are down from `690` to `637`. Now let's check wheter their still duplicates from the previous list of `patient_id` we had queried earlier, let's try the `1182404` `patient_id` string for that matter.

```python
# let's check whether the 1182404 patient_id still has duplication.
df.loc[df['patient_id'] == 1182404]
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

```python
df.isnull().values.any()
```

False

# 5. Visual Analysis

And they say, picture says a thousand words. And I couldn't agree more with the statement, we as a human easily absorb information, through graphs, colors and visualization, in contrast to just plain numbers. In this section, let's try to visualize our findings better.

There are numerous great visualization libraries out there for both Python and Pandas, but I've been experimenting with Seaborn for awhile, and found them somewhat easier to implement to our objectives. Here are some of the benefit of having Seaborn as your library of choice for visualtization as taken from the official homepage:

> Seaborn aims to make visualization a central part of exploring and understanding data. Its dataset-oriented plotting > functions operate on dataframes and arrays containing whole datasets and internally perform the necessary semantic
> mapping and statistical aggregation to produce informative plots.

Here is some of the functionality that seaborn offers:

- A dataset-oriented API for examining [relationships](https://seaborn.pydata.org/examples/scatter_bubbles.html#scatter-bubbles) between [multiple variables](https://seaborn.pydata.org/examples/faceted_lineplot.html#faceted-lineplot)
- Specialized support for using categorical variables to show [observations](https://seaborn.pydata.org/examples/jitter_stripplot.html#jitter-stripplot) or [aggregate statistics](https://seaborn.pydata.org/examples/pointplot_anova.html#pointplot-anova)
- Options for visualizing [univariate](https://seaborn.pydata.org/examples/distplot_options.html#distplot-options) or [bivariate](https://seaborn.pydata.org/examples/joint_kde.html#joint-kde) distributions and for [comparing](https://seaborn.pydata.org/examples/horizontal_boxplot.html#horizontal-boxplot) them between subsets of data
- Automatic estimation and plotting of [linear regression](https://seaborn.pydata.org/examples/anscombes_quartet.html#anscombes-quartet) models for different kinds [dependent](https://seaborn.pydata.org/examples/logistic_regression.html#logistic-regression) variables
- Convenient views onto the overall [structure](https://seaborn.pydata.org/examples/scatterplot_matrix.html#scatterplot-matrix) of complex datasets
- High-level abstractions for structuring [multi-plot grids](https://seaborn.pydata.org/examples/faceted_histogram.html#faceted-histogram) that let you easily build [complex](https://seaborn.pydata.org/examples/pair_grid_with_kde.html#pair-grid-with-kde) visualizations
- Concise control over matplotlib figure styling with several [built-in themes](https://seaborn.pydata.org/tutorial/aesthetics.html#aesthetics-tutorial)
- Tools for choosing [color palettes](https://seaborn.pydata.org/tutorial/color_palettes.html#palette-tutorial) that faithfully reveal patterns in your data

## 5.2 Patients for Each Doctor?

Ever wonder how many patients each doctor handled from the dataset? We know for sure, we have 4 doctors from the dataset, but haven't got some perfect ideas on how many patients each doctor is handling them. So why don't we try to visualize them, to see how many patients for each doctor needs to handle from the dataset?

```python
df['doctor_name'].value_counts()
```

doctor_name
Dr. Doe 167
Dr. Lee 165
Dr. Smith 164
Dr. Wong 141
Name: count, dtype: int64

```python
df['patient_id'].count()
```

637

```python
# updated code
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

# Define the dimensions of the figure
fig_dims = (12, 6)
fig, ax = plt.subplots(figsize=fig_dims)

# Set titles and labels
ax.set_title("How Many Patients Per Doctor", fontsize=14)
ax.set_xlabel('Doctor Name', fontsize=12)
ax.set_ylabel('Patients', fontsize=12)

# Set Seaborn style and despine the plot
sns.set_style('whitegrid')
sns.despine()

# Create countplot
sns.countplot(x='doctor_name', hue='doctor_name', palette='RdBu_r', data=df, dodge=False, legend=False)

# Show the plot
plt.show()
```

![png](images/output_62_0.png)

- `Dr. Doe` :167
- `Dr. Lee` :165
- `Dr. Smith`:164
- `Dr. Wong` :141

## 5.3 Class Cases For Each Doctor?

As mentioned on the earlier sections, we have a column name `class`, which basically contains the value of either `benign` and `malignant`. We wish to understand further whether a person's tumor is `malignant` (cancerous) or `benign` (not cancerous). With that being said, let's get down to business and try to visualize them further down below.

```python
# Print the column names of the DataFrame to check for the correct column name
print(df.columns)
```

Index(['patient_id', 'clump_thickness', 'cell_size_uniformity',
'cell_shape_uniformity', 'marginal_adhesion', 'single_ep_cell_size',
'bare_nuclei', 'bland_chromatin', 'normal_nucleoli', 'mitoses', 'class',
'doctor_name'],
dtype='object')

```python
df.columns = df.columns.str.strip()
print(df.columns)
```

```python
# class_by_doctor = df[("class")].value_counts()
# class_by_doctor

if 'class' in df.columns:
class_by_doctor = df['class'].value_counts()
print(class_by_doctor)
else:
print("The 'class' column is not present in the DataFrame.")
```

class
benign 407
malignant 230
Name: count, dtype: int64

```python
import matplotlib.pyplot as plt
import seaborn as sns

# updated code
fig_dims = (12, 6)
fig, ax = plt.subplots(figsize=fig_dims)

ax.set_title("Patient with Cancer Syndrome Per Doctor", fontsize=15)
ax.set_xlabel('X_axis', fontsize=12)
ax.set_ylabel('Y_axis', fontsize=12)

sns.despine()
sns.set_style('whitegrid')
sns.barplot(x="class", y="patient_id", hue="doctor_name", errorbar=None, palette='RdBu_r', data=df)

plt.show()

```

![png](images/output_68_0.png)

```python
# This time, let's do them horizontally

# updated code
ax.set_title("Class of Patient Per Doctor", fontsize=15)
ax.set_xlabel('Doctor Name', fontsize=12)
ax.set_ylabel('Patients', fontsize=12)

sns.despine()
sns.set_style('whitegrid')
sns.barplot(x="patient_id", y="class", hue="doctor_name", errorbar=None, palette='RdBu_r', data=df)

plt.show()
```

![png](images/output_69_0.png)

## 5.4 Class Case Per Doctor?

As mentioned on the earlier sections, we have a column name `class`, which basically contains the value of `benign` and `malignant`. We wish to understand further whether a person's tumor is `malignant` (cancerous) or `benign` (not cancerous). With that being said, let's get down to business and try to visualize them further down below.

```python
# udpated code
import matplotlib.pyplot as plt
import seaborn as sns

# Define the figure dimensions
fig_dims = (12, 6)
fig, ax = plt.subplots(figsize=fig_dims)

# Set plot title and labels
ax.set_title("Class Per Doctor", fontsize=15)
ax.set_xlabel('Doctor Name', fontsize=12)
ax.set_ylabel('Patients', fontsize=12)

# Set style and despine
sns.set_style('whitegrid')
sns.despine()

# Create the barplot with the updated parameter
sns.barplot(x='doctor_name', y='patient_id', hue='class', errorbar=None, palette='RdBu_r', data=df)

# Show the plot
plt.show()
```

![png](images/output_71_0.png)

```python
# updated code
fig_dims = (12, 6)
fig, ax = plt.subplots(figsize=fig_dims)

ax.set_title("Horizontal Classes Per Doctor", fontsize=15)
ax.set_xlabel('Doctor Name', fontsize=12)
ax.set_ylabel('Patients', fontsize=12)

sns.despine()
sns.set_style('whitegrid')
sns.barplot(y='doctor_name', x='patient_id', hue='class', errorbar=None, palette='RdBu_r', data=df)

plt.show()
```

![png](images/output_72_0.png)

```python
df.isnull().values.any()
df.isnull().sum().sum()
```

---

# 6. One Hot Encoding

Now that we've gone through the previous topic of visualizing our dataset, let's continue to the next section of preparing them in a way that our machine learning algorithms, by which will be using them near the end of this article, would be able to pick them up and run them through our `predictive model` easily. You may ask, "Of all the previous process, they're not enough?". Well apparently, it's not sufficient enough to meet the standards.

As among one of the challenges that we're facing is still within the dataset itself. We'll be better off by modifying them to meet the requirements. Our dataset still consist some categorical values in them, the `doctors_name` and `class` columns are two of good examples. And Machine Learning algorithm don't normally like them. We need to modify these two columns, so that it would make it easier and less confusing for the machine learning model to process through. I came across this [great example](https://medium.com/@contactsunny/label-encoder-vs-one-hot-encoder-in-machine-learning-3fc273365621) on how to deal with the similar situation.

## 6.1 `doctor_name` column.

Let's first try to deal with the `doctor_name` column. This particular consist of 4 distinct values in them and how Pandas would handle them would probably as an object rather than an integer. Let's have our work around for this particular area. Will create another variable and call it `doctors_hotEncoded` and use the `get_dummies` method to transform them to an encoded one.

```python
doctors_hotEncoded = pd.get_dummies(df['doctor_name'])
```

```python
doctors_hotEncoded
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

Dr. Doe
Dr. Lee
Dr. Smith
Dr. Wong

0
True
False
False
False

1
False
False
True
False

2
False
True
False
False

3
False
False
True
False

4
False
False
False
True

...
...
...
...
...

693
False
True
False
False

694
False
True
False
False

695
False
False
True
False

696
False
True
False
False

697
False
True
False
False

637 rows × 4 columns

```python
combined_doctors_hotEncoded_df = pd.concat([df, doctors_hotEncoded], axis=1)
```

```python
combined_doctors_hotEncoded_df
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bare_nuclei
bland_chromatin
normal_nucleoli
mitoses
class
doctor_name
Dr. Doe
Dr. Lee
Dr. Smith
Dr. Wong

0
1000025
5.0
1.0
1
1
2
1
3.0
1.0
1
benign
Dr. Doe
True
False
False
False

1
1002945
5.0
4.0
4
5
7
10
3.0
2.0
1
benign
Dr. Smith
False
False
True
False

2
1015425
3.0
1.0
1
1
2
2
3.0
1.0
1
benign
Dr. Lee
False
True
False
False

3
1016277
6.0
8.0
8
1
3
4
3.0
7.0
1
benign
Dr. Smith
False
False
True
False

4
1017023
4.0
1.0
1
3
2
1
3.0
1.0
1
benign
Dr. Wong
False
False
False
True

...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...

693
763235
3.0
1.0
1
1
2
1
2.0
1.0
2
benign
Dr. Lee
False
True
False
False

694
776715
3.0
1.0
1
1
3
2
1.0
1.0
1
benign
Dr. Lee
False
True
False
False

695
841769
2.0
1.0
1
1
2
1
1.0
1.0
1
benign
Dr. Smith
False
False
True
False

696
888820
5.0
10.0
10
3
7
3
8.0
10.0
2
malignant
Dr. Lee
False
True
False
False

697
897471
4.0
8.0
6
4
3
4
10.0
6.0
1
malignant
Dr. Lee
False
True
False
False

637 rows × 16 columns

```python
# Now let's drop the 'doctor_name' varibale
combined_doctors_hotEncoded_df = combined_doctors_hotEncoded_df.drop(columns=['doctor_name'])
```

```python
# This is how it would look like.
combined_doctors_hotEncoded_df
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bare_nuclei
bland_chromatin
normal_nucleoli
mitoses
class
Dr. Doe
Dr. Lee
Dr. Smith
Dr. Wong

0
1000025
5.0
1.0
1
1
2
1
3.0
1.0
1
benign
True
False
False
False

1
1002945
5.0
4.0
4
5
7
10
3.0
2.0
1
benign
False
False
True
False

2
1015425
3.0
1.0
1
1
2
2
3.0
1.0
1
benign
False
True
False
False

3
1016277
6.0
8.0
8
1
3
4
3.0
7.0
1
benign
False
False
True
False

4
1017023
4.0
1.0
1
3
2
1
3.0
1.0
1
benign
False
False
False
True

...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...

693
763235
3.0
1.0
1
1
2
1
2.0
1.0
2
benign
False
True
False
False

694
776715
3.0
1.0
1
1
3
2
1.0
1.0
1
benign
False
True
False
False

695
841769
2.0
1.0
1
1
2
1
1.0
1.0
1
benign
False
False
True
False

696
888820
5.0
10.0
10
3
7
3
8.0
10.0
2
malignant
False
True
False
False

697
897471
4.0
8.0
6
4
3
4
10.0
6.0
1
malignant
False
True
False
False

637 rows × 15 columns

```python
combined_doctors_hotEncoded_df.isnull().values.any()
combined_doctors_hotEncoded_df.isnull().sum().sum()
```

## 6.2 `class` column.

```python
# How to convert benign & malingant to 0 and 1
change_class_numeric = {'benign':0, 'malignant':1}
combined_doctors_hotEncoded_df['class'] = combined_doctors_hotEncoded_df['class'].map(change_class_numeric)
```

```python
combined_doctors_hotEncoded_df
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bare_nuclei
bland_chromatin
normal_nucleoli
mitoses
class
Dr. Doe
Dr. Lee
Dr. Smith
Dr. Wong

0
1000025
5.0
1.0
1
1
2
1
3.0
1.0
1
0
True
False
False
False

1
1002945
5.0
4.0
4
5
7
10
3.0
2.0
1
0
False
False
True
False

2
1015425
3.0
1.0
1
1
2
2
3.0
1.0
1
0
False
True
False
False

3
1016277
6.0
8.0
8
1
3
4
3.0
7.0
1
0
False
False
True
False

4
1017023
4.0
1.0
1
3
2
1
3.0
1.0
1
0
False
False
False
True

...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...

693
763235
3.0
1.0
1
1
2
1
2.0
1.0
2
0
False
True
False
False

694
776715
3.0
1.0
1
1
3
2
1.0
1.0
1
0
False
True
False
False

695
841769
2.0
1.0
1
1
2
1
1.0
1.0
1
0
False
False
True
False

696
888820
5.0
10.0
10
3
7
3
8.0
10.0
2
1
False
True
False
False

697
897471
4.0
8.0
6
4
3
4
10.0
6.0
1
1
False
True
False
False

637 rows × 15 columns

```python
#Making a new column based on a nuemrical calcualtion of other columns in the df
combined_doctors_hotEncoded_df['new_column'] = df.normal_nucleoli * df.mitoses
```

```python
combined_doctors_hotEncoded_df.head()
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

```python
combined_doctors_hotEncoded_df.isnull().values.any()
combined_doctors_hotEncoded_df.isnull().sum().sum()
```

# 7. Feature Generation

This is among the crucial aspect area of `Machine Learning` model in the article, as this article [point out](https://towardsdatascience.com/data-wrangling-with-pandas-5b0be151df4e) an individual might be classified as having a cancer if meet the following condtion:

- Their `cell_size_uniformity` is greater than 5, and
- Their `cell_shape_uniformity` is greater than 5.

Based on this information, we could create another Feature from them.

```python
# Feature building:
def celltypelabel(x):
if ((x['cell_size_uniformity'] > 5) & (x['cell_shape_uniformity'] > 5)):
return('1')
else:
return('0')
```

The code provided defines a function `celltypelabel` that takes a dictionary `x` as input and returns a string value based on the values of two specific keys within that dictionary: `'cell_size_uniformity'` and `'cell_shape_uniformity'`.

Here’s a step-by-step explanation of the function:

1. **Function Definition**:

````python
def celltypelabel(x):
```
This defines a function named `celltypelabel` that takes one argument, `x`.

````

2. **Conditional Statement**:

````python
if ((x['cell_size_uniformity'] > 5) & (x['cell_shape_uniformity'] > 5)):
```
This is a conditional statement that checks two conditions:
- The value of `'cell_size_uniformity'` in the dictionary `x` must be greater than 5.
- The value of `'cell_shape_uniformity'` in the dictionary `x` must also be greater than 5.

````

3. **Logical AND Operator (`&`)**:
The logical AND operator (`&`) is used to combine these two conditions. Both conditions must be true for this part of the if-statement to evaluate to True.

4. **Return Statement**:

```python
return('1')
```

If both conditions are met (i.e., both `'cell_size_uniformity'` and `'cell_shape_uniformity'` are greater than 5), the function returns the string `'1'`.

5. **Else Clause**:
```python
else:
return('0')
```
If either or both conditions are not met (i.e., either `'cell_size_uniformity'` or `'cell_shape_uniformity'` is not greater than 5), the function returns the string `'0'`.

Then we use the pandas apply function to run the `celltypelabel(x)` function on the dataframe.

```python
combined_doctors_hotEncoded_df['cell_type_label'] = combined_doctors_hotEncoded_df.apply(lambda x: celltypelabel(x), axis=1)
```

The code snippet provided is used to apply the `celltypelabel` function to each row of the `combined_doctors_hotEncoded_df` DataFrame. Here’s a breakdown of what this code does:

```python
combined_doctors_hotEncoded_df['cell_type_label'] = combined_doctors_hotEncoded_df.apply(lambda x: celltypelabel(x), axis=1)
```

### 7.1 Explanation:

1. **Apply Method**: The `.apply()` method is used to apply a function to each row or column of a DataFrame.
2. **Lambda Function**: A lambda function is used to define the function to be applied. In this case, it calls the `celltypelabel` function.
3. **Axis=1**: The `axis=1` parameter specifies that the function should be applied to each row (as opposed to each column, which would be `axis=0`).

### 7.2 Step-by-Step Breakdown:

1. **Define the Lambda Function**:

```python
lambda x: celltypelabel(x)
```

This defines a lambda function that takes a dictionary `x` and calls the `celltypelabel` function.

2. **Apply the Lambda Function to Each Row**:

```python
combined_doctors_hotEncoded_df.apply(lambda x: celltypelabel(x), axis=1)
```

This applies the lambda function to each row of the DataFrame. The result of this application is a new Series where each element is the output of the `celltypelabel` function for that corresponding row.

3. **Assign Result to New Column**:
```python
combined_doctors_hotEncoded_df['cell_type_label'] = ...
```
The result of applying the lambda function is assigned to a new column named `'cell_type_label'` in the DataFrame.

```python
combined_doctors_hotEncoded_df[['patient_id', 'cell_type_label']]
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
cell_type_label

0
1000025
0

1
1002945
0

2
1015425
0

3
1016277
1

4
1017023
0

...
...
...

693
763235
0

694
776715
0

695
841769
0

696
888820
1

697
897471
1

637 rows × 2 columns

```python
combined_doctors_hotEncoded_df
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bare_nuclei
bland_chromatin
normal_nucleoli
mitoses
class
Dr. Doe
Dr. Lee
Dr. Smith
Dr. Wong
new_column
cell_type_label

0
1000025
5.0
1.0
1
1
2
1
3.0
1.0
1
0
True
False
False
False
1.0
0

1
1002945
5.0
4.0
4
5
7
10
3.0
2.0
1
0
False
False
True
False
2.0
0

2
1015425
3.0
1.0
1
1
2
2
3.0
1.0
1
0
False
True
False
False
1.0
0

3
1016277
6.0
8.0
8
1
3
4
3.0
7.0
1
0
False
False
True
False
7.0
1

4
1017023
4.0
1.0
1
3
2
1
3.0
1.0
1
0
False
False
False
True
1.0
0

...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...

693
763235
3.0
1.0
1
1
2
1
2.0
1.0
2
0
False
True
False
False
2.0
0

694
776715
3.0
1.0
1
1
3
2
1.0
1.0
1
0
False
True
False
False
1.0
0

695
841769
2.0
1.0
1
1
2
1
1.0
1.0
1
0
False
False
True
False
1.0
0

696
888820
5.0
10.0
10
3
7
3
8.0
10.0
2
1
False
True
False
False
20.0
1

697
897471
4.0
8.0
6
4
3
4
10.0
6.0
1
1
False
True
False
False
6.0
1

637 rows × 17 columns

```python
combined_doctors_hotEncoded_df.isnull().values.any()
combined_doctors_hotEncoded_df.isnull().sum().sum()
```

```python
combined_doctors_hotEncoded_df.describe()
```

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}

patient_id
clump_thickness
cell_size_uniformity
cell_shape_uniformity
marginal_adhesion
single_ep_cell_size
bland_chromatin
normal_nucleoli
mitoses
class
new_column

count
6.370000e+02
637.000000
637.000000
637.000000
637.000000
637.000000
637.000000
637.000000
637.000000
637.000000
637.000000

mean
1.076689e+06
4.488226
3.210361
3.298273
2.897959
3.284144
3.516484
2.971743
1.629513
0.361068
7.197802

std
6.408652e+05
2.855856
3.080628
3.0

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/leonism/breast-cancer

Awesome Lists containing this project

README