{"id":20303589,"url":"https://github.com/elahe-dastan/shovel","last_synced_at":"2025-03-04T07:12:21.983Z","repository":{"id":113152858,"uuid":"308933903","full_name":"elahe-dastan/shovel","owner":"elahe-dastan","description":"Digging for Data :pick:","archived":false,"fork":false,"pushed_at":"2021-06-08T14:52:44.000Z","size":8659,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-14T11:14:32.397Z","etag":null,"topics":["data-mining","dbscan-algorithm","dbscan-clustering","kmeans-algorithm","kmeans-clustering","machine-learning","machine-learning-algorithms","ml","regression","titanic-dataset","visualization"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elahe-dastan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-31T17:17:53.000Z","updated_at":"2021-06-08T14:52:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"dbe9cde9-b219-4dbb-af82-60d773687b33","html_url":"https://github.com/elahe-dastan/shovel","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elahe-dastan%2Fshovel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elahe-dastan%2Fshovel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elahe-dastan%2Fshovel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elahe-dastan%2Fshovel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elahe-dastan","download_url":"https://codeload.github.com/elahe-dastan/shovel/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241801257,"owners_count":20022389,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-mining","dbscan-algorithm","dbscan-clustering","kmeans-algorithm","kmeans-clustering","machine-learning","machine-learning-algorithms","ml","regression","titanic-dataset","visualization"],"created_at":"2024-11-14T16:39:27.446Z","updated_at":"2025-03-04T07:12:21.963Z","avatar_url":"https://github.com/elahe-dastan.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# shovel\n\nThis repository contains my data mining course homeworks. The homeworks of this course seem useful and interesting so I\ndecided to tell you what is going on so you can practice too\n\n# HW1\n\nThere is a file named covid.csv that contains information about people suffering from COVID-19 in south korea.\u003cbr/\u003e\n\n1. I read this file using **pandas** library\n2. This dataset is small and contains only 176 records, these are the columns\n\n|id     |  sex   |birth_year |country  | region  |infection_reason |infected_by |confirmed_date | state  |\n|-------|:------:|:---------:|:-------:|:-------:|:---------------:|:----------:|:-------------:|-------:|\n|nominal| nominal| interval  | nominal | nominal |    nominal      |    ratio   |    interval   | nominal|\n| 0 nan | 0 nan  |  10 nan   | 0 nan   | 10 nan  |     81 nan      |  134 nan   |      0 nan    |  0 nan |\n\n3. It is asked to find max, mean and std of the column birth_year\u003cbr/\u003e\n   the max is 2009\u003cbr/\u003e\n   Let me talk about finding mean, this column has null values and I can have different strategies facing them I\n   calculated the mean using two ways:\u003cbr/\u003e\n   first: We can think that the null values don't exist and calculate the mean, mean() function of pandas dataframe does\n   that, the mean is 1973.3855\u003cbr/\u003e\n   second: I can change the pandas dataframe to a numpy array and then calculate the mean but the mean() function of\n   numpy array ignore the null values so I have to substitute them with a number for example zero, the mean is\n   1861.2613.\u003cbr/\u003e\n   everything I said about mean remains the same for std. I calculated std just using pandas dataframe std function\u003cbr/\u003e\n   std: 17.0328\n\n4.yes, null values exist in the dataset.\u003cbr/\u003e\nThe question is to remove the null values by a proper method but what the proper method?? if the dataset was huge and\nthe null values were few I would remove the records which have null values but our case is completely the opposite so I\nshould substitute the null values with a value. **pay attention**: sometimes a column contains null values that I don't\nwant to select in feature selection or I sometimes even use a method that can handle null values but in this question we\nassume that we don't want to put aside any column and our method cannot handle null values. In my opinion the best way\nto substitute null values of **numerical** columns is to put the median value instead of them **Note**: I prefer to use\nmedian over mean cause it's more resistant to outliers. For **nominal** columns I substitute the null values with the\nmost frequent value.\u003cbr/\u003e\n**trouble alert**: There is column in our dataset that has date time values, it's logical to use median strategy for\nthis column but the SimpleImputer class that I use considers this column as string and cannot find the median I thought\nof two solutions, I can find the median `df['confirmed_date'].astype('datetime64[ns]').quantile(.5)` and then use\nSimpleImputer with constant strategy or I can convert the column to timestamp before passing it to SimpleImputer, the\nsecond approach is easier.\n\n5. visualize data\u003cbr/\u003e\n   First, I want to plot the histogram of some columns, I think the birth year and infected by columns are the most\n   appropriate and plotting the histogram of the other columns don't give us any information (for example id column :\n   joy:).\u003cbr/\u003e\n   Second, I'd like to plot a scatter plot so I need to choose two columns, let's find the correlation between\n   birth_year and confirmed_date\n\n6. Here we like to detect and remove outliers, even if you have not studied a single book you may think of sorting or\n   visualizing the data and easily seen and remove outliers for datasets like the one we have in here this approach\n   really works but most of the time the dataset is huge and you may prefer more automatic ways like:\u003cbr/\u003e\n1. Inter Quartile Range (IQR): Look at the code below\n\n```sh\nimport numpy as np\nQ1_Q2 = np.quantile(data,0.25)\nQ3 = np.quantile(data,0.75)\nIQR = Q3 - Q1_Q2\n```\n\n2. Z-Score\u003cbr/\u003e\n   In any distribution, about 95% of values will be within 2 standard deviations of the mean and 99.7% of the data\n   within 3. Based on this, any absolute value of z-score above 3 is considered as an outlier.\u003cbr/\u003e\n   Z-score is calculated by substracting the mean and dividing by std\u003cbr/\u003e\n   I treat with outliers like null values and substitute them with median\n\n### linear regression\n\nWhile reading the dataset using pandas I found out there are ';' instead of ',' so I wrote a bash script to solve this\nproblem\u003cbr/\u003e\n\n1. I extract G3 as Y\n2. I split the dataset to train and test 3 fit a linear regression model (as easy as a piece of cake) **Note**: we have\n   nominal columns in our dataset which obviously linear regression cannot handle so we should transform these nominal\n   values to numerical value before fitting the model\n\n##### Encoding\n\nLet me explain how I think completely, our nominal columns are school, sex, address, femsize, Pstatus, Mjob, Fjob,\nreason, guardian, schoolsup, famsup, paid, activities, nursery, higher, internet, romantic.\u003cbr/\u003e\nthe categorical attributes generally fall into three groups (it's my grouping):\u003cbr/\u003e\n\n1. binary: they can have only two possible values we can consider one them as zero and the other as one, in our dataset\n   these are binary: school, sex, address, femsize, Pstatus, schoolsup, famsup, paid, activities, nursery, higher,\n   internet, romantic\n2. ordinal: they can have multiple categorical values but you can see an order among them for example Like,Like\n   Somewhat,Neutral,Dislike Somewhat,Dislike, it's obviously seen that 'Like' is much closer to 'Like Somewhat' than\n   'Dislike' so the differentiate of the numbers I assign to 'Like' and 'Like Somewhat' should be less than the\n   differentiate of 'Like' and 'Dislike' (you get the point!) none of our attributes is ordinal\n3. nothing : they can have multiple categorical values but there is no order, in this case it's not reasonable to assign\n   numerical values to the values cause the values which get closer numbers will have lower distance from each other\n   though it's not correct.Mjob, Fjob, reason and guardian of our dataset are form this group **Solution**: we can use\n   OneHotEncoding, this strategy converts each category value into a new column and assigns a 1 or 0 (True/False) value\n   to the column\n\n\n4. Predict test data\n5. Find accuracy\u003cbr/\u003e\n   I use mean squared error for this purpose, the mse was 5.7495\n\n# HW2 Q6\n\nWorking with titanic dataset.\n\n### 1. Read the dataset using pandas library\n\n![](images/titanic.png)\n\n### 2. Do something about the null values\n\nI may change my opinion in future but with the knowledge I have right now I guess the best way is to replace the null\nvalues of the embarked column with the mode of the column (it has only 2 null values and replacing mode can be a good\nguess), It's hard to say which column is more important in our prediction at the moment but I guess age can affect our\nprediction a lot so I try not to remove the column and I use median values for the null ones but the null values of\nthe \"cabin\" column is so many that I think the column doesn't worth keeping.\n\n### 3. Get deeper to the dataset\n\n#### Non numerical columns and decision tree\n\nWe'd like to use a decision tree to classify passengers and the decision tree classifier from sklearn library cannot\nwork with categorical values so we have to transform the categorical columns to numerical. I divide categorical columns\ninto two groups, the ones which we can arrange in an order and the ones which we cannot. I use ordinal encoder for the\nfirst group and one hot encoder for the second one. Now let's take a look at the categorical columns:\u003cbr/\u003e\n\n###### Name :\n\nAt first I thought of dropping this column, how can names affect our prediction??!! no body stays alive because of\nhis/her name but it's a little trickier I found two points hidden in names first: we can find families using names and\ncause families travel together it's probable to say they all survive together or they all not, second: some words like\nMiss. and Mrs. are specified in this column which gives us a sense to estimate his/her age so we can fill the null\nvalues in the age column in a more proper way. We Asians may not be familiar with western names I've searched for it and\nwrite some points for you.\u003cbr/\u003e\nLook at this example:\u003cbr/\u003e\n**Baclini,Mrs.Solomon (Latifa Qurban)**\u003cbr/\u003e\u003cbr/\u003e\n**Mrs.** indicates that she is married\u003cbr/\u003e\u003cbr/\u003e\n**Solomon** is the name of her husband. This is an old-ish custom where wives can be referred to by therir husband\nnames. For instance, if Jane Smith was married to John Smith, she could be referred to as Mrs.John Smith.\u003cbr/\u003e\u003cbr/\u003e\n**Latifa** is her first name.\u003cbr/\u003e\u003cbr/\u003e\n**Qurban** is her \"maiden\" name. This is the last name that she had before getting married.\u003cbr/\u003e\u003cbr/\u003e\n**Baclini** is her married last name(the last name of her husband)\u003cbr/\u003e\u003cbr/\u003e\nI take another example:\u003cbr/\u003e\u003cbr/\u003e\n**Baclini,Miss.Marie Catherine**\u003cbr/\u003e\u003cbr/\u003e\nMiss indicates that she is unmarried.\u003cbr/\u003e\u003cbr/\u003e\nIn this **Marie** is her first name, **Catherine** is her middle name and **Baclini** is the last name.\u003cbr/\u003e\u003cbr/\u003e\n**Mr.** (for men of all the ages)\u003cbr/\u003e\u003cbr/\u003e\n**Master.** (for male children)\u003cbr/\u003e\u003cbr/\u003e\nWe have other words like Dr., Sir., Col. and ... it's hard to separate each of them apart so I call all of them\nprofessional and I guess their age should be relatively high.\u003cbr/\u003e\nLet's sum it up I think first name and middle name have nothing to do with the passenger's survival but the last name\ncan help us find out families and the Mr. etc. words help us fill null ages.\u003cbr/\u003e\u003cbr/\u003e\n**Note**: Finding last name for married women is tricky actually they have two last names and those are both important\ncause they may have a trip with their husband or parents. I may change my code in future but to make it simple I just\nconsider their married last name.\n\u003c!-- I don't separate last names for now this will make so many columns --\u003e\n\nSex : There is no sex order but this column takes only two values so I prefer to use ordinal encoder over one hot\nencoder which increases the number of my columns.\u003cbr/\u003e\n\nTicket : Tickets have 1. an optional string prefix and 2. a number except for the special cases Ticket='LINE. Ticket\nprefix tells you who the issuing ticket office and/or embarkation point was. Ticket number can be compared for equality\nthat tells you who were sharing a cabin or travelling together, or compared for closeness. The ticket = LINE have been\nassigned to a group of American Line employees for free\u003cbr/\u003e\n\nEmbarked: This column can take three values 'C', 'Q', 'S' which I think have no order, so I use one hot encoder\n\n### 4. Missing values in the test dataset\n\nI strongly recommend splitting your dataset to train and test as soon as possible and don't look at the test data at\nall, if you do that you may have null values in the test data. To remove them I used the data in the training dataset\nfor example if age is null in the test data I filled it with the average of age in the train data.\n\n### 5. Score in kaggle\n\n![](images/kaggle_titanic.png)\n\n# HW2 Q7\n\nWorking on heart-disease-uci dataset.\n\n### Taking a look at the columns\n\n**age**\u003cbr/\u003e\n**sex**: (1 = male, 0 = female)\u003cbr/\u003e\n**cp** : chest pain type\u003cbr/\u003e\n**trestbps** : resting blood pressure (in mm Hg on admission to the hospital)\u003cbr/\u003e\n**chol** : serum cholestoral in mg/dl\u003cbr/\u003e\n**fbs** : fasting blood sugar \u0026gt; 120 mg/dl (1 = true; 0 = false)\u003cbr/\u003e\n**restecg** : resting electrocardiographic results\u003cbr/\u003e\n**thalach** : maximum heart rate achieved\u003cbr/\u003e\n**exang** : exercise induced angina (1 = yes, 0 = no)\u003cbr/\u003e\n**oldpeak** : ST depression induced by exercise relative to rest\u003cbr/\u003e\n**slope** : the slope of the peak exercise ST segment\u003cbr/\u003e\n**ca** : number of major vessels (0-3) colored by flourosopy\u003cbr/\u003e\n**thal** : 3 = normal, 6 = fixed defect, 7 = reversable defect\u003cbr/\u003e\n**target** : 0 or 1\n\n### Normalize data\n\nNormalizing data is important when we want to calculate distance among data records, for example normalizing is not\nimportant in decision tree models cause we are not going to calculate any distance but it's too important in models like\nregression and knn.\u003cbr/\u003e\nNormalizing means standardizing features by removing the mean and scaling to unit variance.\u003cbr/\u003e\nTo normalize the dataset I used StandardScaler() to **fit and transform** the data then I used the StandardScaler object\nI had gotten to **just transform** the test data.\u003cbr/\u003e\nAccuracy reached to 86% from 65% just by normalizing the data.\n\n[comment]: \u003c\u003e (# dependant columns)\n\n[comment]: \u003c\u003e (an idea I have not used till now is to check if columns depend on each other and use it to drop one )\n\n[comment]: \u003c\u003e (### Correlation)\n\n[comment]: \u003c\u003e (correlation shows if two features have a linear relationship)\n\n[comment]: \u003c\u003e (### Entropy)\n\n[comment]: \u003c\u003e (Average information of a variable)\n\n[comment]: \u003c\u003e (### Mutual Information)\n\n[comment]: \u003c\u003e (MI = H\u0026#40;x\u0026#41; + H\u0026#40;y\u0026#41; - H\u0026#40;x, y\u0026#41;)\n\n[comment]: \u003c\u003e (I checked to see the MI between columns here are the ones which have MI bigger than 1)\n\n[comment]: \u003c\u003e (![]\u0026#40;MI.png\u0026#41;)\n\n[comment]: \u003c\u003e (It seems that chol and thalach are so much dependant)\n\n[comment]: \u003c\u003e (we reached to the place that I don't know what to do any further of course there are other things we should which I may )\n\n[comment]: \u003c\u003e (even have the knowlodege but I don't know how to use them so I search about this dataset and tell you what I found)\n\n### Wrong data\n\nAt first, I thought this dataset doesn't contain any NaN values BUT this dataset has some wrong values which we should\nput NaN instead.\u003cbr/\u003e\n\nLet's see the number of unique values in each column\u003cbr/\u003e\n![](images/unique.png)\n\nLook!!!, there are two columns that seem strange:\u003cbr/\u003e\nafter [investigating the columns](#taking-a-look-at-the-columns) we know that **'ca'** ranges from 0 to 3, so it should\nhave only 4 unique values, but it has five :flushed: so there should be a wrong value which needs to be cleaned.\n\n```sh\nX_train['ca'].unique()\n```\n\nthe above code shows that this column has an unaccepted value '4', so I substituted the value '4' with NaN.\u003cbr/\u003e\n\nThe same thing happens for 'thal' column too [this header](#taking-a-look-at-the-columns) says this column can only take\nvalues from 1 to 3 but, this column contains 4 unique values so like what I did for 'ca' column every value other than 1\nto 3 should be changed to null.\n\n### Explain thal column a little more\n\nBasically it is a radioactive element injected into the bloodstream of the patient. Then the blood flow of the patient\nis observed while they are doing exercise and resting.\n\n- 0 maps to null in the original dataset\n- 1 maps to 6 in the original dataset. This means that a fixed defect was found.\n- 2 maps to 3 in the original dataset. This means that the blood flow was normal.\n- 3 maps to 7 in the original dataset. This means that a reversible defect was found.\n\n### Check for duplicate rows\n\n```shell\nX_train.drop_duplicates()\n```\n\n# Deal with outliers\n\nLet's use box plots\n\n```shell\nX_train.plot(kind='box', subplots=True, layout=(2, 7), sharex=False,\n             sharey=False, figsize=(20, 10), color='deeppink')\n\nplt.show()\n```\n\n![](images/boxplot.png)\n\nI can either drop the outliers or assign a new value. I chose the first option.\n\n### Models\n\nAccuracy on knn model was 83% Accuracy on gaussian naive bayes was 90%\n\n### Different naive bayes models\n\nThere are two naive bayes models explained below\n\n#### Multinomial Naive Bayes\n\nused for discrete data\n\n#### Gaussian Naive Bayes\n\nused for continuous data\n\n# HW3 Q8\n\nI am going to work with titanic dataset again, I have done the preprocessing in [HW2 Q6](hw2-q6) so I can simply use my\nnew model. I want to use random forest tree model.\n\n1. max_depth=5, criterion=gini\u003cbr/\u003e\n   ![](images/max_depth_5_gini.png)\n2. max_depth=5, criterion=entropy\u003cbr/\u003e\n   ![](images/max_depth_5_entropy.png)\n3. max_depth=10, criterion=gini\u003cbr/\u003e\n   ![](images/max_depth_50_gini.png)\n4. max_depth=10, criterion=entropy\u003cbr/\u003e\n   ![](images/max_depth_50_entropy.png)\n\n**Comparing random forest model with decision tree:** In [HW2 Q6](hw2-q6) my decision tree had max_depth=5 and\ncriterion= gini, the accuracy was 77%, the forest classifier with the same parameters gave accuracy= 77%.\n\n**Comparing speed of learning:**\u003cbr/\u003e\nI used time.start and time.end to calculate the time of learning.\u003cbr/\u003e\ndecision tree learning time was 0.0022056102752685547\u003cbr/\u003e\nrandom forest learning time was 0.10478830337524414\n\n# HW3 Q9\n\nI'm going to work with titanic dataset again. In this part I wanna use SVM for classification.\n\n1. SVM with linear kernel function:\u003cbr/\u003e\n   ![](images/linear_svm.png)\n\nAs you see, the accuracy of linear SVM model is less than the decision tree and random forest models, this shows that\nthe data we have is not linearly separable.\n\n2.SVM with non linear kernel function:\u003cbr/\u003e\n2-1. poly\u003cbr/\u003e\n![](images/poly_svm.png)\n2-2. rbf\n![](images/poly_svm.png)\n\n# HW4 Q1\n\nI want to implement k-means algorithm.\u003cbr/\u003e\n\n1. Illustrating datasets\n\n![dataset1 illustration](images/dataset1_illustration.png)\n\n![dataset2 illustration](images/dataset2_illustration.png)\n\nLook at the pictures above, it's obvious that k-means can perform well on dataset1 but is not a good choice for\ndataset2, and for dataset2 we should use algorithms like DBSCAN.\n\n2. Implement k-means algorithm\n\n![k-means on dataset1 with k = 2](images/dataset1k2.png)\n![k-means on dataset1 with k = 3](images/dataset1k3.png)\n![k-means on dataset1 with k = 4](images/dataset1k4.png)\n\nWOW!!! if you want to implement the k-means algorithm notice that using plus-plus algorithm for initializing centroids\nis as important as hell and DOES increase the performance.\n\n3-4. calculating clustering error This is the result of one run.\n\n```shell\ncluster error for cluster blue is 0.3248827689004398\ncluster error for cluster green is 0.3160088195773474\ncluster error for cluster orange is 0.3415297385741179\ncluster error for cluster purple is 0.31782962165200457\nclustering error is 0.32506273717597745\n```\n\n5.\n\n![clustering error for different number of clusters](images/elbow.png)\n\n6.\n\naccording to the above picture the best number of clusters is 10.\n\n7.\n\n![](images/dataset2k3.png)\n\nThe shape of the clusters are not globular so k-means cannot perform well.\n\n# HW4 Q2\n\nThis is the original picture\n\n![original image](images/sample_img1.png)\n\nand this is the image after color reduction using k-means with k=64\n\n![color reduction using k-means with k=64](images/color_reduction64.png)\n\nSomething important I want to mention is that it took me 1033.5089447498322 seconds to do the compression for 64 clusters\nand that's really long for a compression task, AND I also think the plus plus part of the k-means algorithm plays an \nimportant role in the time.\n# One time for always solve the problem SettingWithCopyWarning\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felahe-dastan%2Fshovel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felahe-dastan%2Fshovel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felahe-dastan%2Fshovel/lists"}