{"id":15009306,"url":"https://github.com/kwokhing/yandexcatboost-python-demo","last_synced_at":"2025-04-09T17:14:26.920Z","repository":{"id":46733318,"uuid":"108992216","full_name":"KwokHing/YandexCatBoost-Python-Demo","owner":"KwokHing","description":"Demo on the capability of Yandex CatBoost gradient boosting classifier on a fictitious IBM HR dataset obtained from Kaggle. Data exploration, cleaning, preprocessing and model tuning are performed on the dataset","archived":false,"fork":false,"pushed_at":"2019-12-05T11:49:12.000Z","size":761,"stargazers_count":30,"open_issues_count":0,"forks_count":16,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-09T17:14:19.359Z","etag":null,"topics":["catboost","data-analysis","data-preprocessing","data-science","feature-selection","gradient-boosting","gradient-boosting-classifier","one-hot-encode","pandas","pearson-correlation","python","python27","seaborn","variance-analysis","visualization","yandex-catboost"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KwokHing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-31T12:25:44.000Z","updated_at":"2024-03-01T11:11:50.000Z","dependencies_parsed_at":"2022-08-25T12:24:23.691Z","dependency_job_id":null,"html_url":"https://github.com/KwokHing/YandexCatBoost-Python-Demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KwokHing%2FYandexCatBoost-Python-Demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KwokHing%2FYandexCatBoost-Python-Demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KwokHing%2FYandexCatBoost-Python-Demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KwokHing%2FYandexCatBoost-Python-Demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KwokHing","download_url":"https://codeload.github.com/KwokHing/YandexCatBoost-Python-Demo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248074925,"owners_count":21043490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["catboost","data-analysis","data-preprocessing","data-science","feature-selection","gradient-boosting","gradient-boosting-classifier","one-hot-encode","pandas","pearson-correlation","python","python27","seaborn","variance-analysis","visualization","yandex-catboost"],"created_at":"2024-09-24T19:24:21.380Z","updated_at":"2025-04-09T17:14:26.885Z","avatar_url":"https://github.com/KwokHing.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## Exploration of Yandex CatBoost in Python ##\n\nThis demo will provide a brief introduction in  \n- performing data exploration and preprocessing \n- feature subset selection: low variance filter\n- feature subset selection: high correlation filter\n- catboost model tuning \n- importance of data preprocessing: data normalization\n- exploration of catboost's feature importance ranking  \n\n## Getting started\nOpen `YandexCatBoost-Demo.ipynb` on a jupyter notebook environment, or Google colab. The notebook consists of further technical details.\n\n## Future Improvements ##\n\nResults from the feature importance ranking shows that attribute ‘MaritalStatus’ impacts minimally in class label prediction and could potential be a noise attribute. Removing it might increase model’s accuracy.  \n\n\n## Codes Walkthrough\n\nInstalling the open source Yandex CatBoost package\n\n\n```python\npip install catboost\n```\n\nImporting the required packaged: Numpy, Pandas, Matplotlib, Seaborn, Scikit-learn and CatBoost\n\n\n```python\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n# plt.style.use('ggplot') \nimport seaborn as sns\nfrom catboost import Pool, CatBoostClassifier, cv, CatboostIpythonWidget\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.feature_selection import VarianceThreshold\n```\n\nLoading of [IBM HR Dataset](https://www.kaggle.com/pavansubhasht/ibm-hr-analytics-attrition-dataset/data) into pandas dataframe\n\n\n```python\nibm_hr_df = pd.read_csv(\"IBM-HR-Employee-Attrition.csv\")\n```\n\n### Part 1a: Data Exploration - Summary Statistics ###\n\nGetting the summary statistics of the IBM HR dataset\n\n\n```python\nibm_hr_df.describe()\n```\n\n\n\n\n\u003cdiv\u003e\n\u003ctable class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eAge\u003c/th\u003e\n      \u003cth\u003eDailyRate\u003c/th\u003e\n      \u003cth\u003eDistanceFromHome\u003c/th\u003e\n      \u003cth\u003eEducation\u003c/th\u003e\n      \u003cth\u003eEmployeeCount\u003c/th\u003e\n      \u003cth\u003eEmployeeNumber\u003c/th\u003e\n      \u003cth\u003eEnvironmentSatisfaction\u003c/th\u003e\n      \u003cth\u003eHourlyRate\u003c/th\u003e\n      \u003cth\u003eJobInvolvement\u003c/th\u003e\n      \u003cth\u003eJobLevel\u003c/th\u003e\n      \u003cth\u003e...\u003c/th\u003e\n      \u003cth\u003eRelationshipSatisfaction\u003c/th\u003e\n      \u003cth\u003eStandardHours\u003c/th\u003e\n      \u003cth\u003eStockOptionLevel\u003c/th\u003e\n      \u003cth\u003eTotalWorkingYears\u003c/th\u003e\n      \u003cth\u003eTrainingTimesLastYear\u003c/th\u003e\n      \u003cth\u003eWorkLifeBalance\u003c/th\u003e\n      \u003cth\u003eYearsAtCompany\u003c/th\u003e\n      \u003cth\u003eYearsInCurrentRole\u003c/th\u003e\n      \u003cth\u003eYearsSinceLastPromotion\u003c/th\u003e\n      \u003cth\u003eYearsWithCurrManager\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003ecount\u003c/th\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.0\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e...\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.0\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n      \u003ctd\u003e1470.000000\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emean\u003c/th\u003e\n      \u003ctd\u003e36.923810\u003c/td\u003e\n      \u003ctd\u003e802.485714\u003c/td\u003e\n      \u003ctd\u003e9.192517\u003c/td\u003e\n      \u003ctd\u003e2.912925\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e1024.865306\u003c/td\u003e\n      \u003ctd\u003e2.721769\u003c/td\u003e\n      \u003ctd\u003e65.891156\u003c/td\u003e\n      \u003ctd\u003e2.729932\u003c/td\u003e\n      \u003ctd\u003e2.063946\u003c/td\u003e\n      \u003ctd\u003e...\u003c/td\u003e\n      \u003ctd\u003e2.712245\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n      \u003ctd\u003e0.793878\u003c/td\u003e\n      \u003ctd\u003e11.279592\u003c/td\u003e\n      \u003ctd\u003e2.799320\u003c/td\u003e\n      \u003ctd\u003e2.761224\u003c/td\u003e\n      \u003ctd\u003e7.008163\u003c/td\u003e\n      \u003ctd\u003e4.229252\u003c/td\u003e\n      \u003ctd\u003e2.187755\u003c/td\u003e\n      \u003ctd\u003e4.123129\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003estd\u003c/th\u003e\n      \u003ctd\u003e9.135373\u003c/td\u003e\n      \u003ctd\u003e403.509100\u003c/td\u003e\n      \u003ctd\u003e8.106864\u003c/td\u003e\n      \u003ctd\u003e1.024165\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e602.024335\u003c/td\u003e\n      \u003ctd\u003e1.093082\u003c/td\u003e\n      \u003ctd\u003e20.329428\u003c/td\u003e\n      \u003ctd\u003e0.711561\u003c/td\u003e\n      \u003ctd\u003e1.106940\u003c/td\u003e\n      \u003ctd\u003e...\u003c/td\u003e\n      \u003ctd\u003e1.081209\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e0.852077\u003c/td\u003e\n      \u003ctd\u003e7.780782\u003c/td\u003e\n      \u003ctd\u003e1.289271\u003c/td\u003e\n      \u003ctd\u003e0.706476\u003c/td\u003e\n      \u003ctd\u003e6.126525\u003c/td\u003e\n      \u003ctd\u003e3.623137\u003c/td\u003e\n      \u003ctd\u003e3.222430\u003c/td\u003e\n      \u003ctd\u003e3.568136\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emin\u003c/th\u003e\n      \u003ctd\u003e18.000000\u003c/td\u003e\n      \u003ctd\u003e102.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e30.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e...\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e25%\u003c/th\u003e\n      \u003ctd\u003e30.000000\u003c/td\u003e\n      \u003ctd\u003e465.000000\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e491.250000\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e48.000000\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e...\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e6.000000\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e50%\u003c/th\u003e\n      \u003ctd\u003e36.000000\u003c/td\u003e\n      \u003ctd\u003e802.000000\u003c/td\u003e\n      \u003ctd\u003e7.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e1020.500000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e66.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e2.000000\u003c/td\u003e\n      \u003ctd\u003e...\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e10.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e5.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e75%\u003c/th\u003e\n      \u003ctd\u003e43.000000\u003c/td\u003e\n      \u003ctd\u003e1157.000000\u003c/td\u003e\n      \u003ctd\u003e14.000000\u003c/td\u003e\n      \u003ctd\u003e4.000000\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e1555.750000\u003c/td\u003e\n      \u003ctd\u003e4.000000\u003c/td\u003e\n      \u003ctd\u003e83.750000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e...\u003c/td\u003e\n      \u003ctd\u003e4.000000\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e15.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e9.000000\u003c/td\u003e\n      \u003ctd\u003e7.000000\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e7.000000\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emax\u003c/th\u003e\n      \u003ctd\u003e60.000000\u003c/td\u003e\n      \u003ctd\u003e1499.000000\u003c/td\u003e\n      \u003ctd\u003e29.000000\u003c/td\u003e\n      \u003ctd\u003e5.000000\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e2068.000000\u003c/td\u003e\n      \u003ctd\u003e4.000000\u003c/td\u003e\n      \u003ctd\u003e100.000000\u003c/td\u003e\n      \u003ctd\u003e4.000000\u003c/td\u003e\n      \u003ctd\u003e5.000000\u003c/td\u003e\n      \u003ctd\u003e...\u003c/td\u003e\n      \u003ctd\u003e4.000000\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n      \u003ctd\u003e3.000000\u003c/td\u003e\n      \u003ctd\u003e40.000000\u003c/td\u003e\n      \u003ctd\u003e6.000000\u003c/td\u003e\n      \u003ctd\u003e4.000000\u003c/td\u003e\n      \u003ctd\u003e40.000000\u003c/td\u003e\n      \u003ctd\u003e18.000000\u003c/td\u003e\n      \u003ctd\u003e15.000000\u003c/td\u003e\n      \u003ctd\u003e17.000000\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e8 rows × 26 columns\u003c/p\u003e\n\u003c/div\u003e\n\n\n\nZooming in on the summary statistics of irrelevant attributes __*EmployeeCount*__ and __*StandardHours*__\n\n\n```python\nirrList = ['EmployeeCount', 'StandardHours'] \nibm_hr_df[irrList].describe()\n```\n\n\n\n\n\u003cdiv\u003e\n\u003ctable class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eEmployeeCount\u003c/th\u003e\n      \u003cth\u003eStandardHours\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003ecount\u003c/th\u003e\n      \u003ctd\u003e1470.0\u003c/td\u003e\n      \u003ctd\u003e1470.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emean\u003c/th\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003estd\u003c/th\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emin\u003c/th\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e25%\u003c/th\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e50%\u003c/th\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e75%\u003c/th\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emax\u003c/th\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e80.0\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\nZooming in on the summary statistics of irrelevant attribute __*Over18*__ \n\n\n```python\nibm_hr_df[\"Over18\"].value_counts()\n```\n\n\n\n\n    Y    1470\n    Name: Over18, dtype: int64\n\n\n\nFrom the summary statistics, one could see that attributes __*EmployeeCount*__, __*StandardHours*__ and __*Over18*__ holds only one single value for all of the 1470 records \u003cbr\u003e\n\n__*EmployeeCount*__ only holds a single value - 1.0 \u003cbr\u003e\n__*StandardHours*__ only holds a single value - 80.0 \u003cbr\u003e\n__*Over18*__        only holds a single value - 'Y'  \u003cbr\u003e\n\nThese irrelevant attributes are duely dropped from the dataset\n\n### Part 1b: Data Exploration - Missing Values and Duplicate Records ###\n\nChecking for 'NA' and missing values in the dataset.\n\n\n```python\nibm_hr_df.isnull().sum(axis=0)\n```\n\n\n\n\n    Age                         0\n    Attrition                   0\n    BusinessTravel              0\n    DailyRate                   0\n    Department                  0\n    DistanceFromHome            0\n    Education                   0\n    EducationField              0\n    EmployeeCount               0\n    EmployeeNumber              0\n    EnvironmentSatisfaction     0\n    Gender                      0\n    HourlyRate                  0\n    JobInvolvement              0\n    JobLevel                    0\n    JobRole                     0\n    JobSatisfaction             0\n    MaritalStatus               0\n    MonthlyIncome               0\n    MonthlyRate                 0\n    NumCompaniesWorked          0\n    Over18                      0\n    OverTime                    0\n    PercentSalaryHike           0\n    PerformanceRating           0\n    RelationshipSatisfaction    0\n    StandardHours               0\n    StockOptionLevel            0\n    TotalWorkingYears           0\n    TrainingTimesLastYear       0\n    WorkLifeBalance             0\n    YearsAtCompany              0\n    YearsInCurrentRole          0\n    YearsSinceLastPromotion     0\n    YearsWithCurrManager        0\n    dtype: int64\n\n\n\nWell, we got lucky here, there isn't any missing values in this dataset\n\nNext, let's check for the existence of duplicate records in the dataset\n\n\n```python\nibm_hr_df.duplicated().sum()\n```\n\n\n\n\n    0\n\n\n\nThere are also no duplicate records in the dataset\n\nConverting __*OverTime*__ binary categorical attribute to {1, 0}\n\n\n```python\nibm_hr_df['OverTime'].replace(to_replace=dict(Yes=1, No=0), inplace=True)\n```\n\n### Part 2a: Data Preprocessing - Removal of Irrelevant Attributes ###\n\n\n```python\nibm_hr_df = ibm_hr_df.drop(['EmployeeCount', 'StandardHours', 'Over18'], axis=1)\n```\n\n### Part 2b: Data Preprocessing - Feature Subset Selection - Low Variance Filter  ###\n\nPerforming variance analysis to aid in feature selection\n\n\n```python\nvariance_x = ibm_hr_df.drop('Attrition', axis=1)\nvariance_one_hot = pd.get_dummies(variance_x)\n```\n\n\n```python\n#Normalise the dataset. This is required for getting the variance threshold\nscaler = MinMaxScaler()\nscaler.fit(variance_one_hot)\nMinMaxScaler(copy=True, feature_range=(0, 1))\nscaled_variance_one_hot = scaler.transform(variance_one_hot)\n```\n\n\n```python\n#Set the threshold values and run VarianceThreshold \nthres = .85* (1 - .85)\nsel = VarianceThreshold(threshold=thres)\nsel.fit(scaled_variance_one_hot)\nvariance = sel.variances_\n```\n\n\n```python\n#Sorting of the score in acsending orders for plotting\nindices = np.argsort(variance)[::-1]\nfeature_list = list(variance_one_hot)\nsorted_feature_list = []\nthres_list = []\nfor f in range(len(variance_one_hot.columns)):\n    sorted_feature_list.append(feature_list[indices[f]])\n    thres_list.append(thres)\n```\n\n\n```python\nplt.figure(figsize=(14,6))\nplt.title(\"Feature Variance: %f\" %(thres), fontsize = 14)\nplt.bar(range(len(variance_one_hot.columns)), variance[indices], color=\"c\")\nplt.xticks(range(len(variance_one_hot.columns)), sorted_feature_list, rotation = 90)\nplt.xlim([-0.5, len(variance_one_hot.columns)])\nplt.plot(range(len(variance_one_hot.columns)), thres_list, \"k-\", color=\"r\")\nplt.tight_layout()\nplt.show()\n```\n\n\n![png](images/output_30_0.png)\n\n\n### Part 2c: Data Preprocessing - Feature Subset Selection - High Correlation Filter  ###\n\nPerforming Pearson correlation analysis between attributes to aid in feature selection\n\n\n```python\nplt.figure(figsize=(16,16))\nsns.heatmap(ibm_hr_df.corr(), annot=True, fmt=\".2f\")\n\nplt.show()\n```\n\n\n![png](images/output_24_0.png)\n\n\n\n### Part 3: Mutli-Class Label Generation ###\n\n\n```python\nrAttrList = ['Department', 'OverTime', 'HourlyRate',\n             'StockOptionLevel', 'DistanceFromHome',\n             'YearsInCurrentRole', 'Age']\n```\n\n\n```python\n#keep only the attribute list on rAttrList\nlabel_hr_df = ibm_hr_df[rAttrList]\n```\n\n\n```python\n#convert continous attribute DistanceFromHome to Catergorical\n#: 1: near, 2: mid distance, 3: far\nmaxValues = label_hr_df['DistanceFromHome'].max()\nminValues = label_hr_df['DistanceFromHome'].min()\nintervals = (maxValues - minValues)/3\nbins = [0, (minValues + intervals), (maxValues - intervals), maxValues]\ngroupName = [1, 2, 3]\nlabel_hr_df['CatDistanceFromHome'] = pd.cut(label_hr_df['DistanceFromHome'], bins, labels = groupName)\n```\n\n\n```python\n# convert col type from cat to int64\nlabel_hr_df['CatDistanceFromHome'] = pd.to_numeric(label_hr_df['CatDistanceFromHome']) \nlabel_hr_df.drop(['DistanceFromHome'], axis = 1, inplace = True)\n```\n\n\n```python\n#replace department into 0 \u0026 1, 0: R\u0026D, and 1: Non-R\u0026D\nlabel_hr_df['Department'].replace(['Research \u0026 Development', 'Human Resources', 'Sales'],\n                                  [0, 1, 1], inplace = True)\n```\n\n\n```python\n#normalise data\nlabel_hr_df_norm = (label_hr_df - label_hr_df.min()) / (label_hr_df.max() - label_hr_df.min())\n```\n\n\n```python\n#create a data frame for the function value and class labels\nvalue_df = pd.DataFrame(columns = ['ClassValue'])\n```\n\n\n```python\n#compute the class value\nfor row in range (0, ibm_hr_df.shape[0]):\n    if label_hr_df_norm['Department'][row] == 0:\n        value = 0.3 * label_hr_df_norm['HourlyRate'][row] - 0.2 * label_hr_df_norm['OverTime'][row] + \\\n            - 0.2 * label_hr_df_norm['CatDistanceFromHome'][row] + 0.15 * label_hr_df_norm['StockOptionLevel'][row] + \\\n            0.1 * label_hr_df_norm['Age'][row] - 0.05 * label_hr_df_norm['YearsInCurrentRole'][row]\n    \n    else:\n        value = 0.2 * label_hr_df_norm['HourlyRate'][row] - 0.3 * label_hr_df_norm['OverTime'][row] + \\\n            - 0.15 * label_hr_df_norm['CatDistanceFromHome'][row] + 0.2 * label_hr_df_norm['StockOptionLevel'][row] + \\\n            0.05 * label_hr_df_norm['Age'][row] - 0.1 * label_hr_df_norm['YearsInCurrentRole'][row]\n    value_df.loc[row] = value\n```\n\n\n```python\n# top 500 highest class value is satisfied with their job\nv1 = value_df.sort_values('ClassValue', ascending = False).reset_index(drop = True)\\\n        ['ClassValue'][499]\n# next top 500 is neutral\nv2 = value_df.sort_values('ClassValue', ascending = False).reset_index(drop = True)\\\n        ['ClassValue'][999]\n# rest is unsatisfied\n```\n\n\n```python\nlabel_df = pd.DataFrame(columns = ['ClassLabel'])\n```\n\n\n```python\n#compute the classlabel\nfor row in range (0, value_df.shape[0]):\n    if value_df['ClassValue'][row] \u003e= v1:\n        cat = \"Satisfied\"\n    elif value_df['ClassValue'][row] \u003e= v2:\n        cat = \"Neutral\"\n    else:\n        cat = \"Unsatisfied\"\n    label_df.loc[row] = cat\n```\n\n\n```python\ndf = pd.concat([ibm_hr_df, label_df], axis = 1)\n```\n\n### Part 4: Classification with CatBoost ###\n\n\n```python\ndf = df[['Age', 'Department', 'DistanceFromHome', 'HourlyRate', 'OverTime', 'StockOptionLevel', \n         'MaritalStatus', 'YearsInCurrentRole', 'EmployeeNumber', 'ClassLabel']]\n```\n\nSplit dataset into attributes/features __*X*__ and label/class __*y*__\n\n\n```python\nX = df.drop('ClassLabel', axis=1)\ny = df.ClassLabel\n```\n\nReplacing label/class value from __*'Satisfied'*__, __*'Neutral'*__ and *__'Unsatisfied'__* to *__2__*, __*1*__ and __*0*__\n\n\n```python\ny.replace(to_replace=dict(Satisfied=2, Neutral=1, Unsatisfied=0), inplace=True)\n```\n\nPerforming __'one hot encoding'__ method\n\n\n```python\none_hot = pd.get_dummies(X)\n```\n\n```python\ncategorical_features_indices = np.where(one_hot.dtypes != np.float)[0]\n```\n\n### Part 5: Model training with CatBoost ###\nNow lets split our data to train (70%) and test (30%) set:\n\n\n```python\nfrom sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(one_hot, y, train_size=0.7, random_state=1234)\n```\n\n\n```python\nmodel = CatBoostClassifier(\n    custom_loss = ['Accuracy'],\n    random_seed = 100,\n    loss_function = 'MultiClass'\n)\n```\n\n\n```python\nmodel.fit(\n    X_train, y_train,\n    cat_features = categorical_features_indices,\n    verbose = True,  # you can uncomment this for text output\n    #plot = True\n)\n```\n\n\n```python\ncm = pd.DataFrame()\ncm['Satisfaction'] = y_test\ncm['Predict'] = model.predict(X_test)\n```\n\n\n```python\nmappingSatisfaction = {0:'Unsatisfied', 1: 'Neutral', 2: 'Satisfied'}\nmappingPredict = {0.0:'Unsatisfied', 1.0: 'Neutral', 2.0: 'Satisfied'}\ncm = cm.replace({'Satisfaction': mappingSatisfaction, 'Predict': mappingPredict})\n```\n\n\n```python\npd.crosstab(cm['Satisfaction'], cm['Predict'], margins=True)\n```\n\n\n\n\n\u003cdiv\u003e\n\u003ctable class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003ePredict\u003c/th\u003e\n      \u003cth\u003eNeutral\u003c/th\u003e\n      \u003cth\u003eSatisfied\u003c/th\u003e\n      \u003cth\u003eUnsatisfied\u003c/th\u003e\n      \u003cth\u003eAll\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eSatisfaction\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003eNeutral\u003c/th\u003e\n      \u003ctd\u003e143\u003c/td\u003e\n      \u003ctd\u003e8\u003c/td\u003e\n      \u003ctd\u003e8\u003c/td\u003e\n      \u003ctd\u003e159\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eSatisfied\u003c/th\u003e\n      \u003ctd\u003e20\u003c/td\u003e\n      \u003ctd\u003e123\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e144\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eUnsatisfied\u003c/th\u003e\n      \u003ctd\u003e18\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e120\u003c/td\u003e\n      \u003ctd\u003e138\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eAll\u003c/th\u003e\n      \u003ctd\u003e181\u003c/td\u003e\n      \u003ctd\u003e131\u003c/td\u003e\n      \u003ctd\u003e129\u003c/td\u003e\n      \u003ctd\u003e441\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n\n```python\nmodel.score(X_test, y_test)\n```\n\n\n\n\n    0.87528344671201819\n\n\n### Part 6: CatBoost Classifier Tuning ###\n\n\n```python\nmodel = CatBoostClassifier(\n    l2_leaf_reg = 5,\n    iterations = 1000,\n    fold_len_multiplier = 1.1,\n    custom_loss = ['Accuracy'],\n    random_seed = 100,\n    loss_function = 'MultiClass'\n)\n```\n\n\n```python\nmodel.fit(\n    X_train, y_train,\n    cat_features = categorical_features_indices,\n    verbose = True,  # you can uncomment this for text output\n    #plot = True\n)\n```\n\n\n```python\ncm = pd.DataFrame()\ncm['Satisfaction'] = y_test\ncm['Predict'] = model.predict(X_test)\n```\n\n\n```python\nmappingSatisfaction = {0:'Unsatisfied', 1: 'Neutral', 2: 'Satisfied'}\nmappingPredict = {0.0:'Unsatisfied', 1.0: 'Neutral', 2.0: 'Satisfied'}\ncm = cm.replace({'Satisfaction': mappingSatisfaction, 'Predict': mappingPredict})\n```\n\n\n```python\npd.crosstab(cm['Satisfaction'], cm['Predict'], margins=True)\n```\n\n\n\n\n\u003cdiv\u003e\n\u003ctable class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003ePredict\u003c/th\u003e\n      \u003cth\u003eNeutral\u003c/th\u003e\n      \u003cth\u003eSatisfied\u003c/th\u003e\n      \u003cth\u003eUnsatisfied\u003c/th\u003e\n      \u003cth\u003eAll\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eSatisfaction\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003eNeutral\u003c/th\u003e\n      \u003ctd\u003e142\u003c/td\u003e\n      \u003ctd\u003e9\u003c/td\u003e\n      \u003ctd\u003e8\u003c/td\u003e\n      \u003ctd\u003e159\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eSatisfied\u003c/th\u003e\n      \u003ctd\u003e17\u003c/td\u003e\n      \u003ctd\u003e126\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e144\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eUnsatisfied\u003c/th\u003e\n      \u003ctd\u003e12\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e126\u003c/td\u003e\n      \u003ctd\u003e138\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eAll\u003c/th\u003e\n      \u003ctd\u003e171\u003c/td\u003e\n      \u003ctd\u003e135\u003c/td\u003e\n      \u003ctd\u003e135\u003c/td\u003e\n      \u003ctd\u003e441\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n\n```python\nmodel.score(X_test, y_test)\n```\n\n\n\n\n    0.89342403628117917\n\n\n### Part 7: Data Preprocessing: Attributes Value Normalization ###\n\n\nNormalization of features, after realizing that tuning no longer improve model's accuracy \n\n\n```python\none_hot = (one_hot - one_hot.mean()) / (one_hot.max() - one_hot.min())\n```\n\n\n```python\ncategorical_features_indices = np.where(one_hot.dtypes != np.float)[0]\n```\n\n\n```python\nfrom sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(one_hot, y, train_size=0.7, random_state=1234)\n```\n\n\n```python\nmodel = CatBoostClassifier(\n    l2_leaf_reg = 5,\n    iterations = 1000,\n    fold_len_multiplier = 1.1,\n    custom_loss = ['Accuracy'],\n    random_seed = 100,\n    loss_function = 'MultiClass'\n)\n```\n\n\n```python\nmodel.fit(\n    X_train, y_train,\n    cat_features = categorical_features_indices,\n    verbose = True,  # you can uncomment this for text output\n    #plot = True\n)\n```\n\n\n```python\nfeature_score = pd.DataFrame(list(zip(one_hot.dtypes.index, model.get_feature_importance(Pool(one_hot, label=y, cat_features=categorical_features_indices)))),\n                columns=['Feature','Score'])\n```\n\n\n```python\nfeature_score = feature_score.sort_values(by='Score', ascending=False, inplace=False, kind='quicksort', na_position='last')\n```\n\n\n```python\nplt.rcParams[\"figure.figsize\"] = (12,7)\nax = feature_score.plot('Feature', 'Score', kind='bar', color='c')\nax.set_title(\"Catboost Feature Importance Ranking\", fontsize = 14)\nax.set_xlabel('')\n\nrects = ax.patches\n\n# get feature score as labels round to 2 decimal\nlabels = feature_score['Score'].round(2)\n\nfor rect, label in zip(rects, labels):\n    height = rect.get_height()\n    ax.text(rect.get_x() + rect.get_width()/2, height + 0.35, label, ha='center', va='bottom')\n\nplt.show()\n```\n\n\n![png](images/output_69_0.png)\n\n\n\n```python\ncm = pd.DataFrame()\ncm['Satisfaction'] = y_test\ncm['Predict'] = model.predict(X_test)\n```\n\n\n```python\nmappingSatisfaction = {0:'Unsatisfied', 1: 'Neutral', 2: 'Satisfied'}\nmappingPredict = {0.0:'Unsatisfied', 1.0: 'Neutral', 2.0: 'Satisfied'}\ncm = cm.replace({'Satisfaction': mappingSatisfaction, 'Predict': mappingPredict})\n```\n\n\n```python\npd.crosstab(cm['Satisfaction'], cm['Predict'], margins=True)\n```\n\n\n\n\n\u003cdiv\u003e\n\u003ctable class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003ePredict\u003c/th\u003e\n      \u003cth\u003eNeutral\u003c/th\u003e\n      \u003cth\u003eSatisfied\u003c/th\u003e\n      \u003cth\u003eUnsatisfied\u003c/th\u003e\n      \u003cth\u003eAll\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eSatisfaction\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003eNeutral\u003c/th\u003e\n      \u003ctd\u003e146\u003c/td\u003e\n      \u003ctd\u003e11\u003c/td\u003e\n      \u003ctd\u003e2\u003c/td\u003e\n      \u003ctd\u003e159\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eSatisfied\u003c/th\u003e\n      \u003ctd\u003e7\u003c/td\u003e\n      \u003ctd\u003e137\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e144\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eUnsatisfied\u003c/th\u003e\n      \u003ctd\u003e8\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003e130\u003c/td\u003e\n      \u003ctd\u003e138\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eAll\u003c/th\u003e\n      \u003ctd\u003e161\u003c/td\u003e\n      \u003ctd\u003e148\u003c/td\u003e\n      \u003ctd\u003e132\u003c/td\u003e\n      \u003ctd\u003e441\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n\n```python\nmodel.score(X_test, y_test)\n```\n\n\n\n\n    0.93650793650793651\n\n\n  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwokhing%2Fyandexcatboost-python-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkwokhing%2Fyandexcatboost-python-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwokhing%2Fyandexcatboost-python-demo/lists"}