{"id":22492264,"url":"https://github.com/rita94105/salary_predict","last_synced_at":"2025-03-27T20:25:38.152Z","repository":{"id":244395005,"uuid":"802803984","full_name":"Rita94105/Salary_predict","owner":"Rita94105","description":"The analysis revealed that the job title is the most significant factor influencing salaries. This insight is incredibly valuable for both job seekers and employers in the data field, aiding in career planning and recruitment strategies.","archived":false,"fork":false,"pushed_at":"2024-06-07T04:08:24.000Z","size":1973,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-01T22:47:51.661Z","etag":null,"topics":["ai","data-engineering","data-science","salary-prediction"],"latest_commit_sha":null,"homepage":"https://rita94105.shinyapps.io/DS_2024/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rita94105.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-19T10:12:59.000Z","updated_at":"2024-06-14T10:55:25.000Z","dependencies_parsed_at":"2024-06-14T13:10:29.573Z","dependency_job_id":"993326d1-32b9-4eb2-b35a-220651f9f26b","html_url":"https://github.com/Rita94105/Salary_predict","commit_stats":null,"previous_names":["rita94105/salary_predict"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rita94105%2FSalary_predict","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rita94105%2FSalary_predict/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rita94105%2FSalary_predict/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rita94105%2FSalary_predict/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rita94105","download_url":"https://codeload.github.com/Rita94105/Salary_predict/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245918263,"owners_count":20693675,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","data-engineering","data-science","salary-prediction"],"created_at":"2024-12-06T18:16:55.804Z","updated_at":"2025-03-27T20:25:38.130Z","avatar_url":"https://github.com/Rita94105.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [Data Engineer Salary Predict](https://rita94105.shinyapps.io/DS_2024/)\n\nSince the advent of ChatGPT, AI applications have flourished, driving a surge in demand for data-related positions. This phenomenon sparked our interest in the salary structures within the data industry. On Kaggle, we discovered a dataset on Data Engineer salaries, which led us to undertake a project aimed at predicting salary ranges in the data field based on features such as region, job title, years of experience, and company size.\n\nOur research findings indicate that job title is the most critical factor in determining salaries. This insight holds significant value for both job seekers and employers in the data field, aiding in career planning and recruitment strategies.\n\nThis project not only reveals the factors influencing salaries in the data industry but also provides valuable information for those looking to enter the field, helping them make more informed career decisions.\n\n## [Data](https://www.kaggle.com/datasets/chopper53/data-engineer-salary-in-2024)\n\n### About Dataset\n\nThis dataset provides insights into data engineer salaries and employment attributes for the year 2024. It includes information such as salary, job title, experience level, employment type, employee residence, remote work ratio, company location, and company size.\n\nThe dataset allows for analysis of salary trends, employment patterns, and geographic variations in data engineering roles. It can be used by researchers, analysts, and organizations to understand the evolving landscape of data engineering employment and compensation.\n\n### Feature Description\n\n|     | Feature Name   | Type        | Description             |\n|--------|--------|-----------|----------------------|\n| 1   | work_year | [❌deprecated] | The year in which the data was collected (2024). |\n| 2   |  experience_level | ordinal |    The experience level of the employee, categorized as SE (Senior Engineer), MI (Mid-Level Engineer), or EL (Entry-Level Engineer).   |\n| 3   |  employment_type| ordinal |   The type of employment, such as full-time (FT), part-time (PT), contract (C), or freelance (F).    |\n| 4   |  job_title| nominal |   The title or role of the employee within the company, for example, AI Engineer.    |\n| 5   |  salary| [❌deprecated] |    The salary of the employee in the local currency (e.g., 202,730 USD).   |\n| 6   |  salary_currency| [❌deprecated] |   The currency in which the salary is denominated (e.g., USD).    |\n| 7   |  salary_in_usd| numerical \u003cbr\u003e (🔍predicted) |    The salary converted to US dollars for standardization purposes.   |\n| 8   |  employee_residence| nominal |    The country of residence of the employee.   |\n| 9   |  remote_ratio| ordinal |    The ratio indicating the extent of remote work allowed in the position (0 for no remote work, 1 for fully remote).   |\n| 10   |  company_location| nominal |    The location of the company where the employee is employed.   |\n| 11   |  company_size| ordinal |    The size of the company, often categorized by the number of employees (S for small, M for medium, L for large).   |\n\n### Our Target\nWe try to use these features of the dataset to predict the salaries of data science-related jobs.\n\n## Exploratory Data Analysis(EDA)\n\n| Graph Type   | Horizontal Axis (x)  | Vertical Axis (y)  | Note |\n|--------|-----------|----------------------|--------|\n| Boxplot   | categorical feature  | salary_in_usd   | order by median descending |\n| Barplot   | categorical feature  | avg. of salary_in_usd or counts   | order by value descending |\n| Histogram | bins of salary_in_usd  | frequency   |  |\n| Histogram | bins of signedlog10(salary_in_usd)  | frequency   |  |\n\n## Data Preprocessing\n\n### Library\n\n1.  [smotfamily](https://cran.r-project.org/web/packages/smotefamily/smotefamily.pdf)\n2.  [ggplot2](https://cran.r-project.org/web/packages/ggplot2/ggplot2.pdf)\n3.  [dataPreparation](https://cran.r-project.org/web/packages/dataPreparation/index.html)\n4.  [data.table](https://cran.r-project.org/web/packages/data.table/index.html)\n\n### Process\n\n1.  Checking the type of data\n```\nsummary(data)\nstr(data)\n```\n2.  Checking data distribution\n```\nclass_distribution \u003c- table(data$company_size)\n```\n3.  Checking the missing or null value\n```\nmissing_counts \u003c- colSums(is.na(data))\n```\n4.  Checking Outliers\n```\nboxplot(data$salary_in_usd, main = \"Boxplot\")\n\nprocess_outliers \u003c- function(data, columns) {\n  for (col in columns) {\n    Q1 \u003c- quantile(data[[col]], 0.25, na.rm = TRUE)\n    Q3 \u003c- quantile(data[[col]], 0.75, na.rm = TRUE)\n    IQR \u003c- Q3 - Q1\n    lower_bound \u003c- Q1 - 1.5 * IQR\n    upper_bound \u003c- Q3 + 1.5 * IQR\n    \n    median_val \u003c- median(data[[col]], na.rm = TRUE)\n    \n    print(paste(\"---\", col))\n    print(paste(\"lower bound:\",lower_bound))\n    print(paste(\"upper bound:\", upper_bound))\n    print(paste(\"median\", median_val))\n    outliers \u003c- data[[col]][data[[col]] \u003c lower_bound | data[[col]] \u003e upper_bound]\n    print(paste(\"outliers count:\", length(outliers)))\n    \n    data[[col]][data[[col]] \u003c lower_bound] \u003c- median_val\n    data[[col]][data[[col]] \u003e upper_bound] \u003c- median_val\n    \n    outliers_after \u003c- data[[col]][data[[col]] \u003c lower_bound | data[[col]] \u003e upper_bound]\n    print(paste(\"outliers_after count:\", length(outliers_after)))\n    \n  }\n  return(data)\n}\n\ndata \u003c- process_outliers(data, c(\"salary_in_usd\"))\n```\n5.  Ordinal Encoding\n```\n# ordinal: experience_level, company_size\n\nordinal_encoding \u003c- function(data, old_col, new_col, order) {\n  data[[new_col]] \u003c- factor(data[[old_col]], levels = order, ordered = TRUE)\n  data[[new_col]] \u003c- as.numeric(data[[new_col]])\n  return(data)\n}\n\nexperience_levels \u003c- c(\"EN\", \"MI\", \"SE\", \"EX\")\ndata \u003c- ordinal_encoding(data, \"experience_level\", \"experience_level_encoded\", experience_levels)\ndata$experience_level_encoded\n\ncompany_size_levels \u003c- c(\"S\", \"M\", \"L\")\ndata \u003c- ordinal_encoding(data, \"company_size\", \"company_size_encoded\", company_size_levels)\ndata$company_size_encoded\n```\n6.  Target Encoding\n```\ntarget_encoding \u003c- function(data, col, target) {\n  target_encode_tmp \u003c- build_target_encoding(data, cols_to_encode = col,\n                                           target_col = target, functions = c(\"mean\"))\n  encode_result \u003c- target_encode(data, target_encoding = target_encode_tmp)\n  return(encode_result[[length(encode_result)]])\n}\n\n# employment_type: Target encoding\ndata$employment_type_encoded \u003c-target_encoding(data, \"employment_type\", target_encode_col_name)\n\n# job_title: Target encoding\ndata$job_title_encoded \u003c- target_encoding(data, \"job_title\", target_encode_col_name)\n\n# employee_residence: Target encoding\ndata$employee_residence_encoded \u003c- target_encoding(data, \"employee_residence\", target_encode_col_name)\n\n# company_location: Target encoding\ndata$company_location_encoded \u003c- target_encoding(data, \"company_location\", target_encode_col_name)\n```\n7.  One-hot Encoding\n\n- During testing, it was found that One-hot Encoding led to an excessive number of features, resulting in decreased accuracy and increased training time. Therefore, it was ultimately not adopted.\n\n```\ncols_encoding \u003c- c(\"experience_level\",\"employment_type\",\"job_title\",\"employee_residence\",\n                   \"company_location\",\"company_size\")\n                   \ndo_one_hot_encoding \u003c- function(data, col_name) {\n  category_name \u003c- col_name\n  formula_str \u003c- paste(\"~\", category_name, \"- 1\")\n  formula_obj \u003c- as.formula(formula_str)\n  encoded_df \u003c- model.matrix(formula_obj, data = data)\n  colnames(encoded_df) \u003c- make.names(colnames(encoded_df))\n  \n  combined_data \u003c- cbind(data, encoded_df)\n  return(combined_data)\n}\n\nfor (col in cols_encoding) {\n  data \u003c- do_one_hot_encoding(data, col)\n}\n```\n8.  Splitting the data into training and testing sets\n\n- The original plan was to use K-means to create new features. However, since all the features within the dataset are categorical in nature, the K-means method is not feasible.\n\n```\ndo_kmeans \u003c- function(data, cluster_count) {\n  k \u003c- cluster_count\n  kmeans_result \u003c- kmeans(data$salary_in_usd, centers = k)\n  return(as.factor(kmeans_result$cluster))\n}\n\ndata$salary_in_usd_cluster \u003c- do_kmeans(data, 10)\ncolnames(data)\n```\n\n- Therefore, in the end, 80% of the data was selected for training using an index-based approach, while 20% was reserved for testing.\n\n```\ndata$i \u003c- runif(nrow(data))\ntrain_data \u003c- subset(data, i \u003e= 0.2)\ntest_data \u003c- subset(data, i \u003c 0.2)\ntrain_data$i \u003c- NULL\ntest_data$i \u003c- NULL\n```\n9.  Transforming Transforming the target variable\n``` r\n# sin log function\nsignedlog10 \u003c- function(x) {\n  ifelse(abs(x) \u003c= 1, 0, sign(x) * log10(abs(x)))\n}\n```\n\n## Model Training and Results\n\n### Target Encoding\n\nWe use target encoding to calculate the mean of **signedlog10(salary_in_usd)** for each feature.\n\nAfter target encoding, we use [ggcorrplot](https://cran.r-project.org/web/packages/ggcorrplot/ggcorrplot.pdf) to generate the correlation matrix for the features.\n\n![](/output/correlation_matrix.png)\n\n``` r\nlibrary(ggcorrplot)\ncorrelation_matrix \u003c- cor(data %\u003e% select(-salary_in_usd))\nggcorrplot(correlation_matrix, lab = T)+\n  labs(title = \"correlation matrix\",\n       x = \"\",\n       y = \"\")+\n  theme_bw()+\n  theme(axis.text.x = element_text(\"BL\", \"bold\", \"black\", size = 10, angle = 45, hjust = 1),\n        axis.text.y = element_text(\"BL\", \"bold\", \"black\", size = 10, hjust = 1),\n        title = element_text(\"BL\", \"bold\", size = 10))\n```\n\n### Training\n\n1. We use [randomForest](https://cran.r-project.org/web/packages/randomForest/index.html) and gradient boosting ([gbm](https://cran.r-project.org/web/packages/caret/gbm.html)) to build an ensemble learning model to predict **signedlog10(salary_in_usd)**.\n\n2. Ensemble Learning: Use [caret](https://cran.r-project.org/web/packages/caret/index.html) and [caretEnsemble](https://cran.r-project.org/web/packages/caretEnsemble/index.html)\n- Functions for creating ensembles of caret models: caretList() and caretStack(). caretList() is a convenience function for fitting multiple caret::train() models to the same dataset. caretStack() will make linear or non-linear combinations of these models, using a caret::train() model as a meta-model, and caretEnsemble() will make a robust linear combination of models using a GLM.\n\n4. We also use cross-validation (5-fold) and grid search to find the best hyperparameters.\n5. We use **RMSE of salary_in_usd** to evaluate the performance of model.\n\n- null model: use linear regression(only intercept)\n```r\nnull_model \u003c- lm(salary_in_usd~1, data = train_data)\nsummary(null_model)\n\nnull_rmse \u003c- RMSE(train_data$salary_in_usd %\u003e% arcsignedlog10,\n                  null_model$fitted.values %\u003e% arcsignedlog10)\n```\n\n- ensemble model = random forest + gradient boosting\n``` r\nlibrary(randomForest)\nlibrary(gbm)\n\nlibrary(caret)\nlibrary(caretEnsemble)\n\n# cross validation\nctrl \u003c- trainControl(\n  method = \"cv\",                \n  number = 5,                   \n  savePredictions = \"final\",    \n  returnData = TRUE,            \n  returnResamp = \"final\",       \n  verboseIter = TRUE            \n)\n\n# grid search\nrf_grid \u003c- expand.grid(\n  mtry = c(2)\n)\n\ngbm_grid \u003c- expand.grid(\n  n.trees = 150,\n  interaction.depth = 3,\n  shrinkage = 0.1,\n  n.minobsinnode = 10\n)\n\n# model_list\nmodel_list \u003c- caretList(\n  salary_in_usd~.,\n  data = train_data, \n  metric = \"RMSE\",\n  verbose = T,\n  trControl = ctrl,\n  tuneList = list(\n    rf = caretModelSpec(\"rf\", tuneGrid = rf_grid),\n    gbm = caretModelSpec(\"gbm\", tuneGrid = gbm_grid)\n  )\n)\n\n# ensemble model\nens_model \u003c- caretEnsemble(model_list)\n```\n\n### Feature Importance\n\n| feature_name\t| overall\t| rf\t| gbm |\n|---|---|---|---|\n| job_title👑\t| 41.94337184\t| 41.65463988\t| 44.24573465 |\n| employee_residence\t| 29.53998588\t| 29.47609937\t| 30.04942006 |\n| experience_level\t| 25.94155302\t| 26.09243964\t| 24.73837574 |\n| company_size\t| 1.397332208\t| 1.513988644\t| 0.467108035 |\n| remote_ratio\t| 1.177757053\t| 1.262832468\t| 0.499361506 |\n| employment_type\t| 0\t| 0\t| 0 |\n\n### Measurement\n\n| Train / Test           | RMSE             |\n|--------|----------------------|\n| Train  | 46728.5916975186       |\n\nscatter plot: true vs pred\n![](/output/scatterplot_train_data.png)\n\n### Prediction\n\n| Train / Test         | RMSE             |\n|--------|----------------------|\n| Test | 47087.3351748348        |\n\nscatter plot: true vs pred\n![](/output/scatterplot_test_data.png)\n\n### Compare with other results on Kaggle\n1. Our Performance👑\n\n| Train / Test           | RMSE             |\n|--------|----------------------|\n| Train |  46728.5916975186      |\n| Test |  47087.3351748348        |\n\n2. [AIML salaries 2022-2024 AutoViz+CatBoost+SHAP](https://www.kaggle.com/code/dima806/aiml-salaries-2022-2024-autoviz-catboost-shap)\n\n- RMSE score for train 51.4 kUSD/year, and for test 52.0 kUSD/year\n\n3. [Neural Network Regression Models](https://www.kaggle.com/code/samuelmason23/neural-network-regression-models)\n\n- test RMSE: 57857.07162184822\n\n## Conclusion\nOur accuracy is already significantly better than other results on Kaggle. \n\nHowever, this is a passable result. We need more data or features to optimize our model training to achieve results that most people can accept. \n\nNonetheless, during the training process, we found that job_title is the most important feature, indicating that job titles play a crucial role in predicting salaries.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frita94105%2Fsalary_predict","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frita94105%2Fsalary_predict","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frita94105%2Fsalary_predict/lists"}