{"id":26662667,"url":"https://github.com/ndomah1/data-cleaning-in-mysql","last_synced_at":"2025-03-25T14:17:41.706Z","repository":{"id":283565675,"uuid":"952187775","full_name":"ndomah1/Data-Cleaning-in-MySQL","owner":"ndomah1","description":"This project cleans and standardizes a global dataset of tech layoffs using MySQL, transforming raw data into an analysis-ready format.","archived":false,"fork":false,"pushed_at":"2025-03-20T22:44:41.000Z","size":216,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-20T23:21:28.362Z","etag":null,"topics":["data-cleaning","data-preparation","layoffs","mysql","sql"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ndomah1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-20T21:52:59.000Z","updated_at":"2025-03-20T22:44:44.000Z","dependencies_parsed_at":"2025-03-20T23:21:29.720Z","dependency_job_id":"1a92a079-a743-41b4-aadc-dce1a84d6537","html_url":"https://github.com/ndomah1/Data-Cleaning-in-MySQL","commit_stats":null,"previous_names":["ndomah1/data-cleaning-in-mysql"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndomah1%2FData-Cleaning-in-MySQL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndomah1%2FData-Cleaning-in-MySQL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndomah1%2FData-Cleaning-in-MySQL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndomah1%2FData-Cleaning-in-MySQL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ndomah1","download_url":"https://codeload.github.com/ndomah1/Data-Cleaning-in-MySQL/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245476718,"owners_count":20621699,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-cleaning","data-preparation","layoffs","mysql","sql"],"created_at":"2025-03-25T14:17:40.005Z","updated_at":"2025-03-25T14:17:41.683Z","avatar_url":"https://github.com/ndomah1.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Cleaning in MySQL - World Layoffs Dataset\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Goals and Key Questions](#goals-and-key-questions)\n- [Dataset](#dataset)\n- [Data Cleaning Steps](#data-cleaning-steps)\n  - [1. Removing Duplicates](#1-removing-duplicates)\n  - [2. Standardizing Data](#2-standardizing-data)\n  - [3. Handling Missing Values](#3-handling-missing-values)\n  - [4. Changing Data Types](#4-changing-data-types)\n- [SQL Scripts Used](#sql-scripts-used)\n- [Future Recommendations](#future-recommendations)\n- [Usage Instructions](#usage-instructions)\n\n## **Overview**\n\nThis project focuses on cleaning a dataset of worldwide layoffs using MySQL. The raw dataset contains inconsistencies, missing values, and formatting errors that must be addressed before conducting meaningful analysis. By applying best practices in data cleaning, we ensure that the dataset is well-structured, accurate, and ready for further exploration.\n\n## **Goals and Key Questions**\n\n- How can we remove duplicate records while preserving data integrity?\n- What inconsistencies exist in the dataset, and how can we standardize them?\n- How should missing values be handled (imputation vs. removal)?\n- Are all data types correctly assigned for analysis?\n\n## **Dataset**\n\nTech firms worldwide have been laying off employees due to economic challenges like slow consumer spending, rising interest rates, and a strong dollar. This dataset tracks tech industry layoffs from **March 11, 2020, to July 20, 2024**, based on reports from Bloomberg, TechCrunch, The New York Times, and other sources.\n\nThe dataset includes:\n\n- **Company Name:** The organization that issued the layoffs.\n- **Location:** The city or country where the layoffs occurred.\n- **Industry:** The sector in which the company operates.\n- **Number of Employees Laid Off:** The total number of employees affected.\n- **Percentage of Workforce Laid Off:** The proportion of the company’s workforce impacted.\n- **Date of Layoffs:** The specific date of the layoff event.\n- **Funding Raised (in Millions):** The total financial backing the company has received.\n\n## **Data Cleaning Steps**\n\n### **1. Removing Duplicates**\n\n- Used row numbering and partitioning techniques to identify duplicate records.\n- Applied filtering strategies to eliminate redundant duplicate rows while retaining one accurate entry.\n\n**SQL Query:**\n\n```sql\nDELETE FROM layoffs_staging\nWHERE id NOT IN (\n    SELECT MIN(id) FROM layoffs_staging\n    GROUP BY company, location, industry, total_laid_off, percentage_laid_off, date, stage, country, funds_raised_millions\n);\n```\n\n### **2. Standardizing Data**\n\n- Trimmed extra spaces in company names and locations.\n- Unified industry categories for consistency (e.g., combining “Crypto” and “Cryptocurrency” into one category).\n- Standardized country names by removing unnecessary punctuation (e.g., “United States.” → “United States”).\n\n**SQL Query:**\n\n```sql\n-- Trim spaces from company names\nUPDATE layoffs_staging\nSET company = TRIM(company);\n\n-- Standardize industry names\nUPDATE layoffs_staging\nSET industry = 'Crypto'\nWHERE industry IN ('Cryptocurrency', 'crypto', 'CryptoCurrency');\n\n-- Remove trailing periods from country names\nUPDATE layoffs_staging\nSET country = TRIM(TRAILING '.' FROM country);\n```\n\n### **3. Handling Missing Values**\n\n- Identified missing values in critical fields such as industry and layoff counts.\n- Filled in missing industry data by referencing existing company records.\n- Evaluated whether specific missing values should be removed based on their impact on data quality.\n\n**SQL Query:**\n\n```sql\n-- Populate missing industry names using existing company records\nUPDATE layoffs_staging t1\nJOIN layoffs_staging t2 ON t1.company = t2.company AND t1.location = t2.location\nSET t1.industry = t2.industry\nWHERE t1.industry IS NULL AND t2.industry IS NOT NULL;\n```\n\n### **4. Changing Data Types**\n\n- Converted the date column from text format to a proper **DATE** type to support time-based analysis.\n- Ensured numerical columns were stored in appropriate **INTEGER** or **FLOAT** data types for accurate calculations.\n\n**SQL Query:**\n\n```sql\n-- Convert date column from TEXT to DATE format\nUPDATE layoffs_staging\nSET date = STR_TO_DATE(date, '%m/%d/%Y');\n\n-- Modify column type to store converted DATE values\nALTER TABLE layoffs_staging\nMODIFY COLUMN date DATE;\n```\n\n## **SQL Scripts Used**\n\nAll data cleaning operations were performed using SQL queries, which are included in the `data_cleaning.sql` script. This script covers:\n\n- Table creation and dataset import\n- Duplicate removal logic\n- Standardization queries\n- Handling of missing values\n- Data type conversions\n\n## **Future Recommendations**\n\n- **Exploratory Data Analysis (EDA):** Now that the data is clean, further analysis can help uncover trends in layoffs over time, across industries, and based on funding levels.\n- **Visualization Dashboards:** Tools like **Tableau** or **Power BI** can be used to create interactive reports and visualizations.\n- **Predictive Modeling:** Machine learning techniques could be applied to predict future layoffs based on industry trends and economic conditions.\n\n## **Usage Instructions**\n\n1. Run the `data_cleaning.sql` script in **MySQL Workbench** or any compatible database management tool.\n2. Verify the cleaned dataset by executing:\n    \n    ```sql\n    SELECT * FROM layoffs_staging;\n    ```\n    \n3. Use the cleaned data for further analysis, visualization, or reporting.\n\nThis project demonstrates how structured data cleaning can transform messy datasets into valuable assets for business insights and decision-making.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndomah1%2Fdata-cleaning-in-mysql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fndomah1%2Fdata-cleaning-in-mysql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndomah1%2Fdata-cleaning-in-mysql/lists"}