{"id":21184557,"url":"https://github.com/aravind2060/hr_data_analysis_with_spark_structured_api","last_synced_at":"2026-04-29T08:35:03.138Z","repository":{"id":263491587,"uuid":"870836670","full_name":"aravind2060/HR_Data_Analysis_with_Spark_Structured_API","owner":"aravind2060","description":"This assignment helps students learn how to use filtering, grouping, aggregation operations","archived":false,"fork":false,"pushed_at":"2024-11-18T20:17:30.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-14T22:29:16.611Z","etag":null,"topics":["apache-spark","docker","docker-compose","filtering","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aravind2060.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-10T18:54:02.000Z","updated_at":"2024-11-18T20:17:34.000Z","dependencies_parsed_at":"2024-11-19T06:21:05.307Z","dependency_job_id":null,"html_url":"https://github.com/aravind2060/HR_Data_Analysis_with_Spark_Structured_API","commit_stats":null,"previous_names":["aravind2060/hr_data_analysis_with_spark_structured_api"],"tags_count":0,"template":true,"template_full_name":null,"purl":"pkg:github/aravind2060/HR_Data_Analysis_with_Spark_Structured_API","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aravind2060%2FHR_Data_Analysis_with_Spark_Structured_API","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aravind2060%2FHR_Data_Analysis_with_Spark_Structured_API/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aravind2060%2FHR_Data_Analysis_with_Spark_Structured_API/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aravind2060%2FHR_Data_Analysis_with_Spark_Structured_API/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aravind2060","download_url":"https://codeload.github.com/aravind2060/HR_Data_Analysis_with_Spark_Structured_API/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aravind2060%2FHR_Data_Analysis_with_Spark_Structured_API/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273317763,"owners_count":25084044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","docker","docker-compose","filtering","spark"],"created_at":"2024-11-20T18:09:20.424Z","updated_at":"2026-04-29T08:35:03.108Z","avatar_url":"https://github.com/aravind2060.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Employee Engagement Analysis with Spark\n\n## 1. Introduction\n\nThis project aims to analyze employee data using Apache Spark's Structured APIs. You will perform various operations such as filtering, grouping, and aggregation to uncover insights that can help improve employee retention, satisfaction, and compensation at XYZ Corporation.\n\n## 2. Prerequisites\n\nYou can run this project either locally using PySpark or via Docker. Below are the installation and execution steps for both options:\n\n### Option 1: Local Setup with PySpark\n1. Install PySpark using pip:\n   ```bash\n   pip install pyspark\n   ```\n\n2. To run the analysis:\n   ```bash\n   python src/task.py\n   ```\n\n### Option 2: Docker Setup\n1. Docker compose up:\n   ```bash\n   docker compose up -d\n   ```\n2. Copy your source code into containers \n\n3. Execute the script:\n   ```bash\n   spark-submit task1_identify_departments_high_satisfaction.py\n   ```\n\n## 3. Overview \u0026 Objectives\n\n### Overview\n\nThe goal of this project is to use Spark Structured APIs to analyze HR-related datasets and provide actionable insights. You will manipulate and analyze employee and department data, focusing on key metrics such as satisfaction, tenure, and performance.\n\n### Objectives\n\n- Utilize Spark Structured APIs for data manipulation and analysis.\n- Perform advanced filtering, grouping, and aggregation operations.\n- Derive insights on employee retention, satisfaction, and compensation.\n\n### Dataset Introduction\n\nYou are provided with two CSV files:\n\n1. **employees.csv**: Contains details about employees, including their department, age, tenure, satisfaction, salary, and whether they have left the company.\n2. **departments.csv**: Contains information about the departments and their respective managers.\n\n## 4. Assignment Tasks \u0026 Expected Output\n\n### Task 1: Identify Top 2 Highest Paid Employees in Each Department\n\n- Join the `employees` and `departments` datasets on `department_id`.\n- Group by `department_name`, then order employees by `salary` within each group.\n- Select the top 2 highest-paid employees in each department.\n\n**Expected Output**:\n```\ndepartment_name  | employee_id | name          | salary\n---------------------------------------------------------\nMarketing        | 3           | Carol Williams| 72000\nMarketing        | 7           | Grace Wilson  | 70000\nSales            | 8           | Henry Moore   | 95000\nSales            | 5           | Emma Davis    | 90000\n...\n```\n\n### Task 2: Calculate Average Tenure of Employees Who Left vs. Those Who Stayed\n\n- Group by `department_name` and `left_company` status.\n- Calculate the average tenure for both groups.\n\n**Expected Output**:\n```\ndepartment_name  | left_company | average_tenure\n------------------------------------------------\nMarketing        | 0            | 4.1\nMarketing        | 1            | N/A\nSales            | 0            | 8.5\n...\n```\n\n### Task 3: Determine Departments with Significant Gender-Based Salary Disparities\n\n- Group by `department_name` and `gender`.\n- Calculate the average salary by gender and identify departments where the salary difference exceeds 10%.\n\n**Expected Output**:\n```\ndepartment_name | gender | average_salary | salary_diff_percentage\n-------------------------------------------------------------------\nMarketing       | F      | 70000          | -\nMarketing       | M      | N/A            | -\nSales           | F      | 90000          | 0%\n...\n```\n\n### Task 4: Identify Employees Eligible for Promotion Based on Tenure and Performance Score\n\n- Filter employees with tenure ≥ 3 years and performance score ≥ 4.5.\n- List the eligible employees along with their details.\n\n**Expected Output**:\n```\nemployee_id | name          | department_name | tenure | performance_score\n---------------------------------------------------------------------------\n3           | Carol Williams| Marketing       | 5.1    | 4.8\n5           | Emma Davis    | Sales           | 7.0    | 4.9\n...\n```\n\n### Task 5: Identify Departments with the Highest Employee Turnover Rate and Average Salaries\n\n- Group by `department_name` to calculate the turnover rate and average salary.\n- Identify the departments with the highest turnover rates.\n\n**Expected Output**:\n```\ndepartment_name | turnover_rate (%) | average_salary\n----------------------------------------------------\nSales           | 15.0              | 90000\nIT              | 10.0              | 64500\n...\n```\n\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faravind2060%2Fhr_data_analysis_with_spark_structured_api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faravind2060%2Fhr_data_analysis_with_spark_structured_api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faravind2060%2Fhr_data_analysis_with_spark_structured_api/lists"}