{"id":30559356,"url":"https://github.com/pramithamj/job-analysis-mapreduce","last_synced_at":"2025-08-28T09:34:33.589Z","repository":{"id":297601974,"uuid":"996359316","full_name":"PramithaMJ/job-analysis-MapReduce","owner":"PramithaMJ","description":"Technical Skills Analysis using MapReduce - hadoop","archived":false,"fork":false,"pushed_at":"2025-06-06T10:01:52.000Z","size":904,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-06T11:18:54.942Z","etag":null,"topics":["hadoop","mapreduce"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PramithaMJ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-04T20:49:41.000Z","updated_at":"2025-06-06T10:01:54.000Z","dependencies_parsed_at":"2025-06-06T11:29:18.328Z","dependency_job_id":null,"html_url":"https://github.com/PramithaMJ/job-analysis-MapReduce","commit_stats":null,"previous_names":["pramithamj/job-analysis-mapreduce"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/PramithaMJ/job-analysis-MapReduce","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2Fjob-analysis-MapReduce","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2Fjob-analysis-MapReduce/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2Fjob-analysis-MapReduce/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2Fjob-analysis-MapReduce/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PramithaMJ","download_url":"https://codeload.github.com/PramithaMJ/job-analysis-MapReduce/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2Fjob-analysis-MapReduce/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272476909,"owners_count":24940943,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hadoop","mapreduce"],"created_at":"2025-08-28T09:34:32.718Z","updated_at":"2025-08-28T09:34:33.571Z","avatar_url":"https://github.com/PramithaMJ.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hadoop \u0026 Spark Job Market Analysis\n\n## University of Ruhuna - Faculty of Engineering\n\n### Assignment 1 - Semester 7 (May 2025)\n\n### Module: Cloud Computing (EC7205)\n\n## Team Members\n\n1. Jayasooriya L.P.M (EG/2020/3990)\n2. Jayathilake H.A.C.P (EG/2020/3994)\n3. Jayawardhana M.V.T.I (EG/2020/3996)\n\n## 🔗 Project Links\n\n- Website: [Hadoop Cluster Manager](https://pramithamj.github.io/hadoop-cluster-manager/)\n- GitHub Repository: [Hadoop Cluster Manager GitHub](https://github.com/pramithamj/hadoop-cluster-manager)\n- Kaggle Dataset: [Job Description Dataset](https://www.kaggle.com/datasets/ravindrasinghrana/job-description-dataset/data)\n\n## Project Overview\n\nThis comprehensive project analyzes job market data using both Hadoop MapReduce and Apache Spark frameworks. The analysis focuses on two key aspects:\n\n1. **Technical Skills Analysis**: Identifies the most in-demand technical skills in the job market by extracting and counting skills mentioned in job descriptions.\n2. **Salary Analysis**: Analyzes salary information across different job titles to identify trends, high-paying roles, and salary distributions.\n\n## Apache Hadoop Cluster Installation Guide\n\nTo simplify running Hadoop clusters, we developed a custom web-based tool: **Hadoop Cluster Manager**.\n\n- Easily install Hadoop\n- Manage clusters via a user-friendly interface\n- Automates common admin operations\n\nFor full instructions and live demo: [Website Link](https://pramithamj.github.io/hadoop-cluster-manager/)\n\ngitHub Link : [hadoop-cluster-manager](https://github.com/pramithamj/hadoop-cluster-manager)\n\n## AWS cluster setup for hadoop\n\nTo run the hadoop cluster on aws with one master node and 3 data nodes, the terraform automation is done to provision and setup the resources in folder [aws-cloud-deployment](aws-cloud-deployment)\n\n\n![alt text](screenshots_and_logs/aws.png)\n\nEC2s provisionsed in AWS\n\n\n![alt text](\u003cscreenshots_and_logs/master install.png\u003e)\n\nInstalling Hadoop on master\n\n\n![alt text](screenshots_and_logs/workerinstall.png)\n\nInstall Hadoop on worker\n\n\n![alt text](screenshots_and_logs/mastersetup.png)\n\nConfigure Hadoop on master node\n\n\n![alt text](screenshots_and_logs/workersetup.png)\n\nConfigure Hadoop on worker node\n\n\n![alt text](screenshots_and_logs/master.png)\n\nHadoop running on the master node\n\n\n![alt text](screenshots_and_logs/worker.png)\n\nHadoop running on a worker node\n\n\n## Repository Structure\n\n- `src/` - Java source code for MapReduce implementation\n\n  - `SkillsAnalyzer.java` - Driver class for the MapReduce job\n  - `SkillsMapper.java` - Mapper implementation for extracting skills\n  - `SkillsReducer.java` - Reducer implementation for aggregating skill counts\n- `input/` - Input data directory\n\n  - `job_descriptions.csv` - Dataset with job postings\n- `conf/` - configuration of hadoop\n- `python_version/` - python version of the hadoop mapreducer\n- `screenshots_and_log` - sceenshot and log of the analysis\n- `output/` - Generated output from MapReduce job\n- `build_salary.md` - Builds the salary analysis MapReduce job\n- `build_spark.sh` - Builds the Spark analysis jobs\n- `build.sh` - Script to compile code and build JAR\n- `run_local.sh` - Script to run the job in local mode\n- `generate_comprehensive_report.sh` - Script to generate conmprehensive report\n- `generate_report.sh` - Script to generate a summary report of results\n- `generate_salary_report.sh` - Generates a basic report for the salary analysis\n- `generate_spark_report_simple.sh` - Generates reports for Spark-based analysis\n- `generate_spark_report.sh` - Generates reports for Spark-based analysis\n- `capture_evidence.sh` - Script to capture evidence of execution\n- `PROJECT_README.md` - Detailed technical documentation\n- `PROJECT_SUMMARY.md` - Summary of project and findings\n- `COMPREHENSIVE_SALARY_REPORT.md` - Detailed salary analysis from MapReduce\n- `RESULTS_ANALYSIS.md` - Analysis of the skills frequency results\n- `PROJECT_STRUCTURE.md` - Details Stucture of the project\n- `RESULTS_ANALYSIS.md` - In-depth analysis of results\n- `SPARK_salary_ANALYSIS_REPORT.md` - Spark implementation of the salary analysis\n- `SPARK_SKILLS_ANALYSIS_REPORT.md` - Spark implementation of the skill analysis\n- `USAGE_GUIDE.md` - Comprehensive usage instructions for all scripts\n- `execution_log.txt` - Log of MapReduce job execution\n- `skills-analyzer.jar` - Compiled MapReduce application\n- `SALARY_ANALYSIS_REPORT.md` - Basic salary statistics report\n- `run_local.sh` - Runs the skills analysis in local mode\n- `run_salary_analysis.sh` - Comprehensive script for running the salary analysis with detailed output\n- `run_salary.sh` - Comprehensive script for running the salary analysis with detailed output\n- `run_spark_analysis.sh` - Runs either salary or skills analysis in Spark\n- `run_spark_simulation.sh` - Generates reports for Spark-based analysis\n\n## Quick Start Guide\n\n### Prerequisites\n\n- Java 11 or higher\n- Hadoop 3.x\n- Apache Spark 3.x (optional, simulation mode available)\n\n### Running the Analysis\n\n#### Hadoop MapReduce Analysis\n\n```bash\n# Build the salary analyzer\n./build_salary.sh\n\n# Run the salary analysis\n./run_salary_analysis.sh input/job_descriptions.csv output/full_salary_analysis\n\n# Generate comprehensive report\n./generate_comprehensive_report.sh output/full_salary_analysis\n```\n\n#### Apache Spark Analysis (with Spark installed)\n\n```bash\n# Build the Spark analyzer\n./build_spark.sh\n\n# Run the Spark salary analysis\n./run_spark_analysis.sh salary input/job_descriptions.csv output/spark_salary_analysis\n\n# Generate the Spark report\n./generate_spark_report_simple.sh salary output/spark_salary_analysis\n```\n\n#### Apache Spark Analysis (simulation mode)\n\n```bash\n# Run the Spark simulation\n./run_spark_simulation.sh salary input/job_descriptions.csv output/spark_salary_analysis\n\n# Generate the Spark report\n./generate_spark_report_simple.sh salary output/spark_salary_analysis\n```\n\n### Viewing Reports\n\nThe reports are generated in markdown format and can be viewed in any markdown viewer:\n\n- `COMPREHENSIVE_SALARY_REPORT.md`: Hadoop-based salary analysis\n- `SPARK_SALARY_ANALYSIS_REPORT.md`: Spark-based salary analysis\n- `SPARK_SKILLS_ANALYSIS_REPORT.md`: Spark-based skills analysis\n\n## Detailed Documentation\n\nFor comprehensive documentation on all scripts and workflows:\n\n- `USAGE_GUIDE.md`: Complete usage instructions for all scripts\n- `PROJECT_SUMMARY.md`: Technical overview and results summary\n- `PROJECT_README.md`: Detailed technical implementation details\n\n## Key Features\n\n- Dual implementation using both Hadoop MapReduce and Apache Spark\n- Comprehensive salary analysis across job titles\n- Technical skills frequency analysis with filtering\n- Detailed report generation\n- Simulation mode for environments without Spark\n- Bash scripts for automation of the entire workflow\n\n## Screenshots and Evidence\n\nThe `screenshots/` directory contains evidence of execution, including:\n\n- Terminal outputs\n- Directory structures\n- File listings\n- Sample results\n\n## License\n\nThis project is developed for educational purposes as part of the EC7205 Cloud Computing module.\n\n## Dataset Description\n\nWe used a job descriptions dataset containing detailed information about job postings, including:\n\n- Job titles and descriptions\n- Required skills and qualifications\n- Company information\n- Salary ranges\n- Job locations and other metadata\n\nThe dataset contains over 100,000 records and is approximately 1.7GB in size, stored in CSV format. Key columns used in our analysis include:\n\n- Column 14: Job Title\n- Column 17: Job Description\n- Column 19: Skills\n\n## How to Run\n\n### Prerequisites\n\n- Java JDK 11 or higher\n- Hadoop 3.4.1 installed and configured\n- CSV dataset in the input directory\n\n### Steps\n\n1. **Build the project**\n\n   ```bash\n   ./build.sh\n   ```\n2. **Run in local mode**\n\n   ```bash\n   ./run_local.sh\n   ```\n3. **Generate a report**\n\n   ```bash\n   ./generate_report.sh\n   ```\n4. **Capture execution evidence**\n\n   ```bash\n   ./capture_evidence.sh\n   ```\n\n\n---\n## Results analysis\n\nThe results analysis is done in the following files:\n\n[COMPREHENSIVE_SALARY_REPORT](COMPREHENSIVE_SALARY_REPORT.md)\n\n[RESULTS_ANALYSIS](RESULTS_ANALYSIS.md)\n\n[SALARY_ANALYSIS_REPORT](SALARY_ANALYSIS_REPORT.md)\n\n[SPARK_salary_ANALYSIS_REPORT](SPARK_salary_ANALYSIS_REPORT.md)\n\n[SPARK_SKILLS_ANALYSIS_REPORT](SPARK_SKILLS_ANALYSIS_REPORT.md)\n\n\n*This project was completed as part of the Cloud Computing (EC7205) course at the University of Ruhuna, Faculty of Engineering.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpramithamj%2Fjob-analysis-mapreduce","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpramithamj%2Fjob-analysis-mapreduce","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpramithamj%2Fjob-analysis-mapreduce/lists"}