{"id":15060634,"url":"https://github.com/branb97/jobstreet-data-eng-project","last_synced_at":"2026-02-12T18:02:17.293Z","repository":{"id":249987807,"uuid":"833099815","full_name":"BranB97/jobstreet-data-eng-project","owner":"BranB97","description":"Building a data pipeline to deliver job listing data from Jobstreet for analysis.","archived":false,"fork":false,"pushed_at":"2024-07-31T09:35:49.000Z","size":197,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-02T11:34:40.308Z","etag":null,"topics":["airflow","bigquery","data-engineering","etl-pipeline","google-cloud","looker-studio","python","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BranB97.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-24T10:57:18.000Z","updated_at":"2024-11-28T10:13:08.000Z","dependencies_parsed_at":"2024-10-13T01:40:31.666Z","dependency_job_id":"1cdb225c-31b0-425a-bb93-a3e46a35c259","html_url":"https://github.com/BranB97/jobstreet-data-eng-project","commit_stats":null,"previous_names":["branb97/jobstreet-data-eng-project"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BranB97/jobstreet-data-eng-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BranB97%2Fjobstreet-data-eng-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BranB97%2Fjobstreet-data-eng-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BranB97%2Fjobstreet-data-eng-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BranB97%2Fjobstreet-data-eng-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BranB97","download_url":"https://codeload.github.com/BranB97/jobstreet-data-eng-project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BranB97%2Fjobstreet-data-eng-project/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29375603,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T08:51:36.827Z","status":"ssl_error","status_checked_at":"2026-02-12T08:51:26.849Z","response_time":55,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","bigquery","data-engineering","etl-pipeline","google-cloud","looker-studio","python","sql"],"created_at":"2024-09-24T23:01:47.811Z","updated_at":"2026-02-12T18:02:17.275Z","avatar_url":"https://github.com/BranB97.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Jobstreet Data Engineering Project\n## Introduction\nIn this project, I developed an end-to-end data pipeline to process and load data from a dataset containing job listings scraped from Jobstreet. The core objectives for the project are as follow:\n* Extract job listing data from Jobstreet dataset.\n* Clean and transform the data as needed (e.g. handle missing values, convert data types).\n* Load the data into Google Cloud Storage for further processing.\n* Save the data into BigQuery and organize it using the Star Schema modeling approach.\n* Orchestrate the data pipeline.\n* Visualize the data by creating an interactable dashboard which connects to the database in BigQuery.\n\nThe following sections will elaborate upon the details on the tools and processes involved to create this pipeline.\n\n## Table of Contents\n1. [Dataset Used](#dataset-used)\n2. [Tools Used](#tools-used)\n3. [Data Pipeline Architecture](#data-pipeline-architecture)\n4. [Data Modeling](#data-modeling)\n5. [Infrastructure Setup](#infrastructure-setup)\n6. [ETL Process](#etl-process)\n7. [Dashboard](#dashboard)\n\n## Dataset Used\nThe dataset used for this project are publicly accessible job listings scraped from Jobstreet. This is achieved by developing a script to scrape the job listings based on the keyword specified in a standard parameter file.\n\nMore info about the dataset and web scraping scripts can be found in the following links:\n* Website: https://www.jobstreet.com.my/\n* Data Dictionary: [jobstreet_jobs.csv](https://github.com/BranB97/jobstreet-data-eng-project/blob/main/data/jobstreet_jobs.csv)\n* Raw Data: [jobstreet_datadict.csv](https://github.com/BranB97/jobstreet-data-eng-project/blob/main/jobstreet_datadict.csv)\n* Web Scraping Scripts: [scrape_jobstreet.py](https://github.com/BranB97/jobstreet-data-eng-project/blob/main/job-scraper/scrape_jobstreet.py)\n\n## Tools Used\n* Language: Python, SQL\n* Storage: Google Cloud Storage\n* Data Warehouse: Google BigQuery\n* Workflow Orchestration: Apache Airflow\n* Visualisation: Looker Studio\n* Libraries: Pandas, Selenium\n\n## Data Pipeline Architecture\n![Jobstreet_de_architecture drawio(2)](https://github.com/user-attachments/assets/b18cfc03-4348-49f1-8a26-b2acc9f59d71)\n\nA walkthrough of the pipeline:\n1. The ETL process is orchestrated via Airflow - [dag_jobstreet_etl.py](https://github.com/BranB97/jobstreet-data-eng-project/blob/main/dags/dag_jobstreet_etl.py)\n2. The cleaned data is loaded into the BigQuery data warehouse - [create_dwh_tbls.sql](https://github.com/BranB97/jobstreet-data-eng-project/blob/main/sql/create_dwh_tbls.sql)\n3. Data from BigQuery is visualized via Looker Studio to facilitate analytics - [Jobstreet_Dashboard.pdf](https://github.com/BranB97/jobstreet-data-eng-project/blob/main/Jobstreet_Dashboard.pdf)\n\n## Data Modeling\nThe data warehouse is designed using the Star Schema modeling approach (fact and dim tables).\n\n![Jobstreet Data Modelling_2024-07-17T00_19_56 208Z](https://github.com/user-attachments/assets/b6ef2e0e-a14c-4fa8-b2d9-6a2e2c28c5d5)\n\n\n## Infrastructure Setup\n### Install Windows Subsystem for Linux (WSL)\n1. Open PowerShell or Windows Command Prompt in administrator mode by right-clicking and selecting \"Run as administrator\" and then enter the `wsl --install command`.\n    ```powershell\n     wsl --install\n    ``` \n2. Once installation is complete, restart your machine.\n3. After restart, the WSL can be opened by running Ubuntu. (Note: Ubuntu is installed as the default Linux distribution. If you want to change the distro, please refer [here](https://learn.microsoft.com/en-us/windows/wsl/basic-commands#install) )\n\n   ![image](https://github.com/user-attachments/assets/2a69c2c6-f683-4db2-a330-04415d96747d)\n\n   ![image](https://github.com/user-attachments/assets/5cd59236-42e9-4f54-834f-1e04f1ad4819)\n\n### Install Python\n1. Open WSL and update the local package repositories.\n   ```shell\n    sudo apt update\n   ```\n2. Run the below command to install Python with Pandas library.\n   ```shell\n    sudo apt-get install python3-pandas\n   ```\n3. Verify if its successful by running the below command.\n   ```shell\n    python3 --version\n   ```\n\n### Install Airflow\n1. Open WSL and install `apache-airflow` via pip with constraints that is compatible with `AIRFLOW_VERSION` and `PYTHON_VERSION` to ensure no incompatibility errors arise.\n   ```shell\n    AIRFLOW_VERSION=2.9.3\n    PYTHON_VERSION=\"$(python3 -c 'import sys; print(f\"{sys.version_info.major}.{sys.version_info.minor}\")')\"\n    CONSTRAINT_URL=\"https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt\"\n    pip install \"apache-airflow[google]==${AIRFLOW_VERSION}\" --constraint \"${CONSTRAINT_URL}\"\n   ```\n2. Set `AIRFLOW_HOME` based on your current directory.\n   ```shell\n    export AIRFLOW_HOME=$(pwd)\n   ```\n3. Setup Airflow metadata database based on `AIRFLOW_HOME`.\n   ```shell\n    airflow db migrate\n   ```\n4. Create a user account to login to Airflow UI.\n   ```shell\n    USERNAME=admin\n    FIRSTNAME=data\n    LASTNAME=engineer\n    PASSWORD=\u003cENTER YOUR PASSWORD\u003e\n    EMAIL=\u003cENTER YOUR EMAIL\u003e\n    \n    airflow users create --username USERNAME --firstname FIRSTNAME --lastname LASTNAME --role Admin --password PASSWORD --email EMAIL\n   ```\n5. Start Airflow webserver. \n   ```shell\n    airflow webserver --port 8080\n   ```\n   ![start airflow webserver](https://github.com/user-attachments/assets/074e94b9-d84c-4ae9-ae24-9bcf709aeb21)\n\n6. Open another WSL session and start Airflow scheduler.\n   ```shell\n    airflow scheduler\n   ```   \n   ![start airflow scheduler](https://github.com/user-attachments/assets/5de0ff43-0d53-4042-ac35-83ae957e028a)\n\n7. Access the Airflow UI by visiting `http://localhost:8080/home` in your web browser. Use the account username and password that was created previously to login.\n8. Alternatively, you can run [install.sh](https://github.com/BranB97/jobstreet-data-eng-project/blob/main/install.sh) to streamline the installation process.\n\n### Setup Google Cloud Infrastructure\n1. Open [Google Cloud](https://github.com/BranB97/jobstreet-data-eng-project/blob/main/install.sh) and create a new project.\n2. Go [here](https://console.cloud.google.com/apis/library/browse) and activate the following APIs:\n   * BigQuery API\n   * Cloud Storage API\n3. Go to BigQuery Studio and create your dataset.\n4. Go to Cloud Storage and create your bucket.\n5. Create a [Service Account](https://cloud.google.com/iam/docs/service-account-overview) with the following roles:\n   * `BigQuery Admin`\n   * `Storage Admin`\n6. Download the service account credentials and store the json file in the `AIRFLOW_HOME` directory. This will be used by Airflow to connect to GCP.\n\n### Setup Airflow Connection\n1. Open `http://localhost:8080/connection/list/` and add a connection.\n2. Enter the following fields and then save the connection:\n   * `Connection Id` - Your connection name\n   * `Connection Type` -  Google Cloud\n   * `Description` - Describe what your connection does\n   * `Project Id` - Your GCP project ID\n   * `Keyfile Path` - The path to your service account json key\n   * `Scopes (comma separated)` - https://www.googleapis.com/auth/cloud-platform\n   * `Number of Retries` - 2\n\n### Setup Airflow Global Variables\n1. Open `http://localhost:8080/variable/list/` and add the following variables:\n   * `BASE_PATH` - `AIRFLOW_HOME` directory\n   * `BUCKET_NAME` - The name of the bucket you created in GCS\n   * `DATASET_ID` - The name of the dataset you created in BigQuery\n   * `GOOGLE_CLOUD_CONN_ID` - The name of the Airflow connection you created to interface with GCP\n\n\n## ETL Process\nThis section will breakdown the ETL workflow orchestration process. Each of the process is divided by their individual DAG tasks.\n\n![DAG ETL PIPELINE AIRFLOW](https://github.com/user-attachments/assets/a21e2911-51c6-4bc7-9ba1-f07bfb818f4f)\n\n### Extract and Transform\nIn the `transformer` task, the job listing data is extracted from the raw CSV file that is placed inside the `DATA_PATH` which is the subdirectory of the `AIRFLOW_HOME` directory. Once the raw data is extracted, data cleaning and transformation is carried out prior to loading the data into GCS.\n\nHere is the specific cleaning/transformation task that is involved:\n* Replace certain sentences with more generic terms. e.g.  If the salary range of a certain record does not indicate salary info, it will replace with `No Info for Salary` instead.\n* Split string and string-to-integer data type conversion. e.g. Salary range indicates `RM 7,000 - RM 10,000 per month`, the numbers will be split from the string and stored into min and maximum salary columns respectively.\n\n### Load\nIn the `staging_in_gcs` task, the file containing the cleaned dataset is transferred and stored in GCS, specifically the bucket that was created previously.\n\n![staging](https://github.com/user-attachments/assets/a03f9cf8-667a-4cd8-8357-9895b34170c6)\n\nAfter that, the `load_in_dwh` task will begin and the cleaned data will be loaded into a staging table in BigQuery.\n\nThe process ends with the `load_fact_dim` task which will generate the fact and dim tables from the staging table.\n\n![bigquery loaded tables](https://github.com/user-attachments/assets/db249f9c-d820-4f99-adc0-6dfbffa20626)\n\n## Dashboard\nOnce the data warehouse is established, a dashboard is created via Looker Studio and connected to a customized [dataset](https://github.com/BranB97/jobstreet-data-eng-project/blob/main/sql/create_analysis_tbl.sql) for visualization. You can view the dashboard [here](https://lookerstudio.google.com/reporting/2d07c793-705b-46a2-87a7-d62ded2b842a).\n\n![dashboard1](https://github.com/user-attachments/assets/13859ccd-1cb1-48ed-8e59-9befa5b3379b)\n![dashboard2](https://github.com/user-attachments/assets/0bedcc05-3b5d-4cde-b011-a69fae125013)\n![dashboard3](https://github.com/user-attachments/assets/ef7035eb-ca97-4322-9757-28db8f653457)\n![dashboard4](https://github.com/user-attachments/assets/e530fdeb-4c94-4743-91e0-230a86ed4639)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbranb97%2Fjobstreet-data-eng-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbranb97%2Fjobstreet-data-eng-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbranb97%2Fjobstreet-data-eng-project/lists"}