{"id":22148528,"url":"https://github.com/mitgar14/etl-workshop-1","last_synced_at":"2026-04-11T07:44:14.937Z","repository":{"id":254185153,"uuid":"844200184","full_name":"mitgar14/etl-workshop-1","owner":"mitgar14","description":"Workshop #1 (Data Engineer) for the ETL course using Pandas, Matplotlib, SQLAlchemy and Power BI for the creation of the dashboard.","archived":false,"fork":false,"pushed_at":"2025-02-18T17:36:02.000Z","size":5174,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T12:48:48.453Z","etag":null,"topics":["data-engineer","data-visualization","etl","pandas","postgresql","powerbi","python","sqlalchemy"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mitgar14.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-18T17:19:46.000Z","updated_at":"2025-02-18T17:36:05.000Z","dependencies_parsed_at":"2024-08-28T17:38:15.547Z","dependency_job_id":"6c991c2e-a030-41b5-b3fe-a9080fcd6d46","html_url":"https://github.com/mitgar14/etl-workshop-1","commit_stats":null,"previous_names":["mitgar14/etl-workshop-1"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mitgar14/etl-workshop-1","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitgar14%2Fetl-workshop-1","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitgar14%2Fetl-workshop-1/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitgar14%2Fetl-workshop-1/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitgar14%2Fetl-workshop-1/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mitgar14","download_url":"https://codeload.github.com/mitgar14/etl-workshop-1/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitgar14%2Fetl-workshop-1/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31673067,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T17:19:37.612Z","status":"online","status_checked_at":"2026-04-11T02:00:05.776Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineer","data-visualization","etl","pandas","postgresql","powerbi","python","sqlalchemy"],"created_at":"2024-12-01T23:28:18.126Z","updated_at":"2026-04-11T07:44:14.923Z","avatar_url":"https://github.com/mitgar14.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Workshop #1: Data Engineer \u003cimg src=\"https://github.com/user-attachments/assets/e50b269a-fd97-4ec3-a1e9-e9629cef94ae\" alt=\"Data Icon\" width=\"30px\"/\u003e\n\nRealized by **Martín García** ([@mitgar14](https://github.com/mitgar14)).\n\n## Overview ✨\n\nIn this workshop we use randomly generated data on candidates stored in a CSV. With this data we run loading, cleaning and transformation processes to find interesting insights using the following tools:\n\n* Python 3.12 ➜ [Download site](https://www.python.org/downloads/)\n* Jupyter Notebook ➜ [VS Code tool for using notebooks](https://youtu.be/ZYat1is07VI?si=BMHUgk7XrJQksTkt)\n* PostgreSQL ➜ [Download site](https://www.postgresql.org/download/)\n* Power BI (Desktop version) ➜ [Download site](https://www.microsoft.com/es-es/power-platform/products/power-bi/desktop)\n\nThe libraries needed for Python are\n\n* Pandas\n* Matplotlib\n* Seaborn\n* SQLAlchemy\n* Dotenv\n\nThese libraries are included in the Poetry project config file (*pyproject.toml*). The step-by-step installation will be described later.\n\n## Dataset Information \u003cimg src=\"https://github.com/user-attachments/assets/5fa5298c-e359-4ef1-976d-b6132e8bda9a\" alt=\"Dataset\" width=\"30px\"/\u003e\n\n\nThe dataset used (*candidates.csv*) has 50,000 rows and 10 columns describing each candidate registered for the recruitment process. \nThis dataset is further transformed to be better processed by the visual analysis tool.\nInitially, the column names of the dataset and their respective Dtype are:\n\n* First Name ➜ Object\n* Last Name ➜ Object\n* Email ➜ Object\n* Country ➜ Object\n* Application Date ➜ Object\n* YOE (years of experience) ➜ Integer\n* Seniority ➜ Object\n* Technology ➜ Object\n* Code Challenge Score ➜ Integer\n* Technical Interview Score ➜ Integer\n\n## Run the project \u003cimg src=\"https://github.com/user-attachments/assets/99bffef1-2692-4cb8-ba13-d6c8c987c6dd\" alt=\"Running code\" width=\"30px\"/\u003e\n\n\u003e Although in this case the example is done with Ubuntu using WSL, this process can be done for any operating system (OS).\n\n### Clone the repository\n\nExecute the following command to clone the repository:\n\n```bash\n  git clone https://github.com/mitgar14/etl-workshop-1.git\n```\n\n#### Demonstration of the process\n\n![Git Clone](https://github.com/user-attachments/assets/0885fcdd-d4d8-4774-98bc-ac34914d9a94)\n\n\n### Enviromental variables\n\n\u003e From now on, the steps will be done in VS Code.\n\nTo establish the connection to the database, we use a module called *connection.py*. In this Python script we call a file where our environment variables are stored, this is how we will create this file:\n\n1. We create a directory named ***env*** inside our cloned repository.\n\n2. There we create a file called ***.env***.\n\n3. In that file we declare 6 enviromental variables. Remember that the variables in this case go without double quotes, i.e. the string notation (`\"`):\n   ```python\n    PG_HOST = # host address, e.g. localhost or 127.0.0.1\n    PG_PORT = # PostgreSQL port, e.g. 5432\n\n    PG_USER = # your PostgreSQL user\n    PG_PASSWORD = # your user password\n    \n    PG_DRIVER = postgresql+psycopg2\n    PG_DATABASE = # your database name, e.g. postgres\n   ```\n\n#### Demonstration of the process\n\n![Env Variables](https://github.com/user-attachments/assets/9e756e5e-db1a-4953-8d7c-1fa0a92d0500)\n\n\n### Installing the dependencies with *Poetry*\n\n\u003e To install Poetry follow [this link](https://elcuaderno.notion.site/Poetry-8f7b23a0f9f340318bbba4ef36023d60?pvs=4).\n\n\n1. Enter the Poetry shell with `poetry shell`.\n\n2. Once the virtual environment is created, execute `poetry install` to install the dependencies. In some case of error with the *.lock* file, just execute `poetry lock` to fix it.\n\n3. Now you can execute the notebooks!\n\n#### Demonstration of the process\n\n![Poetry](https://github.com/user-attachments/assets/3e683921-df9d-4e85-bb3f-c3761a8e3c73)\n\n\n### Running the notebooks\n\nWe execute the 3 notebooks following the next order. You can run it just pressing the \"Execute All\" button:\n\n   1. *001_rawDataLoad.ipynb*\n   2. *002_candidatesEDA.ipynb*\n   3. *003_cleanDataLoad.ipynb*\n\n![image](https://github.com/user-attachments/assets/7599de5a-3330-4d1d-ac08-ced17639c320)\n  \nRemember to choose **the right Python kernel** at the time of running the notebook and **install the *ipykernel*** to support Jupyter notebooks in VS Code.\n\n### Connecting the database with Power BI\n\n1. Open Power BI Desktop and create a new dashboard. Select the *Get data* option - be sure you choose the \"PostgreSQL Database\" option.\n\n![Power BI](https://github.com/user-attachments/assets/a53ef992-d5b9-468e-b227-94e72179a591)\n\n\n2. Insert the PostgreSQL Server and Database Name.\n\n![image](https://github.com/user-attachments/assets/ebe02754-44e8-498c-891f-e1a0038d351d)\n\n\n3. Fill in the following fields with your credentials.\n\n![image](https://github.com/user-attachments/assets/18748b7f-7d5c-4c21-891a-70e77dd21d69)\n\n\n4. If you manage to connect to the database the following tables will appear:\n\n![image](https://github.com/user-attachments/assets/2b4aa913-5fa3-4dc3-8704-1d56a4354a89)\n\n\n5. Choose the candidates_hired table and start making your own visualizations!\n\n![image](https://github.com/user-attachments/assets/4cdff11e-215f-4387-8a2f-ab383dbe436d)\n\n\n## Thank you! 💕🐍\n\nThanks for visiting my project. Any suggestion or contribution is always welcome 👄.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmitgar14%2Fetl-workshop-1","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmitgar14%2Fetl-workshop-1","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmitgar14%2Fetl-workshop-1/lists"}