https://github.com/ffatahillah7/etl-with-python-pandapostgresql

Tranform dataset from Csv to Postgresql using Python Panda Sql
https://github.com/ffatahillah7/etl-with-python-pandapostgresql

create-engine etl jupyter-notebook pandasql postgresql python sql sqlalchemy-python

Last synced: about 2 months ago
JSON representation

Tranform dataset from Csv to Postgresql using Python Panda Sql

• Host: GitHub
• URL: https://github.com/ffatahillah7/etl-with-python-pandapostgresql
• Owner: ffatahillah7
• Created: 2024-11-05T04:40:00.000Z (2 months ago)
• Default Branch: main
• Last Pushed: 2024-11-21T08:50:13.000Z (about 2 months ago)
• Last Synced: 2024-11-21T09:32:39.659Z (about 2 months ago)
• Topics: create-engine, etl, jupyter-notebook, pandasql, postgresql, python, sql, sqlalchemy-python
• Language: Jupyter Notebook
• Homepage:
• Size: 201 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# ETL With Python PandaPostgreSQL

Tranform dataset from Csv to Postgresql using Python-PandaSql

The purpose of this project is to extract dataset from csv, tranform using python pandasql and load to PostgreSQL.

Download Dataset file : https://drive.google.com/file/d/1TURPIDouu6_MpVBZ-mRGYnBwiE0ZJ1yK/view?usp=sharing

## Goals of the Project:

1.  Extract dataset like csv,excel,etc.

2.  Tranform using python (I used jupyter notebook) and pandaSQL as Library. PandaSQL have a query like common query in SQL Languages on Python.

3.  Load the result to PostgreSQL using SQLAlchemy library on Python.

## Materials and Methods

1.  Bussiness Understanding : The Business Understanding phase focuses on understanding the objectives and requirements of the project. In this project, we should know about what's happening, why it's happening and who is involved in the dataset.

2.  Data Understanding : Next is the Data Understanding phase. Adding to the foundation of Business Understanding, it drives the focus to identify, collect, and analyze the data sets that can help you accomplish the project goals. The data is consist of Name of covid cases in all around the world.

3.  Data Preparation : This phase, which is often referred to as “data munging”, prepares the final data set(s) for modeling. In this methods , we reading the data frame using dataframe head or subscribe. We make sure that the table, the columns and the data values are on right things.

## Lesson Results

1.  The covid dataset successfully extract from csv to panda dataframe on python. Using pd.read_csv and pointed to our directory, we can read the dataset clearly.

2.  The transformation phase is to confirm the data, and check the column names, values ​​and data format.

3.  We can learn about connection on The Load Phase. The connection build from dataframe to database, for this case is PostgreSQL, using to_sql code on python-pandas library. In this stage, we defined the table name, the connection name, and any other configuration such as if there is an existing table, what should we do?