https://github.com/fabsdevx/file-format-converter-handout
Data Engineering project for learning purposes. Credits to itversity
https://github.com/fabsdevx/file-format-converter-handout
csv csv-import data data-engineering database pandas python
Last synced: about 1 year ago
JSON representation
Data Engineering project for learning purposes. Credits to itversity
- Host: GitHub
- URL: https://github.com/fabsdevx/file-format-converter-handout
- Owner: FabSDevX
- Created: 2025-01-29T01:24:36.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-29T01:28:43.000Z (about 1 year ago)
- Last Synced: 2025-01-29T02:27:19.931Z (about 1 year ago)
- Topics: csv, csv-import, data, data-engineering, database, pandas, python
- Homepage:
- Size: 1.95 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# File Format Converter Handout
The objective of this project is to develop solutions based on the design provided. In this case, the source data was obtained in the form of CSV files from a MySQL DB.
To improve the efficiency of our data engineering pipelines, we need to convert these CSV files into JSON files, since JSON is better to use in downstream applications than CSV files. The scope of this project involves converting CSV files into JSON files.
### Data Model Details
### Design
### Setup Instructions
1. Setup the Project Using VSCode
2. Make sure you have set up a virtual environment (creating venv, requirements.txt, etc.,) and installed dependencies for the project.
3. It is essential that you deploy the application with the core logic.
4. Run the project after setting all the environment variables.
5. Take appropriate steps to handle the exception
### Validation Steps
* You should check whether the data in the files has been converted properly.
* Make sure the target folder has been created and populated with JSON files and confirm that the schema structure was accurately reflected from the CSV file. (Hint: Refer to schemas.json)
* Take the count of records in the CSV files and compare it to the number of records in the JSON files.
``` Python
import pandas as pd
# ###### Read orders JSON File using PANDAS
orders_data_json= pd.read_json(
'data/retail_db/orders_json/part-00000',
lines=True
)
# To find count of rows
orders_data_json.count()
# ###### Read order_items JSON File using PANDAS
order_items_data_json= pd.read_json(
'data/retail_db/order_items_json/part-00000',
lines=True
)
# To find count of rows
order_items_data_json.count()
```
### Technologies Used
* Programming Language – Python
* Pandas – For Converting CSV to Dataframe and then Dataframe into JSON.