https://github.com/daniel-elston/json_to_pgsql

This project utilises Python control flows and Pandas to parse JSON data efficiently. The parsed data is stored in a PostgreSQL database. Basic SQL commands are used to query and export data from the DB.
https://github.com/daniel-elston/json_to_pgsql

control-flow json json-parsing postgresql python sql

Last synced: 5 days ago
JSON representation

This project utilises Python control flows and Pandas to parse JSON data efficiently. The parsed data is stored in a PostgreSQL database. Basic SQL commands are used to query and export data from the DB.

• Host: GitHub
• URL: https://github.com/daniel-elston/json_to_pgsql
• Owner: Daniel-Elston
• Created: 2023-06-16T16:19:57.000Z (over 1 year ago)
• Default Branch: master
• Last Pushed: 2023-09-09T15:13:58.000Z (over 1 year ago)
• Last Synced: 2023-09-09T15:43:30.984Z (over 1 year ago)
• Topics: control-flow, json, json-parsing, postgresql, python, sql
• Language: Jupyter Notebook
• Homepage:
• Size: 448 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
![pgAdminResult](pgAdmin_result.png)



# Python JSON Parsing and PostgreSQL Integration

The purpose of this project is to develop an understanding of JSON file formats and how unstructured text data can be stored in a PostgreSQL database, and used in Python. 

For JSON parsing code, please see: 


 `https://github.com/Daniel-Elston/JSON_to_PGSQL/blob/master/Notebooks/B1_JSON_Exploration/json_exploration_3.ipynb`

#### -- Project Status: [Complete]



## Project Objective

Textual data is often unstructured and can be extremely messy. Having the ability to appropriately store this form of data is essential for ML model building and generating insights.

The first stage of this project will raw unstructured data in .JSON format will be parsed using Python then stored in a PostgreSQL database. Once the data has been stored in an organised manner, PostgreSQL queries will used to export data ready for processing in Python.

## Raw Data

https://www.reddit.com/r/all.json

## Technologies

* Python (JSON data handling)

* PostGreSQL

* Libraries: Pandas, NumPy

## Methodologies

* Parsing and handling JSON data

- Database design and management with PostgreSQL

- Data processing and analysis using Python libraries (Pandas, NumPy)



## Contributing Members

**Team Lead: [Daniel Elston](https://github.com/Daniel-Elston)**

|Name     |  GitHub Handles   |  

|---------|-----------------|

| Daniel Elston | [GitHub D. Elston](https://github.com/Daniel-Elston)   |

Please feel free to contact me if you have any questions, require any further information or wish to contribute.




Email: [email protected]