{"id":35322707,"url":"https://github.com/cc59chong/data-modeling-with-postgres","last_synced_at":"2026-04-06T08:01:35.216Z","repository":{"id":148025443,"uuid":"374233490","full_name":"cc59chong/Data-Modeling-with-Postgres","owner":"cc59chong","description":"This project builds a star schema in Postgres with fact and dimension tables for analytics. A Python ETL pipeline transfers data from local JSON files into these tables.","archived":false,"fork":false,"pushed_at":"2025-09-16T18:05:03.000Z","size":141,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-16T21:08:44.768Z","etag":null,"topics":["database","etl","json","modeling","postgres","python","sql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cc59chong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-06-05T23:56:04.000Z","updated_at":"2025-09-16T18:05:06.000Z","dependencies_parsed_at":"2025-09-16T20:22:21.285Z","dependency_job_id":null,"html_url":"https://github.com/cc59chong/Data-Modeling-with-Postgres","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cc59chong/Data-Modeling-with-Postgres","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cc59chong%2FData-Modeling-with-Postgres","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cc59chong%2FData-Modeling-with-Postgres/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cc59chong%2FData-Modeling-with-Postgres/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cc59chong%2FData-Modeling-with-Postgres/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cc59chong","download_url":"https://codeload.github.com/cc59chong/Data-Modeling-with-Postgres/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cc59chong%2FData-Modeling-with-Postgres/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31464102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T21:22:52.476Z","status":"online","status_checked_at":"2026-04-06T02:00:07.287Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","etl","json","modeling","postgres","python","sql"],"created_at":"2025-12-30T23:02:57.699Z","updated_at":"2026-04-06T08:01:35.210Z","avatar_url":"https://github.com/cc59chong.png","language":"Jupyter Notebook","readme":"# Project: Data Modeling with Postgres\n## Introduction\nA startup called Sparkify wants to analyze the data they've been collecting on songs and user activity on their new music streaming app. The analytics team is particularly interested in understanding what songs users are listening to. Currently, they don't have an easy way to query their data, which resides in a directory of JSON logs on user activity on the app, as well as a directory with JSON metadata on the songs in their app. \u003cbr\u003e\nCreating a Postgres database with tables designed to optimize queries on song play analysis. Creating a database schema and ETL pipeline for this analysis and test the database and ETL pipeline by running queries given to you by the analytics team from Sparkify and compare the results with their expected results.\n## Project Description\nThis project implements a data modeling and ETL pipeline solution using PostgreSQL. It features a star schema design with clearly defined fact and dimension tables optimized for specific analytical purposes. The Python-based ETL pipeline efficiently processes and transfers data from JSON files stored in two local directories into the PostgreSQL database, combining Python's processing capabilities with SQL for seamless data integration. This complete solution transforms raw JSON data into an analysis-ready database structure.\n## Project Datasets\n### Song Dataset\nEach file is in JSON format and contains **metadata** about a song and the artist of that song. The files are partitioned by the first three letters of each song's track ID. For example, here are filepaths to two files in this dataset. \u003cbr\u003e\n---**Metadata** is data that provides information about other data.For example, in a database, metadata could describe the schema (table names, column names, data types)\n```\nsong_data/A/B/C/TRABCEI128F424C983.json\nsong_data/A/A/B/TRAABJL12903CDCF1A.json\n```\nAnd below is an example of what a single song file, TRAABJL12903CDCF1A.json, looks like.\n``` JSON\n{\n  \"num_songs\": 1,\n  \"artist_id\": \"ARJIE2Y1187B994AB7\",\n  \"artist_latitude\": null,\n  \"artist_longitude\": null,\n  \"artist_location\": \"\",\n  \"artist_name\": \"Line Renaud\",\n  \"song_id\": \"SOUPIRU12A6D4FA1E1\",\n  \"title\": \"Der Kleine Dompfaff\",\n  \"duration\": 152.92036,\n  \"year\": 0\n}\n```\n### Log Dataset\nThe log files are from a music streaming app based on specified configurations. For example, here are filepaths to two files in this dataset.\u003cbr\u003e\n```\nlog_data/2018/11/2018-11-12-events.json\nlog_data/2018/11/2018-11-13-events.json\n```\n```JSON\n{\n  \"artist\": \"Pavement\",\n  \"auth\": \"Logged In\",\n  \"firstName\": \"Sylvie\",\n  \"gender\": \"F\",\n  \"itemInSession\": 0,\n  \"lastName\": \"Cruz\",\n  \"length\": 99.16036,\n  \"level\": \"free\",\n  \"location\": \"Washington-Arlington-Alexandria, DC-VA-MD-WV\",\n  \"method\": \"PUT\",\n  \"page\": \"NextSong\",\n  \"registration\": 1540266185796.0,\n  \"sessionId\": 345,\n  \"song\": \"Mercy:The Laundromat\",\n  \"status\": 200,\n  \"ts\": 1541990258796,\n  \"userAgent\": \"\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.77.4 (KHTML, like Gecko) Version/7.0.5 Safari/537.77.4\\\"\",\n  \"userId\": \"10\"\n}\n```\n## Data Modeling\nUsing the Star Schema: one fact table consist of the measures associated with each event songplays, and referencing four dimensional tables songs, artists, users and time, each with a primary key that is being referenced from the fact table.\u003cbr\u003e\n![image](https://github.com/MengyaCao/Data-Modeling-with-Postgres/blob/main/ER_Diagram_DM.JPG)\n## Project Template\nThe data files, the project includes seven files:\u003cbr\u003e\n`create_tables.py`: drops and creates your tables. You run this file to reset your tables before each time you run your ETL scripts.\u003cbr\u003e\n`etl.ipynb`: reads and processes a single file from song_data and log_data and loads the data into your tables. This notebook contains detailed instructions on the ETL process for each of the tables.\u003cbr\u003e\n`etl.py`: reads and processes files from song_data and log_data and loads them into your tables. You can fill this out based on your work in the ETL notebook.\u003cbr\u003e\n`README.md`: provides discussion on this project.\u003cbr\u003e\n`ER_Diagram.png`: ERD for star schema of this project\u003cbr\u003e\n`sql_queries.py`: contains all your sql queries, and is imported into the first three files above.\u003cbr\u003e\n`test.ipynb`: displays the first few rows of each table to let us check on the database.\n## How to Run\nRun `create_tables.py` to create the database and tables.\u003cbr\u003e\nRun `etl.py` to build ETL pipeline for loading, extracting and inserting the data.\u003cbr\u003e\nRun `test.ipynb` to confirm the creation of database and columns.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcc59chong%2Fdata-modeling-with-postgres","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcc59chong%2Fdata-modeling-with-postgres","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcc59chong%2Fdata-modeling-with-postgres/lists"}