{"id":17471105,"url":"https://github.com/mikeacosta/data-model-songplays","last_synced_at":"2026-05-02T22:34:06.114Z","repository":{"id":150326735,"uuid":"224956288","full_name":"mikeacosta/data-model-songplays","owner":"mikeacosta","description":"Data modeling and ETL pipeline using Python and PostgreSQL","archived":false,"fork":false,"pushed_at":"2019-11-30T18:27:50.000Z","size":424,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-10T19:06:17.034Z","etag":null,"topics":["data-model","etl","jupyter-notebook","postgresql","python","sql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mikeacosta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-30T03:53:52.000Z","updated_at":"2019-11-30T18:27:53.000Z","dependencies_parsed_at":"2023-07-26T23:47:15.305Z","dependency_job_id":null,"html_url":"https://github.com/mikeacosta/data-model-songplays","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikeacosta%2Fdata-model-songplays","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikeacosta%2Fdata-model-songplays/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikeacosta%2Fdata-model-songplays/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikeacosta%2Fdata-model-songplays/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mikeacosta","download_url":"https://codeload.github.com/mikeacosta/data-model-songplays/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240154953,"owners_count":19756548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-model","etl","jupyter-notebook","postgresql","python","sql"],"created_at":"2024-10-18T16:26:37.727Z","updated_at":"2026-05-02T22:34:06.078Z","avatar_url":"https://github.com/mikeacosta.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# data-model-songplays\n\n## Background\n\nThe analytics team for music streaming startup Spakify wants to anaylze the song-listening activity of their users.  This analysis will be based on JSON user activity logs and song metadata that exist on their mobile app. \n\n## Objective\n\nThe goal of this project is to design a database schema and create an ETL pipeline that enters data from the activity and song metadata JSON files, and deliver tables in a Postgres database against which the analtyics team can run optimized queries for performing song play analysis.\n\n## Data model\n\n\u003cimg src=\"songplays.png\" width=\"75%\" height=\"75%\" /\u003e\n\n## ETL summary\n\n\u003col\u003e\n\u003cli\u003eProcess song files\u003c/li\u003e\n\u003col\u003e\n\u003cli\u003eInsert unique \u003cb\u003esongs\u003c/b\u003e records\u003c/li\u003e\n\u003cli\u003eInsert unique \u003cb\u003eartists\u003c/b\u003e records\u003c/li\u003e\n\u003c/ol\u003e\n\u003c/li\u003e\n\u003cli\u003eProcess log files\u003c/li\u003e\n\u003col\u003e\n\u003cli\u003eFilter by \"NextSong\" action\u003c/li\u003e\n\u003cli\u003eInsert \u003cb\u003etime\u003c/b\u003e records\u003c/li\u003e\n\u003cli\u003eInsert unique user records\u003c/li\u003e\n\u003cli\u003eInsert \u003cb\u003esongplays\u003c/b\u003e records, getting songid and artistid from \u003cb\u003esongs\u003c/b\u003e and \u003cb\u003eartists\u003c/b\u003e tables, respetively\u003c/li\u003e\n\u003c/ol\u003e\n\u003c/ol\u003e \n\n## Project files\n\n- `sql_queries.py` - queries for creating tables and entering data\n- `create_tables.py` - drops and creates tables, used to reset tables prior to running ETL scripts\n- `etl.ipynb` - notebook for developing ETL process, runs insert queries with sample data from  `song_data` and `log_data`\n- `test.ipynb` - selects and displays data from each table to ensure data is correctly entered \n- `etl.py` - primary ETL file, populates tables based on all activity and song metadata files\n\n## Steps to run project\n\n1. Create tables\n\n```\npython create_tables.py\n```\n\n2. Execute ETL pipeline\n\n```\npython etl.py\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikeacosta%2Fdata-model-songplays","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmikeacosta%2Fdata-model-songplays","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikeacosta%2Fdata-model-songplays/lists"}