{"id":15056811,"url":"https://github.com/vuanhtuan1012/data-modeling-with-cassandra","last_synced_at":"2026-02-19T00:33:58.256Z","repository":{"id":155259337,"uuid":"321465281","full_name":"vuanhtuan1012/data-modeling-with-cassandra","owner":"vuanhtuan1012","description":"Design an Apache Cassandra database which can create queries on song play data to answer the questions of the analysis team of a music streaming application.","archived":false,"fork":false,"pushed_at":"2020-12-31T19:58:52.000Z","size":839,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-26T07:51:30.599Z","etag":null,"topics":["apache-cassandra","cassandra","etl-pipeline","jupyter-notebook","music-streaming-application","python3"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vuanhtuan1012.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-14T20:30:46.000Z","updated_at":"2020-12-31T19:58:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"d352e0f7-62af-4c93-bb54-e7f9b201084d","html_url":"https://github.com/vuanhtuan1012/data-modeling-with-cassandra","commit_stats":{"total_commits":13,"total_committers":1,"mean_commits":13.0,"dds":0.0,"last_synced_commit":"6b3522b6cce49a6ee02f174dd3b7bf4432c05570"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vuanhtuan1012/data-modeling-with-cassandra","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vuanhtuan1012%2Fdata-modeling-with-cassandra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vuanhtuan1012%2Fdata-modeling-with-cassandra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vuanhtuan1012%2Fdata-modeling-with-cassandra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vuanhtuan1012%2Fdata-modeling-with-cassandra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vuanhtuan1012","download_url":"https://codeload.github.com/vuanhtuan1012/data-modeling-with-cassandra/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vuanhtuan1012%2Fdata-modeling-with-cassandra/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29599373,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T22:25:43.180Z","status":"ssl_error","status_checked_at":"2026-02-18T22:25:42.766Z","response_time":162,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-cassandra","cassandra","etl-pipeline","jupyter-notebook","music-streaming-application","python3"],"created_at":"2024-09-24T21:56:37.521Z","updated_at":"2026-02-19T00:33:58.227Z","avatar_url":"https://github.com/vuanhtuan1012.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data modeling with Cassandra\n\nA startup called Sparkify wants to analyze the data they've been collecting on songs and user activity on their new music streaming app. The analysis team is particularly interested in understanding what songs users are listening to. Currently, there is no easy way to query the data to generate the results, since the data reside in a directory of CSV files on user activity on the app.\n\nIn this project I will create an Apache Cassandra database which can create queries on song play data to answer the questions.\n\n## Datasets\n\nThe dataset `event_data` is stored in the directory [event_data](event_data/)   which contains CSV files partioned by date. Here are examples of filepaths to two files in the dataset:\n\n```\nevent_data/2018-11-08-events.csv\nevent_data/2018-11-09-events.csv\n```\n\n## ETL Pipeline\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/pipeline.svg\"/\u003e\n\u003c/p\u003e\n\nThe figure above presents the ETL pipeline of project. It consists of three steps:\n\n1. **E**xtract: Process CSV files in the directory [event_data](event_data/) to create a new CSV file.\n2. **T**ransform: Create an Apache Cassandra database and transform data from the new CSV file into tables.\n3. **L**oad: Do queries on database.\n\n### 1. Process CSV files\n\nIn this part, we reads all CSV files in the directory [event_data](event_data/) then create a smaller event data file called `event_datafile_new.csv` that will be used to insert data into the Apache Cassandra tables.\n\nThe image below is a screenshot of what the denormalized data should appear like in the `event_datafile_new.csv` after the processing of CSV files.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/image_event_datafile_new.jpg\"/\u003e\n\u003c/p\u003e\n\n### 2. Transform data into Cassandra database\n\nThe analysis team needs to query the database to answer the following three questions:\n1. Get the artist, song title and song's length in the music app history that was heard during a session id and an item in section, for example, `sessionId = 338`, and `itemInSession = 4`.\n2. Get only the following: name of artist, song (sorted by `itemInSession`) and user (first and last name) for a given user id and session id, for example, `userid = 10`, `sessionid = 182`.\n3. Get every user name (first and last) in my music app history who listened to a given song, for example, who listened the song `All Hands Against His Own`.\n\n#### Question 1: Get the artist, song title and song’s length in the music app history that was heard during a session id and an item in section.\n\nTo answer this question, we will create the table  `artist_song_session`  which consists of 5 columns:\n\n-   `session_id`: stores the session id, data type: int\n-   `item_in_session`: stores the item number in session, data type: int\n-   `artist`: stores the artist name, data type: text\n-   `song`: stores the song title, data type: text\n-   `length`: stores the song's length, data type: float\n\nThe partition key is  `session_id`, and the column key is  `item_in_session`.\n\n```Python\n# create table\nquery = \"CREATE TABLE IF NOT EXISTS artist_song_session \"\nquery += \"(session_id int, item_in_session int, artist text, song text, length double, \\\nPRIMARY KEY (session_id, item_in_session))\"\ntry:\n    session.execute(query)\nexcept Exception as e:\n    print(e)\n\n# insert data into table\nwith open(new_datafile, encoding = 'utf8') as f:\n    csvreader = csv.reader(f)\n    next(csvreader) # skip header\n    for line in csvreader:\n        query = \"INSERT INTO artist_song_session (session_id, item_in_session, artist, song, length)\"\n        query = query + \" VALUES (%s, %s, %s, %s, %s)\"\n        session.execute(query, (int(line[8]), int(line[3]), line[0], line[9], float(line[5])))\n```\n\n#### Question 2: Get only the following: name of artist, song (sorted by `itemInSession`) and user (first and last name) for a given user id and session id.\n\nTo answer this question, we will create the table  `song_playlist_session`  which consists of 7 columns:\n\n-   `user_id`: stores the user id, data type: int\n-   `session_id`: stores the session id, data type: int\n-   `item_in_session`: stores the item number in session, data type: int\n-   `artist`: stores the artist name, data type: text\n-   `song`: stores the song title, data type: text\n-   `first_name`: stores the first name of user, data type: text\n-   `last_name`: stores the last name of, data type: text\n\nThe partition keys are  `user_id` and `session_id`. We use both user_id and session_id as primary keys so that sessions from the same user are stored in the same nodes.\n\nThe column key is `item_in_session` as we need songs sorted by item in section.\n\n```Python\n# create table\nquery = \"CREATE TABLE IF NOT EXISTS song_playlist_session \"\nquery += \"(user_id int, session_id int, item_in_session int, artist text, song text, \\\nfirst_name text, last_name text, PRIMARY KEY ((user_id, session_id), item_in_session))\"\ntry:\n    session.execute(query)\nexcept Exception as e:\n    print(e)\n\n# insert data into table\nwith open(new_datafile, encoding = 'utf8') as f:\n    csvreader = csv.reader(f)\n    next(csvreader) # skip header\n    for line in csvreader:\n        query = \"INSERT INTO song_playlist_session (user_id, session_id, item_in_session, artist, song, \\\n                first_name, last_name)\"\n        query = query + \" VALUES (%s, %s, %s, %s, %s, %s, %s)\"\n        session.execute(query, (int(line[10]), int(line[8]), int(line[3]), line[0], line[9], line[1], line[4]))\n```\n\n#### Question 3: Get every user name (first and last) in my music app history who listened to a given song.\n\nTo answer this question, we will create the table  `user_song`  which consists of 4 columns:\n\n-   `song`: stores the song title, data type: text\n-   `user_id`: stores the user id, data type: int\n-   `first_name`: stores the first name of user, data type: text\n-   `last_name`: stores the last name of, data type: text\n\nThe partition key is  `song`, and the column key is  `user_id`. We add  `user_id`  to primary key to ensure the row data is unique.\n\n```Python\n# create table\nquery = \"CREATE TABLE IF NOT EXISTS user_song \"\nquery += \"(song text, user_id int, first_name text, last_name text, PRIMARY KEY (song, user_id))\"\ntry:\n    session.execute(query)\nexcept Exception as e:\n    print(e)\n\n# insert data into table\nwith open(new_datafile, encoding = 'utf8') as f:\n    csvreader = csv.reader(f)\n    next(csvreader) # skip header\n    for line in csvreader:\n        query = \"INSERT INTO user_song (song, user_id, first_name, last_name)\"\n        query = query + \" VALUES (%s, %s, %s, %s)\"\n        session.execute(query, (line[9], int(line[10]), line[1], line[4]))\n```\n\n### 3. Query data from database\n\n#### Question 1: Give me the artist, song title and song's length in the music app history that was heard during `sessionId = 338`, and `itemInSession = 4`.\n\n```Python\nquery = \"SELECT artist, song, length FROM artist_song_session WHERE session_id=338 and item_in_session=4\"\ntry:\n    rows = session.execute(query)\nexcept Exception as e:\n    print(e)\n\ndata = PrettyTable()\ndata.field_names = [\"Artist\", \"Song\", \"Length\"]\ndata.align = \"l\"\ndata.align[\"Length\"] = \"r\"\nfor row in rows:\n    data.add_row([row.artist, row.song, row.length])\nprint(data)\n```\n\nThe result:\n\n| Artist | Song | Length |\n| :--- | :--- | ---: |\n| Faithless | Music Matters (Mark Knight Dub) | 495.3073 |\n\n#### Question 2: Give me only the following: name of artist, song (sorted by `itemInSession`) and user (first and last name) for `userid = 10`, `sessionid = 182`.\n\n```Python\nquery = \"SELECT item_in_session, artist, song, first_name, last_name FROM song_playlist_session \\\nWHERE user_id=10 and session_id=182\"\ntry:\n    rows = session.execute(query)\nexcept Exception as e:\n    print(e)\n\ndata = PrettyTable()\ndata.field_names = [\"Artist\", \"Song\", \"Item in session\", \"First name\", \"Last name\"]\ndata.align = \"l\"\ndata.align[\"Item in session\"] = \"r\"\nfor row in rows:\n    data.add_row([row.artist, row.song, row.item_in_session, row.first_name, row.last_name])\nprint(data)\n```\n\nThe result:\n\n| Artist | Song | Item in session | First name | Last name |\n| :--- | :--- | ---: | :--- | :--- |\n| Down To The Bone | Keep On Keepin' On | 0 | Sylvie | Cruz |\n| Three Drives | Greece 2000 | 1 | Sylvie | Cruz |\n| Sebastien Tellier | Kilometer | 2 | Sylvie | Cruz |\n| Lonnie Gordon | Catch You Baby (Steve Pitron \u0026 Max Sanna Radio Edit) | 3 | Sylvie | Cruz |\n\n#### Question 3: Give me every user name (first and last) in my music app history who listened to the song `All Hands Against His Own`.\n\n```Python\nquery = \"SELECT first_name, last_name FROM user_song WHERE song='All Hands Against His Own'\"\ntry:\n    rows = session.execute(query)\nexcept Exception as e:\n    print(e)\n\ndata = PrettyTable()\ndata.field_names = [\"First name\", \"Last name\"]\ndata.align = \"l\"\nfor row in rows:\n    data.add_row([row.first_name, row.last_name])\nprint(data)\n```\n\nThe result:\n\n| First name | Last name |\n| :--- | :--- |\n| Jacqueline | Lynch |\n| Tegan | Levine |\n| Sara | Johnson |\n\nThe complete source code is in the notebook [project](project.ipynb).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvuanhtuan1012%2Fdata-modeling-with-cassandra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvuanhtuan1012%2Fdata-modeling-with-cassandra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvuanhtuan1012%2Fdata-modeling-with-cassandra/lists"}