{"id":14983741,"url":"https://github.com/flavienbwk/neo4j-example-spotifylike","last_synced_at":"2025-04-10T19:41:26.680Z","repository":{"id":88049420,"uuid":"125384285","full_name":"flavienbwk/Neo4j-Example-Spotifylike","owner":"flavienbwk","description":"Based on the idea of Spotify : a concrete example to understand how graph databases work with Neo4j. The challenge is to create a music recommendation algorithm using a very large database of songs (Million Song Challenge Dataset) with an API to interact with (Symfony).","archived":false,"fork":false,"pushed_at":"2018-03-22T21:55:37.000Z","size":20603,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T17:21:17.465Z","etag":null,"topics":["api","graph-database","neo4j","php","python2","song-dataset","symfony3"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flavienbwk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-15T15:02:21.000Z","updated_at":"2025-03-17T01:07:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"f6ef0792-c51a-4e2b-acfa-615f29c6e56d","html_url":"https://github.com/flavienbwk/Neo4j-Example-Spotifylike","commit_stats":{"total_commits":92,"total_committers":2,"mean_commits":46.0,"dds":0.4565217391304348,"last_synced_commit":"4c41f92405a4d857c97a743e107f273d49e9eba5"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2FNeo4j-Example-Spotifylike","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2FNeo4j-Example-Spotifylike/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2FNeo4j-Example-Spotifylike/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2FNeo4j-Example-Spotifylike/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flavienbwk","download_url":"https://codeload.github.com/flavienbwk/Neo4j-Example-Spotifylike/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248281415,"owners_count":21077423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","graph-database","neo4j","php","python2","song-dataset","symfony3"],"created_at":"2024-09-24T14:07:51.619Z","updated_at":"2025-04-10T19:41:26.657Z","avatar_url":"https://github.com/flavienbwk.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Neo4j Example : Spotifylike\nBased on the idea of Spotify : a concrete example to understand how graph databases work, with Neo4j. \nThe challenge is to create a music recommendation algorithm, using a very large database of songs (Million Song Dataset) with a graphical interface (Symfony).\n\n**Take the time to read what I wrote here so you will understand what you do.**\n\n# Formatting the data for Neo4j\n\nDatasets are a bunch of data in a certain form, but we have to convert them to match what Neo4j wants to ingest.\n\n## Dataset :\nYou can first download the data list of the songs. Then, extract it.\n\n\u003cpre\u003e\n$\u003e wget http://static.echonest.com/millionsongsubset_full.tar.gz\n$\u003e tar -xvzf millionsongsubset_full.tar.gz\n\u003c/pre\u003e\n\nThis will create a `./MillionSongSubset` directory.\nThis is a ressource allowing us to get the list of the titles of the songs, along with many (many) data such as the bitrates of the musics or the related artists of the artist who created the music.\n\nEach data of a song is stored inside its `.h5` file.\nThis file format contains a folder and file tree containing data. \nRoughly speaking, these files contain batches of data in tables (as in Excel).\n\n_You can use the\"HDFView\" software to see what these files actually contain._\n\n## Processing the .h5 files :\n\n### 1. Congregate all the files in a single directory.\n\nThe dataset downloaded previously comes with the `.h5` files stored under multiple directories.\nWe store everything in the same directory to make the following scripts easier to process.\n\n\u003cpre\u003e\n$\u003e find -name \"*.h5\" -exec cp {} ../tools/DATASET_PROCESS/H5_FILES/ \\;\n\u003c/pre\u003e\n\nThe dataset provides the data of exactly `10 000` songs.\nTo be sure everything is in there, execute :\n\n\u003cpre\u003e\n$\u003e ls ./tools/DATASET_PROCESS/H5_FILES/ | wc -l\n\u003c/pre\u003e\n\n### 2. Convert .h5 files in a ASCII format text file.\n\nA Python script allows us to extract the data from the dataset, with the information we want (title, artists related, play time etc...).\nExecute `h5_to_ascii.sh` to run the script that translate `.h5` files to human-readable ASCII files.\n\n\u003cpre\u003e\n$\u003e sh ./tools/h5_to_ascii.sh\n\u003c/pre\u003e\nEverything will be stored under `./tools/DATASET_PROCESS/ASCII_FILES/`.\n\n### 3. Convert ASCII format text file to JSON (and then CSV).\n\nNeo4j allows to import CSV files.\nBut as the first script outputs only a ASCII text file, we have to format it in JSON and then in CSV.\n\n:warning: This script is experimental. You might experience an invalid JSON file as some songs have no title or special characters.\n\u003cpre\u003e\n$\u003e sh ./tools/ascii_to_json.sh\n\u003c/pre\u003e\nEverything will be stored under `./tools/DATASET_PROCESS/JSON_FILES/`.\n\nThis script as well concatenate the JSON files into a single file **(in `./tools/DATASET_PROCESS/JSON_FILES/ALL_DATA_JSON.json`)** so we will easily convert it to CSV.\n\n### 4. Convert JSON to CSV.\n\nTo convert the JSON we've outputed to CSV, we use an excellent website :\n- `https://codebeautify.org/json-to-csv`\n\nClick the \"Browse\" button, select `./ALL_DATA_JSON.json` and click \"Download\".\n\nDon't forget to add the file on your server with the name : `ALL_DATA_CSV.csv`.\n\n### 5. Get the list of the artist IDs in a single CSV file.\n\nGet these files we've compiled inside :\n\u003cpre\u003e\n./data/processed/artists_ids.csv\n./data/processed/genres.csv\n\u003c/pre\u003e\n\nWe've stepped into several problems while importing the csv data with our first algorithm.\nSo to have a cleaner and slimmer import, we had to list the artists IDs in a single node to then link the _similar artists_ of a music, to the music.\nSame thing for the genres.\n\n**Here are the steps to get the artist IDs:**\n\nInside the downloaded song list directory : (by default `./MillionSongSubset/`), is a file named `./MillionSongSubset/subset_artist_term.db`.\nThis file is a SQLite database file.\n\nWe've just browsed this database with the [SQLite browser](http://sqlitebrowser.org/) and used the function \"export\", selecting only the `artist_id` column.\n\n\u003chr/\u003e\n\n# Installing Neo4j\n\nYou can follow [this official tutorial](https://neo4j.com/docs/operations-manual/current/installation/linux/debian/) to install Neo4j for your Debian machine.\n\n## Configuration of Neo4j :\n\nInside `/etc/neo4j/neo4j.conf` :\n\u003cpre\u003e\n# Uncomment :\ndbms.security.auth_enabled=false\ndbms.security.allow_csv_import_from_file_urls=true\n\n# Comment :\n#dbms.directories.import=/var/lib/neo4j/import\n\u003c/pre\u003e\n\nRestart Neo4j :\n\u003cpre\u003e\n$\u003e service neo4j restart\n\u003c/pre\u003e\n\n# Fill the database :\n\nAll the queries below are written in _Cypher_.\n_Cypher_ is to _Neo4j_ what _SQL_ is to _MySQL_.\n\nAccess your browser instance of Neo4j with the following link.\nReplace _localhost_ by your IP address if it is necessary.\n\n\u003cpre\u003e\nhttp://localhost:7474/browser\n\u003c/pre\u003e\n\nJust before continuing, we have to increase the limit of 300 nodes display for Neo4j using this command in the Neo4j console :\nChange `1000` by the number you want. But careful : it may make your browser crash.\n\u003cpre\u003e\n:config initialNodeDisplay: 1000\n\u003c/pre\u003e\n\n## Import the *artists_id.csv* file.\n\nReplace */home/user* by the absolute file where you've cloned this git repository.\n\u003cpre\u003e\nLOAD CSV WITH HEADERS FROM \"file:/home/user/Neo4j-Example-Spotifylike/data/processed/artists_id.csv\" AS csvLine\nCREATE (a:Artist { artist_id: csvLine.artist_id })\n\u003c/pre\u003e\n\n## Import the *genres.csv* file.\n\nReplace */home/user* by the absolute file where you've cloned this git repository.\n\u003cpre\u003e\nLOAD CSV WITH HEADERS FROM \"file:/home/user/Neo4j-Example-Spotifylike/data/processed/genres.csv\" AS csvLine\nCREATE (g:Genre { name: csvLine.mbtag })\n\u003c/pre\u003e\n\n## Linking artists' gender to artists.\n\nFor this, we will use the `./data/processed/artist_genre.csv` file.\n\nReplace */home/user* by the absolute file where you've cloned this git repository.\n\u003cpre\u003e\nLOAD CSV WITH HEADERS FROM \"file:/artist_genre.csv\" AS csvLine\nMATCH (a:Artist {artist_id:csvLine.artist_id}), (g:Genre {name: csvLine.mbtag})\nMERGE (a)-[:HAS_GENRE]-\u003e(g)\n\u003c/pre\u003e\n\n## Import the songs data.\n\nWell, it is not just about importing the song list. The fact is that each music has a \"similar_artists\" property, which is really heavy and will overload our server for no reason.\nTo fix this, we will use the `Artist` nodes, and add a relation : `music OWNED_BY artist`. \n\n\u003cpre\u003e\nLOAD CSV WITH HEADERS FROM \"file:/ALL_DATA_CSV.csv\" AS csvLine\n\n// Creating the music node.\nMERGE (m:Music {title: csvLine.title, duration: csvLine.duration})\nWITH m, csvLine\nMATCH (a:Artist {artist_id: csvLine.artist_id})\nMERGE (a)-[:OWNS]-\u003e(m)\nSET a.name = csvLine.artist_name\n\nMERGE (y:Year {year: csvLine.year})\nMERGE (m)-[:RELEASED_IN]-\u003e(y)\n\nMERGE (al:Album {name: csvLine.album})\nMERGE (m)-[:IN]-\u003e(al)\nMERGE (a)-[:CREATED]-\u003e(al)\n\nWITH a, m, csvLine\n\nUNWIND split(csvLine.all_terms, ',') as genre_instance\nMATCH (g:Genre {name: genre_instance})\nMERGE (m)-[:HAS_GENRE]-\u003e(g)\n\nWITH a, m, csvLine\n\nUNWIND split(csvLine.similar_artists, ',') as asi\nMATCH (as:Artist {artist_id: asi})\nMERGE (a)-[:SIMILAR_TO]-\u003e(as)\n\nRETURN count(*)\n\n// LIMIT 5; // Limit the query if you computer is not really powerful.\n\u003c/pre\u003e\n\n:information_source: You might experience some problems while importing a large quantity of data. \nUse the following command at the beginning of the previous command to make it work. It will persist the data every `50` entity processed.\n\u003cpre\u003e\nUSING PERIODIC COMMIT 50\n\u003c/pre\u003e\n\n:information_source: You might experience bugs of memory while importing the data. In `/etc/neo4j/neo4j.conf`, uncomment and modify the following line.\n\u003cpre\u003e\ndbms.memory.heap.max_size=1024m\n\u003c/pre\u003e\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"https://i.imgur.com/UzHJN6s.png\"/\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflavienbwk%2Fneo4j-example-spotifylike","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflavienbwk%2Fneo4j-example-spotifylike","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflavienbwk%2Fneo4j-example-spotifylike/lists"}