{"id":19001823,"url":"https://github.com/albertopirillo/smbud-project-2022","last_synced_at":"2026-04-30T11:33:57.930Z","repository":{"id":171281848,"uuid":"557442692","full_name":"albertopirillo/smbud-project-2022","owner":"albertopirillo","description":"Implementation of multiple NoSQL database technologies (Neo4j, MongoDB and Apache Spark) to handle a large scale data set of scientific publications.","archived":false,"fork":false,"pushed_at":"2022-12-21T10:46:13.000Z","size":22544,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-07T07:07:03.165Z","etag":null,"topics":["apache-spark","mongodb","neo4j","nosql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/albertopirillo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-25T17:42:52.000Z","updated_at":"2024-04-01T12:07:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"a103859b-456c-4285-855c-3ab907360dcd","html_url":"https://github.com/albertopirillo/smbud-project-2022","commit_stats":null,"previous_names":["albertopirillo/smbud-project-2022"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/albertopirillo/smbud-project-2022","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albertopirillo%2Fsmbud-project-2022","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albertopirillo%2Fsmbud-project-2022/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albertopirillo%2Fsmbud-project-2022/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albertopirillo%2Fsmbud-project-2022/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/albertopirillo","download_url":"https://codeload.github.com/albertopirillo/smbud-project-2022/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albertopirillo%2Fsmbud-project-2022/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264032253,"owners_count":23546802,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","mongodb","neo4j","nosql"],"created_at":"2024-11-08T18:12:55.762Z","updated_at":"2025-10-30T05:05:00.316Z","avatar_url":"https://github.com/albertopirillo.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# System and Methods for Big and Unstructured Data - Project\n\nThe aim of the project is to compare different NoSQL database technologies (in particular Neo4j, MongoDB and Apache Spark). \nThis was done by implementing a bibliographic database storage solution capable of supporting a large scale data set containing different types of publications ranging from scientific papers, books, articles and so on.\n\nThe complete report of the project is available [here](/spark/report/Third%20Delivery%20SMBUD%20-%20Group%2038.pdf)\n\n\n## Pre-processing\nA lot of pre-processing was performed, in order to make the downloaded data sets a good fit for our project.\nWe have used Python and in particular the [Pandas](https://pandas.pydata.org/) libarry. All the scripts and the notebooks we used are in this repository.\n\nThe project report contains detailed instructions on how to use such scripts to generate the exact same data sets that we used and on how to perform the exact same queries we executed.\n\n\n## Setup\nInstall all the required Python packages with:\n\n    pip install -r requirements.txt\n\n\n## First delivery \nThe first delivery was about Neo4j, a graph database.\nWe used a data set downloaded from [AMiner](https://lfs.aminer.cn/misc/dblp.v11.zip). \nAfter some additional pre-processing, the data set was uploaded into Neo4j and some queries and commands were executed.\n\nThe report of this delivery is available [here](/neo4j/report/First%20Delivery%20SMBUD%20-%20Group%2038.pdf)\n\n## Second delivery\nThe second delivery was about MongoDB, a document-oriented database.\nWe used the same data set downloaded from [AMiner](https://lfs.aminer.cn/misc/dblp.v11.zip) and some additional data sets to highlight the capabilities of MongoDB at handling sub-documents.  \nAfter some additional pre-processing, the data set was uploaded into MongoDB and some queries and commands were executed.\n\nThe report of this delivery is available [here](/mongodb/report/Second%20Delivery%20SMBUD%20-%20Group%2038.pdf)\n\n## Third delivery\nThe third delivery was about Apache Spark, a framework for large-scale data processing.\nWe used the same data set downloaded from [AMiner](https://lfs.aminer.cn/misc/dblp.v11.zip).  \nAfter some additional pre-processing, the data set was uploaded into Apache Spark and some queries and commands were executed.\n\nThe report of this delivery is available [here](/spark/report/Third%20Delivery%20SMBUD%20-%20Group%2038.pdf)\n\n\n## Software\n- [Draw.io](https://app.diagrams.net/)\n- [Neo4j](https://neo4j.com/)\n- [MongoDB](https://www.mongodb.com/)\n- [Apache Spark](https://spark.apache.org/)\n- [Overleaf](https://www.overleaf.com/)\n- [Python](https://www.python.org/)\n- [PyCharm](https://www.jetbrains.com/pycharm/)\n- [Google Colaboratory](https://colab.research.google.com/)\n\n## License\nLicensed under [MIT License](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falbertopirillo%2Fsmbud-project-2022","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falbertopirillo%2Fsmbud-project-2022","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falbertopirillo%2Fsmbud-project-2022/lists"}