{"id":24005164,"url":"https://github.com/ophiase/graph-database-ifncy020","last_synced_at":"2026-04-17T06:33:31.455Z","repository":{"id":269591808,"uuid":"903023978","full_name":"Ophiase/Graph-DataBase-IFNCY020","owner":"Ophiase","description":"🎥 Graph DataBase Course: A project on graph database creation and management using Neo4j and PostgreSQL.","archived":false,"fork":false,"pushed_at":"2025-01-20T00:12:03.000Z","size":3765,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-03T20:41:04.566Z","etag":null,"topics":["cypher","neo4j"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Ophiase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-13T19:06:49.000Z","updated_at":"2025-01-20T00:12:04.000Z","dependencies_parsed_at":"2025-01-19T15:35:06.170Z","dependency_job_id":"d465d632-c41d-4a33-b604-40c2d6b7fca6","html_url":"https://github.com/Ophiase/Graph-DataBase-IFNCY020","commit_stats":null,"previous_names":["ophiase/graph-database-ifncy020"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Ophiase/Graph-DataBase-IFNCY020","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ophiase%2FGraph-DataBase-IFNCY020","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ophiase%2FGraph-DataBase-IFNCY020/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ophiase%2FGraph-DataBase-IFNCY020/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ophiase%2FGraph-DataBase-IFNCY020/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Ophiase","download_url":"https://codeload.github.com/Ophiase/Graph-DataBase-IFNCY020/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ophiase%2FGraph-DataBase-IFNCY020/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271786709,"owners_count":24820747,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-24T02:00:11.135Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cypher","neo4j"],"created_at":"2025-01-08T02:34:22.608Z","updated_at":"2026-04-17T06:33:31.417Z","avatar_url":"https://github.com/Ophiase.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🎥 [Graph DataBase - IFNCY020](https://github.com/Ophiase/Graph-DataBase-IFNCY020)\n\nThis project involves the creation and management of a graph database using Neo4j and PostgreSQL. \\\nThe dataset includes information about works, persons, genres, episodes, and their relationships.\n\n## Table of Contents\n\n- [Loading the Dataset](#loading-the-dataset)\n- [Executing Queries](#executing-queries)\n- [Directory Structure](#directory-structure)\n- [Benchmark](#benchmark)\n- [Conclusion](#conclusion)\n\n## Loading the Dataset\n\nTo load the dataset into PostgreSQL and Neo4j, follow the instructions in the [load/README.md](load/README.md) file. \\\nThis includes steps for setting up the databases and importing the data.\n\n## Executing Queries\n\nThe project includes a set of predefined queries for both Neo4j and PostgreSQL. These queries can be executed using the provided scripts.\n\nTo execute the queries, navigate to the `requests` directory and run the following commands:\n\n```bash\ncd requests\npython3 requests_neo4j.py\npython3 requests_psql.py\n\n# Statistics\npython3 statistics.py\n# Named Graph Statistics (require the named graph)\npython3 named_statistics.py\n```\n\nThese scripts will execute the queries located in the `requests/queries_neo4j` and `requests/queries_psql` directories, respectively.\n\n## Directory Structure\n\n- 🟢 `load/`: Contains scripts for loading the dataset into PostgreSQL and Neo4j.\n  - 🔵 `data/`: Directory for storing processed data files.\n  - 🔵 `resources/`: Directory for storing resource files such as images and configuration files.\n  - 🔵 `default/`: Directory for storing default configuration and data files.\n    - 🟠 `dump.txt`: Default dump file for loading data.\n  - 🟠 `imbd_to_psql.py`: Script to generate SQL scripts from raw TSV files.\n  - 🟠 `psql_to_neo4j.py`: Script to migrate data from PostgreSQL to Neo4j.\n  -\n  - 🟠 `named_graph.py`: Script to recreate the named graph.\n  - 🟠 `load_dump.py`\n  - 🟠 `dump.py`\n  - 🟠 `Makefile`: Makefile to automate the loading process.\n- 🟢 `requests/`: Contains scripts and query files for executing predefined queries.\n  - 🟠 `requests_neo4j.py`: Script to execute Neo4j queries.\n  - 🔵 `queries_neo4j/`: Contains `.cypher` files.\n  - 🟠 `requests_psql.py`: Script to execute PostgreSQL queries.\n  - 🔵 `queries_psql/`: Contains `.sql` files.\n  - 🟠 `statistics.py`: Script to generate statistics from the dataset.\n  - 🔵 `queries_statistics/`: Contains `.cypher` files.\n  - 🟠 `named_statistics.py`: Script to generate statistics for named graphs.\n  - 🔵 `queries_named_statistics/`: Contains `.cypher` files.\n\n\n### Benchmark\n\nPostgres Queries:\n\n```bash\n➜  requests git:(main) ✗ python3 requests_psql.py\n# Responses are cutted them from this REPORT\nExecuting query 'get_all_episodes.sql':\nExecution time: 0.0011 seconds\n----------------------------------------\nExecuting query 'data_graph_topology.sql':\nExecution time: 0.1502 seconds\n----------------------------------------\nExecuting query 'optional_match.sql':\nExecution time: 0.0008 seconds\n----------------------------------------\nExecuting query 'negative_filter.sql':\nExecution time: 0.0006 seconds\n----------------------------------------\nExecuting query 'get_all_persons.sql':\nExecution time: 0.0004 seconds\n----------------------------------------\nExecuting query 'get_all_genres.sql':\nExecution time: 0.0005 seconds\n----------------------------------------\nExecuting query 'get_all_works.sql':\nExecution time: 0.0005 seconds\n```\n\nNeo4J Queries:\n\n```bash\n➜  requests git:(main) ✗ python3 requests_neo4j.py          \n# Responses are cutted them from this REPORT\n----------------------------------------\nExecuting query 'shortest_path.cypher':\nExecution time: 0.0459 seconds\n----------------------------------------\nExecuting query 'collect_unwind.cypher':\nExecution time: 0.0060 seconds\n----------------------------------------\nExecuting query 'data_graph_topology.cypher':\nExecution time: 0.0884 seconds\n----------------------------------------\nExecuting query 'negative_filter.cypher':\nExecution time: 0.0020 seconds\n----------------------------------------\nExecuting query 'with_aggregate_filter.cypher':\nExecution time: 0.0060 seconds\n----------------------------------------\nExecuting query 'post_union_filter.cypher':\nExecution time: 0.0028 seconds\n----------------------------------------\nExecuting query 'with_read_update.cypher':\nExecution time: 0.0146 seconds\n----------------------------------------\nSkip weighted_dijkstra.cypher\n----------------------------------------\nExecuting query 'quantified_patterns.cypher':\nExecution time: 0.0054 seconds\n----------------------------------------\nExecuting query 'optional_match.cypher':\nExecution time: 0.0023 seconds\n----------------------------------------\nExecuting query 'reduce_list.cypher':\nExecution time: 0.0078 seconds\n----------------------------------------\nExecuting query 'predicate_functions.cypher':\nExecution time: 0.0083 seconds\n----------------------------------------\n```\n\nStatistics on the graph:\n\n```bash\n➜  requests git:(main) ✗ python3 statistics.py\nExecution times:\n most_common_genre.cypher: 0.0155 seconds\n average_runtime.cypher: 0.0110 seconds\n most_active_person.cypher: 0.0093 seconds\n total_writers.cypher: 0.0063 seconds\n total_directors.cypher: 0.0081 seconds\n degree_distribution.cypher: 0.0140 seconds\n\nResults:\nMost Common Genre: {'genre': 'Drama', 'count': 12911}\nAverage Runtime: {'average_runtime': 61.746802904697546}\nMost Active Person: {'person': 'Robert Ellis', 'count': 8}\nTotal Writers: {'total_writers': 420}\nTotal Directors: {'total_directors': 304}\nDegree Distribution: {'degree': None, 'count': 180032}\n```\n\nStatistics that require a named graph (Graph Data Science Library):\n\n```bash\n➜  requests git:(main) ✗ python3 named_statistics.py\nExecution times:\n average_path_length.cypher: 1.1397 seconds\n some_path_lengths_over_100.cypher: 1.5135 seconds\n undirected_double_bfs.cypher: 0.5850 seconds\n\nResults:\nAverage Path Length: {'avgLength': 27.476}\nSome Path Lengths Over 100: {'pathLengths': [106, 123, 105, 112, 112, 111, 113, 109, 108, 105, 109, 110, 110, 111, 109, 111, 109, 105, 110, 108]}\nUndirected Double Bfs: {'length(path)': 17}\n```\n\n## Conclusion\n\n- Loading is way faster in postgres\n  - Load every data in Postgres : few seconds\n  - Load a little part of the data in Neo4J : 1 hour\n- Queries faster in Neo4J\n  - ``data_graph_topology.cypher`` : 0.0884 seconds\n  - ``data_graph_topology.sql`` : 0.1502 seconds\n- Queries faster in Postgres\n  - ``optional_match.cypher`` : 0.0023 seconds\n  - ``optional_match.sql``: 0.0008 seconds\n  - ``negative_filter.cypher``: 0.0020 seconds\n  - ``negative_filter.sql`` : 0.0006 seconds\n\nThis project demonstrates the power and flexibility of graph databases for managing and analyzing complex datasets. By leveraging both PostgreSQL and Neo4j, we can efficiently store, query, and analyze data with rich relationships and complex structures. \n\nWhile loading data in PostgreSQL is significantly faster compared to Neo4j, with PostgreSQL handling large datasets in seconds and Neo4j taking much longer for smaller datasets, the Cypher query language in Neo4j offers a more natural and intuitive way to express complex graph queries. Thus we have way more intersting graph queries to show using Neo4J.\n\n## Future Work\n\n- **Data Enrichment**: Integrate additional datasets to enrich the existing data.\n- **Performance Optimization**: Further optimize queries and indexing strategies.\n- **Advanced Analytics**: Implement advanced graph analytics and machine learning algorithms.\n- **Visualization**: Develop visualization tools to explore the graph data interactively.\n- **Double BFS** : Estimation of the diameter \n  - The `undirected_double_bfs.cypher` query returns a path length of 17.\n    - We know from other queries that it should be at least 120. \n  - This discrepancy arises because the graph is directed, and Double BFS is not suited for directed graphs. \n  - To address this, we either need to modify BFS (java) to ignore the direction or complete the graph, which could be a lengthy process.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fophiase%2Fgraph-database-ifncy020","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fophiase%2Fgraph-database-ifncy020","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fophiase%2Fgraph-database-ifncy020/lists"}