{"id":24380660,"url":"https://github.com/ophiase/knowledge-graph-ifncy020","last_synced_at":"2025-07-17T14:06:21.324Z","repository":{"id":270836420,"uuid":"903024608","full_name":"Ophiase/Knowledge-Graph-IFNCY020","owner":"Ophiase","description":"🧠 Knowledge Graph Course: A data engineering project focused on creating, managing, and manipulating knowledge graphs, showcasing essential skills in SPARQL.","archived":false,"fork":false,"pushed_at":"2025-01-05T22:53:56.000Z","size":78,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-12T15:32:05.628Z","etag":null,"topics":["knowledge-graph","sparql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Ophiase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-13T19:08:35.000Z","updated_at":"2025-01-05T22:53:59.000Z","dependencies_parsed_at":"2025-03-12T15:42:15.603Z","dependency_job_id":null,"html_url":"https://github.com/Ophiase/Knowledge-Graph-IFNCY020","commit_stats":null,"previous_names":["ophiase/knowledge-graph-ifncy020"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/Ophiase/Knowledge-Graph-IFNCY020","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ophiase%2FKnowledge-Graph-IFNCY020","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ophiase%2FKnowledge-Graph-IFNCY020/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ophiase%2FKnowledge-Graph-IFNCY020/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ophiase%2FKnowledge-Graph-IFNCY020/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Ophiase","download_url":"https://codeload.github.com/Ophiase/Knowledge-Graph-IFNCY020/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ophiase%2FKnowledge-Graph-IFNCY020/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265614271,"owners_count":23798419,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["knowledge-graph","sparql"],"created_at":"2025-01-19T08:27:42.884Z","updated_at":"2025-07-17T14:06:21.307Z","avatar_url":"https://github.com/Ophiase.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🧠 [Knowledge Graph IFNCY020](https://github.com/Ophiase/Knowledge-Graph-IFNCY020)\n\nThis project involves creating a basic knowledge graph for a university assignment. The knowledge graph integrates data from various sources, including Geonames, Wikidata, DBpedia, Open-Drug, and PubMed. The data is downloaded, transformed into RDF format, and merged into a single RDF file. The ontology defines the structure of the knowledge graph, ensuring consistency and enabling reasoning over the data. The project includes scripts for data downloading, transformation, and merging, as well as examples of basic queries and reasoning.\n\n## Summary\n\n- [Installation](#installation)\n  - [Setup a server](#setup-a-server)\n- [Usage](#usage)\n  - [Run Basic Queries](#run-basic-queries)\n  - [Ontology](#ontology)\n- [Documentation](#documentation)\n  - [Data](#data)\n    - [Geonames](#geonames)\n    - [Wikidata](#wikidata)\n    - [DBpedia](#dbpedia)\n    - [Open-Drug](#open-drug)\n    - [PubMed](#pubmed)\n  - [Data Loading and Transformation](#data-loading-and-transformation)\n  - [Challenges](#challenges)\n  - [Conclusion](#conclusion)\n  - [Resources](#resources)\n  - [TODO](#todo)\n\n## Installation\n\n```bash\n# pip dependancies\nmake pip \n\n# download, transform, merge\nmake download \nmake transform\nmake merge\n```\n### Setup a server\n\nIt's not necessary to set up a server; the Python scripts can directly operate on the .rdf files.\n\nRequires: Apache Jena, Riot, Fuseki\n\n```bash\n# For scripts requiring org.apache.jena,\n# do not forget to add JENA to the Java class path in .zshrc or .bashrc\nexport CLASSPATH=$CLASSPATH:$JENAROOT/lib/*\n```\n\n```bash\n# Set up the server\nmake setup_server\nmake start_server\nmake load_rdf # requires make transform\n# Stop the server\nmake stop_server\n```\n\n## Usage\n\n### Run Basic Queries\n\nBy default, the scripts will be launched directly on the .rdf files. \\\nIf you want to use them on the JENA server, \\\nset `USE_JENA_SERVER = True` in `basic_queries_geonames.py`.\n```\n\n```bash\nmake basic_queries\n```\n\n### Ontology\n\n```bash\nmake reasoning\n```\n\n## Documentation\n\n### Data\n\n#### 🔵 Geonames\nGeonames provides geographical data such as cities, administrative divisions, and time zones. The data is downloaded with [download_geonames.py](./download_geonames.py) as tab-delimited text files, transformed into RDF format using [transform_geonames.py](./transform_geonames.py), and integrated into the knowledge graph. The transformation includes mapping columns to RDF properties and creating triples for each entity. For example:\n- **city**: `\u003ccity\u003e \u003cname\u003e \u003cascii_name\u003e \u003clatitude\u003e \u003clongitude\u003e \u003cpopulation\u003e \u003ccountry_code\u003e \u003ctimezone\u003e`\n- **country_info**: `\u003ccountry\u003e \u003ciso\u003e \u003ciso3\u003e \u003ccountry\u003e \u003ccapital\u003e \u003cpopulation\u003e \u003cContinent\u003e \u003ccurrency_code\u003e`\n\n#### 🔴 Wikidata *(Sparql)*\nWikidata offers structured data about various entities. We fetch data about cities and countries, including labels and relationships, using SPARQL queries in [download_sparql.py](./download_sparql.py). The data is saved into RDF format and includes properties such as population, area, and coordinates. The transformation involves converting JSON results from SPARQL queries into RDF triples. For example:\n- **cities**: `\u003ccity\u003e \u003cname\u003e \u003ccountry\u003e \u003cpopulation\u003e \u003carea\u003e \u003clatitude\u003e \u003clongitude\u003e`\n- **countries**: `\u003ccountry\u003e \u003cname\u003e \u003cpopulation\u003e \u003carea\u003e \u003ccapital\u003e`\n\n#### 🔴 DBpedia *(Sparql)*\nDBpedia extracts structured information from Wikipedia. We fetch data about cities and countries, including labels and relationships, using SPARQL queries in [download_sparql.py](./download_sparql.py). The data is saved into RDF format and includes properties such as population, area, and coordinates. The transformation involves converting JSON results from SPARQL queries into RDF triples. For example:\n- **cities**: `\u003ccity\u003e \u003cname\u003e \u003ccountry\u003e \u003cpopulation\u003e \u003carea\u003e \u003clatitude\u003e \u003clongitude\u003e`\n- **countries**: `\u003ccountry\u003e \u003cname\u003e \u003cpopulation\u003e \u003carea\u003e \u003ccapital\u003e`\n\n#### 🟢 [Open-Drug](https://www.kaggle.com/datasets/mannbrinson/open-drug-knowledge-graph) *(Kaggle)*\nOpen-Drug provides data about drugs, conditions, interactions, and manufacturers. The data is downloaded from Kaggle with [download_kaggle.py](./download_kaggle.py), transformed into RDF format using [transform_open_drug.py](./transform_open_drug.py), and integrated into the knowledge graph. The transformation includes mapping columns to RDF properties and creating triples for each entity. For example:\n- **drugs**: `\u003cdrug\u003e \u003cname\u003e \u003cwiki_url\u003e \u003cdrugbank_url\u003e`\n- **conditions**: `\u003ccondition\u003e \u003cname\u003e \u003csource_id\u003e \u003curl\u003e`\n- **interactions**: `\u003cinteraction\u003e \u003csource_drug_id\u003e \u003ctarget_drug_id\u003e`\n- **manufacturers**: `\u003cmanufacturer\u003e \u003cname\u003e`\n\n#### 🟢 [PubMed](https://www.kaggle.com/datasets/krishnakumarkk/pubmed-knowledge-graph-dataset) *(Kaggle)*\nPubMed provides a large dataset of biomedical literature. Due to its size (50GB), it is not included in this project.\n\n### Data Loading and Transformation\n\n1. **Download**: Data is downloaded from various sources using [download_kaggle.py](./download_kaggle.py), [download_geonames.py](./download_geonames.py), and [download_sparql.py](./download_sparql.py).\n2. **Transform**: Data is transformed into RDF format using [transform_open_drug.py](./transform_open_drug.py) and [transform_geonames.py](./transform_geonames.py). \n    - The transformation scripts map columns to RDF properties and create triples for each entity.\n3. **Merge**: All RDF files are merged into a single RDF file using [merge_rdf.py](./merge_rdf.py).\n\n### Ontology\n\nThe ontology defines the structure of the knowledge graph, including classes, properties, and relationships. It ensures consistency and enables reasoning over the data. The ontology is defined in [ontology.ttl](./ontology.ttl) and includes classes such as city, country, and Drug, and properties such as located_in, population, and name.\n\n### Challenges\n\n#### Federated Query Challenges\n\nIn `basic_queries_geonames.py`, the federated query is particularly challenging because the `rdfs:label` property in DBpedia does not exactly correspond to the `geo:name` property in Geonames. For example, in Geonames, a city might have `geo:name = \"Seaford\"`, but in DBpedia, the `rdfs:label` might be `Seaford, Atlanta` or `Seaford, New York`.\n\nThis discrepancy makes it difficult to match cities between the two datasets directly. To address this, the query needs to include additional logic to handle these differences, which can make the query more complex and less efficient. This issue highlights the importance of having a consistent ontology and naming conventions across different datasets.\n\n### Reasoning with RDFS\n\nTo enhance the knowledge graph, we applied RDFS reasoning to infer new knowledge from the existing data. The ontology defines the relationships between classes and properties, allowing us to infer new information based on these relationships.\n\n#### Ontology Definition\n\nThe ontology is defined in [ontology.ttl](./ontology.ttl) and includes the following elements:\n\n- **Classes**: `location`, `city`, `country`, `drug`, `condition`, `interaction`, `manufacturer`\n- **Properties**: `capital_of`, `located_in`, `country_code`, `population`, `source_drug_id`, `target_drug_id`, `manufacturer`\n- **Unification of Properties**: `dbp:region` and `geo:in_country` are unified under `ex:located_in`\n\n#### Reasoning Process\n\n1. **Load Graph**: Load the RDF data and ontology into a graph.\n2. **Apply Reasoning**: Apply RDFS reasoning to infer new knowledge based on the ontology.\n3. **Query Graph**: Execute SPARQL queries on the graph before and after reasoning to compare the results.\n\nThe reasoning process is implemented in [reasoning.py](./reasoning.py). The script applies RDFS reasoning and compares the results before and after reasoning.\n\n### Conclusion\nThe integration of SPARQL queries with RDFS reasoning has proven to be an effective method for exploiting the integrated datasets in our knowledge graph. By defining a clear and consistent ontology, we were able to infer new relationships and properties, enriching the data and providing more comprehensive query results. The use of RDFS reasoning allowed us to automatically classify entities into higher-level categories and extend relationships, making the data more interconnected and valuable.\n\nHowever, there are some limitations. Loading the graph in Python is quite slow, which can be a bottleneck when dealing with large datasets. Additionally, the requests on Geonames that need to call DBpedia are extremely slow, impacting the overall performance of the system.\n\nFuture improvements to this project could include optimizing the data loading process and improving the efficiency of federated queries. Integrating additional datasets could further enrich the knowledge graph and provide more comprehensive insights.\n\n### Resources\n\n- [Datasets](#data)\n  - Geonames\n    - [Geonames Readme.txt](https://download.geonames.org/export/dump/readme.txt)\n  - Wikidata\n    - [Wikidata Ontology, City](https://www.wikidata.org/wiki/Q515)\n  - BDpedia\n    - [BDPedia Ontology](https://dbpedia.org/ontology/)\n    - [BDPedia Ontology, City](https://dbpedia.org/page/City)\n    - [BDPedia Ontology, Country](https://dbpedia.org/page/Country)\n  - Open Drug\n    - [Open Drug Kaggle](https://www.kaggle.com/datasets/krishnakumarkk/pubmed-knowledge-graph-dataset)\n    - [Open Drug Paper](https://ceur-ws.org/Vol-2873/paper10.pdf)\n  - PubMed\n    - I finally didn't use it.\n\n### TODO\n\n- **Wikidata and DBpedia**: Enhance the data fetching to include more properties such as population, area, and coordinates.\n- **Reasoning**: Apply more advanced reasoning techniques to infer new knowledge from the data.\n- **Queries**: Develop more complex SPARQL queries to demonstrate the integration of different datasets.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fophiase%2Fknowledge-graph-ifncy020","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fophiase%2Fknowledge-graph-ifncy020","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fophiase%2Fknowledge-graph-ifncy020/lists"}