{"id":13605345,"url":"https://github.com/HKUST-KnowComp/EFO-1-QA-benchmark","last_synced_at":"2025-04-12T05:32:35.383Z","repository":{"id":97807686,"uuid":"407852312","full_name":"HKUST-KnowComp/EFO-1-QA-benchmark","owner":"HKUST-KnowComp","description":"Benchmark for Answering Existential First Order Queries with Single Free Variable (NeurIPS dataset and benchmark 2021)","archived":false,"fork":false,"pushed_at":"2023-05-03T12:30:30.000Z","size":43212,"stargazers_count":19,"open_issues_count":1,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-07T10:41:04.457Z","etag":null,"topics":["knowledge-graph","knowledge-graph-reasoning","logic-query","query-answering","question-answering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HKUST-KnowComp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-09-18T12:18:59.000Z","updated_at":"2024-03-08T02:37:52.000Z","dependencies_parsed_at":"2023-09-16T04:01:38.549Z","dependency_job_id":null,"html_url":"https://github.com/HKUST-KnowComp/EFO-1-QA-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HKUST-KnowComp%2FEFO-1-QA-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HKUST-KnowComp%2FEFO-1-QA-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HKUST-KnowComp%2FEFO-1-QA-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HKUST-KnowComp%2FEFO-1-QA-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HKUST-KnowComp","download_url":"https://codeload.github.com/HKUST-KnowComp/EFO-1-QA-benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248524180,"owners_count":21118609,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["knowledge-graph","knowledge-graph-reasoning","logic-query","query-answering","question-answering"],"created_at":"2024-08-01T19:00:57.636Z","updated_at":"2025-04-12T05:32:34.876Z","avatar_url":"https://github.com/HKUST-KnowComp.png","language":"Python","funding_links":[],"categories":[":wrench: Implementations"],"sub_categories":["Dataset tools"],"readme":"# EFO-1-QA Benchmark for Complex Query Answering on Knowledge Graphs\n\nThis repository contains an entire pipeline for the EFO-1-QA benchmark. EFO-1 stands for the Existential First Order Queries with Single Free Varibale. The related paper has been accepted by the NeurIPS 2021 track on dataset and benchmark (see [OpenReview Link](https://openreview.net/forum?id=pX4x8f6Km5T)), and appeared on [arXiv](https://arxiv.org/abs/2109.08925).\n\nThe data used in the evaluation can be found in this open link https://hkustconnect-my.sharepoint.com/:f:/g/personal/zwanggc_connect_ust_hk/EpaFL1PUoOFBuCc7hclIM30B8c21e-Tnv1gL11jw_z_SQQ?e=m8RJb5. The compressed version is about 5 GB.\n\n\nIf this work helps you, please cite\n```\n@inproceedings{wang2021benchmarking,\ntitle={Benchmarking the Combinatorial Generalizability of Complex Query Answering on Knowledge Graphs},\nauthor={Zihao Wang and Hang Yin and Yangqiu Song},\nbooktitle={NeurIPS Datasets and Benchmarks Track},\nyear={2021},\nurl={https://openreview.net/forum?id=pX4x8f6Km5T}\n}\n```\n\n## The pipeline overview.\n\n![alt text](figures/pipeline.png)\n\n1. **Query type generation and normalization** The query types are generated by the DFS iteration of the context free grammar with the bounded negation hypothesis. The generated types are also normalized to several normal forms\n2. **Query grounding and answer sampling** The queries are grounded on specific knowledge graphs and the answers that are non-trivial are sampled.\n3. **Model training and estimation** We train and evaluate the specific query structure \n\n## Query type generation and normalization\nThe OpsTree is represented in the nested objects of `FirstOrderSetQuery` class in `fol/foq_v2.py`. \nWe first generate the specific OpsTree and then store it by the `formula` property of `FirstOrderSetQuery`.\n\nThe OpsTree is generated by `binary_formula_iterator` in `fol/foq_v2.py`. The overall process is managed in `formula_generation.py`.\n\nTo generate the formula, just run\n```bash\npython formula_generation.py\n```\n\nThen the file formula csv is generated in the `outputs` folder.\nIn this paper, we use the file in `outputs/test_generated_formula_anchor_node=3.csv`\n\n## Query grounding and answer sampling\n\nWe first prepare the KG data and then run the sampling code\n\nThe KG data (FB15k, FB15k-237, NELL995) should be put into under 'data/' folder. We use the [data](http://snap.stanford.edu/betae/KG_data.zip) provided in the [KGReasoning](https://github.com/snap-stanford/KGReasoning).\n\nThe structure of the data folder should be at least\n\n```\ndata\n\t|---FB15k-237-betae\n\t|---FB15k-betae\n\t|---NELL-betae\t\n```\n\nThen we can run the benchmark sampling code on specific knowledge graph by\n```bash\npython benchmark_sampling.py --knowledge_graph FB15k-237 --num_samples 5000\npython benchmark_sampling.py --knowledge_graph FB15k --num_samples 8000\npython benchmark_sampling.py --knowledge_graph NELL --num_samples 4000\n```\n\n**Append new forms to existing data**\nOne can append new forms to the existing dataset by\n```bash\npython append_new_normal_form.py --knowledge_graph FB15k-237 \n```\n\n## Model training and estimation\n\n**Models**\n\n- [x] [BetaE](https://arxiv.org/abs/2010.11465)\n- [x] [Query2box](https://arxiv.org/abs/2002.05969)\n- [x] [NewLook](http://tonghanghang.org/pdfs/kdd21_newlook.pdf)\n- [x] [LogicE](https://arxiv.org/abs/2103.00418)\n\n**Examples**\n\nThe detailed setting of hyper-parameters or the knowledge graph to choose are in `config` folder,\nyou can modify those configurations to create your own, all the experiments are on FB15k-237 by default.\n\n\nBesides, the generated benchmark, one can also use the BetaE dataset after \nconverting to our format by running:\n\n```bash\npython transform_beta_data.py\n```\n\nUse one of the commands in the following to do training, depending on the choice of models and choices of data type:\n\n```bash\npython main.py --config config/{data_type}_{model_name}.yaml\n```\n\n- The `data_type` includes `EFO-1` and `beta`\n- The `model_name` includes `BetaE`, `LogicE`, `NewLook` and `Query2Box`\n\nIf you need to evaluate on the EFO-1-QA benchmark, be sure to load from existing model checkpoint, you can\ntrain one on your own or download\nfrom [here](https://drive.google.com/drive/folders/13S3wpcsZ9t02aOgA11Qd8lvO0JGGENZ2?usp=sharing):\n\n```bash\npython main.py --config config/evaluate_BetaE.yaml --checkpoint_path ckpt/FB15k-237/Beta_full\npython main.py --config config/evaluate_NewLook.yaml --checkpoint_path ckpt/FB15k-237/NLK_full --load_step 450000\npython main.py --config config/evaluate_LogicE.yaml --checkpoint_path ckpt/FB15k-237/Logic_full --load_step 450000\n```\nWe note that the BetaE checkpoint above is trained from [KGReasoning](https://github.com/snap-stanford/KGReasoning)\n\n## Paper/Repository Checklist\n\n1. For all authors..\n\n\t(a)  Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? **Yes**\n\n\t(b) Have you read the ethics review guidelines and ensured that your paper conforms to them? **Yes**\n\n\t(c) Did you discuss any potential negative societal impacts of your work? **No**\n\n\t(d) Did you describe the limitations of your work? **Yes**\n\n2. If you are including theoretical results...\n\n\t(a) Did you state the full set of assumptions of all theoretical results? **N/A**\n\n\t(b) Did you include complete proofs of all theoretical results? **N/A**\n\n3. If you ran experiments...\n\n\t(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? **Yes**\n\n\t(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? **Yes**\n\n\t(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? **No**\n\n\t(d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? **No**\n\n4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...\n\n\t(a) If your work uses existing assets, did you cite the creators? **Yes**\n\n\t(b) Did you mention the license of the assets? **No**\n\n\t(c) Did you include any new assets either in the supplemental material or as a URL? **Yes**\n\n\t(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? **N/A**\n\n\t(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? **N/A**\n\n5. If you used crowdsourcing or conducted research with human subjects...\n\n\t(a) Did you include the full text of instructions given to participants and screenshots, if applicable? **N/A**\n\n\t(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? **N/A**\n\n\t(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? **N/A**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHKUST-KnowComp%2FEFO-1-QA-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FHKUST-KnowComp%2FEFO-1-QA-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHKUST-KnowComp%2FEFO-1-QA-benchmark/lists"}