{"id":43717,"url":"https://github.com/neuralgraphdatabases/awesome-logical-query","name":"awesome-logical-query","description":"A collection of resources on the topic of Complex Logical Query Answering","projects_count":413,"last_synced_at":"2026-08-01T13:00:25.899Z","repository":{"id":148382667,"uuid":"523101404","full_name":"neuralgraphdatabases/awesome-logical-query","owner":"neuralgraphdatabases","description":"A collection of resources on the topic of Complex Logical Query Answering","archived":false,"fork":false,"pushed_at":"2023-09-30T03:13:32.000Z","size":453,"stargazers_count":181,"open_issues_count":2,"forks_count":15,"subscribers_count":10,"default_branch":"main","last_synced_at":"2026-07-13T10:06:46.124Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neuralgraphdatabases.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-08-09T20:37:28.000Z","updated_at":"2026-04-30T09:46:23.000Z","dependencies_parsed_at":"2023-09-30T05:02:47.583Z","dependency_job_id":null,"html_url":"https://github.com/neuralgraphdatabases/awesome-logical-query","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/neuralgraphdatabases/awesome-logical-query","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralgraphdatabases%2Fawesome-logical-query","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralgraphdatabases%2Fawesome-logical-query/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralgraphdatabases%2Fawesome-logical-query/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralgraphdatabases%2Fawesome-logical-query/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neuralgraphdatabases","download_url":"https://codeload.github.com/neuralgraphdatabases/awesome-logical-query/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralgraphdatabases%2Fawesome-logical-query/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36157743,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-01T02:00:05.789Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-01-13T21:18:43.837Z","updated_at":"2026-08-01T13:00:25.900Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":[":scroll: Categorization of papers","All Papers on Complex Logical Query Answering (54)","📈 Datasets and Benchmarking","Application Papers (7)","Potentially Relevant",":wrench: Implementations"],"sub_categories":["Graphs | Modalities","Queries | Query Operators","Dataset tools","Inductive Datasets","Inference (datasets)","EFO-1 Queries","Very Large Datasets","Hyper-Relational Datasets","Temporal Datasets","Graphs | Background Semantics","Regex Queries","Real EFO-1 dataset","Type-Aware Datasets"],"readme":"# Complex Logical Query Answering \u0026 Neural Graph Databases #\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2303.14617\"\u003e\u003cimg src=\"http://img.shields.io/badge/Paper-PDF-red.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://mgalkin.medium.com/neural-graph-databases-cc35c9e1d04f\"\u003e\u003cimg src=\"https://img.shields.io/badge/Blog-Medium-03a87c\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.ngdb.org/\"\u003e\n      \u003cimg src=\"https://img.shields.io/badge/Project-website-blue\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nA collection of resources on the topic of Complex Logical Query Answering accompanying the paper [Neural Graph Reasoning: Complex Logical Query Answering Meets Graph Databases](https://arxiv.org/abs/2303.14617). \nFeel free to open PRs and issues to add new papers, datasets, and implementations!\n\nThis repo follows the Neural Query Engine taxonomy proposed in the paper (Figure 9).\n\n![hierarchy](asset/tree.png)\n\n## :scroll: Categorization of papers\n\n### Graphs | Modalities\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eTriple-based KGs (44)\u003c/summary\u003e\n\n  1. [GQE](https://proceedings.neurips.cc/paper/2018/file/ef50c335cca9f340bde656363ebd02fd-Paper.pdf), NeurIPS 2018  \n  2. [GQE+hashing](https://ieeexplore.ieee.org/abstract/document/8970688/?casa_token=gLHFmr7V2ekAAAAA:wwDbUufdkwHTQo68pansuhJsJ2XQAF0P21_mQSu75KVRUkgqARmXBs_VEmFOkFgz_Lq-FXP8OA), ICDM 2019\n  3. [CGA](https://arxiv.org/pdf/1910.00084.pdf), K-CAP 2019\n  4. [TractOR](https://arxiv.org/pdf/2002.10029.pdf), UAI 2020  \n  5. [Query2Box](https://openreview.net/pdf?id=BJgr4kSFDS), ICLR 2020  \n  6. [BetaE](https://proceedings.neurips.cc/paper/2020/file/e43739bba7cdb577e9e3e4e42447f5a5-Paper.pdf), NeurIPS 2020\n  7. [EmQL](https://proceedings.neurips.cc/paper/2020/hash/fe74074593f21197b7b7be3c08678616-Abstract.html), NeurIPS 2020\n  8. [MPQE](https://grlplus.github.io/papers/26.pdf), ICML 2020 Workshop\n  9. [RotatE-Box](https://arxiv.org/abs/2005.00480), AKBC 2021\n  10. [BiQE](https://ojs.aaai.org/index.php/AAAI/article/view/16630), AAAI 2021\n  11. [Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding](https://arxiv.org/pdf/2110.03789.pdf), arxiv 2021\n  12. [CQD](https://openreview.net/forum?id=Mos9F9kDwkz), ICLR 2021\n  13. [HypE](https://dl.acm.org/doi/abs/10.1145/3442381.3449974?casa_token=tQx20rBUtMwAAAAA:GRqp4JBze6ybzZzeSdCc9cNGlqN0wkAP0BVBPctAgtUVviVSoNiUQcNxBbEuGVGZxnCzJyLqfMmiYg), WWW 2021\n  14. [NewLook](http://tonghanghang.org/pdfs/kdd21_newlook.pdf), KDD 2021\n  15. [ConE](https://proceedings.neurips.cc/paper/2021/hash/a0160709701140704575d499c997b6ca-Abstract.html), NeurIPS 2021\n  16. [PERM](https://proceedings.neurips.cc/paper/2021/hash/c4d2ce3f3ebb5393a77c33c0cd95dc93-Abstract.html), NeurIPS 2021\n  17. [Neural-symbolic Approach for Ontology-mediated Query Answering](https://arxiv.org/pdf/2106.14052.pdf), arxiv 2021\n  18. [LogicE](https://arxiv.org/pdf/2103.00418.pdf), arxiv 2021\n  19. [MLPMix](https://openreview.net/forum?id=tgcAoUVHRIB), ICLR 2022\n  20. [FuzzQE](https://ojs.aaai.org/index.php/AAAI/article/view/20310), AAAI 2022\n  21. [GNN-QE](https://arxiv.org/abs/2205.10128), ICML 2022\n  22. [SMORE](https://arxiv.org/abs/2110.14890), KDD 2022\n  23. [kgTransformer](https://keg.cs.tsinghua.edu.cn/jietang/publications/KDD22-Liu-et-al-KG-Transformer.pdf), KDD 2022\n  24. [LinE](https://dl.acm.org/doi/pdf/10.1145/3534678.3539338?casa_token=_jPlNJj2TlYAAAAA:pKAA42_lrZ2JIHc1YZV0fchIlRiIcqCy8oCBL2UU3Gm84MOeTSYLfQn31DKtXBbU2yqzC7LUsYvREBw), KDD 2022\n  25. [Query2Particles](https://arxiv.org/abs/2204.12847), NAACL 2022\n  26. [TAR](https://arxiv.org/abs/2205.14591), arxiv 2022\n  27. [TeMP](https://arxiv.org/abs/2205.00782), arxiv 2022\n  28. [FLEX](https://arxiv.org/abs/2205.11039), arxiv 2022\n  29. [TFLEX](https://arxiv.org/pdf/2205.14307.pdf), arxiv 2022\n  30. [GNNQ](https://scholar.archive.org/work/sswyx3mbc5danaxosfnyr45cw4/access/wayback/https://s3-eu-west-1.amazonaws.com/pstorage-oxford-0703704596/36647646/GNNQ_full.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIA3OGA3B5WHKHW6LWV/20220814/eu-west-1/s3/aws4_request\u0026X-Amz-Date=20220814T213541Z\u0026X-Amz-Expires=10\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=1bf627735fbce377c2478f514fbac1dfcf49423980d372cc71e2bed732d6ae2c), ISWC 2022\n  31. [ENeSy](https://arxiv.org/pdf/2209.08779.pdf), NeurIPS 2022\n  32. [NodePiece-QE](https://arxiv.org/abs/2210.08008), NeurIPS 2022\n  33. [GammaE](https://arxiv.org/pdf/2210.15578.pdf), EMNLP 2022\n  34. [NMP-QEM](https://aclanthology.org/2022.emnlp-main.194.pdf), EMNLP 2022\n  35. [SignalE](https://link.springer.com/content/pdf/10.1007/978-3-031-10983-6_20.pdf?pdf=inline%20link), KSEM 2022\n  36. [Query2Geom](https://link.springer.com/chapter/10.1007/978-3-031-26438-2_6), AICS 2022\n  37. [LMPNN](https://openreview.net/pdf?id=SoyOsp7i_l), ICLR 2023\n  38. [QTO](https://arxiv.org/pdf/2212.09567.pdf), arxiv 2023\n  39. [Var2Vec](https://ora.ox.ac.uk/objects/uuid:27c9b239-20d0-449f-b425-b5065eb128fe/download_file?file_format=application%2Fpdf\u0026safe_filename=Wang_et_al_2022_Efficient_Embeddings_ofAM.pdf\u0026type_of_work=Conference+item), AAAI 2023\n  40. [CQD A](https://arxiv.org/pdf/2301.12313.pdf), arxiv 2023\n  41. [SQE](https://arxiv.org/pdf/2302.13114.pdf), arxiv 2023\n  42. [NRN](https://arxiv.org/abs/2306.01399) KDD 2023\n  43. [FIT](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023  \n  44. [WFRE](https://arxiv.org/pdf/2305.04034.pdf), ACL 2023\n\n\n  \u003c/details\u003e \n\n\u003cdetails\u003e\n  \u003csummary\u003eHyper-relational KGs (2)\u003c/summary\u003e\n\n  1. [StarQE](https://arxiv.org/abs/2106.08166), ICLR 2022\n  2. [NQE](https://arxiv.org/pdf/2211.13469.pdf), AAAI 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eHyper-graphs and Multi-modal graphs (0)\u003c/summary\u003e\n\n  0. None as of March 2023\n  \n\u003c/details\u003e\n\n\n### Graphs | Reasoning Domain\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eDiscrete (Entities only) (45) \u003c/summary\u003e\n\n  1. [GQE](https://proceedings.neurips.cc/paper/2018/file/ef50c335cca9f340bde656363ebd02fd-Paper.pdf), NeurIPS 2018  \n  2. [GQE+hashing](https://ieeexplore.ieee.org/abstract/document/8970688/?casa_token=gLHFmr7V2ekAAAAA:wwDbUufdkwHTQo68pansuhJsJ2XQAF0P21_mQSu75KVRUkgqARmXBs_VEmFOkFgz_Lq-FXP8OA), ICDM 2019\n  3. [CGA](https://arxiv.org/pdf/1910.00084.pdf), K-CAP 2019\n  4. [TractOR](https://arxiv.org/pdf/2002.10029.pdf), UAI 2020  \n  5. [Query2Box](https://openreview.net/pdf?id=BJgr4kSFDS), ICLR 2020  \n  6. [BetaE](https://proceedings.neurips.cc/paper/2020/file/e43739bba7cdb577e9e3e4e42447f5a5-Paper.pdf), NeurIPS 2020\n  7. [EmQL](https://proceedings.neurips.cc/paper/2020/hash/fe74074593f21197b7b7be3c08678616-Abstract.html), NeurIPS 2020\n  8. [MPQE](https://grlplus.github.io/papers/26.pdf), ICML 2020 Workshop\n  9. [RotatE-Box](https://arxiv.org/abs/2005.00480), AKBC 2021\n  10. [BiQE](https://ojs.aaai.org/index.php/AAAI/article/view/16630), AAAI 2021\n  11. [Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding](https://arxiv.org/pdf/2110.03789.pdf), arxiv 2021\n  12. [CQD](https://openreview.net/forum?id=Mos9F9kDwkz), ICLR 2021\n  13. [HypE](https://dl.acm.org/doi/abs/10.1145/3442381.3449974?casa_token=tQx20rBUtMwAAAAA:GRqp4JBze6ybzZzeSdCc9cNGlqN0wkAP0BVBPctAgtUVviVSoNiUQcNxBbEuGVGZxnCzJyLqfMmiYg), WWW 2021\n  14. [NewLook](http://tonghanghang.org/pdfs/kdd21_newlook.pdf), KDD 2021\n  15. [ConE](https://proceedings.neurips.cc/paper/2021/hash/a0160709701140704575d499c997b6ca-Abstract.html), NeurIPS 2021\n  16. [PERM](https://proceedings.neurips.cc/paper/2021/hash/c4d2ce3f3ebb5393a77c33c0cd95dc93-Abstract.html), NeurIPS 2021\n  17. [Neural-symbolic Approach for Ontology-mediated Query Answering](https://arxiv.org/pdf/2106.14052.pdf), arxiv 2021\n  18. [LogicE](https://arxiv.org/pdf/2103.00418.pdf), arxiv 2021\n  19. [MLPMix](https://openreview.net/forum?id=tgcAoUVHRIB), ICLR 2022\n  20. [StarQE](https://arxiv.org/abs/2106.08166), ICLR 2022\n  21. [FuzzQE](https://ojs.aaai.org/index.php/AAAI/article/view/20310), AAAI 2022\n  22. [GNN-QE](https://arxiv.org/abs/2205.10128), ICML 2022\n  23. [CBR-SUBG](https://proceedings.mlr.press/v162/das22a.html), ICML 2022\n  24. [SMORE](https://arxiv.org/abs/2110.14890), KDD 2022\n  25. [kgTransformer](https://keg.cs.tsinghua.edu.cn/jietang/publications/KDD22-Liu-et-al-KG-Transformer.pdf), KDD 2022\n  26. [LinE](https://dl.acm.org/doi/pdf/10.1145/3534678.3539338?casa_token=_jPlNJj2TlYAAAAA:pKAA42_lrZ2JIHc1YZV0fchIlRiIcqCy8oCBL2UU3Gm84MOeTSYLfQn31DKtXBbU2yqzC7LUsYvREBw), KDD 2022\n  27. [Query2Particles](https://arxiv.org/abs/2204.12847), NAACL 2022\n  28. [TAR](https://arxiv.org/abs/2205.14591), arxiv 2022\n  29. [TeMP](https://arxiv.org/abs/2205.00782), arxiv 2022\n  30. [FLEX](https://arxiv.org/abs/2205.11039), arxiv 2022\n  31. [GNNQ](https://scholar.archive.org/work/sswyx3mbc5danaxosfnyr45cw4/access/wayback/https://s3-eu-west-1.amazonaws.com/pstorage-oxford-0703704596/36647646/GNNQ_full.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIA3OGA3B5WHKHW6LWV/20220814/eu-west-1/s3/aws4_request\u0026X-Amz-Date=20220814T213541Z\u0026X-Amz-Expires=10\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=1bf627735fbce377c2478f514fbac1dfcf49423980d372cc71e2bed732d6ae2c), ISWC 2022\n  32. [ENeSy](https://arxiv.org/pdf/2209.08779.pdf), NeurIPS 2022\n  33. [NodePiece-QE](https://arxiv.org/abs/2210.08008), NeurIPS 2022\n  34. [GammaE](https://arxiv.org/pdf/2210.15578.pdf), EMNLP 2022\n  35. [NMP-QEM](https://aclanthology.org/2022.emnlp-main.194.pdf), EMNLP 2022\n  36. [SignalE](https://link.springer.com/content/pdf/10.1007/978-3-031-10983-6_20.pdf?pdf=inline%20link), KSEM 2022\n  37. [Query2Geom](https://link.springer.com/chapter/10.1007/978-3-031-26438-2_6), AICS 2022\n  38. [LMPNN](https://openreview.net/pdf?id=SoyOsp7i_l), ICLR 2023\n  39. [QTO](https://arxiv.org/pdf/2212.09567.pdf), arxiv 2023\n  40. [Var2Vec](https://ora.ox.ac.uk/objects/uuid:27c9b239-20d0-449f-b425-b5065eb128fe/download_file?file_format=application%2Fpdf\u0026safe_filename=Wang_et_al_2022_Efficient_Embeddings_ofAM.pdf\u0026type_of_work=Conference+item), AAAI 2023\n  41. [NQE](https://arxiv.org/pdf/2211.13469.pdf), AAAI 2023\n  42. [CQD A](https://arxiv.org/pdf/2301.12313.pdf), arxiv 2023\n  43. [SQE](https://arxiv.org/pdf/2302.13114.pdf), TMLR 2023\n  44. [FIT](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023  \n  45. [WFRE](https://arxiv.org/pdf/2305.04034.pdf), ACL 2023\n\n\u003c/details\u003e\n  \n\u003cdetails\u003e\n  \u003csummary\u003eDiscrete Temporal (Entities + Dates) (1)\u003c/summary\u003e\n\n  1. [TFLEX: Temporal Feature-Logic Embedding Framework for Complex Reasoning over Temporal Knowledge Graph](https://arxiv.org/pdf/2205.14307.pdf), arxiv 2022\n  \n\u003c/details\u003e\n\n\u003cdetails\u003e\n \u003csummary\u003eDiscrete + Continuous (Entities + string/numerical Literals) (0)\u003c/summary\u003e\n\n  0. None as of March 2023\n  \n\u003c/details\u003e\n\n\n\n### Graphs | Background Semantics\n\n\u003cdetails\u003e\n  \u003csummary\u003eFacts-only (ABOX) (42) \u003c/summary\u003e\n\n  1. [GQE](https://proceedings.neurips.cc/paper/2018/file/ef50c335cca9f340bde656363ebd02fd-Paper.pdf), NeurIPS 2018  \n  2. [GQE+hashing](https://ieeexplore.ieee.org/abstract/document/8970688/?casa_token=gLHFmr7V2ekAAAAA:wwDbUufdkwHTQo68pansuhJsJ2XQAF0P21_mQSu75KVRUkgqARmXBs_VEmFOkFgz_Lq-FXP8OA), ICDM 2019\n  3. [TractOR](https://arxiv.org/pdf/2002.10029.pdf), UAI 2020  \n  4. [Query2Box](https://openreview.net/pdf?id=BJgr4kSFDS), ICLR 2020  \n  5. [BetaE](https://proceedings.neurips.cc/paper/2020/file/e43739bba7cdb577e9e3e4e42447f5a5-Paper.pdf), NeurIPS 2020\n  6. [EmQL](https://proceedings.neurips.cc/paper/2020/hash/fe74074593f21197b7b7be3c08678616-Abstract.html), NeurIPS 2020\n  7. [MPQE](https://grlplus.github.io/papers/26.pdf), ICML 2020 Workshop\n  8. [RotatE-Box](https://arxiv.org/abs/2005.00480), AKBC 2021\n  9.  [BiQE](https://ojs.aaai.org/index.php/AAAI/article/view/16630), AAAI 2021\n  10. [Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding](https://arxiv.org/pdf/2110.03789.pdf), arxiv 2021\n  11. [CQD](https://openreview.net/forum?id=Mos9F9kDwkz), ICLR 2021\n  12. [HypE](https://dl.acm.org/doi/abs/10.1145/3442381.3449974?casa_token=tQx20rBUtMwAAAAA:GRqp4JBze6ybzZzeSdCc9cNGlqN0wkAP0BVBPctAgtUVviVSoNiUQcNxBbEuGVGZxnCzJyLqfMmiYg), WWW 2021\n  13. [NewLook](http://tonghanghang.org/pdfs/kdd21_newlook.pdf), KDD 2021\n  14. [ConE](https://proceedings.neurips.cc/paper/2021/hash/a0160709701140704575d499c997b6ca-Abstract.html), NeurIPS 2021\n  15. [PERM](https://proceedings.neurips.cc/paper/2021/hash/c4d2ce3f3ebb5393a77c33c0cd95dc93-Abstract.html), NeurIPS 2021\n  16. [LogicE](https://arxiv.org/pdf/2103.00418.pdf), arxiv 2021\n  17. [MLPMix](https://openreview.net/forum?id=tgcAoUVHRIB), ICLR 2022\n  18. [FuzzQE](https://ojs.aaai.org/index.php/AAAI/article/view/20310), AAAI 2022\n  19. [GNN-QE](https://arxiv.org/abs/2205.10128), ICML 2022\n  20. [CBR-SUBG](https://proceedings.mlr.press/v162/das22a.html), ICML 2022\n  21. [SMORE](https://arxiv.org/abs/2110.14890), KDD 2022\n  22. [kgTransformer](https://keg.cs.tsinghua.edu.cn/jietang/publications/KDD22-Liu-et-al-KG-Transformer.pdf), KDD 2022\n  23. [LinE](https://dl.acm.org/doi/pdf/10.1145/3534678.3539338?casa_token=_jPlNJj2TlYAAAAA:pKAA42_lrZ2JIHc1YZV0fchIlRiIcqCy8oCBL2UU3Gm84MOeTSYLfQn31DKtXBbU2yqzC7LUsYvREBw), KDD 2022\n  24. [Query2Particles](https://arxiv.org/abs/2204.12847), NAACL 2022\n  25. [FLEX](https://arxiv.org/abs/2205.11039), arxiv 2022\n  26. [TFLEX](https://arxiv.org/pdf/2205.14307.pdf), arxiv 2022\n  27. [GNNQ](https://scholar.archive.org/work/sswyx3mbc5danaxosfnyr45cw4/access/wayback/https://s3-eu-west-1.amazonaws.com/pstorage-oxford-0703704596/36647646/GNNQ_full.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIA3OGA3B5WHKHW6LWV/20220814/eu-west-1/s3/aws4_request\u0026X-Amz-Date=20220814T213541Z\u0026X-Amz-Expires=10\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=1bf627735fbce377c2478f514fbac1dfcf49423980d372cc71e2bed732d6ae2c), ISWC 2022\n  28. [ENesy](https://arxiv.org/pdf/2209.08779.pdf), NeurIPS 2022\n  29. [NodePiece-QE](https://arxiv.org/abs/2210.08008), NeurIPS 2022\n  30. [GammaE](https://arxiv.org/pdf/2210.15578.pdf), EMNLP 2022\n  31. [NMP-QEM](https://aclanthology.org/2022.emnlp-main.194.pdf), EMNLP 2022\n  32. [SignalE](https://link.springer.com/content/pdf/10.1007/978-3-031-10983-6_20.pdf?pdf=inline%20link), KSEM 2022\n  33. [Query2Geom](https://link.springer.com/chapter/10.1007/978-3-031-26438-2_6), AICS 2022\n  34. [LMPNN](https://openreview.net/pdf?id=SoyOsp7i_l), ICLR 2023\n  35. [QTO](https://arxiv.org/pdf/2212.09567.pdf), arxiv 2023\n  36. [Var2Vec](https://ora.ox.ac.uk/objects/uuid:27c9b239-20d0-449f-b425-b5065eb128fe/download_file?file_format=application%2Fpdf\u0026safe_filename=Wang_et_al_2022_Efficient_Embeddings_ofAM.pdf\u0026type_of_work=Conference+item), AAAI 2023\n  37. [NQE](https://arxiv.org/pdf/2211.13469.pdf), AAAI 2023\n  38. [CQD A](https://arxiv.org/pdf/2301.12313.pdf), arxiv 2023\n  39. [SQE](https://arxiv.org/pdf/2302.13114.pdf), TMLR 2023\n  40. [NRN](https://arxiv.org/abs/2306.01399) KDD 2023\n  41. [FIT](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023  \n  42. [WFRE](https://arxiv.org/pdf/2305.04034.pdf), ACL 2023\n  \n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eClass Hierarchy (3) \u003c/summary\u003e\n\n  1. [CGA](https://arxiv.org/pdf/1910.00084.pdf), K-CAP 2019\n  2. [TeMP](https://arxiv.org/abs/2205.00782), arxiv 2022\n  3. [TAR](https://arxiv.org/abs/2205.14591), arxiv 2022\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eComplex axioms (TBOX) (1) \u003c/summary\u003e\n\n  1. [Neural-symbolic Approach for Ontology-mediated Query Answering](https://arxiv.org/pdf/2106.14052.pdf), arxiv 2021\n\u003c/details\u003e\n\n\n\n### Modeling | Encoder \n\n\u003cdetails\u003e\n  \u003csummary\u003eShallow Embedding (32) \u003c/summary\u003e\n\n  1. [GQE](https://proceedings.neurips.cc/paper/2018/file/ef50c335cca9f340bde656363ebd02fd-Paper.pdf), NeurIPS 2018  \n  2. [GQE+hashing](https://ieeexplore.ieee.org/abstract/document/8970688/?casa_token=gLHFmr7V2ekAAAAA:wwDbUufdkwHTQo68pansuhJsJ2XQAF0P21_mQSu75KVRUkgqARmXBs_VEmFOkFgz_Lq-FXP8OA), ICDM 2019 \n  3. [CGA](https://arxiv.org/pdf/1910.00084.pdf), K-CAP 2019 \n  4. [TractOR](https://arxiv.org/pdf/2002.10029.pdf), UAI 2020 \n  5. [Query2Box](https://openreview.net/pdf?id=BJgr4kSFDS), ICLR 2020  \n  6. [BetaE](https://proceedings.neurips.cc/paper/2020/file/e43739bba7cdb577e9e3e4e42447f5a5-Paper.pdf), NeurIPS 2020\n  7. [EmQL](https://proceedings.neurips.cc/paper/2020/hash/fe74074593f21197b7b7be3c08678616-Abstract.html), NeurIPS 2020\n  8. [Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding](https://arxiv.org/pdf/2110.03789.pdf), arxiv 2021\n  9. [RotatE-Box](https://arxiv.org/abs/2005.00480), AKBC 2021\n  10. [Neural-symbolic Approach for Ontology-mediated Query Answering](https://arxiv.org/pdf/2106.14052.pdf), arxiv 2021\n  11. [HypE](https://dl.acm.org/doi/abs/10.1145/3442381.3449974?casa_token=tQx20rBUtMwAAAAA:GRqp4JBze6ybzZzeSdCc9cNGlqN0wkAP0BVBPctAgtUVviVSoNiUQcNxBbEuGVGZxnCzJyLqfMmiYg), WWW 2021\n  12. [NewLook](http://tonghanghang.org/pdfs/kdd21_newlook.pdf), KDD 2021\n  13. [CQD](https://openreview.net/forum?id=Mos9F9kDwkz), ICLR 2021\n  14. [ConE](https://proceedings.neurips.cc/paper/2021/hash/a0160709701140704575d499c997b6ca-Abstract.html), NeurIPS 2021\n  15. [PERM](https://proceedings.neurips.cc/paper/2021/hash/c4d2ce3f3ebb5393a77c33c0cd95dc93-Abstract.html), NeurIPS 2021\n  16. [LogicE](https://arxiv.org/pdf/2103.00418.pdf), arxiv 2021\n  17. [FuzzQE](https://ojs.aaai.org/index.php/AAAI/article/view/20310), AAAI 2022\n  18. [SMORE](https://arxiv.org/abs/2110.14890), KDD 2022\n  19. [LinE](https://dl.acm.org/doi/pdf/10.1145/3534678.3539338?casa_token=_jPlNJj2TlYAAAAA:pKAA42_lrZ2JIHc1YZV0fchIlRiIcqCy8oCBL2UU3Gm84MOeTSYLfQn31DKtXBbU2yqzC7LUsYvREBw), KDD 2022\n  20. [TAR](https://arxiv.org/abs/2205.14591), arxiv 2022\n  21. [Query2Particles](https://arxiv.org/abs/2204.12847), NAACL 2022\n  22. [FLEX](https://arxiv.org/abs/2205.11039), arxiv 2022\n  23. [TFLEX](https://arxiv.org/pdf/2205.14307.pdf), arxiv 2022\n  24. [GammaE](https://arxiv.org/pdf/2210.15578.pdf), EMNLP 2022\n  25. [NMP-QEM](https://aclanthology.org/2022.emnlp-main.194.pdf), EMNLP 2022\n  26. [SignalE](https://link.springer.com/content/pdf/10.1007/978-3-031-10983-6_20.pdf?pdf=inline%20link), KSEM 2022\n  27. [Query2Geom](https://link.springer.com/chapter/10.1007/978-3-031-26438-2_6), AICS 2022\n  28. [QTO](https://arxiv.org/pdf/2212.09567.pdf), arxiv 2023\n  29. [Var2Vec](https://ora.ox.ac.uk/objects/uuid:27c9b239-20d0-449f-b425-b5065eb128fe/download_file?file_format=application%2Fpdf\u0026safe_filename=Wang_et_al_2022_Efficient_Embeddings_ofAM.pdf\u0026type_of_work=Conference+item), AAAI 2023\n  30. [CQD A](https://arxiv.org/pdf/2301.12313.pdf), arxiv 2023\n  31. [FIT](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023  \n  32. [WFRE](https://arxiv.org/pdf/2305.04034.pdf), ACL 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eTransductive Encoder (9) \u003c/summary\u003e\n\n  1. [MPQE](https://grlplus.github.io/papers/26.pdf), ICML 2020 Workshop\n  2. [BiQE](https://ojs.aaai.org/index.php/AAAI/article/view/16630), AAAI 2021\n  3. [kgTransformer](https://keg.cs.tsinghua.edu.cn/jietang/publications/KDD22-Liu-et-al-KG-Transformer.pdf), KDD 2022\n  4.  [MLPMix](https://openreview.net/forum?id=tgcAoUVHRIB), ICLR 2022\n  5.  [StarQE](https://arxiv.org/abs/2106.08166), ICLR 2022\n  6.  [ENeSy](https://arxiv.org/pdf/2209.08779.pdf) NeurIPS 2022\n  7.  [LMPNN](https://openreview.net/pdf?id=SoyOsp7i_l) ICLR 2023\n  8.  [NQE](https://arxiv.org/pdf/2211.13469.pdf), AAAI 2023\n  9.  [SQE](https://arxiv.org/pdf/2302.13114.pdf), TMLR 2023\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eInductive Encoder (4) \u003c/summary\u003e\n\n  1. (TeMP) [Type-aware embeddings for multi-hop reasoning over knowledge graphs](https://arxiv.org/abs/2205.00782), arxiv 2022\n  2. (GNN-QE) [Neural-Symbolic Models for Logical Queries on Knowledge Graphs](https://arxiv.org/abs/2205.10128), ICML 2022\n  3. (GNNQ) [GNNQ: A Neuro-Symbolic Approach for Query Answering over Incomplete Knowledge Graphs](https://scholar.archive.org/work/sswyx3mbc5danaxosfnyr45cw4/access/wayback/https://s3-eu-west-1.amazonaws.com/pstorage-oxford-0703704596/36647646/GNNQ_full.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIA3OGA3B5WHKHW6LWV/20220814/eu-west-1/s3/aws4_request\u0026X-Amz-Date=20220814T213541Z\u0026X-Amz-Expires=10\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=1bf627735fbce377c2478f514fbac1dfcf49423980d372cc71e2bed732d6ae2c), ISWC 2022\n  4. (NodePiece-QE) [Inductive Logical Query Answering in Knowledge Graphs](https://arxiv.org/abs/2210.08008), NeurIPS 2022\n\n\u003c/details\u003e\n\n\n### Modeling | Processor\n\n\u003cdetails\u003e\n  \u003csummary\u003eAny Processor (2) \u003c/summary\u003e\n\n  1. [TeMP](https://arxiv.org/abs/2205.00782), arxiv 2022 \n  2. [NodePiece-QE](https://arxiv.org/abs/2210.08008), NeurIPS 2022\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eEnd-to-end Neural (15) \u003c/summary\u003e\n\n  1. [GQE](https://proceedings.neurips.cc/paper/2018/file/ef50c335cca9f340bde656363ebd02fd-Paper.pdf), NeurIPS 2018  \n  2. [GQE+hashing](https://ieeexplore.ieee.org/abstract/document/8970688/?casa_token=gLHFmr7V2ekAAAAA:wwDbUufdkwHTQo68pansuhJsJ2XQAF0P21_mQSu75KVRUkgqARmXBs_VEmFOkFgz_Lq-FXP8OA), ICDM 2019\n  3. [CGA](https://arxiv.org/pdf/1910.00084.pdf), K-CAP 2019\n  4. [MPQE](https://grlplus.github.io/papers/26.pdf), ICML 2020 Workshop\n  5. [BiQE](https://ojs.aaai.org/index.php/AAAI/article/view/16630), AAAI 2021\n  6. [MLPMix](https://openreview.net/forum?id=tgcAoUVHRIB), ICLR 2022\n  7. [StarQE](https://arxiv.org/abs/2106.08166), ICLR 2022\n  8. [kgTransformer](https://keg.cs.tsinghua.edu.cn/jietang/publications/KDD22-Liu-et-al-KG-Transformer.pdf), KDD 2022\n  9. [Query2Particles](https://arxiv.org/abs/2204.12847), NAACL 2022\n  10. [SMORE](https://arxiv.org/abs/2110.14890), KDD 2022\n  11. [GNNQ](https://scholar.archive.org/work/sswyx3mbc5danaxosfnyr45cw4/access/wayback/https://s3-eu-west-1.amazonaws.com/pstorage-oxford-0703704596/36647646/GNNQ_full.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIA3OGA3B5WHKHW6LWV/20220814/eu-west-1/s3/aws4_request\u0026X-Amz-Date=20220814T213541Z\u0026X-Amz-Expires=10\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=1bf627735fbce377c2478f514fbac1dfcf49423980d372cc71e2bed732d6ae2c), ISWC 2022\n  12. [SignalE](https://link.springer.com/content/pdf/10.1007/978-3-031-10983-6_20.pdf?pdf=inline%20link), KSEM 2022\n  13. [LMPNN](https://openreview.net/pdf?id=SoyOsp7i_l) ICLR 2023\n  14. [SQE](https://arxiv.org/pdf/2302.13114.pdf), TMLR 2023\n  15. [WFRE](https://arxiv.org/pdf/2305.04034.pdf), ACL 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eNeuro-Symbolic | Geometric (8) \u003c/summary\u003e\n\n  1. [Query2Box](https://openreview.net/pdf?id=BJgr4kSFDS), ICLR 2020\n  2. [Neural-symbolic Approach for Ontology-mediated Query Answering](https://arxiv.org/pdf/2106.14052.pdf), arxiv 2021\n  3. [Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding](https://arxiv.org/pdf/2110.03789.pdf), arxiv 2021\n  4. [RotatE-Box](https://arxiv.org/abs/2005.00480), AKBC 2021\n  5. [NewLook](http://tonghanghang.org/pdfs/kdd21_newlook.pdf), KDD 2021\n  6. [HypE](https://dl.acm.org/doi/abs/10.1145/3442381.3449974?casa_token=tQx20rBUtMwAAAAA:GRqp4JBze6ybzZzeSdCc9cNGlqN0wkAP0BVBPctAgtUVviVSoNiUQcNxBbEuGVGZxnCzJyLqfMmiYg), WWW 2021\n  7. [ConE](https://proceedings.neurips.cc/paper/2021/hash/a0160709701140704575d499c997b6ca-Abstract.html), NeurIPS 2021\n  8. [Query2Geom](https://link.springer.com/chapter/10.1007/978-3-031-26438-2_6), AICS 2022\n   \n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eNeuro-Symbolic | Probabilistic (5) \u003c/summary\u003e\n\n  1. [BetaE](https://proceedings.neurips.cc/paper/2020/file/e43739bba7cdb577e9e3e4e42447f5a5-Paper.pdf), NeurIPS 2020\n  2. [PERM](https://proceedings.neurips.cc/paper/2021/hash/c4d2ce3f3ebb5393a77c33c0cd95dc93-Abstract.html), NeurIPS 2021\n  3. [LinE](https://dl.acm.org/doi/pdf/10.1145/3534678.3539338?casa_token=_jPlNJj2TlYAAAAA:pKAA42_lrZ2JIHc1YZV0fchIlRiIcqCy8oCBL2UU3Gm84MOeTSYLfQn31DKtXBbU2yqzC7LUsYvREBw), KDD 2022\n  4. [GammaE](https://arxiv.org/pdf/2210.15578.pdf), EMNLP 2022\n  5. [NMP-QEM](https://aclanthology.org/2022.emnlp-main.194.pdf), EMNLP 2022\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eNeuro-Symbolic | Fuzzy Logic (16) \u003c/summary\u003e\n\n  1. [EmQL](https://proceedings.neurips.cc/paper/2020/hash/fe74074593f21197b7b7be3c08678616-Abstract.html), NeurIPS 2020\n  2. [TractOR](https://arxiv.org/pdf/2002.10029.pdf), UAI 2020   \n  3. [CQD](https://openreview.net/forum?id=Mos9F9kDwkz), ICLR 2021\n  4. [LogicE](https://arxiv.org/pdf/2103.00418.pdf), arxiv 2021\n  5. [FuzzQE](https://ojs.aaai.org/index.php/AAAI/article/view/20310), AAAI 2022\n  6. [TAR](https://arxiv.org/abs/2205.14591), arxiv 2022\n  7. [FLEX](https://arxiv.org/abs/2205.11039), arxiv 2022\n  8. [TFLEX](https://arxiv.org/pdf/2205.14307.pdf), arxiv 2022\n  9. [GNN-QE](https://arxiv.org/abs/2205.10128), ICML 2022\n  10. [ENeSy](https://arxiv.org/pdf/2209.08779.pdf) NeurIPS 2022\n  11. [QTO](https://arxiv.org/pdf/2212.09567.pdf), arxiv 2023\n  12. [NQE](https://arxiv.org/pdf/2211.13469.pdf), AAAI 2023\n  13. [Var2Vec](https://ora.ox.ac.uk/objects/uuid:27c9b239-20d0-449f-b425-b5065eb128fe/download_file?file_format=application%2Fpdf\u0026safe_filename=Wang_et_al_2022_Efficient_Embeddings_ofAM.pdf\u0026type_of_work=Conference+item), AAAI 2023\n  14. [CQD A](https://arxiv.org/pdf/2301.12313.pdf), arxiv 2023\n  15. [FIT](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023\n  16. [WFRE](https://arxiv.org/pdf/2305.04034.pdf), ACL 2023\n\n\u003c/details\u003e\n\n### Modeling | Decoder\n\n\u003cdetails\u003e\n  \u003csummary\u003eNon-Parametric (all) \u003c/summary\u003e\n\n  All existing models up to March 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eParametric (0) \u003c/summary\u003e\n\n  1. None as of March 2023\n\n\u003c/details\u003e\n\n### Queries | Query Operators\n\nProgressive scale of supported operators. That is, all models listed under the \"NOT\" category also support JOIN and UNION.\n\n\u003cdetails\u003e\n  \u003csummary\u003ePROJECTION + JOIN (intersection) (10) \u003c/summary\u003e\n\n  1. [GQE](https://proceedings.neurips.cc/paper/2018/file/ef50c335cca9f340bde656363ebd02fd-Paper.pdf), NeurIPS 2018  \n  2. [GQE+hashing](https://ieeexplore.ieee.org/abstract/document/8970688/?casa_token=gLHFmr7V2ekAAAAA:wwDbUufdkwHTQo68pansuhJsJ2XQAF0P21_mQSu75KVRUkgqARmXBs_VEmFOkFgz_Lq-FXP8OA), ICDM 2019\n  3. [CGA](https://arxiv.org/pdf/1910.00084.pdf), K-CAP 2019 \n  4. [TractOR](https://arxiv.org/pdf/2002.10029.pdf), UAI 2020\n  5. [MPQE](https://grlplus.github.io/papers/26.pdf), ICML 2020 Workshop\n  6. [Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding](https://arxiv.org/pdf/2110.03789.pdf), arxiv 2021\n  7. [BiQE](https://ojs.aaai.org/index.php/AAAI/article/view/16630), AAAI 2021\n  8. [StarQE](https://arxiv.org/abs/2106.08166), ICLR 2022\n  9. [SMORE](https://arxiv.org/abs/2110.14890), KDD 2022\n  10. [GNNQ](https://scholar.archive.org/work/sswyx3mbc5danaxosfnyr45cw4/access/wayback/https://s3-eu-west-1.amazonaws.com/pstorage-oxford-0703704596/36647646/GNNQ_full.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIA3OGA3B5WHKHW6LWV/20220814/eu-west-1/s3/aws4_request\u0026X-Amz-Date=20220814T213541Z\u0026X-Amz-Expires=10\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=1bf627735fbce377c2478f514fbac1dfcf49423980d372cc71e2bed732d6ae2c), ISWC 2022\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e + UNION (9) \u003c/summary\u003e\n\n  1. [Query2Box](https://openreview.net/pdf?id=BJgr4kSFDS), ICLR 2020\n  2. [EmQL](https://proceedings.neurips.cc/paper/2020/hash/fe74074593f21197b7b7be3c08678616-Abstract.html), NeurIPS 2020\n  3. [HypE](https://dl.acm.org/doi/abs/10.1145/3442381.3449974?casa_token=tQx20rBUtMwAAAAA:GRqp4JBze6ybzZzeSdCc9cNGlqN0wkAP0BVBPctAgtUVviVSoNiUQcNxBbEuGVGZxnCzJyLqfMmiYg), WWW 2021\n  4. [NewLook](http://tonghanghang.org/pdfs/kdd21_newlook.pdf), KDD 2021\n  5. [PERM](https://proceedings.neurips.cc/paper/2021/hash/c4d2ce3f3ebb5393a77c33c0cd95dc93-Abstract.html), NeurIPS 2021\n  6. [CQD](https://openreview.net/forum?id=Mos9F9kDwkz) ICLR’21\n  7. [Neural-symbolic Approach for Ontology-mediated Query Answering](https://arxiv.org/pdf/2106.14052.pdf), arxiv 2021\n  8. [kgTransformer](https://keg.cs.tsinghua.edu.cn/jietang/publications/KDD22-Liu-et-al-KG-Transformer.pdf), KDD 2022\n  9. [Query2Geom](https://link.springer.com/chapter/10.1007/978-3-031-26438-2_6), AICS 2022\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e + NOT (negation) (23) \u003c/summary\u003e\n\n  1. [BetaE](https://proceedings.neurips.cc/paper/2020/file/e43739bba7cdb577e9e3e4e42447f5a5-Paper.pdf), NeurIPS 2020\n  2. [ConE](https://proceedings.neurips.cc/paper/2021/hash/a0160709701140704575d499c997b6ca-Abstract.html), NeurIPS 2021\n  3. [LogicE](https://arxiv.org/pdf/2103.00418.pdf), arxiv 2021\n  4. [MLPMix](https://openreview.net/forum?id=tgcAoUVHRIB), ICLR 2022\n  5. [FuzzQE](https://ojs.aaai.org/index.php/AAAI/article/view/20310), AAAI 2022\n  6. [GNN-QE](https://arxiv.org/abs/2205.10128), ICML 2022\n  7. [LinE](https://dl.acm.org/doi/pdf/10.1145/3534678.3539338?casa_token=_jPlNJj2TlYAAAAA:pKAA42_lrZ2JIHc1YZV0fchIlRiIcqCy8oCBL2UU3Gm84MOeTSYLfQn31DKtXBbU2yqzC7LUsYvREBw), KDD 2022\n  8. [Query2Particles](https://arxiv.org/abs/2204.12847), NAACL 2022\n  9. [GammaE](https://arxiv.org/pdf/2210.15578.pdf), EMNLP 2022\n  10. [NMP-QEM](https://aclanthology.org/2022.emnlp-main.194.pdf), EMNLP 2022\n  11. [TAR](https://arxiv.org/abs/2205.14591), arxiv 2022\n  12. [FLEX](https://arxiv.org/abs/2205.11039), arxiv 2022\n  13. [TFLEX](https://arxiv.org/pdf/2205.14307.pdf), arxiv 2022\n  14. [ENeSy](https://arxiv.org/pdf/2209.08779.pdf), NeurIPS 2022\n  15. [SignalE](https://link.springer.com/content/pdf/10.1007/978-3-031-10983-6_20.pdf?pdf=inline%20link), KSEM 2022\n  16. [QTO](https://arxiv.org/pdf/2212.09567.pdf), arxiv 2023\n  17. [LMPNN](https://openreview.net/pdf?id=SoyOsp7i_l), ICLR 2023\n  18. [NQE](https://arxiv.org/pdf/2211.13469.pdf), AAAI 2023\n  19. [Var2Vec](https://ora.ox.ac.uk/objects/uuid:27c9b239-20d0-449f-b425-b5065eb128fe/download_file?file_format=application%2Fpdf\u0026safe_filename=Wang_et_al_2022_Efficient_Embeddings_ofAM.pdf\u0026type_of_work=Conference+item), AAAI 2023\n  20. [CQD A](https://arxiv.org/pdf/2301.12313.pdf), arxiv 2023\n  21. [SQE](https://arxiv.org/pdf/2302.13114.pdf), TMLR 2023\n  22. [FIT](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023  \n  23. [WFRE](https://arxiv.org/pdf/2305.04034.pdf), ACL 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Kleene Plus (1) \u003c/summary\u003e\n\n  1. [RotatE-Box](https://arxiv.org/abs/2005.00480), AKBC 2021\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e FILTER (0) \u003c/summary\u003e\n\n  0. None as of March 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e AGGREGATIONS (GROUP BY, ORDER BY, etc) (0) \u003c/summary\u003e\n\n  0. None as of March 2023\n\n\u003c/details\u003e\n\n### Queries | Query Patterns\n\n\u003cdetails\u003e\n  \u003csummary\u003eTree-structured (47) \u003c/summary\u003e\n\n  1. All existing processors as of March 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eArbitrary DAGs (1) \u003c/summary\u003e\n\n  1. [FIT](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eCyclic Queries (1) \u003c/summary\u003e\n\n  1. [FIT](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023\n\n\u003c/details\u003e\n\n\n### Queries | Projected Variables\n\n\u003cdetails\u003e\n  \u003csummary\u003e Zero Projected Vars (ASK queries) (0) \u003c/summary\u003e\n\n  0. None as of March 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e One Projected Variable (all) \u003c/summary\u003e\n\n  1. All processors as of March 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Multiple Projected Variables (1) \u003c/summary\u003e\n \n  1. (EFOk-CQA) [EFOk-CQA: Towards Knowledge Graph Complex Query Answering beyond Set Operation](https://arxiv.org/abs/2307.13701) arxiv 2023\n\n\u003c/details\u003e\n\n\n\n### Metrics\n\n- Original metrics: ROC AUC and Average Percentile Rank over 1000 negative samples. \n  - Proposed by original [GQE](https://proceedings.neurips.cc/paper/2018/file/ef50c335cca9f340bde656363ebd02fd-Paper.pdf) (NeurIPS 2018), used by [GQE+hashing](https://ieeexplore.ieee.org/abstract/document/8970688/?casa_token=gLHFmr7V2ekAAAAA:wwDbUufdkwHTQo68pansuhJsJ2XQAF0P21_mQSu75KVRUkgqARmXBs_VEmFOkFgz_Lq-FXP8OA), [CGA](https://arxiv.org/pdf/1910.00084.pdf), and [TractOR](https://arxiv.org/pdf/2002.10029.pdf). **Not used after**.\n- Generalization: predicting hard answers (MRR / Hits@k).\n  - Introduced by [Query2Box](https://openreview.net/pdf?id=BJgr4kSFDS) (ICLR 2020). **Standard metric**.\n- Generalization: from ranking to binary classification\n  - Proposed in [Approximate knowledge graph query answering: from ranking to binary classification](https://arxiv.org/abs/2102.11389)\n- Entailment: faithfulness - ability to recover easy answers (no link prediction) (MRR / Hits@k)\n  - Proposed by [EmQL](https://proceedings.neurips.cc/paper/2020/hash/fe74074593f21197b7b7be3c08678616-Abstract.html) (NeurIPS 2020)\n- Estimating the cardinality of answer set size (Spearman's rank correlation, MAPE)\n  - Used in [GNN-QE](https://arxiv.org/abs/2205.10128), [QTO](https://arxiv.org/pdf/2212.09567.pdf)\n- Predicting easy answers before hard answers (ROC-AUC)\n  - Used in [NodePiece-QE](https://arxiv.org/abs/2210.08008)\n- Multiple variable queries, (multiply / marginal / joint) x (MRR/ HITs@k)\n  - Proposed in [EFOk-CQA](https://arxiv.org/abs/2307.13701) arxiv 2023\n\n\n## 📈 Datasets and Benchmarking\n\n### Inference (datasets)\n\n\u003cdetails\u003e\n  \u003csummary\u003eTransductive datasets (15) \u003c/summary\u003e\n\n  1. (GQE datasets) [GQE](https://proceedings.neurips.cc/paper/2018/file/ef50c335cca9f340bde656363ebd02fd-Paper.pdf), NeurIPS 2018\n  2. (Query2Box datasets) [Query2Box](https://openreview.net/pdf?id=BJgr4kSFDS), ICLR 2020  \n  3. (BetaE datasets) [BetaE](https://proceedings.neurips.cc/paper/2020/file/e43739bba7cdb577e9e3e4e42447f5a5-Paper.pdf), NeurIPS 2020\n  4. (Regex datasets) [Regex Queries](https://arxiv.org/abs/2005.00480), AKBC 2021\n  5. (BiQE dataset) [BiQE](https://ojs.aaai.org/index.php/AAAI/article/view/16630), AAAI 2021\n  6. (Query2Onto datasets) [Neural-symbolic Approach for Ontology-mediated Query Answering](https://arxiv.org/pdf/2106.14052.pdf), arxiv 2021\n  7. (EFO-1 dataset) [Benchmarking the Combinatorial Generalizability of Complex Query Answering on Knowledge Graphs](https://arxiv.org/abs/2109.08925), NeurIPS 2021 (Datasets and Benchmarks)\n  8. (SMORE datasets) [SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs](https://arxiv.org/abs/2110.14890), KDD 2022\n  9. (StarQE dataset) [Query Embedding on Hyper-relational Knowledge Graphs](https://arxiv.org/abs/2106.08166) ICLR 2022,\n  10. (TFLEX dataset) [TFLEX: Temporal Feature-Logic Embedding Framework for Complex Reasoning over Temporal Knowledge Graph](https://arxiv.org/pdf/2205.14307.pdf), arxiv 2022\n  11. (WD50K-NFOL dataset) [NQE: N-ary Query Embedding for Complex Query Answering over Hyper-relational Knowledge Graphs](https://arxiv.org/pdf/2211.13469.pdf), AAAI 2023\n  12. (SQE dataset) [Sequential Query Encoding For Complex Query Answering on Knowledge Graphs](https://arxiv.org/pdf/2302.13114.pdf)\n  13. (Real EFO-1) [On Existential First Order Queries Inference on Knowledge Graphs](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023\n  14. (Numerical CQA dataset) [Knowledge Graph Reasoning over Entities and Numerical Values](https://arxiv.org/abs/2306.01399) KDD 2023\n  15. (EFOk-CQA) [EFOk-CQA: Towards Knowledge Graph Complex Query Answering beyond Set Operation](https://arxiv.org/abs/2307.13701) arxiv 2023\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eInductive datasets (3) \u003c/summary\u003e\n\n  1. (TeMP datasets) [Type-aware embeddings for multi-hop reasoning over knowledge graphs](https://arxiv.org/abs/2205.00782), arxiv 2022\n  2. (InductiveQE datasets) [Inductive Logical Query Answering in Knowledge Graphs](https://arxiv.org/abs/2210.08008) NeurIPS 2022\n  3. (GNNQ dataset) [GNNQ: A Neuro-Symbolic Approach for Query Answering over Incomplete Knowledge Graphs](https://scholar.archive.org/work/sswyx3mbc5danaxosfnyr45cw4/access/wayback/https://s3-eu-west-1.amazonaws.com/pstorage-oxford-0703704596/36647646/GNNQ_full.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIA3OGA3B5WHKHW6LWV/20220814/eu-west-1/s3/aws4_request\u0026X-Amz-Date=20220814T213541Z\u0026X-Amz-Expires=10\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=1bf627735fbce377c2478f514fbac1dfcf49423980d372cc71e2bed732d6ae2c) ISWC 2022\n\n\u003c/details\u003e\n\n### GQE Datasets\n\nAre Bio and Reddit available at all? Introduced in GQE, used in 4 papers overall (GQE, GQE+hashing, CGA, TractOR).\n\n### BetaE Datasets\n\nThe main difference with Query2Box datasets: queries in the BetaE datasets have less than 100 answers. Has queries with negation.\n\nIntroduced in [Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs](https://proceedings.neurips.cc/paper/2020/file/e43739bba7cdb577e9e3e4e42447f5a5-Paper.pdf), NeurIPS 2020\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n\n  | Dataset | Entities | Relations | Training Edges | Validation Edges | Test Edges | Total Edges\n  |-----|----:|----:|----:|----:|----:|----:|\n  | FB15k    | 14,951 | 1,345 | 483,142 | 50,000 | 59,071 | 592,213 |\n  | FB15k237 | 14,505 | 237   | 272,115 | 17,526 | 20,438 | 310,079 |\n  | NELL995  | 63,361 | 200   | 114,213 | 14,324 | 14,267 | 142,804 | \n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n\n  | Queries | Training | Training |  Validation | Validation |  Test | Test |\n  |---------|---------:|-----------:|-----:|---------:|-----------:|-----:|\n  | Dataset | 1p/2p/3p/2i/3i  | 2in/3in/inp/pin/pni  | 1p | others | 1p | others |\n  | FB15k    | 273,710 | 27,371 | 59,097 | 8,000 | 67,016 | 8,000 |\n  | FB15k237 | 149,689 | 14,968 | 20,101 | 5,000 | 22,812 | 5,000 |\n  | NELL995  | 107,982 | 10,798 | 16,927 | 4,000 | 17,034 | 4,000 | \n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Average Number of Answers \u003c/summary\u003e\n\n  | Dataset | 1p | 2p |  3p | 2i |  3i | ip | pi | 2u | up | 2in | 3in | inp | pin | pni |\n  |---------|---:|---:|----:|---:|----:|---:|---:|---:|---:|---:|----:|---:|---:|---:|\n  | FB15k   | 1.7 | 19.6 | 24.4 | 8.0 | 5.2 | 18.3 | 12.5 | 18.9 | 23.8 | 15.9 | 14.6 | 19.8 | 21.6 | 16.9 |\n  | FB15k237 | 1.7 | 17.3 | 24.3 | 6.9 | 4.5 | 17.7 | 10.4 | 19.6 | 24.3 | 16.3 | 13.4 | 19.5 | 21.7 | 18.2 |\n  | NELL995 | 1.6 | 14.9 | 17.5 | 5.7 | 6.0 | 17.4 | 11.9 | 14.9 | 19.0 | 12.9 | 11.1 | 12.9 | 16.0 | 13.0 | \n  \n\u003c/details\u003e\n\n### Query2Box Datasets\n\nIntroduced in [Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box Embeddings](https://openreview.net/pdf?id=BJgr4kSFDS), ICLR 2020. \n\nEPFO queries are considered **easier** than BetaE datasets. Doesn't have queries with negations.\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n\n  | Dataset | Entities | Relations | Training Edges | Validation Edges | Test Edges | Total Edges\n  |-----|----:|----:|----:|----:|----:|----:|\n  | FB15k    | 14,951 | 1,345 | 483,142 | 50,000 | 59,071 | 592,213 |\n  | FB15k237 | 14,505 | 237   | 272,115 | 17,526 | 20,438 | 310,079 |\n  | NELL995  | 63,361 | 200   | 114,213 | 14,324 | 14,267 | 142,804 | \n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n\n  | Queries | Training | Training |  Validation | Validation |  Test | Test |\n  |---------|---------:|-----------:|-----:|---------:|-----------:|-----:|\n  | Dataset | 1p  | others  | 1p | others | 1p | others |\n  | FB15k    | 273,710 | 273,710 | 59,097 | 8,000 | 67,016 | 8,000 |\n  | FB15k237 | 149,689 | 149,689 | 20,101 | 5,000 | 22,812 | 5,000 |\n  | NELL995  | 107,982 | 107,982 | 16,927 | 4,000 | 17,034 | 4,000 | \n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Average Number of Answers \u003c/summary\u003e\n\n  | Dataset | 1p | 2p |  3p | 2i |  3i | ip | pi | 2u | up | \n  |---------|---:|---:|----:|---:|----:|---:|---:|---:|---:|\n  | FB15k   | 10.8 | 255.6 | 250.0 | 90.3 | 64.1 | 593.8 | 190.1 | 27.8 | 227.0 | \n  | FB15k237 | 13.3 | 131.4 | 215.3 | 69.0 | 48.9 | 593.8 | 257.7 | 35.6 | 127.7 | \n  | NELL995 | 8.5 | 56.6 | 65.3 | 30.3 | 15.9 | 310.0 | 144.9 | 14.4 | 62.5 | \n  \n\u003c/details\u003e\n\n### CGA Datasets\n\nGQE-like patterns mined on subsets of DBpedia and Wikidata. The datasets are DB18 and WikiGeo19, introduced in [CGA](https://arxiv.org/pdf/1910.00084.pdf), K-CAP 2019.\n\n**As of Sept 2022: not available**.\n\n### Regex Queries\n\nQueries emulating property paths in SPARQL with variable length of relation paths (up to length 5). Queries are EPFO queries, i.e., no negation.\nNew operators over relations resemble those from SPARQL:\n* $r_1 / r_2 / \\dots$ - relational path, aka classic projection queries\n* $r_1 \\lor r_2$ - a union of decomposed patterns $(e, r_1, ?) \\lor (e, r_2, ?)$\n* Kleene plus $r^{+}$ - one or more occurence of relation $r$, eg, $r_1/r_2^{+}$ corresponds to $r_1 / r_2$, $r_1 / r_2 / r_2$, $r_1 / r_2 / r_2 / r_2 / \\dots$ up to some final depth. Those *can* be cyclic patterns. \n\nTwo datasets:\n\n* FB15k-Regex is based on Freebase, queries have less than 50 answers, 21 query types\n* Wiki100-Regex is based on query logs from the official Wikidata SPARQL endpoint, 5 query types. \n\nIntroduced in [RotatE-Box](https://arxiv.org/abs/2005.00480), AKBC 2021.\n\nRepo: [GitHub](https://github.com/dair-iitd/kbi-regex) - no actual data dumps are present :(\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n\n  | Dataset | Entities | Relations | Training Edges | Validation Edges | Test Edges | Total Edges\n  |-----|----:|----:|----:|----:|----:|----:|\n  | FB15k    | 14,951 | 1,345 | 483,142 | 50,000 | 59,071 | 592,213 |\n  | Wiki100 | 41,291 | 100   | 389,795 | 21,655 | 21,656 | 433,106 |\n\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n\n  #### FB15k-Regex\n\n  | Query type | Train | Valid | Test |\n  |-----------:|------:|------:|-----:|\n  |  $(e_1, r_1^+, ?)$    | 24,476 | 4,614 | 8,405 |\n  |  $(e_1, r_1/r_2, ?)$  | 25,378 | 4,927  | 8,844 |\n  | $(e_1, r_1^+/r_2^+, ?)$  | 26,391  | 4,978  | 9,028 |\n  | $(e_1, r_1^+/r_2^+/r_3^+, ?)$ | 25,470  | 4,878 | 8,816 |\n  | $(e_1, r_1/r_2^+, ?)$  | 26,335 | 5,007 | 9,062 |\n  | $(e_1, r_1^+/r_2, ?)$  | 27,614 | 5,229 | 9,429 |\n  | $(e_1, r_1^+/r_2^+/r_3, ?)$ | 27,865 | 5,283 | 9,509 |\n  | $(e_1, r_1^+/r_2/r_3^+, ?)$ | 26,366 | 5,058 | 9,159 |\n  | $(e_1, r_1/r_2^+/r_3^+, ?)$ | 26,366 | 5,045 | 9,099 |\n  | $(e_1, r_1/r_2/r_3^+, ?)$   | 26,703 | 5,155 | 9,313 |\n  | $(e_1, r_1/r_2^+/r_3, ?)$   | 28,005 | 5,380 | 9,688 |\n  | $(e_1, r_1^+/r_2/r_3, ?)$   | 27,884 | 5,338 | 9,632 |\n  | $(e_1, r_1\\lor r_2, ?)$ | 30,080 | 5,828 | 9,664 |\n  | $(e_1, (r_1\\lor r_2)/r_3, ?)$ | 31,559 | 6,606 | 10,974 |\n  | $(e_1, r_1/(r_2\\lor r_3), ?)$ | 41,886 | 7,755 | 13,611 |\n  | $(e_1, r_1^+\\lor r_2^+, ?)$   | 23,109 | 4,469 | 8,367  |\n  | $(e_1, (r_1\\lor r_2)/r_3^+, ?)$ | 27,658 | 5,738 | 9,711 |\n  | $(e_1, (r_1^+\\lor r_2^+)/r_3, ?)$ | 24,462 | 4,865 | 8,863 |\n  | $(e_1, r_1^+/(r_2\\lor r_3), ?)$ | 27,676 | 5,340 | 9,267 |\n  | $(e_1, r_1/(r_2^+\\lor r_3^+), ?)$ | 28,542 | 5,475 | 9,436 |\n  | $(e_1, (r_1\\lor r_2)^+, ?)$  | 26,260 | 5,523 | 10,360 |\n  |  **Total**                   | 580,085 | 112,491 | 200,237 |\n\n\n  #### Wiki100-Regex\n\n  | Query type | Train | Valid | Test |\n  |-----------:|------:|------:|-----:|\n  |  $(e_1, r_1^+, ?)$    | 490,562 | 24,878 | 23,443 |\n  | $(e_1, r_1^+/r_2^+, ?)$  | 6,945  | 620  | 772 |\n  | $(e_1, r_1/r_2^+, ?)$  | 85,253 | 10,013 | 8,377 |\n  | $(e_1, r_1\\lor r_2, ?)$ | 274,012 | 14,900 | 14,915 |\n  | $(e_1, (r_1\\lor r_2)^+, ?)$  | 348,274 | 15,720 | 15,311 |\n  |  **Total**                   | 1,205,046 | 66,131 | 62,818 |\n\n\u003c/details\u003e\n\n### DAG Queries\n\nConjunctive queries (w/o union) not limited to 9 patterns from Query2Box/BetaE datasets. The task is to predict *all* intermediate entities, not just final leaf nodes. Query depth: 2-5; max 3 intersecting branches.\n\nIntroduced in [Answering complex queries in knowledge graphs with bidirectional sequence encoders](https://ojs.aaai.org/index.php/AAAI/article/view/16630), AAAI 2021.\n\nNew FB15K-237-CQ and WN18RR-CQ datasets have two variations:\n* CQ (conjunctive queries) - Training on triples + paths + DAGs, Validation/Test on **DAGs only**;\n* Paths - Training on triples + paths, Validation/Test on **paths only**\n\n**Sept 2022: the datasets are not publicly available.**\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n  \n  | Dataset | FB15K-237-CQ | FB15K-237-CQ | FB15K-237-CQ | WN18RR-CQ | WN18RR-CQ | WN18RR-CQ |\n  |-----|----:|----:|----:|----:|----:|----:|\n  | Dataset | Train | Validation | Test | Train | Validation | Test |\n  | Entities | 14,505 | - | - | 40,943 | - | - |\n  | Relations | 237 | 237 | 237 | 11 | 11 | 11 |\n  | Triples  | 272,115 | - | - | 86,835 | - | - |\n  | Paths | 50,000 | - | - | 10,000 | - | - |\n  | DAGs | 48,865 | 2,785 | 2,599 | 9,465 | 112 | 95 |\n  | Avg Masks | 1.86 | 5.91 | 6.05 | 1.84 | 5.13 | 4.91 |\n  | Avg Query Len (Tokens) | 152 | 460 | 479 | 71 | 198 | 199 |   \n\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n  \n  No detailed breakdown by query type is available, only the DAGs stats from the main table.\n\n  | Dataset | FB15K-237-CQ | FB15K-237-CQ | FB15K-237-CQ | WN18RR-CQ | WN18RR-CQ | WN18RR-CQ |\n  |-----|----:|----:|----:|----:|----:|----:|\n  | Dataset | Train | Validation | Test | Train | Validation | Test |\n  | Paths | 50,000 | - | - | 10,000 | - | - |\n  | DAGs | 48,865 | 2,785 | 2,599 | 9,465 | 112 | 95 |\n  | Avg Masks | 1.86 | 5.91 | 6.05 | 1.84 | 5.13 | 4.91 |\n  | Avg Query Len (Tokens) | 152 | 460 | 479 | 71 | 198 | 199 | \n\n\u003c/details\u003e\n\n###  EFO-1 Queries\n\nExistential First-Order queries  with Single Free Variable, extended from BetaE. The goal is to evaluate the combinatorial generalizability. \n\nIntroduced in [Benchmarking the Combinatorial Generalizability of Complex Query Answering on Knowledge Graphs](https://arxiv.org/abs/2109.08925), NeurIPS 2021 (Datasets and Benchmarks)\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n\n  | Queries | Training | Training |  Validation | Validation |  Test | Test |\n  |---------|---------:|-----------:|-----:|---------:|-----------:|-----:|\n  | Dataset | 1p/2p/3p/2i/3i  | 2in/3in/inp/pin/pni  | 1p | others | 1p | others |\n  | FB15k    | 273,710 | 27,371 | 59,097 | 8,000 | 67,016 | 8,000 |\n  | FB15k237 | 149,689 | 14,968 | 20,101 | 5,000 | 22,812 | 5,000 |\n  | NELL995  | 107,982 | 10,798 | 16,927 | 4,000 | 17,034 | 4,000 | \n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n\nCannot list all the 301 query types. Details can be found in a summarization excel file [here](https://docs.google.com/spreadsheets/d/18RJL1puHpbGZJKLkffqKEbzRmRFdSnGFQvekApFWc3Q/edit?usp=sharing).\n\n\u003c/details\u003e\n\n### Real EFO-1 dataset\n\nRethinking the EFO-1 formulation by introducing leaf nodes, multi edge, and cycle.\nFor standard FB15k, FB15k-237, and NELL - 9 new query types (10 with reworked `pni` type) including:\n-  `l` - queries with existentially quantified variables as leaf nodes (2il, 3il)\n-  `m` - queries with multiple relation projection edges from one variable to another (2m, 2nm, 3mp, 3pm, im)\n-  `c` - queries with cycles (3c, cm)\n\nAll new query have 5000 instances in three KGs. Introduced in [On Existential First Order Queries Inference on Knowledge Graphs](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023. The dataset can be downloaded from [here](https://drive.google.com/drive/folders/17bPr6_ESqh5D0LgWNgpE4mY8gpg2iC5o?usp=sharing).\n\n#### EPFO queries with Literals\n\nBased on a variation of the FB15k-237 dataset with entity attributes (12,390 entities, 237 relations, 115 attributes, 29,229 (?) triples).\nLiterals are restricted to numerical values, three additional filter functions (less than, equal, greater then).\n\nThe dataset includes standard 9 EPFO query types and adds 8 more variations of those patterns enriched with literals:\n- 5 query types where literals are in queries, but the answer is an entity (ai, 2ai, pai, aip, au)\n- 3 query types where literals are in queries, and the answer is a mean of relevant literal values (1ap, 2ap, 3ap)\n\nIntroduced in [LitCQD: Multi-Hop Reasoning in Incomplete Knowledge Graphs with Numeric Literals](https://arxiv.org/pdf/2304.14742.pdf), arxiv 2023\n\n###  SQE Queries\n\nExistential First-Order queries aimed at evaluating compositional generalization to OOD query patterns (29 in-distribution types, 29 out-of-distribution). In contrast to BetaE datasets, does **not** have restrictions on the number of answers per query, long tails are possible.  \n\nIntroduced in [Sequential Query Encoding For Complex Query Answering on Knowledge Graphs](https://openreview.net/pdf?id=ERqGqZzSu5)\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n\n  | Queries | Training | Training |  Validation | Test |\n  |---------|---------:|-----------:|-----:|---------:|\n  | Dataset | 1p | others  | all | all |\n  | FB15k    | 273,710 | 821,130 | 8,000 | 8,000 |\n  | FB15k237 | 149,689 | 449,067 | 5,000 | 5,000 |\n  | NELL995  | 107,982 | 323,946 | 4,000 | 4,000 | \n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n\n58 query types, refer to Appendix A in the paper for the full list of patterns.\n\n\u003c/details\u003e\n\n### EFOk queries\n\nIntroduced by (EFOk-CQA) [EFOk-CQA: Towards Knowledge Graph Complex Query Answering beyond Set Operation](https://arxiv.org/abs/2307.13701) with 741 query types in total.\n\nFeatured by:\n- existential first-order queries with more than multiple variables.\n- combinatorial space with multi-edge and cyclic queries\n\n### Numerical CQA Queries\n\nThe Numerical CQA queries both include entities and typed numerical attribute values. \n\nIntroduced by [Knowledge Graph Reasoning over Entities and Numerical Values](https://arxiv.org/abs/2306.01399)\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n  \n|  Graphs | Data Split |    1p   |    2p   |    2i   |    3i   |    pi   |    ip   |    2u   |    up   |    All    |\n|:-------:|:----------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:---------:|\n|  FB15K  |  Training  | 304,633 | 138,192 | 226,729 | 288,874 | 260,057 | 233,834 | 284,301 | 284,931 | 2,021,551 |\n|         | Validation |  8,271  |  15,860 |  23,359 |  28,836 |  25,081 |  22,930 |  29,187 |  29,210 |  182,734  |\n|         |   Testing  |  7,969  |  15,431 |  23,346 |  28,865 |  24,810 |  22,232 |  29,212 |  29,274 |  181,139  |\n|  DB15K  |  Training  | 124,851 |  99,698 | 140,427 | 190,413 | 171,353 | 163,687 | 190,364 | 194,244 | 1,275,037 |\n|         | Validation |  3,529  |  10,388 |  9,792  |  13,817 |  14,594 |  16,651 |  19,512 |  19,792 |  108,075  |\n|         |   Testing  |  3,387  |  10,047 |  9,914  |  14,603 |  14,642 |  15,897 |  19,504 |  19,773 |  107,767  |\n| YAGO15K |  Training  |  84,014 |  76,238 | 136,282 | 183,850 | 162,712 | 145,994 | 183,963 | 183,459 | 1,156,512 |\n|         | Validation |  2,833  |  7,986  |  10,757 |  16,884 |  13,485 |  13,899 |  18,444 |  19,105 |  103,393  |\n|         |   Testing  |  2,713  |  7,949  |  10,935 |  17,171 |  13,481 |  13,526 |  18,433 |  18,997 |  103,205  |\n\n\u003c/details\u003e\n\n### Type-Aware Datasets\n\nIn addition to a normal graph of entities (instances) a-la BetaE datasets, the type-aware datasets offer an additional set of classes, classes hierarchy (from a pre-existing ontology), and `instanceOf` links between entities and classes.\n\nThose datasets might include an additional task of predicting types of answer entities (Concept Retrieval).\n\n- LUBM, introduced in [Neuro-Symbolic Ontology-Mediated Query Answering](https://openreview.net/pdf?id=wwVb95CkrFm), OpenReview 2021\n- NELL, introduced in [Neuro-Symbolic Ontology-Mediated Query Answering](https://openreview.net/pdf?id=wwVb95CkrFm), OpenReview 2021. The base graph is the same as in the BetaE datasets, but a few ontological axioms were added.\n- YAGO 4, introduced in [TAR: Neural Logical Reasoning across TBox and ABox](https://github.com/lilv98/TAR)\n- DBpedia, introduced in [TAR: Neural Logical Reasoning across TBox and ABox](https://github.com/lilv98/TAR)\n\nLUBM and NELL employ ontological axioms of the DL-Lite (R) family of Description Logics.\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n\n  **TODO** Figure out Concept Retrieval edges in TAR \n  \n  | Dataset | Entities | Relations | Axioms | Base Graph | Materialized Graph | \n  |-----|----:|----:|----:|----:|----:|\n  | LUBM | 55,684 | 28 | 68 | 284k | 565k | \n  | NELL | 63,361 | 400 | 307 | 285k | 497k | \n\n  Axioms breakdown in ontologies for LUBM and NELL\n\n  | Rules | LUBM | NELL | \n  |------:|-----:|-----:|\n  | $\\mathcal{O}$ (Total) |  68  | 307  |\n  | $A \\sqsubseteq A'$ (Subclass) | 13 | - |\n  | $p \\sqsubseteq s$ | 5 | 92 |\n  | $p^{-} \\sqsubseteq s$ | 28 | 215 |\n  | $\\exists p \\sqsubseteq A$ | 11 | - |\n  | $\\exists p^{-} \\sqsubseteq A$ | 11 | - |\n\n\n\n  | Dataset | Entities | Relations | Classes | Training Edges | Validation Edges | Test Edges | Entity-Class Edges | Class Hierarchy Edges | Total Edges |\n  |-----|----:|----:|----:|----:|----:|----:|---:|---:|---:|\n  | YAGO 4 | 32,465 | 75 | 8,382 | 101,417 | 1,000 | 1,000 | 83,291 | 16,644 | 184,708 |\n  | DBpedia | 28,824 | 327 | 981 | 136,821 | 1,000 | 1,000 | 225,436 | 2,582 | 362,257 |\n  \n\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n  \n  | Dataset | Train / Test | 1p | 2p | 3p | 2i | 3i | ip | pi | 2u | up |\n  |--------:|-------------:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n  | LUBM    | Plain (**Train**)| 110,000 | 110,000 | 110,000 | 110,000 | 110,000 | - | - | - | - |\n  | LUBM    | Generalized (**Train**)| 117,124 | 136,731 | 150,653 | 181,234 | 208,710 | - | - | - | - | \n  | LUBM    | Specialized (**Train**)| 117,780 | 154,851 | 173,678 | 271,532 | 230,085 | - | - | - | - | \n  | LUBM    | Ontological (**Train**)| 116,893 | 166,159 | 333,406 | 212,718 | 491,707 | - | - | - | - |\n  | LUBM    | Induction (w/ missing links in queries) (**Val/Test**)| 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | \n  | LUBM    | Deduction (w/o missing link in training) (**Val/Test**)| 1,241 | 4,701 | 6,472 | 3,829 | 4,746 | 7,393 | 7,557 | 4,986 | 7,122 |\n  | LUBM    | Induction + Deduction (**Val/Test**)| 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | 7,986 | 8,000 |\n  | NELL    | Plain (**Train**)| 107,982 | 107,982 | 107,982 | 107,982 | 107,982 | - | - | - | - |\n  | NELL    | Generalized (**Train**)| 174,310 | 408,842 | 864,268 | 398,412 | 930,787 | - | - | - | - | \n  | NELL    | Specialized (**Train**)| 174,310 | 419,664 | 906,609 | 401,954 | 936,537 | - | - | - | - | \n  | NELL    | Ontological (**Train**)| 114,614 | 542,923 | 864,268 | 629,144 | 930,787 | - | - | - | - |\n  | NELL    | Induction (w/ missing links in queries) (**Val/Test**)| 15,688 | 3,910 | 3,918 | 3,828 | 3,786 | 3,932 | 3,895 |3,940 | 3,966 | \n  | NELL    | Deduction (w/o missing link in training) (**Val/Test**)| 346 | 4,461 | 4,294 | 4,842 | 5,996 | 7,295 | 5,862 | 5,646 | 6,894 |\n  | NELL    | Induction + Deduction (**Val/Test**)| 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | 8,000 | 7,990 | 8,000 |\n\n\n  | Queries | Training | Training |  Validation | Validation |  Test | Test |\n  |---------|---------:|-----------:|-----:|---------:|-----------:|-----:|\n  | Dataset | 1p  | others  | 1p | others | 1p | others |\n  | YAGO 4 (Concept Retrieval)    | 189,338 | 10,000 | 1,000 | 1,000 | 1,000 | 1,000 |\n  | YAGO 4 (Entity Only)    | 101,417 | 10,000 | 1,000 | 1,000 | 1,000 | 1,000 |\n  | YAGO 4 (Entity + Instantiations)    | 184,708 | 10,000 | 1,000 | 1,000 | 1,000 | 1,000 |\n  | DBpedia (Concept Retrieval)    | 473,924 | 10,000 | 1,000 | 1,000 | 1,000 | 1,000 |\n  | DBpedia (Entity Only)    | 136,821 | 10,000 | 1,000 | 1,000 | 1,000 | 1,000 |\n  | DBpedia (Entity + Instantiations)    | 362,257 | 10,000 | 1,000 | 1,000 | 1,000 | 1,000 |\n\n\n\n\u003c/details\u003e\n\n### Very Large Datasets\n\nIntroduced in [SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs](https://arxiv.org/abs/2110.14890), KDD 2022.\n\nTraining queries are sampled on-the-fly during training due to the huge size of underlying graphs. \n\nThe underlying graphs are FB400k (400K nodes), WikiKG 2 (2.5M nodes) [(from OGB)](https://ogb.stanford.edu/docs/linkprop/#ogbl-wikikg2), and full Freebase (86M nodes)\nTODO: confirm with Hongyu the number of validation / test queries.\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n\n  | Dataset | Entities | Relations | Training Edges | Validation Edges | Test Edges | Total Edges\n  |-----|----:|----:|----:|----:|----:|----:|\n  | FB400k    | 409,829 | 918 | 1,075,837 | 537,917 | 537,917 | 2,151,671 |\n  | WikiKG2 | 2,500,604 | 535   | 16,109,182 | 429,456 | 598,543 | 17,137,181 |\n  | Freebase  | 86,054,361 | 14,824  | 304,727,650 | 16,929,318 | 16,929,308 | 338,586,276 | \n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n\n  | Queries |  Validation | Validation |  Test | Test |\n  |---------|------------:|-----------:|------:|-----:|\n  | Dataset |  1p | others | 1p | others |\n  | FB400k    | TODO | TODO | TODO | TODO |\n  | WikiKG2 | TODO | TODO | TODO | TODO |\n  | Freebase  | TODO | TODO | TODO | TODO |\n\n\u003c/details\u003e\n\n### Hyper-Relational Datasets\n\nThe main difference of hyper-relational datasets is that edges are no longer plain triples $(h, r, t)$ but *statements* (in terms of Wikidata or RDF-Star) $\\Big(h, r, t, (q\\_{ri}, q\\_{ei})\\_i\\Big)$ with key-value (relation:entity) qualifiers $(q\\_{r}, q\\_{e})$ over the *main* triple. \nFor example, in the statment `(Albert Einstein, educated at, ETH Zurich, (degree, Bachelor))`, the main triple is `Albert Einstein, educated at, ETH Zurich` and its unique qualifier is `(degree, Bachelor)`. \nQualifiers provide an additional context to the edge - the tail node might change with another qualifier, e.g., `(Albert Einstein, educated at, University of Zurich, (degree, Doctorate))`.\n\nEntities and relations in qualifiers are still legit entities and relations which could be present in main triples. Some entities and relations can be found *only* in qualifiers.\n\nThe WD50K dataset has only conjunctive queries (projection + intersection), neither union nor negation.\n\nIntroduced in [Query Embedding on Hyper-Relational Knowledge Graphs](https://openreview.net/pdf?id=4rLw09TgRw9), ICLR 2022\n\nThe WD50K-NFOL dataset introduced in [NQE: N-ary Query Embedding for Complex Query Answering over Hyper-relational Knowledge Graphs](https://arxiv.org/pdf/2211.13469.pdf)adds unions and negations, as well as possibility of variables at qualifier entity positions. **As of Nov 2022, not openly available)\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graph \u003c/summary\u003e\n\n  The original WD50K graph from the [StarE paper](https://aclanthology.org/2020.emnlp-main.596.pdf) by Galkin et al.\n\n\n  | Dataset | Entities | Relations | Qualifier-only Entities | Qualifier-only Relations | Training Edges | Validation Edges | Test Edges | Total Edges\n  |-----|----:|----:|----:|----:|----:|----:|----:|----:|\n  | WD50K    | 47,156 | 532 | 5460 | 45 | 166,435 | 23,913 | 46,159 | 236,508 |\n\n  32,167 edges have at least one key-value (relation:entity) qualifier.\n \n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n\n  | Split | 1p | 2p | 3p | 2i | 3i | ip | pi |\n  |-----|----:|----:|----:|----:|----:|----:|----:|\n  | train | 24,819 | 313,088 | 5,950,990 | 48,513 | 318,735 | 306,022 | 1,088,539 |\n  | validation | 4,100 | 100,706 | 2,968,315 | 15,648 | 169,195 | 169,438 | 569,957 |\n  | test  | 7,716 | 202,045 | 6,433,476 | 38,207 | 547,272 | 445,007 | 1,267,452 |\n\n  WD50K-NFOL stats are not yet available\n\n\u003c/details\u003e\n\n### Inductive Datasets\n\nAs of March 2023, there are no existing purely inductive datasets such that the training and validation/test graphs are different (validation and test containing new unseen entities) and predictions should only rely on the graph structure w/o external data.\n\n#### Type-based Inductive\n\nAs a bridge between shallow transductive models and inductive inference, [Type-aware Embeddings for Multi-Hop Reasoning over Knowledge Graphs](https://arxiv.org/pdf/2205.00782.pdf) propose to mine entity types as the invariant that remains the same at training and inference.\n\nAs a result, the following datasets assume an existing and known in advance class hierarchy (or a graph of classes). Technically, those can be put in the [Type-Aware Datasets](#type-aware-datasets) category. The query datasets only have EPFO queries (no negation).\n\nInductive splits have been published, see the [GitHub issue](https://github.com/zhiweihu1103/QE-TEMP/issues/2)\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n  \n  The underlying graphs are FB15k-237-V2 and NELL995-V3 from [Inductive relation prediction by subgraph reasoning](https://arxiv.org/abs/1911.06962) by Teru et al, ICML 2020. The original repo and other datasets are [here](https://github.com/kkteru/grail).\n\n  Training is performed on the Train Graph, but at validation/test time the model is fed with a new Inference Graph with completely new nodes. The Inference Graph has missing edges that have to be predicted at validation or test time.\n\n\n  | Dataset | Relations | Types | Train Graph | Train Graph | Inference Graph | Inference Graph | Inference Graph | Inference Graph |\n  |-----|----:|----:|----:|----:|----:|----:|----:|----:|\n  |  |  |  |  Train Entities | Train Edges | Inference Entities | Inference Edges | Validation Edges | Test Edges |\n  | FB15k-237-V2 | 203 | 3851 | 3,000 | 4,245 | 2,000 | 4,145 | 469 | 478 |\n  | NELL995-V3 | 142 | 267 | 4,647 | 16,393 | 4,921 | 8.048 | 811 | 809 |\n\n  The type hierarchy created for those datasets remains unknown.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n\n  | Queries | Training |  Validation | Validation |  Test | Test |\n  |---------|---------:|-----:|---------:|-----------:|-----:|\n  | Dataset | 1p/2p/3p/2i/3i  | 1p | others | 1p | others |\n  | FB15k-237-V2  | 9,964 | 1,738 | 2,000 | 791 | 1,000 |\n  | NELL995-V3 | 12,010 | 2,197 | 2,000 | 1,167 | 1,500 |\n\n\u003c/details\u003e \n\n#### Tree-like Conjunctive Inductive\n\nThe dataset proposed in [GNNQ](https://github.com/KRR-Oxford/GNNQ) frames query answering as node classification. The dataset has 9 tree-like conjunctive queries (6 synthetic from WatDiv and 3 from FB15k237), no unions nor negations. For each query, there are **P** KGs with an answer entity satisfying a query and **N** KGs with negative samplies where an answer does not satisfy a query. \nTest splits have graphs with new entities (but the same query shapes).\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n    Many - each WatDiv query has 2K positive GRAPHS and 700K negative GRAPHS (each of about 100K triples); each FB15k237 query has about 1K positive GRAPHS and 1K negative GRAPHS (each of about 10K triples)\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n\n  Each query in the table has many associated graphs where one node is an answer (positive graph sample) and where nodes are not answers (negative graph samples)\n  \n  | Query | Relations | Num atoms / tree depth |  Train: pos/neg | Test: pos/neg | \n  |---------|---------:|-----:|---------:|-----------:|\n  | WatDiv-Q1 | 158  | 8 / 4 | 2114 / 699699 | 1085 / 349877 |\n  | WatDiv-Q2 | 158  | 8 / 3 | 3258 / 698396 | 1769 / 349119 |\n  | WatDiv-Q3 | 158  | 8 / 3 | 1520 / 700276 | 798 / 350165 | \n  | WatDiv-Q4 | 158  | 10 / 4 | 2397 / 698986 | 1226 / 349546 | \n  | WatDiv-Q5 | 158  | 10 / 4 | 6338 / 693988 | 2866 / 347570 | \n  | WatDiv-Q6 | 158  | 10 / 4 | 7545 / 692439 | 3744 / 346290 | \n  | FB15k237-Q1 | 237  | 7 / 4 | 1185 / 1180 | 395 / 395 | \n  | FB15k237-Q2 | 237  | 7 / 4 | 650 / 660 | 220 / 220 | \n  | FB15k237-Q3 | 237  | 5 / 4 | 860 / 870 | 290 / 290 | \n\u003c/details\u003e\n\n### Temporal Datasets\n\nIntroduced in [TFLEX: Temporal Feature-Logic Embedding Framework for Complex Reasoning over Temporal Knowledge Graph](https://arxiv.org/abs/2205.14307), KDD 2022.\n\nBased on FOL operators, the dataset focuses on temporal reasoning, which includes `after`, `before` and `between` on any timestamp set.\n\n\u003cdetails\u003e\n  \u003csummary\u003e Graphs \u003c/summary\u003e\n\n  | Dataset | Entities | Relations | Timestamps | Training Edges | Validation Edges | Test Edges | Total Edges\n  |-----|----:|----:|----:|----:|----:|----:|----:|\n  | ICEWS14 | 7,128 | 230 | 365 | 72,826 | 8,941 | 8,963 | 90,730 |\n  | ICEWS05-15 | 10,488 | 251 | 4,017 | 386,962 | 46,275 | 46,092 | 479,329 |\n  | GDELT-500  | 500 | 20 | 366 | 2,735,685 | 341,961 | 341,961 | 3,419,607 |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Queries \u003c/summary\u003e\n  \n| Query Name | ICEWS14-Train | Validation | Test | ICES05-15-Train | Validation | Test  | GDELT-500-Train | Validation | Test  |\n|------------|---------------|------------|------|-----------------|------------|-------|-----------------|------------|-------|\n| Pe2        | 72826         | 3482       | 4037 | 368962          | 10000      | 10000 | 2215309         | 10000      | 10000 |\n| Pe3        | 72826         | 3492       | 4083 | 368962          | 10000      | 10000 | 2215309         | 10000      | 10000 |\n| Pe_Pt      | 7282          | 3385       | 3638 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| e2i        | 72826         | 3305       | 3655 | 368962          | 10000      | 10000 | 2215309         | 10000      | 10000 |\n| e3i        | 72826         | 2966       | 3023 | 368962          | 10000      | 10000 | 2215309         | 10000      | 10000 |\n| e2i_Pe     | -             | 2913       | 2913 | -               | 10000      | 10000 | -               | 10000      | 10000 |\n| Pe_e2i     | -             | 2913       | 2913 | -               | 10000      | 10000 | -               | 10000      | 10000 |\n| Pe_t2i     | -             | 2913       | 2913 | -               | 10000      | 10000 | -               | 10000      | 10000 |\n| e2i_NPe       | 7282          | 3061       | 3192 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| e2i_peN       | 7282          | 2971       | 3031 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| Pe_e2i_Pe_NPe | 7282          | 2968       | 3012 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| e2i_N         | 7282          | 2949       | 2975 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| e3i_N         | 7282          | 2913       | 2914 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| e2u           | -             | 2913       | 2913 | -               | 10000      | 10000 | -               | 10000      | 10000 |\n| Pe_e2u        | -             | 2913       | 2913 | -               | 10000      | 10000 | -               | 10000      | 10000 |\n| Pt_lPe     | 7282          | 4976       | 5608 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| Pt_rPe     | 7282          | 3321       | 3621 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| t2i        | 72826         | 5112       | 6631 | 368962          | 10000      | 10000 | 2215309         | 10000      | 10000 |\n| t3i        | 72826         | 3094       | 3296 | 368962          | 10000      | 10000 | 2215309         | 10000      | 10000 |\n| t2i_Pe     | -             | 2913       | 2913 | -               | 10000      | 10000 | -               | 10000      | 10000 |\n| Pt_le2i    | 7282          | 3226       | 3466 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| Pt_re2i    | 7282          | 3236       | 3485 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| t2i_NPt         | 7282          | 4873       | 5464 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| t2i_PtN         | 7282          | 3300       | 3609 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| Pe_t2i_PtPe_NPt | 7282          | 3031       | 3127 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| t2i_N           | 7282          | 3135       | 3328 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| t3i_N           | 7282          | 2924       | 2944 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| t2u        | -             | 2913       | 2913 | -               | 10000      | 10000 | -               | 10000      | 10000 |\n| Pe_t2u     | -             | 2913       | 2913 | -               | 10000      | 10000 | -               | 10000      | 10000 |\n| Pe_aPt     | 7282          | 4134       | 4733 | 68262           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| Pe_bPt     | 7282          | 3970       | 4565 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| Pe_at2i    | 7282          | 4607       | 5338 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| Pe_bt2i    | 7282          | 4583       | 5386 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n| between    | 7282          | 2913       | 2913 | 36896           | 10000      | 10000 | 221530          | 10000      | 10000 |\n\n\u003c/details\u003e\n\n### Dataset tools\n\n- [Graph Query Sampler](https://github.com/miselico/graph_query_sampler): Not a method, rather a dataset generator\n- [EFO-1-QA-benchmark](https://github.com/HKUST-KnowComp/EFO-1-QA-benchmark): Generating combiantorial tree-formed query types and sampling the data.\n- [EFOk-CQA](https://github.com/HKUST-KnowComp/EFOK-CQA): Generating combinatorial existential first order query types with multiple (k) variables and sampling the data.\n\n## :wrench: Implementations\n\n- [KGReasoning](https://github.com/snap-stanford/KGReasoning): GQE, Query2Box, BetaE\n- [CQD](https://github.com/pminervini/KGReasoning): GQE, Query2Box, BetaE, CQD\n- [EFO-1-QA-benchmark](https://github.com/HKUST-KnowComp/EFO-1-QA-benchmark): Query2Box, BetaE, LogicE, NewLook, ConE, FuzzQE\n- [Query2particles](https://github.com/HKUST-KnowComp/query2particles)\n- [StarQE](https://github.com/DimitrisAlivas/StarQE): StarQE\n- [SMORE](https://github.com/google-research/smore): GQE, Query2Box, BetaE + Very Large Datasets\n- [GNN-QE](https://github.com/DeepGraphLearning/GNN-QE): GNN-QE\n- [InductiveQE](https://github.com/DeepGraphLearning/InductiveQE): Inductive QE with NodePiece and GNN-QE\n- [TAR](https://github.com/lilv98/TAR): TAR\n- [QE-TeMP](https://github.com/zhiweihu1103/QE-TEMP): TeMP (based on KGReasoning)\n- [GNNQ](https://github.com/KRR-Oxford/GNNQ): GNNQ\n- [SE-KGE](https://github.com/gengchenmai/se-kge): GQE, CGA, and geospatial model\n- [LARK](https://github.com/Akirato/LLM-KG-Reasoning/): LARK (uses Huggingface LLMs)\n- [WFRE](https://github.com/HKUST-KnowComp/WFRE): WFRE\n- [FIT](https://github.com/HKUST-KnowComp/FIT): FIT\n- [SQE](https://github.com/HKUST-KnowComp/SQE): SQE with Transformer/LSTM/GRU/TCN, Tree-LSTM, Tree-RNN, BetaE, BiQE, ConE, FuzzQE, GQE, HypE, NerualMLP (Mixer), Query2Box, Query2Particles\n- [NRN](https://github.com/HKUST-KnowComp/NRN): NRN with GQE, Query2Box, Query2Particles\n- [EFOk-CQA](https://github.com/HKUST-KnowComp/EFOK-CQA): EFOk\n\n## All Papers on Complex Logical Query Answering (54)\n\n\u003cdetails\u003e\n  \u003csummary\u003eClick to expand \u003c/summary\u003e\n\n  1. (GQE) [Embedding Logical Queries on Knowledge Graphs](https://proceedings.neurips.cc/paper/2018/file/ef50c335cca9f340bde656363ebd02fd-Paper.pdf) NeurIPS 2018\n  2. (GQE + hashing) [Learning to Hash for Efficient Search over Incomplete Knowledge Graphs](https://ieeexplore.ieee.org/abstract/document/8970688/?casa_token=gLHFmr7V2ekAAAAA:wwDbUufdkwHTQo68pansuhJsJ2XQAF0P21_mQSu75KVRUkgqARmXBs_VEmFOkFgz_Lq-FXP8OA) ICDM 2019\n  3. (CGA) [Contextual Graph Attention for Answering Logical Queries over Incomplete Knowledge Graphs](https://arxiv.org/pdf/1910.00084.pdf) K-CAP 2019, GQE + self-attention instead of DeepSet\n  4. (TractOR) [Symbolic querying of vector spaces: Probabilistic databases meets relational embeddings](https://arxiv.org/pdf/2002.10029.pdf), UAI 2020\n  5. (Query2Box) [Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box Embeddings](https://openreview.net/pdf?id=BJgr4kSFDS) ICLR 2020\n  6. (BetaE) [Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs](https://proceedings.neurips.cc/paper/2020/file/e43739bba7cdb577e9e3e4e42447f5a5-Paper.pdf) NeurIPS 2020\n  7. (EmQL) [Faithful embeddings for knowledge base queries](https://proceedings.neurips.cc/paper/2020/hash/fe74074593f21197b7b7be3c08678616-Abstract.html) NeurIPS 2020\n  8. (MPQE) [Message Passing Query Embedding](https://grlplus.github.io/papers/26.pdf) ICML’20 Workshop\n  9. (RotatE-Box)[Regex Queries over Incomplete Knowledge Bases](https://arxiv.org/abs/2005.00480) AKBC’21\n  10. (BiQE) [Answering complex queries in knowledge graphs with bidirectional sequence encoders](https://ojs.aaai.org/index.php/AAAI/article/view/16630), AAAI’21 \n  11. [Approximate knowledge graph query answering: from ranking to binary classification](https://arxiv.org/abs/2102.11389)\n  12. [Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding](https://arxiv.org/pdf/2110.03789.pdf) arxiv, 2021\n  13. (ConE) [Cone: Cone embeddings for multi-hop reasoning over knowledge graphs](https://proceedings.neurips.cc/paper/2021/hash/a0160709701140704575d499c997b6ca-Abstract.html) NeurIPS’21\n  14. (PERM) [Probabilistic entity representation model for reasoning over knowledge graphs](https://proceedings.neurips.cc/paper/2021/hash/c4d2ce3f3ebb5393a77c33c0cd95dc93-Abstract.html) (improv over BetaE) NeurIPS’21\n  15. (CQD) [Complex Query Answering with Neural Link Predictors](https://openreview.net/forum?id=Mos9F9kDwkz) ICLR’21\n  16. (HypE) [Self-Supervised Hyperboloid Representations from Logical Queries over Knowledge Graphs](https://arxiv.org/pdf/2012.13023.pdf), WWW 2021\n  17. (NewLook) [Neural-Answering Logical Queries on Knowledge Graphs](http://tonghanghang.org/pdfs/kdd21_newlook.pdf) (KDD’21) \n  18. [Benchmarking the Combinatorial Generalizability of Complex Query Answering on Knowledge Graphs](https://arxiv.org/abs/2109.08925), NeurIPS 2021 (Datasets and Benchmarks)\n  19. [Neuro-Symbolic Ontology-Mediated Query Answering](https://openreview.net/pdf?id=wwVb95CkrFm) OpenReview 2021\n  20. (LogicE) [Logic Embeddings for Complex Query Answering](https://arxiv.org/pdf/2103.00418.pdf) arxiv 2021\n  21. (StarQE) [Query Embedding on Hyper-relational Knowledge Graphs](https://arxiv.org/abs/2106.08166) ICLR 2022, \n  22. (MLPMix) [Neural Methods for Logical Reasoning over Knowledge Graphs](https://openreview.net/forum?id=tgcAoUVHRIB) ICLR 2022\n  23. (FuzzQE) [Fuzzy Logic Based Logical Query Answering on Knowledge Graphs](https://ojs.aaai.org/index.php/AAAI/article/view/20310), AAAI 2022\n  24. (GNN-QE)  [Neural-Symbolic Models for Logical Queries on Knowledge Graphs](https://arxiv.org/abs/2205.10128), ICML 2022\n  25. (SMORE) [SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs](https://arxiv.org/abs/2110.14890) KDD 2022 \n  26. (kgTransformer) [Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries](https://dl.acm.org/doi/pdf/10.1145/3534678.3539472?casa_token=c_fD5BwP0MoAAAAA:eYO12-vcXophXPFSYro_6go4b2sQRkCTcEYRZRcVgDT6SiDpKa0AvGhqS8spvhn8-y_H0ZmCz72E) KDD 2022\n  27. (Query2Particles) [Query2Particles: Knowledge Graph Reasoning with Particle Embeddings](https://arxiv.org/abs/2204.12847), Findings NAACL’22\n  28. (TAR) [TAR: Neural Logical Reasoning across TBox and ABox](https://arxiv.org/abs/2205.14591) (arxiv, 2022) \n  29. (TeMP) [Type-aware embeddings for multi-hop reasoning over knowledge graphs](https://arxiv.org/abs/2205.00782) (IJCAI-ECAI 2022) \n  30. (FLEX) [FLEX: Feature-Logic Embedding Framework for CompleX Knowledge Graph Reasoning](https://arxiv.org/abs/2205.11039) (arxiv 2022)\n  31. (TFLEX) [TFLEX: Temporal Feature-Logic Embedding Framework for Complex Reasoning over Temporal Knowledge Graph](https://arxiv.org/pdf/2205.14307.pdf) (arxiv, 2022)\n  32. (LinE) [LinE: Logical Query Reasoning over Hierarchical Knowledge Graphs](https://dl.acm.org/doi/pdf/10.1145/3534678.3539338?casa_token=_jPlNJj2TlYAAAAA:pKAA42_lrZ2JIHc1YZV0fchIlRiIcqCy8oCBL2UU3Gm84MOeTSYLfQn31DKtXBbU2yqzC7LUsYvREBw) KDD 2022\n  33. [GNNQ: A Neuro-Symbolic Approach for Query Answering over Incomplete Knowledge Graphs](https://scholar.archive.org/work/sswyx3mbc5danaxosfnyr45cw4/access/wayback/https://s3-eu-west-1.amazonaws.com/pstorage-oxford-0703704596/36647646/GNNQ_full.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIA3OGA3B5WHKHW6LWV/20220814/eu-west-1/s3/aws4_request\u0026X-Amz-Date=20220814T213541Z\u0026X-Amz-Expires=10\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=1bf627735fbce377c2478f514fbac1dfcf49423980d372cc71e2bed732d6ae2c) ISWC 2022\n  34. (ENeSy) [Neural-Symbolic Entangled Framework for Complex Query Answering](https://arxiv.org/pdf/2209.08779.pdf) NeurIPS 2022  \n  35. (NodePiece-QE, InductiveQE) [Inductive Logical Query Answering in Knowledge Graphs](https://arxiv.org/abs/2210.08008) NeurIPS 2022  \n  36. (RoMA) [Reasoning over Multi-view Knowledge Graphs](https://arxiv.org/pdf/2209.13702.pdf) arxiv 2022, some new datasets, but no code/data published  \n  37. (LMPNN) [Logical Message Passing Networks With One-Hop Inference On Atomic Formulas](https://openreview.net/pdf?id=SoyOsp7i_l) ICLR'23  \n  38. (GammaE) [GammaE: Gamma Embeddings for Logical Queries on Knowledge Graphs](https://arxiv.org/pdf/2210.15578.pdf) EMNLP 2022  \n  39. (NMP-QEM) [Neural-based Mixture Probabilistic Query Embedding for Answering FOL queries on Knowledge Graphs](https://aclanthology.org/2022.emnlp-main.194.pdf), EMNLP 2022  \n  40. (NQE) [NQE: N-ary Query Embedding for Complex Query Answering over Hyper-relational Knowledge Graphs](https://arxiv.org/pdf/2211.13469.pdf) AAAI 2023  \n  41. (QTO) [Answering Complex Logical Queries on Knowledge Graphs via Query Computation Tree Optimization](https://arxiv.org/pdf/2212.09567.pdf), ICML'23 submission  \n  42. (SignalE) [Signal Embeddings for Complex Logical Reasoning in Knowledge Graphs](https://link.springer.com/content/pdf/10.1007/978-3-031-10983-6_20.pdf?pdf=inline%20link), KSEM'22  \n  43. (Var2Vec) [Efficient Embeddings of Logical Variables for Query Answering over Incomplete Knowledge Graphs](https://ora.ox.ac.uk/objects/uuid:27c9b239-20d0-449f-b425-b5065eb128fe/download_file?file_format=application%2Fpdf\u0026safe_filename=Wang_et_al_2022_Efficient_Embeddings_ofAM.pdf\u0026type_of_work=Conference+item), AAAI'23\n  44. (CQD-A) [Adapting Neural Link Predictors for Complex Query Answering](https://arxiv.org/pdf/2301.12313.pdf)\n  45. (Query2Geom) [Analysis of Attention Mechanisms in Box-Embedding Systems](https://link.springer.com/chapter/10.1007/978-3-031-26438-2_6), 2023\n  46. (SQE) [Sequential Query Encoding For Complex Query Answering on Knowledge Graphs](https://openreview.net/pdf?id=ERqGqZzSu5), TMLR 2023  \n  47. (CylE) [CylE: Cylinder Embeddings for Multi-hop Reasoning over Knowledge Graphs](https://aclanthology.org/2023.eacl-main.127.pdf), EACL 2023  \n  48. (RoConE) [Modeling Relational Patterns for Logical Query Answering over Knowledge Graphs](https://arxiv.org/pdf/2303.11858.pdf)  \n  49. (FIT) [On Existential First Order Queries Inference on Knowledge Graphs](https://arxiv.org/pdf/2304.07063.pdf), arxiv 2023  \n  50. (LitCQD) [LitCQD: Multi-Hop Reasoning in Incomplete Knowledge Graphs with Numeric Literals](https://arxiv.org/pdf/2304.14742.pdf), arxiv 2023  \n  51. (LARK) [Complex Logical Reasoning over Knowledge Graphs using Large Language Models](https://arxiv.org/pdf/2305.01157.pdf), arxiv 2023  \n  52. (WFRE) [Wasserstein-Fisher-Rao Embedding: Logical Query Embeddings with Local Comparison and Global Transport](https://arxiv.org/pdf/2305.04034.pdf), arxiv 2023\n  53. (NRN) [Knowledge Graph Reasoning over Entities and Numerical Values](https://arxiv.org/abs/2306.01399) KDD 2023\n  54. (EFOk-CQA) [EFOk-CQA: Towards Knowledge Graph Complex Query Answering beyond Set Operation](https://arxiv.org/abs/2307.13701) arxiv 2023\n\n\u003c/details\u003e\n\n## Application Papers (7)\n\u003cdetails\u003e\n  \u003csummary\u003eClick to expand \u003c/summary\u003e\n\n  1. (SE-KGE) [SE-KGE: A Location-Aware Knowledge Graph Embedding Model for Geographic Question Answering and Spatial Semantic Lifting](https://arxiv.org/pdf/2004.14171.pdf), Transactions in GIS 2020, GQE with scalar (x,y) coordinate prediction / encoding\n  2. (LEGO) [Lego: Latent execution-guided reasoning for multi-hop question answering on knowledge graphs](https://proceedings.mlr.press/v139/ren21a.html), ICML 2021\n  3. (CBR-SubG) [Knowledge base question answering by case-based reasoning over subgraphs](https://proceedings.mlr.press/v162/das22a.html) ICML 2022, application to Question Answering, entailment only, custom datasets\n  4. (LogiRec) [Towards High-Order Complementary Recommendation via Logical Reasoning Network](https://arxiv.org/pdf/2212.04966.pdf) Application: BetaE in RecSys, arxiv 2022\n  5. [Context-aware explainable recommendation based on domain knowledge graph](https://www.mdpi.com/1458952), Big Data and Cognitive Computing, 2022\n  6. (PLM4CLQA) [Unifying Structure Reasoning and Language Model Pre-training for Complex Reasoning](https://arxiv.org/pdf/2301.08913.pdf), arxiv 2023\n  7. [Unifying structure reasoning and language model pre-training for complex reasoning](https://arxiv.org/pdf/2301.08913), arxiv 2023\n\u003c/details\u003e\n\n## Potentially Relevant \n\u003cdetails\u003e\n  \u003csummary\u003eClick to expand \u003c/summary\u003e\n\n  1. [Hybrid Structured and Similarity Queries over Wikidata plus Embeddings with Kypher-V](https://www.isi.edu/~hans/publications/WDWS-ISWC-2022.pdf), ISWC 2022\n  2. [Combining RDF Graph Data and Embedding Models for an Augmented Knowledge Graph](https://dl.acm.org/doi/fullHtml/10.1145/3184558.3191527), BigNet 2018 Workshop @ WWW'18\n  3. [TrQuery: An Embedding-based Framework for Recommanding SPARQL Queries](https://arxiv.org/pdf/1806.06205.pdf), 2018\n  4. [Towards Empty Answers in SPARQL: Approximating Querying with RDF Embedding](https://link.springer.com/content/pdf/10.1007/978-3-030-00671-6.pdf), ISWC 2018\n\u003c/details\u003e\n\n## Citation\n\nIf you find this work useful, please cite the original paper:\n\n```bibtex\n@article{ren2023ngdb,\n    title={Neural Graph Reasoning: Complex Logical Query Answering Meets Graph Databases},\n    author={Hongyu Ren and Mikhail Galkin and Michael Cochez and Zhaocheng Zhu and Jure Leskovec},\n    year={2023},\n    eprint={2303.14617},\n    archivePrefix={arXiv},\n}\n```\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/neuralgraphdatabases%2Fawesome-logical-query/projects"}