Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/FreedomIntelligence/ReasoningNLP
paper list on reasoning in NLP
https://github.com/FreedomIntelligence/ReasoningNLP
Last synced: 3 months ago
JSON representation
paper list on reasoning in NLP
- Host: GitHub
- URL: https://github.com/FreedomIntelligence/ReasoningNLP
- Owner: FreedomIntelligence
- Created: 2022-09-08T14:01:50.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-11-13T12:00:00.000Z (about 1 year ago)
- Last Synced: 2024-08-02T09:26:36.817Z (6 months ago)
- Size: 152 KB
- Stars: 166
- Watchers: 7
- Forks: 15
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- Awesome-LLM-Reasoning - ReasoningNLP
README
# ReasoningNLP
Paper list on reasoning in NLP. Here we mainly collect papers about datasets and methods using PLMs (under the progress).
**See our survey on natural language reasoning:**
- [Natural Language Reasoning, A Survey](https://arxiv.org/pdf/2303.14725.pdf)
Here are anohter two related surveys on LLM prompting:
- [Reasoning with Language Model Prompting: A Survey](https://arxiv.org/pdf/2212.09597.pdf) | [resources](https://github.com/zjunlp/Prompt4ReasoningPapers)
- [Towards Reasoning in Large Language Models: A Survey](https://arxiv.org/pdf/2212.10403.pdf) | [resources](https://github.com/jeffhj/LM-reasoning)
Table of Contents
- [Methodology](#1)
- [Reasoning Paradigm](#1.1)
- [End-to-End Reasoning](#1.1.1)
- [Forward Reasoning](#1.1.2)
- [Backward Reasoning](#1.1.3)
- [Learning Paradigm](#1.2)
- [Finetuning](#1.2.1)
- [In-Context Learning](#1.2.2)
- [Natural Language Reasoning](#2)
- [Classical Logical Reasoning](#2.1)
- [Datasets & Benchmarks](#2.1.1)
- [Related Works](#2.1.2)
- [Natural Language Inference](#2.2)
- [Datasets & Benchmarks](#2.2.1)
- [Related Works](#2.2.2)
- [Multi-hop Question Answering](#2.3)
- [Datasets & Benchmarks](#2.3.1)
- [Related Works](#2.3.2)
- [Commonsense Reasoning](#2.4)
- [Datasets & Benchmarks](#2.4.1)
- [Related Works](#2.4.2)
- [Knowledge Bases](#2.4.3)
- [Complex Reasoning](#2.5)
- [Datasets & Benchmarks](#2.5.1)
- [Knowledge Graph Reasoning](#knowledge-graph-reasoning)
- [Mathematical Reasoning](#mathematical-reasoning)
Methodology
Reasoning Paradigm
End-to-End Reasoning
Forward Reasoning
Backward Reasoning
Learning Paradigm
Finetuning
In-Context Learning
1. **Show Your Work: Scratchpads for Intermediate Computation with Language Models** arXiv (2021)
*Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, Augustus Odena* [[pdf](https://arxiv.org/pdf/2112.00114.pdf)]
2. **Chain of Thought Prompting Elicits Reasoning in Large Language Models** arXiv (2022)
*Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou* [[pdf](https://arxiv.org/pdf/2201.11903.pdf)] [[project](https://github.com/jasonwei20/chain-of-thought-prompting)]
3. **Self-Consistency Improves Chain of Thought Reasoning in Language Models** arXiv (2022)
*Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou* [[pdf](https://arxiv.org/pdf/2203.11171.pdf)]
4. **STaR: Bootstrapping Reasoning With Reasoning** arXiv (2022)
*Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman* [[pdf](https://arxiv.org/pdf/2203.14465.pdf)]
5. **Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning** arXiv (2022)
*Antonia Creswell, Murray Shanahan, Irina Higgins* [[pdf](https://arxiv.org/pdf/2205.09712.pdf)]
6. **Least-to-Most Prompting Enables Complex Reasoning in Large Language Models** arXiv (2022)
*Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, Ed Chi* [[pdf](https://arxiv.org/pdf/2205.10625.pdf)]
7. **Large Language Models are Zero-Shot Reasoners** arXiv (2022)
*Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa* [[pdf](https://arxiv.org/pdf/2205.11916.pdf)] [[project](https://github.com/kojima-takeshi188/zero_shot_cot)]
8. **Language models show human-like content effects on reasoning** arXiv (2022)
*Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, Felix Hill* [[pdf](https://arxiv.org/pdf/2207.07051.pdf)]
9. **Language Model Cascades** arXiv (2022)
*David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein, Kevin Murphy, Charles Sutton* [[pdf](https://arxiv.org/pdf/2207.10342.pdf)] [[project](https://model-cascades.github.io/)]
10. **Faithful Reasoning Using Large Language Models** arXiv (2022)
*Antonia Creswell, Murray Shanahan* [[pdf](https://arxiv.org/pdf/2208.14271.pdf)]
11. **Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought** arXiv (2022)
*Abulhair Saparov, He He* [[pdf](https://arxiv.org/pdf/2210.01240.pdf)] [[project](https://github.com/asaparov/prontoqa)]
12. **ThinkSum: Probabilistic reasoning over sets using large language models** arXiv (2022)
*Batu Ozturkler, Nikolay Malkin, Zhen Wang, Nebojsa Jojic* [[pdf](https://arxiv.org/pdf/2210.01293.pdf)]
13. **Measuring and Narrowing the Compositionality Gap in Language Models** arXiv (2022)
*Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis* [[pdf](https://arxiv.org/pdf/2210.03350.pdf)] [[project](https://github.com/ofirpress/self-ask)]
NLP Topics
Classical Logical Reasoning
Some datasets explicitly target at philosophical reasoning types, e.g. deduction, abduction and induction. Thus, we call them as ``classical logical reasoning tasks''. A key characteristic of this topic is that tasks are mostly artificial to study reasoning.
Datasets & Benchmarks
Inference Type
Dataset
Size
Task
Link
Remark
Deductive Reasoning
babI-15
-
extraction
paper
project
synthetic
RuleTaker
>500k
classification
paper
project
synthetic, the first large-scale benchmark. D*, Birds-Electricity, ParaRules
ProofWriter
>500k
classification
paper
project
improvement on RuleTaker, + open-world assumption
PARARULE Plus
400k
classification
paper
project
improvement on ParaRules, addresses the depth imbalance issue
AAC
710k
generation
paper
project
synthetic, syllogistic arguments
LogicInference
200k
generation
paper
project
synthetic, more tasks
FOLIO
1.4k
classification, generation
paper
project
expert-written, annotate with FOL
RobustLR
120k
classification
paper
project
synthetic, robustness on logical semantics
Defeasible Reasoning
Abductive Reasoning
AbductionRules
-
generation
paper
project
variant of RuleTaker
ART
(αNLI, αNLG)
17.8k
classification, generation
paper
project
commonsense, based on ROCStories
Inductive Reasoning
bAbI-16
-
extraction
paper
project
synthetic, induce-then-deduce
CLUTRR
-
extraction
paper
project
synthetic, induce-then-deduce, kinship
DEER
1.2k
generation
paper
project
induce explicit natural language rule (human-authored) from natural language facts (Webs)
Others
defeasibleNLI
43.8k
classification, generation
paper
project
direction on evidence updation, based on the existing datasets
Papers on dataset artifacts:
1. **On the Paradox of Learning to Reason from Data** arXiv (2022)
*Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, Guy Van den Broeck* [[pdf](https://arxiv.org/pdf/2205.11502.pdf)] [[project](https://github.com/joshuacnf/paradox-learning2reason)]
Related Works
Deductive reasoning:
1. **Transformers as Soft Reasoners over Language** IJCAI (2020)
*Peter Clark, Oyvind Tafjord, Kyle Richardson* [[pdf](https://www.ijcai.org/proceedings/2020/0537.pdf)] [[project](https://allenai.org/data/ruletaker)]
2. **PRover: Proof Generation for Interpretable Reasoning over Rules** EMNLP (2020)
*Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, Mohit Bansal* [[pdf](https://aclanthology.org/2020.emnlp-main.9.pdf)] [[project](https://github.com/swarnaHub/PRover)]
3. **multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning** NAACL (2021)
*Swarnadeep Saha, Prateek Yadav, Mohit Bansal* [[pdf](https://aclanthology.org/2021.naacl-main.287.pdf)] [[project](https://github.com/swarnaHub/multiPRover)]
4. **Critical Thinking for Language Models** IWCS (2021)
*Gregor Betz, Christian Voigt, Kyle Richardson* [[pdf](https://aclanthology.org/2021.iwcs-1.7.pdf)] [[project](https://github.com/debatelab/aacorpus)]
5. **Explainable Multi-hop Verbal Reasoning Through Internal Monologue** NAACL (2021)
*Zhengzhong Liang, Steven Bethard, Mihai Surdeanu* [[pdf](https://aclanthology.org/2021.naacl-main.97.pdf)] [[project](https://github.com/clulab/releases/tree/master/naacl2021-evr)]
6. **ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language** ACL findings (2021)
*Oyvind Tafjord, Bhavana Dalvi, Peter Clark* [[pdf](https://aclanthology.org/2021.findings-acl.317.pdf)] [[project](https://allenai.org/data/proofwriter)]
7. **Flexible Generation of Natural Language Deductions** EMNLP (2021)
*Kaj Bostrom, Xinyu Zhao, Swarat Chaudhuri, Greg Durrett* [[pdf](https://aclanthology.org/2021.emnlp-main.506.pdf)] [[project](https://github.com/alephic/ParaPattern)]
8. **FaiRR: Faithful and Robust Deductive Reasoning over Natural Language** ACL (2022)
*Soumya Sanyal, Harman Singh, Xiang Ren* [[pdf](https://aclanthology.org/2022.acl-long.77.pdf)] [[project](https://github.com/INK-USC/FaiRR)]
9. **Interpretable Proof Generation via Iterative Backward Reasoning** NAACL (2022)
*Hanhao Qu, Yu Cao, Jun Gao, Liang Ding, Ruifeng Xu* [[pdf](https://aclanthology.org/2022.naacl-main.216.pdf)] [[project](https://github.com/find-knowledge/IBR)]
10. **Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning** arXiv (2022)
*Antonia Creswell, Murray Shanahan, Irina Higgins* [[pdf](https://arxiv.org/pdf/2205.09712.pdf)]
11. **Generating Natural Language Proofs with Verifier-Guided Search** arXiv (2022)
*Kaiyu Yang, Jia Deng, Danqi Chen* [[pdf](https://arxiv.org/pdf/2205.12443.pdf)] [[project](https://github.com/princeton-nlp/NLProofS)]
12. **ROBUSTLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners** arXiv (2022)
*Soumya Sanyal, Zeyi Liao, Xiang Ren* [[pdf](https://arxiv.org/pdf/2205.12598.pdf)] [[project](https://github.com/INK-USC/RobustLR)]
13. **Language models show human-like content effects on reasoning** arXiv (2022)
*Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, Felix Hill* [[pdf](https://arxiv.org/pdf/2207.07051.pdf)]
14. **Faithful Reasoning Using Large Language Models** arXiv (2022)
*Antonia Creswell, Murray Shanahan* [[pdf](https://arxiv.org/pdf/2208.14271.pdf)]
15. **Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought** arXiv (2022)
*Abulhair Saparov, He He* [[pdf](https://arxiv.org/pdf/2210.01240.pdf)] [[project](https://github.com/asaparov/prontoqa)]
16. **LAMBADA: Backward Chaining for Automated Reasoning in Natural Language** arXiv (2022)
*Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, Deepak Ramachandran* [[pdf](https://arxiv.org/pdf/2212.13894.pdf)]
Defeasible reasoning:
1. **Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision** AAAI (2020)
*Faeze Brahman, Vered Shwartz, Rachel Rudinger, Yejin Choi* [[pdf](https://arxiv.org/pdf/2012.08012.pdf)] [[project](https://github.com/fabrahman/RationaleGen)]
2. **Could you give me a hint ? Generating inference graphs for defeasible reasoning** ACL findings (2021)
*Aman Madaan, Dheeraj Rajagopal, Niket Tandon, Yiming Yang, Eduard H. Hovy* [[pdf](https://aclanthology.org/2021.findings-acl.456.pdf)] [[project](https://tinyurl.com/defeasiblegraphs)]
3. **Think about it! Improving defeasible reasoning by first modeling the question scenario** EMNLP (2021)
*Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, Eduard H. Hovy* [[pdf](https://aclanthology.org/2021.emnlp-main.508.pdf)] [[project](https://github.com/madaan/thinkaboutit)]
4. **Language Models as Inductive Reasoners** arXiv (2022)
*Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, Furu Wei* [[pdf](https://arxiv.org/pdf/2212.10923.pdf)]
Natural Language Inference
Task: given a premise-hypothesis pair, classify it into three classes: entailment, contradiction and neutral.
There are mainly three types of premise-hypothesis pairs in NLI task: paraphrase, compound semantics understanding (CSU), and reasoning. Here we just consider the last one.
Premise
Hypothesis
Paraphrase
Two doctors perform surgery on patient
Doctors are performing surgery
CSU
Two women are embracing while holding to go packages
Two women are holding packages
(Two women are embracing)
Reasoning
A soccer game with multiple males playing
(Soccer is a sport)
Some men are playing a sport
Datasets & Benchmarks
Dataset
Size
Link
Remark
generic
SNLI
570k
paper
project
the first large-scale NLI dataset
one of the most typical
e-SNLI
-
paper
project
annotate natural language explanations for SNLI
MultiNLI
433k
paper
project
cover more styles and topics than SNLI
one of the most typical
DebiasedNLI
7.5k
paper
project
debiased versions of SNLI & MultiNLI
XNLI
7.5k
paper
project
cross-lingual, based on MultiNLI
MPE
10k
paper
project
multiple premises
science
SciTail
27k
paper
project
the first NLI dataset with entirely existing text
SciNLI
107k
paper
project
data from scholarly papers
Recently, some datasets are proposed to model different subjective opinions (on classifying into which class) of crowdworkers.
Subjective Opinions
Dataset
Domain
Size
Link
Remark
UNLI
generic
61k
paper
project
subjective probability assessment (regression rather than binary), based on SNLI
ChaosNLI
generic
464k
paper
project
human opinion distribution, based on SNLI, MultiNLI and αNLI
Some datasets for other language:
Other Languages
Dataset
Language
Link
Remark
NLI-TR
Turkish
paper
project
translate SNLI and MultiNLI
IndoNLI
Indonesian
paper
project
data collection protocol from MultiNLI
Papers on dataset artifacts:
1. **Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment** LREC (2018)
*Masatoshi Tsuchiya* [[pdf](https://aclanthology.org/N19-1101.pdf)]
2. **Annotation Artifacts in Natural Language Inference Data** NAACL (2018)
*Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, Noah A. Smith* [[pdf](https://aclanthology.org/N18-2017.pdf)]
3. **Hypothesis Only Baselines in Natural Language Inference** SEM (2019)
*Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, Benjamin Van Durme* [[pdf](https://aclanthology.org/S18-2023.pdf)]
Related Works
1. **NILE : Natural Language Inference with Faithful Natural Language Explanations** ACL (2020)
*Sawan Kumar, Partha P. Talukdar* [[pdf](https://aclanthology.org/2020.acl-main.771.pdf)] [[project](https://github.com/SawanKumar28/nile)]
2. **Identifying inherent disagreement in natural language inference** NAACL (2021)
*Xinliang Frederick Zhang, Marie-Catherine de Marneffe* [[pdf](https://aclanthology.org/2021.naacl-main.390.pdf)] [[project](https://github.com/FrederickXZhang/FgNLI)]
3. **KACE: Generating Knowledge Aware Contrastive Explanations for Natural Language Inference** ACL (2021)
*Qianglong Chen, Feng Ji, Xiangji Zeng, Feng-Lin Li, Ji Zhang, Haiqing Chen, Yin Zhang* [[pdf](https://aclanthology.org/2021.acl-long.196.pdf)] [[project](https://github.com/AI4NLP/KACE)]
4. **Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference** ACL findings (2021)
*Hai Hu, He Zhou, Zuoyu Tian, Yiwen Zhang, Yina Patterson, Yanting Li, Yixin Nie, Kyle Richardson* [[pdf](https://aclanthology.org/2021.findings-acl.331.pdf)] [[project](https://github.com/huhailinguist/ChineseNLIProbing)]
5. **Enhancing Cross-lingual Natural Language Inference by Prompt-learning from Cross-lingual Templates** ACL (2022)
*Kunxun Qi, Hai Wan, Jianfeng Du, Haolan Chen* [[pdf](https://aclanthology.org/2022.acl-long.134.pdf)] [[project](https://github.com/qikunxun/PCT)]
6. **Generating Intermediate Steps for NLI with Next-Step Supervision** arXiv (2022)
*Deepanway Ghosal, Somak Aditya, Monojit Choudhury* [[pdf](https://arxiv.org/pdf/2208.14641.pdf)]
Multi-hop Question Answering
This topic studies answering the complex questions that require to reason over evidences scattered in different contexts. The term ``hop'' here indicates the number of contexts required to reason. There are two settings on the required contexts: (1) all provided along with some distractors (i.e. distractor), (2) need to be retrieved (i.e. retrieval).
Datasets & Benchmarks
Some datasets annotate the ground supporting evidences (paragraph-level, sentence-level, or triple-level), decomposed sub-questions (and the corresponding evidences), or reasoning paths.
Dataset
Size
Knowledge Source
Setting
Answer Type
Evidence
Link
Remark
generic
WikiHop
51k
Wikipedia
distractor
choice
-
paper
project
one of the most typical
HotpotQA
112k
Wikipedia
distractor, retrieval
span, yes/no
sentence
paper
project
the most popular one
R4C
4.6k
-
-
-
triple
paper
project
annotate atomic facts for HotpotQA
BeerQA
530
Wikipedia
retrieval
span, yes/no
-
paper
project
more hops
2WikiMultiHopQA
192k
Wikipedia
distractor
span
sentence
triple
paper
project
similar to WikiHop
MuSiQue
25k
Wikipedia
distractor
span
paragraph
sub-questions
paper
project
more hops
StrategyQA
2.7k
Wikipedia
retrieval
yes/no
paragraph
sub-questions
paper
project
implicit mult-hop questions
specific domain
MedHop
2.5k
Medline
distractor
choice
-
paper
project
medicine. similar to WikiHop
QASC
9.9k
WorldTree
retrieval
choice
sentence
paper
project
science
eQASC
-
-
-
-
reasoning path
paper
project
annotate reasoning paths for QASC
Papers on dataset artifacts:
1. **Understanding Dataset Design Choices for Multi-hop Reasoning** NAACL (2019)
*Jifan Chen, Greg Durrett* [[pdf](https://aclanthology.org/N19-1405.pdf)]
2. **Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA** ACL (2019)
*Yichen Jiang, Mohit Bansal* [[pdf](https://aclanthology.org/P19-1262.pdf)] [[project](https://github.com/jiangycTarheel-zz/Adversarial-MultiHopQA)]
3. **Compositional Questions Do Not Necessitate Multi-hop Reasoning** ACL (2019)
*Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, Luke Zettlemoyer* [[pdf](https://aclanthology.org/P19-1416.pdf)] [[project](https://github.com/shmsw25/single-hop-rc)]
4. **Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning** EMNLP (2020)
*Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal* [[pdf](https://aclanthology.org/2020.emnlp-main.712.pdf)] [[project](https://github.com/stonybrooknlp/dire)]
Related Works
1. **Dynamically Fused Graph Network for Multi-hop Reasoning** ACL (2019)
*Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, Yong Yu* [[pdf](https://aclanthology.org/P19-1617.pdf)] [[project](https://github.com/woshiyyya/DFGN-pytorch)]
2. **Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs** ACL (2019)
*Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, Bowen Zhou* [[pdf](https://aclanthology.org/P19-1260.pdf)]
3. **Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction** ACL (2019)
*Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, Junji Tomita* [[pdf](https://aclanthology.org/P19-1225.pdf)]
4. **Multi-hop Reading Comprehension through Question Decomposition and Rescoring** ACL (2019)
*Sewon Min, Victor Zhong, Luke Zettlemoyer, Hannaneh Hajishirzi* [[pdf](https://aclanthology.org/P19-1613.pdf)] [[project](https://github.com/shmsw25/DecompRC)]
5. **Differentiable Reasoning over a Virtual Knowledge Base** ICLR (2020)
*Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, William W. Cohen* [[pdf](https://openreview.net/pdf?id=SJxstlHFPH)] [[project](http://www.cs.cmu.edu/~bdhingra/pages/drkit.html)]
6. **Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention** ICLR Poster (2020)
*Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul N. Bennett, Saurabh Tiwary* [[pdf](https://openreview.net/pdf?id=r1eIiCNYwS)] [[project](https://aka.ms/transformer-xh)]
7. **Low-Resource Generation of Multi-hop Reasoning Questions** ACL (2020)
*Jianxing Yu, Wei Liu, Shuang Qiu, Qinliang Su, Kai Wang, Xiaojun Quan, Jian Yin* [[pdf](http://aclanthology.lst.uni-saarland.de/2020.acl-main.601.pdf)]
8. **SRLGRN: Semantic Role Labeling Graph Reasoning Network** EMNLP (2020)
*Chen Zheng, Parisa Kordjamshidi* [[pdf](https://aclanthology.org/2020.emnlp-main.714.pdf)]
9. **Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering** ICLR (2020)
*Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, Caiming Xiong* [[pdf](https://openreview.net/pdf?id=SJgVHkrYDH)] [[project](https://github.com/AkariAsai/learning_to_retrieve_reasoning_paths)]
10. **Robustifying Multi-hop QA through Pseudo-Evidentiality Training** ACL (2021)
*Kyungjae Lee, Seung-won Hwang, Sang-eun Han, Dohyeon Lee* [[pdf](https://aclanthology.org/2021.acl-long.476.pdf)]
11. **Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension** EMNLP (2021)
*Naoya Inoue, Harsh Trivedi, Steven Sinha, Niranjan Balasubramanian, Kentaro Inui* [[pdf](https://aclanthology.org/2021.emnlp-main.490.pdf)] [[project](https://github.com/StonyBrookNLP/suqa)]
12. **Generative Context Pair Selection for Multi-hop Question Answering** EMNLP (2021)
*Dheeru Dua, Cícero Nogueira dos Santos, Patrick Ng, Ben Athiwaratkun, Bing Xiang, Matt Gardner, Sameer Singh* [[pdf](https://aclanthology.org/2021.emnlp-main.561.pdf)] [[project](https://github.com/dDua/JointQA)]
13. **Breadth First Reasoning Graph for Multi-hop Question Answering** NAACL (2021)
*Yongjie Huang, Meng Yang* [[pdf](https://aclanthology.org/2021.naacl-main.464.pdf)]
14. **Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval** ICLR Poster (2021)
*Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick S. H. Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, Barlas Oguz* [[pdf](https://openreview.net/pdf?id=EMHoBG0avc1)] [[project](https://github.com/facebookresearch/multihop_dense_retrieval)]
15. **Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval** NAACL (2021)
*Chen Zhao, Chenyan Xiong, Jordan L. Boyd-Graber, Hal Daumé III* [[pdf](https://aclanthology.org/2021.naacl-main.368.pdf)] [[project](https://github.com/henryzhao5852/BeamDR)]
16. **Unsupervised Multi-hop Question Answering by Question Generation** NAACL (2021)
*Liangming Pan, Wenhu Chen, Wenhan Xiong, Min-Yen Kan, William Yang Wang* [[pdf](https://aclanthology.org/2021.naacl-main.469.pdf)] [[project](https://github.com/teacherpeterpan/Unsupervised-Multi-hop-QA)]
17. **If You Want to Go Far Go Together: Unsupervised Joint Candidate Evidence Retrieval for Multi-hop Question Answering** NAACL (2021)
*Vikas Yadav, Steven Bethard, Mihai Surdeanu* [[pdf](https://aclanthology.org/2021.naacl-main.363.pdf)] [[project](https://github.com/vikas95/WAIR_interpretability)]
18. **Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval** NIPS (2021)
*Omar Khattab, Christopher Potts, Matei A. Zaharia* [[pdf](https://papers.nips.cc/paper/2021/file/e8b1cbd05f6e6a358a81dee52493dd06-Paper.pdf)] [[project](https://github.com/stanford-futuredata/Baleen)]
19. **Modeling Multi-hop Question Answering as Single Sequence Prediction** ACL (2022)
*Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, Nitish Shirish Keskar, Caiming Xiong* [[pdf](https://aclanthology.org/2022.acl-long.69.pdf)]
20. **CQG: A Simple and Effective Controlled Generation Framework for Multi-hop Question Generation** ACL (2022)
*Zichu Fei, Qi Zhang, Tao Gui, Di Liang, Sirui Wang, Wei Wu, Xuanjing Huang* [[pdf](https://aclanthology.org/2022.acl-long.475.pdf)] [[project](https://github.com/sion-zcfei/CQG)]
Commonsense Reasoning
Commonsense reasoning deals with implicit commonsense knowledge, which may be non-trivial to machines since they are difficult to retrieve from the web due to reporting bias. While it is named with "reasoning", the common theme of this topic is commonsense knowledge rather than reasoning. Here, we only list reasoning datasets.
Datasets & Benchmarks
There are mainly three types of reasoning datasets towards "what" (i.e. assertions or events), "what if / why" (e.g. causal and temporal relations between events), and "how" (i.e. actions) respectively.
"What" commonsense reasoning require combining multiple pieces of knowledge that some are from external knowledge sources while others are commonsense knowledge.
"What" Commonsense Reasoning
Dataset
Size
Other Knowledge Type / Source
Task
Link
Rationale
OpenBookQA
6k
science
/ WorldTree
multi-choice QA
paper
project
ground science facts
OpenCSR
20k
science
/ WorldTree, ARC corpus
free-form QA
paper
project
-
CREAK
13k
entity
/ Wikipedia
claim verification
paper
project
explanation
"What if / Why" commonsense reasoning often reasons for causal and temporal relations between events. There are two causal relations: causes and effects, which can be seen as backward causal reasoning and forward causal reasoning respectively.
"What if / Why" Commonsense Reasoning
Dataset
Size
Direction
Task
Link
Remark
ROCStories
50k
temporal
2-choice QA
paper
project
-
SWAG
113k
temporal
multi-choice QA
paper
project
-
HellaSwag
20k
temporal
multi-choice QA
paper
project
an upgraded SWAG
COPA
1k
both
2-choice QA
paper
project
-
Social-IQA
38k
both
multi-choice QA
paper
project
social situations
e-CARE
21k
both
2-choice QA
paper
project
with ground supporting facts
WIQA
40k
forward
multi-choice QA
paper
project
about nature processes
TIMETRAVEL
29k
forward
generation
paper
project
counterfactual reasoning
ART
20k
backward
2-choice/generation
paper
project
abductive commonsense reasoning
TellMeWhy
30k
backward
free-form QA
paper
project
each annotated 3 possible answers
WikiWhy
9k
backward
free-form QA
paper
project
about Wikipedia entities / events
"How" commonsense reasoning is mainly about "how to do it". It is often related to problem-solving or decision-making.
"How" Commonsense Reasoning
Dataset
Size
Source
Task
Link
Remark
WikiHow Goal-Step
1489k
WikiHow, generated
multi-choice
paper
project
goals, steps, and temporal ordering
PIQA
21k
human-authored
2-choice
paper
project
physical
Some datasets involve multiple types of reasoning.
Hybrid Commonsense Reasoning
Dataset
Size
Task
Link
Remark
CSQA
12k
multi-choice QA
paper
project
ConceptNet concepts
CoS-E
-
-
paper
project
annotate explanations for CSQA
ECQA
-
-
paper
project
annotate commonsense facts for CSQA
CSQA2
14k
boolen QA
paper
project
data construction via gamification
CosmosQA
35k
multi-choice QA
paper
project
reading comprehension on blogs
Moral Stories
12k
multi-choice QA
paper
project
situated reasoning with social norms
Related Works
1. **Attention Is (not) All You Need for Commonsense Reasoning** ACL (2019)
*Tassilo Klein, Moin Nabi* [[pdf](https://aclanthology.org/P19-1477.pdf)]
2. **COMET: Commonsense Transformers for Automatic Knowledge Graph Construction** ACL (2019)
*Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, Yejin Choi* [[pdf](https://aclanthology.org/P19-1470.pdf)] [[project](https://github.com/atcbosselut/comet-commonsense)]
3. **Explain Yourself! Leveraging Language Models for Commonsense Reasoning** ACL (2019)
*Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, Richard Socher* [[pdf](https://aclanthology.org/P19-1487.pdf)] [[project](https://github.com/salesforce/cos-e)]
4. **Commonsense Knowledge Mining from Pretrained Models** EMNLP (2019)
*Joe Davison, Joshua Feldman, Alexander M. Rush* [[pdf](https://aclanthology.org/D19-1109.pdf)]
5. **How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG** EMNLP (2019)
*Paul Trichelair, Ali Emami, Adam Trischler, Kaheer Suleman, Jackie Chi Kit Cheung* [[pdf](https://aclanthology.org/D19-1335.pdf)] [[project](https://github.com/ptrichel/How-Reasonable-are-Common-Sense-Reasoning-Tasks)]
6. **Guided Generation of Cause and Effect** IJCAI (2020)
*Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, Benjamin Van Durme* [[pdf](https://www.ijcai.org/proceedings/2020/0502.pdf)] [[project](http://nlp.jhu.edu/causalbank)]
7. **Contrastive Self-Supervised Learning for Commonsense Reasoning** ACL (2020)
*Tassilo Klein, Moin Nabi* [[pdf](https://aclanthology.org/2020.acl-main.671.pdf)] [[project](https://github.com/SAP-samples/acl2020-commonsense/)]
8. **Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning** ACL (2020)
*Alexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot, Louise Naudin* [[pdf](https://aclanthology.org/2020.acl-main.357.pdf)]
9. **Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncoder** ACL (2020)
*Daya Guo, Duyu Tang, Nan Duan, Jian Yin, Daxin Jiang, Ming Zhou* [[pdf](https://aclanthology.org/2020.acl-main.544.pdf)] [[project](https://github.com/microsoft/EA-VQ-VAE)]
10. **Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering** EMNLP (2020)
*Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, Xiang Ren* [[pdf](https://aclanthology.org/2020.emnlp-main.99.pdf)] [[project](https://github.com/INK-USC/MHGRN)]
11. **Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning** EMNLP (2020)
*Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D. Hwang, Ronan Le Bras, Antoine Bosselut, Yejin Choi* [[pdf](https://aclanthology.org/2020.emnlp-main.58.pdf)] [[project](https://github.com/qkaren/unsup_gen_for_cms_reasoning)]
12. **Self-Supervised Knowledge Triplet Learning for Zero-Shot Question Answering** EMNLP (2020)
*Pratyay Banerjee, Chitta Baral* [[pdf](https://aclanthology.org/2020.emnlp-main.11.pdf)]
13. **Unsupervised Commonsense Question Answering with Self-Talk** EMNLP (2020)
*Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi* [[pdf](https://aclanthology.org/2020.emnlp-main.373.pdf)]
14. **Paragraph-level Commonsense Transformers with Recurrent Memory** AAAI (2021)
*Saadia Gabriel, Chandra Bhagavatula, Vered Shwartz, Ronan Le Bras, Maxwell Forbes, Yejin Choi* [[pdf](https://arxiv.org/pdf/2010.01486.pdf)] [[project](https://github.com/skgabriel/paracomet)]
15. **Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering** AAAI (2021)
*Kaixin Ma, Filip Ilievski, Jonathan Francis, Yonatan Bisk, Eric Nyberg, Alessandro Oltramari* [[pdf](https://arxiv.org/pdf/2011.03863.pdf)] [[project](https://github.com/Mayer123/HyKAS-CSKG)]
16. **Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models** ACL (2021)
*Peter West, Ximing Lu, Ari Holtzman, Chandra Bhagavatula, Jena D. Hwang, Yejin Choi* [[pdf](https://aclanthology.org/2021.acl-long.114.pdf)] [[project](https://homes.cs.washington.edu/~pawest/ReflectiveDecoding.html)]
17. **Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models** ACL (2021)
*Mingyue Han, Yinglin Wang* [[pdf](https://aclanthology.org/2021.acl-short.20.pdf)]
18. **Learning Event Graph Knowledge for Abductive Reasoning** ACL (2021)
*Li Du, Xiao Ding, Ting Liu, Bing Qin* [[pdf](https://aclanthology.org/2021.acl-long.403.pdf)] [[project](https://github.com/sjcfr/ege-RoBERTa)]
19. **ExCAR: Event Graph Knowledge Enhanced Explainable Causal Reasoning** ACL (2021)
*Li Du, Xiao Ding, Kai Xiong, Ting Liu, Bing Qin* [[pdf](https://aclanthology.org/2021.acl-long.183.pdf)] [[project](https://github.com/sjcfr/ExCAR)]
20. **Differentiable Open-Ended Commonsense Reasoning** NAACL (2021)
*Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, William W. Cohen* [[pdf](https://aclanthology.org/2021.naacl-main.366.pdf)] [[project](https://open-csr.github.io/)]
21. **QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering** NAACL (2021)
*Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, Jure Leskovec* [[pdf](https://aclanthology.org/2021.naacl-main.45.pdf)] [[project](https://github.com/michiyasunaga/qagnn)]
22. **Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models** EMNLP (2021)
*Tassilo Klein, Moin Nabi* [[pdf](https://aclanthology.org/2021.emnlp-main.688.pdf)] [[project](https://github.com/SAP-samples/emnlp2021-contrastive-refinement)]
23. **Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models** EMNLP (2021)
*Kaixin Ma, Filip Ilievski, Jonathan Francis, Satoru Ozaki, Eric Nyberg, Alessandro Oltramari* [[pdf](https://aclanthology.org/2021.emnlp-main.445.pdf)] [[project](https://github.com/Mayer123/CS_Model_Adaptation)]
24. **Shortcutted Commonsense: Data Spuriousness in Deep Learning of Commonsense Reasoning** EMNLP (2021)
*Ruben Branco, António Branco, João António Rodrigues, João Ricardo Silva* [[pdf](https://aclanthology.org/2021.emnlp-main.113.pdf)] [[project](https://github.com/nlx-group/Shortcutted-Commonsense-Reasoning)]
25. **Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference** EMNLP findings (2021)
*Canming Huang, Weinan He, Yongmei Liu* [[pdf](https://aclanthology.org/2021.findings-emnlp.420.pdf)] [[project](https://github.com/sysuhcm/NLI-KB)]
26. **SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning** NIPS (2021)
*Aaron Chan, Jiashu Xu, Boyuan Long, Soumya Sanyal, Tanishq Gupta, Xiang Ren* [[pdf](https://proceedings.neurips.cc/paper/2021/file/9752d873fa71c19dc602bf2a0696f9b5-Paper.pdf)] [[project](https://github.com/INK-USC/SalKG)]
27. **GreaseLM: Graph REASoning Enhanced Language Models** ICLR Spotlight (2022)
*Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, Jure Leskovec* [[pdf](https://openreview.net/pdf?id=41e9o6cQPj)] [[project](https://github.com/snap-stanford/GreaseLM)]
28. **Generated Knowledge Prompting for Commonsense Reasoning** ACL (2022)
*Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, Hannaneh Hajishirzi* [[pdf](https://aclanthology.org/2022.acl-long.225.pdf)] [[project](https://github.com/liujch1998/GKP)]
29. **JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering** NAACL (2022)
*Yueqing Sun, Qi Shi, Le Qi, Yu Zhang* [[pdf](https://aclanthology.org/2022.naacl-main.372.pdf)] [[project](https://github.com/Yueqing-Sun/JointLK)]
30. **Modularized Transfer Learning with Multiple Knowledge Graphs for Zero-shot Commonsense Reasoning** NAACL (2022)
*Yu Jin Kim, Beong-woo Kwak, Youngwook Kim, Reinald Kim Amplayo, Seung-won Hwang, Jinyoung Yeo* [[pdf](https://aclanthology.org/2022.naacl-main.163.pdf)]
31. **On Curriculum Learning for Commonsense Reasoning** NAACL (2022)
*Yu Jin Kim, Beong-woo Kwak, Youngwook Kim, Reinald Kim Amplayo, Seung-won Hwang, Jinyoung Yeo* [[pdf](https://aclanthology.org/2022.naacl-main.72.pdf)] [[project](https://github.com/adymaharana/curriculum_learning)]
32. **Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference** NAACL (2022)
*Emils Kadikis, Vaibhav Srivastav, Roman Klinger* [[pdf](https://aclanthology.org/2022.naacl-main.441.pdf)] [[project](https://github.com/Vaibhavs10/anli-performance-prediction)]
33. **Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge** NAACL (2022)
*Ian Porada, Alessandro Sordoni, Jackie Chi Kit Cheung* [[pdf](https://aclanthology.org/2022.naacl-main.337.pdf)]
34. **ROCK: Causal Inference Principles for Reasoning about Commonsense Causality** ICML (2022)
*Jiayao Zhang, Hongming Zhang, Weijie J. Su, Dan Roth* [[pdf](https://proceedings.mlr.press/v162/zhang22am/zhang22am.pdf)] [[project](https://github.com/zjiayao/ccr_rock)]
35. **ALERT: Adapting Language Models to Reasoning Tasks** arXiv (2022)
*Ping Yu, Tianlu Wang, Olga Golovneva, Badr AlKhamissy, Gargi Ghosh, Mona T. Diab, Asli Celikyilmaz* [[pdf](https://arxiv.org/pdf/2212.08286.pdf)]
36. **Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations** EMNLP (2022)
*Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, Yejin Choi* [[pdf](https://arxiv.org/pdf/2205.11822.pdf)] [[project](https://github.com/jaehunjung1/)]
37. **Using Commonsense Knowledge to Answer Why-Questions** EMNLP (2022)
*Yash Kumar Lal, Niket Tandon, Tanvi Aggarwal, Horace Liu, Nathanael Chambers, Raymond J. Mooney, Niranjan Balasubramanian* [[pdf](https://www.cs.utexas.edu/users/ml/papers/lal.emnlp22.pdf)] [[project](https://github.com/StonyBrookNLP/knowwhy)]
Knowledge Bases
KB
Type of Knowledge
Format of Knowledge
Link
CYC
generic
LISP-style logic
paper
project
ConceptNet
linguistics
triple
paper
project
ConceptNet 5.5
linguistics
triple
paper
project
GenericsKB
generic
statement
paper
project
Event2Mind
mental state
statement
paper
project
ATOMIC
social causality
statement
paper
project
ATOMIC 2020
+physical and eventive causality
statement
paper
project
Social-Chem-101
rules-of-thumb
statement
paper
project
Complex Reasoning
There are some datasets collected from realistic examinations or tests or explicitly designed to challenge LLMs, which may require domain-specific knowledge and multiple types of reasoning skills.
Datasets & Benchmarks
Realistic Examinations
Dataset
Size
Domain
Source
Task
Link
Remark
AR-LSAT
2k
law
law school admission test
multi-choice QA
paper
project
-
HEAD-QA
6.7k
healthcare
specialized healthcare examination
multi-choice QA
paper
project
-
AI2-ARC
7.7k
science
grade-school standardized test
multi-choice QA
paper
project
-
EntailmentBank
2k
-
-
entailment tree generation
paper
project
reasoning paths to hypotheses from AI2-ARC
ReClor
6k
generic
standardized graduate admission examination
RC + multi-choice QA
paper
project
-
MetaLogic
1k
-
-
logic metagraph generation
paper
project
reasoning graphs for passages in ReClor
LogiQA
8k
generic
national civil servants examination of China
RC + multi-choice QA
paper
project
-
ConTRoL
8k
generic
competitive selection and recruitment test
NLI
paper
project
passage-level
Diagnostic Benchmarks for LLMs
Benchmark
Tasks
Link
Remark
BIG-Bench
204
paper
project
believed to be beyond the capabilities of current PLMs
BBH
23
paper
project
challenging BIG-Bench tasks
MMLU
57
paper
project
across a diverse set of subjects that humans learn
## Knowledge Graph Reasoning
**Knowledge graph completion** task aims to complete the graph, while **multi-hop reasoning over KG** is the task querying in incomplete graphs, both of which require reasoning over knowledge graphs. **Temporal knowledge graph reasoning** aims to predict links in future with the past quadruples.
### Knowledge Graph Completion
1. **Collaborative Policy Learning for Open Knowledge Graph Reasoning** EMNLP (2019)
*Cong Fu, Tong Chen, Meng Qu, Woojeong Jin, Xiang Ren* [[pdf](http://aclanthology.lst.uni-saarland.de/D19-1269.pdf)] [[project](https://github.com/shanzhenren/CPL)]
2. **DIVINE: A Generative Adversarial Imitation Learning Framework for Knowledge Graph Reasoning** EMNLP (2019)
*Ruiping Li, Xiang Cheng* [[pdf](https://aclanthology.org/D19-1266.pdf)] [[project](https://github.com/BUPT-Data-Intelligence-Lab/DIVINE)]
3. **Learning Collaborative Agents with Rule Guidance for Knowledge Graph Reasoning** EMNLP (2020)
*Deren Lei, Gangrong Jiang, Xiaotao Gu, Kexuan Sun, Yuning Mao, Xiang Ren* [[pdf](https://aclanthology.org/2020.emnlp-main.688.pdf)] [[project](https://github.com/derenlei/KG-RuleGuider)]
4. **Incorporating Graph Attention Mechanism into Knowledge Graph Reasoning Based on Deep Reinforcement Learning** EMNLP (2019)
*Heng Wang, Shuangyin Li, Rong Pan, Mingzhi Mao* [[pdf](https://aclanthology.org/D19-1264.pdf)] [[project](https://aclanthology.org/attachments/D19-1264.Attachment.zip)]
5. **Dynamically Pruned Message Passing Networks for Large-scale Knowledge Graph Reasoning** ICLR Poster (2020)
*Xiaoran Xu, Wei Feng, Yunsheng Jiang, Xiaohui Xie, Zhiqing Sun, Zhi-Hong Deng* [[pdf](https://openreview.net/pdf?id=rkeuAhVKvB)] [[project](https://github.com/netpaladinx/DPMPN)]
6. **Inductive Relation Prediction by Subgraph Reasoning** ICML (2020)
*Komal K. Teru, Etienne G. Denis, William L. Hamilton* [[pdf](https://proceedings.mlr.press/v119/teru20a/teru20a.pdf)] [[project](https://github.com/kkteru/grail)]
7. **Adapting Meta Knowledge Graph Information for Multi-Hop Reasoning over Few-Shot Relations** EMNLP (2019)
*Xin Lv, Yuxian Gu, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu* [[pdf](https://aclanthology.org/D19-1334.pdf)] [[project](https://github.com/THU-KEG/MetaKGR)]
8. **Dynamic Anticipation and Completion for Multi-Hop Reasoning over Sparse Knowledge Graph** EMNLP (2020)
*Xin Lv, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Wei Zhang, Yichi Zhang, Hao Kong, Suhui Wu* [[pdf](https://aclanthology.org/2020.emnlp-main.459.pdf)] [[project](https://github.com/THU-KEG/DacKGR)]
9. **UniKER: A Unified Framework for Combining Embedding and Definite Horn Rule Reasoning for Knowledge Graph Inference** EMNLP (2021)
*Kewei Cheng, Ziqing Yang, Ming Zhang, Yizhou Sun* [[pdf](https://aclanthology.org/2021.emnlp-main.769.pdf)]
10. **Is Multi-Hop Reasoning Really Explainable? Towards Benchmarking Reasoning Interpretability** EMNLP (2021)
*Xin Lv, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Yichi Zhang, Zelin Dai* [[pdf](https://aclanthology.org/2021.emnlp-main.700.pdf)] [[project](https://github.com/THU-KEG/BIMR)]
11. **GMH: A General Multi-hop Reasoning Model for KG Completion** EMNLP (2021)
*Yao Zhang, Hongru Liang, Adam Jatowt, Wenqiang Lei, Xin Wei, Ning Jiang, Zhenglu Yang* [[pdf](https://aclanthology.org/2021.emnlp-main.276.pdf)]
12. **Neural-Symbolic Commonsense Reasoner with Relation Predictors** ACL (2021)
*Farhad Moghimifar, Lizhen Qu, Yue Zhuo, Gholamreza Haffari, Mahsa Baktashmotlagh* [[pdf](https://aclanthology.org/2021.acl-short.100.pdf)] [[project](https://github.com/farhadmfar/commonsense_reasoner)]
### Multi-Hop Reasoning over KG
1. **Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box Embeddings** ICLR Poster (2020)
*Hongyu Ren, Weihua Hu, Jure Leskovec* [[pdf](https://openreview.net/pdf?id=BJgr4kSFDS)] [[project](http://snap.stanford.edu/query2box)]
2. **Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs** NIPS (2020)
*Hongyu Ren, Jure Leskovec* [[pdf](https://papers.nips.cc/paper/2020/file/e43739bba7cdb577e9e3e4e42447f5a5-Paper.pdf)] [[project](http://snap.stanford.edu/betae)]
3. **Probabilistic Entity Representation Model for Reasoning over Knowledge Graphs** NIPS (2021)
*Nurendra Choudhary, Nikhil Rao, Sumeet Katariya, Karthik Subbian, Chandan K. Reddy* [[pdf](https://papers.nips.cc/paper/2021/file/c4d2ce3f3ebb5393a77c33c0cd95dc93-Paper.pdf)] [[project](https://github.com/Akirato/PERM-GaussianKG)]
4. **ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs** NIPS (2021)
*Zhanqiu Zhang, Jie Wang, Jiajun Chen, Shuiwang Ji, Feng Wu* [[pdf](https://papers.nips.cc/paper/2021/file/a0160709701140704575d499c997b6ca-Paper.pdf)] [[project](https://github.com/MIRALab-USTC/QE-ConE)]
5. **Complex Query Answering with Neural Link Predictors** ICLR Oral (2021)
*Erik Arakelyan, Daniel Daza, Pasquale Minervini, Michael Cochez* [[pdf](https://openreview.net/pdf?id=Mos9F9kDwkz)] [[project](https://github.com/uclnlp/cqd)]
### Temporal Knowledge Graph Reasoning
1. **Explainable Subgraph Reasoning for Forecasting on Temporal Knowledge Graphs** ICLR Poster (2021)
*Zhen Han, Peng Chen, Yunpu Ma, Volker Tresp* [[pdf](https://openreview.net/pdf?id=pGIHq1m7PU)] [[project](https://github.com/TemporalKGTeam/xERTE)]
2. **Search from History and Reason for Future: Two-stage Reasoning on Temporal Knowledge Graphs** ACL (2021)
*Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng Guo, Yuanzhuo Wang, Xueqi Cheng* [[pdf](https://aclanthology.org/2021.acl-long.365.pdf)]
3. **Complex Evolutional Pattern Learning for Temporal Knowledge Graph Reasoning** ACL (2022)
*Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu, Yong Zhu, Long Bai, Wei Li, Jiafeng Guo, Xueqi Cheng* [[pdf](https://aclanthology.org/2022.acl-short.32.pdf)] [[project](https://github.com/Lee-zix/CEN)]
### Others
1. **Quantum Embedding of Knowledge for Reasoning** NIPS (2019)
*Dinesh Garg, Shajith Ikbal, Santosh K. Srivastava, Harit Vishwakarma, Hima P. Karanam, L. Venkata Subramaniam* [[pdf](https://papers.nips.cc/paper/2019/file/cb12d7f933e7d102c52231bf62b8a678-Paper.pdf)] [[project](https://github.com/IBM/e2r)]
2. **Scalable Neural Methods for Reasoning With a Symbolic Knowledge Base** ICLR Poster (2020)
*William W. Cohen, Haitian Sun, R. Alex Hofer, Matthew Siegler* [[pdf](https://openreview.net/pdf?id=BJlguT4YPr)]
3. **Probabilistic Logic Neural Networks for Reasoning** NIPS (2019)
*Meng Qu, Jian Tang* [[pdf](https://papers.nips.cc/paper/2019/file/13e5ebb0fa112fe1b31a1067962d74a7-Paper.pdf)]
4. **RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs** ICLR Poster (2021)
*Meng Qu, Junkun Chen, Louis-Pascal A. C. Xhonneux, Yoshua Bengio, Jian Tang* [[pdf](https://openreview.net/pdf?id=tGZu6DlbreV)] [[project](https://github.com/DeepGraphLearning/RNNLogic)]
5. **Efficient Probabilistic Logic Reasoning with Graph Neural Networks** ICLR Poster (2020)
*Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, Le Song* [[pdf](https://openreview.net/pdf?id=rJg76kStwH)]
6. **Probabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning** NAACL (2021)
*Xuelu Chen, Michael Boratko, Muhao Chen, Shib Sankar Dasgupta, Xiang Lorraine Li, Andrew McCallum* [[pdf](https://aclanthology.org/2021.naacl-main.68.pdf)] [[project](https://github.com/stasl0217/beurre)]
7. **Multimodal Analogical Reasoning over Knowledge Graphs** ICLR (2023)
*Ningyu Zhang, Lei Li, Xiang Chen, Xiaozhuan Liang, Shumin Deng, Huajun Chen* [[pdf](https://openreview.net/pdf?id=NRHajbzg8y0P)] [[project](https://github.com/zjunlp/MKG_Analogy)]
## Mathematical Reasoning
### Benchmarks & Datasets
1. **Analysing Mathematical Reasoning Abilities of Neural Models** ICLR Poster (2019)
*David Saxton, Edward Grefenstette, Felix Hill, Pushmeet Kohli* [[pdf](https://openreview.net/pdf?id=H1gR5iR5FX)] [[project](https://github.com/deepmind/mathematics_dataset)]
2. **HOList: An Environment for Machine Learning of Higher-Order Theorem Proving** ICML (2019)
*Kshitij Bansal, Sarah M. Loos, Markus N. Rabe, Christian Szegedy, Stewart Wilcox* [[pdf](http://proceedings.mlr.press/v97/bansal19a/bansal19a.pdf)] [[project](http://deephol.org/t)]
3. **DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs** EMNLP (2019)
*Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner* [[pdf](https://aclanthology.org/N19-1246.pdf)] [[project](https://allennlp.org/drop)]
4. **IsarStep: a Benchmark for High-level Mathematical Reasoning** ICLR Poster (2021)
*Wenda Li, Lei Yu, Yuhuai Wu, Lawrence C. Paulson* [[pdf](https://openreview.net/pdf?id=Pzj6fzU6wkj)] [[project](https://github.com/Wenda302/IsarStep)]
5. **Towards Table-to-Text Generation with Numerical Reasoning** ACL (2021)
*Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, Hiroya Takamura* [[pdf](https://aclanthology.org/2021.acl-long.115.pdf)] [[project](https://github.com/titech-nlp/numeric-nlg)]
6. **Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning** ACL (2021)
*Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, Song-Chun Zhu* [[pdf](https://aclanthology.org/2021.acl-long.528.pdf)] [[project](https://lupantech.github.io/inter-gps)]
7. **FINQA: A Dataset of Numerical Reasoning over Financial Data** EMNLP (2021)
*Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R. Routledge, William Yang Wang* [[pdf](https://aclanthology.org/2021.emnlp-main.300.pdf)] [[project](https://github.com/czyssrs/FinQA)]
8. **SciGen: a Dataset for Reasoning-Aware Text Generation from Scientific Tables** NIPS (2021)
*Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, Iryna Gurevych* [[pdf](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/149e9677a5989fd342ae44213df68868-Paper-round2.pdf)] [[project](https://github.com/UKPLab/SciGen)]
9. **MULTIHIERTT: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data** ACL (2022)
*Yilun Zhao, Yunxiang Li, Chenying Li, Rui Zhang* [[pdf](https://aclanthology.org/2022.acl-long.454.pdf)] [[project](https://github.com/psunlpgroup/MultiHiertt)]
10. **NUMGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks** ACL (2022)
*Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Singh Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan* [[pdf](https://aclanthology.org/2022.acl-long.246.pdf)] [[project](https://allenai.org/data/numglue)]
### Papers
1. **Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems** NAACL (2019)
*Ting-Rui Chiang, Yun-Nung Chen* [[pdf](https://aclanthology.org/N19-1272.pdf)] [[project](https://github/MiuLab/E2EMathSolver)]
2. **A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning** EMNLP (2019)
*Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li* [[pdf](https://aclanthology.org/D19-1170.pdf)] [[project](https://github.com/huminghao16/MTMSN)]
3. **NumNet: Machine Reading Comprehension with Numerical Reasoning** EMNLP (2019)
*Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, Zhiyuan Liu* [[pdf](https://aclanthology.org/D19-1251.pdf)] [[project](https://github.com/ranqiu92/NumNet)]
4. **Mathematical Reasoning in Latent Space** ICLR Oral (2020)
*Dennis Lee, Christian Szegedy, Markus N. Rabe, Sarah M. Loos, Kshitij Bansal* [[pdf](https://openreview.net/pdf?id=Ske31kBtPr)]
5. **Neural Module Networks for Reasoning over Text** ICLR Poster (2020)
*Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, Matt Gardner* [[pdf](https://openreview.net/pdf?id=SygWvAVFPr)] [[project](http://cogcomp.org/page/publication_view/899)]
6. **Injecting Numerical Reasoning Skills into Language Models** ACL (2020)
*Mor Geva, Ankit Gupta, Jonathan Berant* [[pdf](https://aclanthology.org/2020.acl-main.89.pdf)] [[project](https://github.com/ag1988/injecting_numeracy)]
7. **Question Directed Graph Attention Network for Numerical Reasoning over Text** EMNLP (2020)
*Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, Le Song, Taifeng Wang, Yuan Qi, Wei Chu* [[pdf](https://aclanthology.org/2020.emnlp-main.549.pdf)]
8. **Mathematical Reasoning via Self-supervised Skip-tree Training** ICLR Spotlight (2021)
*Markus Norman Rabe, Dennis Lee, Kshitij Bansal, Christian Szegedy* [[pdf](https://openreview.net/pdf?id=YmqAnY0CMEy)]
9. **Incorporating External Knowledge to Enhance Tabular Reasoning** NAACL (2021)
*J. Neeraja, Vivek Gupta, Vivek Srikumar* [[pdf](https://aclanthology.org/2021.naacl-main.224.pdf)] [[project](https://github.com/utahnlp/knowledge_infotabs)]
10. **Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning** ACL (2021)
*Piotr Piekos, Mateusz Malinowski, Henryk Michalewski* [[pdf](https://aclanthology.org/2021.acl-short.49.pdf)]
11. **GraphMR: Graph Neural Network for Mathematical Reasoning** ACL (2021)
*Weijie Feng, Binbin Liu, Dongpeng Xu, Qilong Zheng, Yun Xu* [[pdf](https://aclanthology.org/2021.emnlp-main.273.pdf)] [[project](https://github.com/nhpcc502/GraphMR)]
12. **LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning** ICML (2021)
*Yuhuai Wu, Markus N. Rabe, Wenda Li, Jimmy Ba, Roger B. Grosse, Christian Szegedy* [[pdf](http://proceedings.mlr.press/v139/wu21c/wu21c.pdf)] [[project](https://github.com/tonywu95/LIME)]
13. **Numerical reasoning in machine reading comprehension tasks: are we there yet?** EMNLP (2021)
*Hadeel Al-Negheimish, Pranava Madhyastha, Alessandra Russo* [[pdf](https://aclanthology.org/2021.emnlp-main.759.pdf)]
14. **Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction** ACL (2022)
*Zhanming Jie, Jierui Li, Wei Lu* [[pdf](https://aclanthology.org/2022.acl-long.410.pdf)] [[project](https://github.com/allanj/Deductive-MWP)]
15. **FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining** ACL (2022)
*Zhoujun Cheng, Haoyu Dong, Ran Jia, Pengfei Wu, Shi Han, Fan Cheng, Dongmei Zhang* [[pdf](https://aclanthology.org/2022.acl-long.82.pdf)] [[project](https://github.com/microsoft/TUTA_table_understanding)]
16. **Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning** ACL (2022)
*Vivek Gupta, Shuo Zhang, Alakananda Vempala, Yujie He, Temma Choji, Vivek Srikumar* [[pdf](https://aclanthology.org/2022.acl-long.231.pdf)] [[project](https://tabevidence.github.io/)]
17. **Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills** ACL (2022)
*Ori Yoran, Alon Talmor, Jonathan Berant* [[pdf](https://aclanthology.org/2022.acl-long.416.pdf)] [[project](https://github.com/oriyor/turning_tables)]
18. **OPERA: Operation-Pivoted Discrete Reasoning over Text** NAACL (2022)
*Yongwei Zhou, Junwei Bao, Chaoqun Duan, Haipeng Sun, Jiahui Liang, Yifan Wang, Jing Zhao, Youzheng Wu, Xiaodong He, Tiejun Zhao* [[pdf](https://aclanthology.org/2022.naacl-main.119.pdf)] [[project](https://github.com/JD-AI-Research-NLP/OPERA)]
## Contributor
Fei YU
## Reference
```bibtex
@article{yu2023natural,
title={Natural Language Reasoning, A Survey},
author={Yu, Fei and Zhang, Hongbo and Wang, Benyou},
journal={arXiv preprint arXiv:2303.14725},
year={2023}
}
```