https://github.com/zhijing-jin/arts_testset

This is our ARTS test set, an enriched test set to probe Aspect Robustness of ABSA.
https://github.com/zhijing-jin/arts_testset

Last synced: 2 months ago
JSON representation

This is our ARTS test set, an enriched test set to probe Aspect Robustness of ABSA.

• Host: GitHub
• URL: https://github.com/zhijing-jin/arts_testset
• Owner: zhijing-jin
• Created: 2020-06-14T20:06:07.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2024-01-16T17:23:22.000Z (about 1 year ago)
• Last Synced: 2024-01-17T01:58:25.370Z (about 1 year ago)
• Language: Python
• Size: 4.09 MB
• Stars: 41
• Watchers: 3
• Forks: 4
• Open Issues: 4
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
This is the repository for our paper 

"[**Tasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysis**](https://arxiv.org/pdf/2009.07964)" **(EMNLP 2020)**.

Authors: [Xiaoyu Xing](https://scholar.google.ru/citations?user=gC7UghIAAAAJ&hl=en)\*, [Zhijing Jin](https://zhijing-jin.com)\*, [Di Jin](https://jind11.github.io/), [Bingning Wang](https://bingning.wang/research/aboutme), [Qi Zhang](http://qizhang.info/), and [Xuanjing Huang](https://xuanjing-huang.github.io/).

## Data

We provide a **Aspect Robustness Probing** test set for [SemEval 2014](http://alt.qcri.org/semeval2014/task4/) Aspect-Based Sentiment Analysis (ABSA).

- Our new enriched test sets are at [data/arts_testset](data/arts_testset/)

- Our `AspectSet` mentioned in the paper Section 2.3 (Table 4) is provided in [data/aspectset](data/aspectset/)

### Data Generation Process

We generate our new probing test set by **three automatic strategies**: 

- RevTgt (sentence with a red background): Reverse the sentiment of the Target aspect.

- RevNon (sentence with a green background): Reverse the sentiment of the Non-target aspect.

- AddDiff (sentence with a blue background): Add new aspects with Different sentiment.



### Aspect Probing Results

We probed nine ABSA models (as mentioned in our paper). 

- Their outputs on SemEval 2014 are in the [output](output) folder.

## How to Use Our Code

If you have a **new** ABSA dataset, you can run our code to generate you own **aspect robustness probing** test set.

```

python code/main.py -dataset_name laptop

```

### Dependencies

- Version of allennlp package: You can install allennlp-2.5.0 with the Predictor https://s3-us-west-2.amazonaws.com/allennlp/models/elmo-constituency-parser-2020.02.10.tar.gz

### All Trained Models

If needed, see a dump of all the trained models and output files [here](https://edmond.mpg.de/dataset.xhtml?persistentId=doi%3A10.17617%2F3.VKWOI9).

## More Questions

If you have more questions, please feel free to submit a [GitHub issue]([issues](https://github.com/zhijing-jin/ARTS_testset/issues)https://github.com/zhijing-jin/ARTS_testset/issues).