https://github.com/shammur/arabic_offensive_language_detection
arabic offensive language detection model from social media comments and posts
https://github.com/shammur/arabic_offensive_language_detection
Last synced: 2 months ago
JSON representation
arabic offensive language detection model from social media comments and posts
- Host: GitHub
- URL: https://github.com/shammur/arabic_offensive_language_detection
- Owner: shammur
- License: apache-2.0
- Created: 2020-02-12T19:42:18.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2020-02-16T22:05:48.000Z (over 5 years ago)
- Last Synced: 2025-03-24T06:03:58.501Z (3 months ago)
- Language: Jupyter Notebook
- Size: 3.74 MB
- Stars: 4
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Arabic Offensive Language Detection model from social media posts/comments.
This release includes model for offensive language detection for Arabic social media posts, trained using comments from different online platforms and tweets.
## Data Annotation
To train the model, we annotated ~5000 amount of data consists of 2 categories: offensive (OFF) and not offensive (NOT_OFF).
The contents are collected from the following sources:
* Twitter
* Youtube
* Facebook
The annotation of the collected dataset is obtained using Amazon Mechanical Turk (AMT). To ensure the quality of the annotation and language proficiency, we utilized two different evaluation criteria of the annotator. For more details, check the below paper:
Will be available in the proceedings of LREC2020:
```
@inproceedings{shammur2020offensive,
title={A Multi-Platform Arabic News Comment Dataset for Offensive Language Detection},
author={Chowdhury, Shammur Absar and Mubarak, Hamdy and Abdelali, Ahmed and Jung, Soon-gyo and Jansen, Bernard J and Salminen, Joni},
booktitle={Proceedings of the International Conference on Language Resources and Evaluation (LREC'20)},
year={2020}
}
```
In addition to the dataset mentioned in the above paper, we also added 948 data points along with 7000 annotated tweet from the training data of OSACT.
## Model training and evaluation
The model use a traditional SVM designed using character ngrams. The motivation for using Support Vector model is to handle the skewneess present in the dataset (see Table 1, for more details). The model is evaluated using:
* 5-fold cross validation for evaluating in-domain data performance
* Official dev set for OSACT Offensive Language detection competition
### SVM
For the training the classifier with SVM, we used TF-IDF representations for character ngrams (1,8). The reason to choose SVM with TF-IDF is their simplicity and execution time while having comparable performance for such dataset nature.
## Data Format
### Input data format
The input file should have the following fields, including
`\t\t`
however when the model is not used to evaluate the performance, `` is optional field.
### Output data format
The output of the file will include the following fields
* While running the model just for prediction:
`\t\t`
* Output of the model when reference label is mentioned
`\t\t\t`
here predicted_class_label is the output of the model (OFF/NOT_OFF)
## Prediction using the models
To run the classification model please use python version 3.7, install dependencies
To install the requirements:
```
pip install -r requirements.txt
```
The model can be used in two ways, either using batch of data or single data points. Even though for single datapoint the batch processing script can be used, we suggest to use the example provided in `run_ar_offensive_language_detection_models_for_single_text.ipynb`
For batch classification of data:
```
python bin/prediction_model.py -c models/ar_offensive_detection_svm.config -d sample_data/test_instances.tsv -o results/test_instances_batch.tsv
```
For evaluation of batch with reference label, just add
the following flag to `prediction_model.py`
```
--eval yes
```
The results of the model on the given dataset will be printed in the i/o
Example:
```
python bin/prediction_model.py.py -c models/ar_offensive_detection_svm.config -d sample_data/dataset_format_with_ref_labels.tsv -o results/dataset_with_reflab.tsv --eval yes &> logs/result_dataset_with_reflab.log
```
## Classification Results
As mentioned earlier, the performance of the model is tested using 1) 5-fold CV on training data 2) official dev set for OSACT (LREC2020) along with other 3? out of domain data including:
Test Data Sets | F1 OFF | Macro F1 | Weighted F1
-------------------------| :------: | :------: | :------:
5-fold CV | **0.73** | **0.83** | **0.88**
OSACT_2020_Shared_dev_set| **0.74** | **0.84** | **0.91**