Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/liamdugan/coling-2025-workshop-on-mgt-detection-task-3

Repository for submission to the COLING GenAI Content Detection Shared Task 3: "Cross-domain Machine-Generated Text Detection"
https://github.com/liamdugan/coling-2025-workshop-on-mgt-detection-task-3

Last synced: 24 days ago
JSON representation

Repository for submission to the COLING GenAI Content Detection Shared Task 3: "Cross-domain Machine-Generated Text Detection"

Awesome Lists containing this project

README 

        
# Task 3: Cross-Domain Machine-Generated Text Detection





  




[News](#news) | [Competition](#competition) | [Dataset](#dataset) | [Important Dates](#important_dates) | [Data Format](#data_format) | [Evaluation Metrics](#scorer_and_official_evaluation_metrics) | [Baselines](#baselines) | [Contact](#contact)



When applying detectors to machine-generated text in the wild, the dominant emerging paradigm is to use an **open-domain API-based** detector. However, many commonly used detectors exhibit poor cross-domain and cross-model robustness. Thus, it is critical to train our detectors to be able to handle text from many domains with both high accuracy and low false positive rates.



In the COLING Workshop on MGT Detection Task 3, we focus on **cross-domain robustness** of detectors by testing submissions on the [RAID benchmark](https://raid-bench.xyz/). We adopt the same straightforward binary problem formulation as Task 1, that is: given a piece of text, determine whether it is generated by a machine or authored by a human. 



However, in this task the texts will **not be limited to any one domain** and may come from any one of 8 different domains, 11 generative models, and 4 decoding strategies. Your goal will be to create a detector that exhibits high levels of robustness across all of these models, domains, and decoding strategies while maintaining a low false positive rate.



Our domains are:



| Domain              | Source | Dataset Link |

| :---------------- | :------: | :----: |

| Arxiv Abstracts | [arxiv.org](https://arxiv.org) | [(Link)](https://www.kaggle.com/datasets/Cornell-University/arxiv) |

| Book Plot Summaries | [wikipedia.org](https://wikipedia.org) | [(Link)](https://paperswithcode.com/dataset/cmu-book-summary-dataset) |

| BBC News Articles |  [bbc.com/news](https://www.bbc.com/news) | [(Link)](https://github.com/derekgreene/bbc-datasets) |

| Poems |  [poemhunter.com](https://www.poemhunter.com/)   | [(Link)](https://www.kaggle.com/datasets/michaelarman/poemsdataset) |

| Reddit Posts | [reddit.com](https://www.reddit.com/) | [(Link)](https://huggingface.co/datasets/sentence-transformers/reddit-title-body) |

| Recipes | [allrecipes.com](https://www.allrecipes.com/) | [(Link)](https://recipenlg.cs.put.poznan.pl/) |

| IMDb Movie Reviews | [imdb.com](https://www.imdb.com/) | [(Link)](https://ieee-dataport.org/open-access/imdb-movie-reviews-dataset) |

| Wikipedia Articles | [wikipedia.org](https://www.wikipedia.org/) | [(Link)](https://huggingface.co/datasets/aadityaubhat/GPT-wiki-intro) |



There are two subtasks:

- Subtask A: Non-Adversarial Cross-Domain MGT detection.

- Subtask B: Adversarial Cross-Domain MGT detection.



## NEWS 



### 18 Sep 2024



We have released our instructions and training set.



### 24 Sep 2024



We have released our format checking script.



### 22 Oct 2024



The deadline for the competition has been extended to Nov 2nd 2024.



## Competition



The competition will be held on the [RAID Website](https://raid-bench.xyz/). We will be releasing a separate leaderboard specifically for the shared task that will exist alongside the main RAID leaderboard and will be populated with results after the task finishes.



To submit to the shared task, you must first get predictions for your detector on the test set. Please consult the [RAID Leaderboard Submission Instructions](https://github.com/liamdugan/raid?tab=readme-ov-file#leaderboard-submission) for more details on how to get the `predictions.json` file for your detector.



After you have the `predictions.json` file you must then write a metadata file for your submission. Your metadata file should use the template found in this repository at `submissions/template-metadata.json`.



Finally, fork this repository. Add your files to `submissions/YOUR-DETECTOR-NAME/predictions.json` and your metadata file to `submissions/YOUR-DETECTOR-NAME/metadata.json` and make a pull request to this repository. We have provided an example submission of the OpenAI Roberta Large classifier under `submissions/openai-roberta-large`.



> [!NOTE]

> Please **DO NOT SUBMIT** to the main RAID leaderboard during the duration of the shared task. If you do so, **you will be disqualified.**



## Dataset

For this task we will be using the RAID dataset.

**Download RAID** by consulting the [RAID Github Repository](https://github.com/liamdugan/raid?tab=readme-ov-file#download-raid).



## Important Dates



- 18th September, 2024: Training & test set release

- **2nd November, 2024** ~~25th October, 2024~~: Submission phase closes

- **5th November, 2024** ~~28th October, 2024~~: Leaderboard to be public

- 15th November, 2024: System description paper submission



## Prediction File Format and Format Checkers



In order to run our automatic evaluation, your submission must include a file named `predictions.json`.

This file should be valid JSON and should be of the following format:

```

[

  {"id": "64005577-3d63-4583-8945-7541d3e53e7d", "score": 0.0021110873541056},

  {"id": "c2b9df67-4e29-45ca-bdcc-7065fb907b77", "score": 0.9116235922302712},

  ...

]

```

The provided `run_detection` function from the [RAID Pypi package](https://pypi.org/project/raid-bench/) will output predictions in this format. 

If you would like to use your own code, you can run something like the below snippet to output in the correct format.

```py

with open(output_path, "w") as f:

  json.dump(df[["id", "score"]].to_dict(orient="records"), f)

```



To check your submission's correctness please run our provided format checker as follows:

```

$ python format_check.py --results_path .json

```



## Scorer and Official Evaluation Metrics



The **official evaluation metric** is **TPR @ FPR=5%**. That is, the accuracy of the model on detecting machine-generated text at a fixed false positive rate of 5%.

To calculate this, our scorer uses the model predictions on human data to search a classification threshold that results in 5% FPR for each domain.



To run the scorer, first run `pip install raid-bench` then use the RAID pip package as follows:



```py

from raid import run_detection, run_evaluation

from raid.utils import load_data



# Define your detector function

def my_detector(texts: list[str]) -> list[float]:

    pass



# Download & Load the RAID dataset

train_df = load_data(split="train")



# Run your detector on the dataset

predictions = run_detection(my_detector, train_df)



# Evaluate your detector predictions

evaluation_result = run_evaluation(predictions, train_df)

```



## Baselines



We have run a number of publicly available open-source detectors on RAID. [Binoculars](https://arxiv.org/abs/2401.12070) gets **79.0%**, [RADAR](https://huggingface.co/TrustSafeAI/RADAR-Vicuna-7B) gets **65.6%**, and [roberta-base-openai-detector](https://huggingface.co/openai-community/roberta-base-openai-detector) gets **59.2%** on the non-adversarial RAID test set.



We will also be releasing some simple baseline trained models on the RAID dataset. These will be released shortly.



## Contact



Website: [https://genai-content-detection.gitlab.io](https://genai-content-detection.gitlab.io)  

Email: [email protected] or directly to [email protected]