{"id":13754319,"url":"https://github.com/doc2dial/sharedtask-dialdoc2021","last_synced_at":"2025-05-09T22:31:56.924Z","repository":{"id":45097627,"uuid":"334679772","full_name":"doc2dial/sharedtask-dialdoc2021","owner":"doc2dial","description":"doc2dial data includes a set of documents from multiple domains; and conversations between an assisting agent and an end user that are grounded in the associated documents.","archived":false,"fork":false,"pushed_at":"2022-01-08T03:15:11.000Z","size":5783,"stargazers_count":40,"open_issues_count":3,"forks_count":12,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-16T07:33:23.546Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/doc2dial.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-31T14:50:48.000Z","updated_at":"2023-12-24T13:26:19.000Z","dependencies_parsed_at":"2022-09-16T13:47:07.985Z","dependency_job_id":null,"html_url":"https://github.com/doc2dial/sharedtask-dialdoc2021","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doc2dial%2Fsharedtask-dialdoc2021","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doc2dial%2Fsharedtask-dialdoc2021/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doc2dial%2Fsharedtask-dialdoc2021/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doc2dial%2Fsharedtask-dialdoc2021/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/doc2dial","download_url":"https://codeload.github.com/doc2dial/sharedtask-dialdoc2021/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253335811,"owners_count":21892739,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T09:01:54.624Z","updated_at":"2025-05-09T22:31:51.876Z","avatar_url":"https://github.com/doc2dial.png","language":"Python","funding_links":[],"categories":["NLP语料和数据集"],"sub_categories":["大语言对话模型及数据"],"readme":"# DialDoc21: Shared Task on Doc2Dial Dataset\n\n[DialDoc21 Shared Task](https://doc2dial.github.io/workshop2021/shared.html) at [ACL 2021](https://2021.aclweb.org) includes two subtasks for building goal-oriented document-grounded dialogue systems. The first subtask is to predict the grounding in the given document for next agent response; the second subtask is to generate agent response in natural language given document-based and dialogue-based contexts.\n\n## Data\n\nThis shared task is based on Doc2Dial v1.0.1 in folder [`data/doc2dial`](data/doc2dial). For more information about the dataset, please refer to [README](data/doc2dial/DATA_README.md), [paper](https://arxiv.org/abs/2011.06623) and [Doc2Dial Project Page](https://doc2dial.github.io/).\n\n**Note**: you can choose to utilize other public datasets in addition to Doc2Dial data for training. Please examples [here](https://mrqa.github.io/2019/shared).\n\n## Shared Task\n\n### Subtask 1\n\nThe task is to predict the knowledge grounding in form of document span for the next agent response given dialogue history and the associated document.\n\n- **Input**: the associated document and dialogue history.\n\n- **Output**: the grounding text.\n\n- **Evaluation**: `exact match` and `F1 scores`. Please refer to [script](scripts/sharedtask_utils.py) for more details.\n\n### Subtask 2\n\nThe task is to generate the next agent response in natural language given dialogue-based and document-based contexts.\n\n- **Input**: the associated document and dialogue history.\n\n- **Output**: dialog utterance.\n\n- **Evaluation**: `sacrebleu` and `human evaluation`. Please refer to [script](scripts/sharedtask_utils.py) for more details. Stay tuned for more details about human evaluations.\n\n\n## Baselines\n\n### **Environment Setup**\n\nCreate a virtual environment\n\n```bash\nconda create -n ENV_NAME python=3.7\nconda activate ENV_NAME\n````\n\nInstall PyTorch\n\n```bash\nconda install pytorch cudatoolkit=10.2 -c pytorch\n```\n\nInstall Huggingface Transformers, Datasets and a few more dependencies\n\n```bash\npip install -r requirements.txt\n```\n\nInstall NVIDIA/apex\n\n```bash\nconda install -c conda-forge nvidia-apex \n```\n\n### **Load Dataset**\n\nYou can use [Huggingface Dataset](https://huggingface.co/docs/datasets/loading_datasets.html) to load Doc2Dial datasets. The latest [source code](https://github.com/huggingface/datasets/tree/master/datasets/doc2dial) includes the code for loading Doc2Dial v1.0.1.  \n\nThe [script](scripts/sharedtask_utils.py) shows how to obtain the ground truth of the given IDs for evaluations of Subtask 1 and Subtask 2. IDs are `{dial_id}_{turn_id}`, where `turn_id` is of the turn right before the next agent turn for grounding prediction (Subtask 1) or generation (Subtask 2). For the withheld test set for the [challenge](https://eval.ai/web/challenges/challenge-page/793/overview), the data was collected in the same process as training and validation sets; the ground truth would be obtained the same way as in the [script](scripts/sharedtask_utils.py).\n\n### **Run Baseline for Subtask 1**\n\n\u003e Run [HuggingFace QA](https://github.com/huggingface/transformers/tree/master/examples/pytorch/question-answering) on Doc2Dial\n\n- For fine-tuning Bert on Doc2Dial,\n\n    ```bash\n    cd sharedtask-dialdoc2021/scripts/subtask1\n    ./run_qa.sh\n    ```\n\n- Results on validation set:\n\n    ```bash\n    # bert-base-uncased\n    f1 = 56.29 \n    exact_match = 39.73\n    # bert-large-uncased-whole-word-masking\n    f1 = 62.98\n    exact_match = 50.50\n    ```\n\n\u003e Evaluating your model output\n\n- Output format and sample file\n\n    Please see the format in [sample file](scripts/sample_files/sample_prediction_subtask1.json).\n\n- Evaluation script\n\n    Please refer to [`script`](scripts/sharedtask_utils.py) for evaluating your model predictions.\n\n    ```bash\n    python sharedtask_utils.py --task subtask1 --prediction_json sample_files/sample_prediction_subtask1.json\n    ```\n\n### **Run Baseline for Subtask 2**\n\n\u003e Run [HuggingFace Seq2Seq](https://github.com/huggingface/transformers/tree/master/examples/seq2seq) on Doc2Dial\n\n- For generating input files,\n\n    We first create source and target files. Please see run [script](scripts/subtask2/seq2seq_utils.py) with required parameters along with other default values.\n\n    ```bash\n    cd scripts/subtask2\n    python seq2seq_utils.py --split validation --output_dir seq2seq_files\n    ```\n\n- For fine-tuning bart on Doc2Dial,\n\n    ```bash\n    cd scripts/subtask2\n    ./run_seq2seq.sh\n    ```\n\n- Results on validation set:\n\n    ```bash\n    # bart-large-cnn\n    val_bleu = 17.72\n    ```\n\n\u003e Evaluating your model output\n\n- Output format and sample file\n\n    Please see the format in [sample file](scripts/sample_files/sample_prediction_subtask2.json).\n\n- Evaluation script\n\n    Please refer to [script](scripts/sharedtask_utils.py) for evaluating your model predictions.\n\n    ```bash\n    python sharedtask_utils.py --task subtask2 --prediction_json sample_files/sample_prediction_subtask2.json\n    ```\n\n## About Participation\n\nFor more up-to-date information about participating DialDoc21 Shared Task, please refer to our [workshop page](https://doc2dial.github.io/workshop2021/shared.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoc2dial%2Fsharedtask-dialdoc2021","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoc2dial%2Fsharedtask-dialdoc2021","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoc2dial%2Fsharedtask-dialdoc2021/lists"}