{"id":18950176,"url":"https://github.com/salesforce/dialfact","last_synced_at":"2025-04-16T00:31:31.911Z","repository":{"id":66001656,"uuid":"415207397","full_name":"salesforce/DialFact","owner":"salesforce","description":"We construct and introduce DIALFACT, a testing benchmark dataset crowd-annotated conversational claims, paired with pieces of evidence from Wikipedia.","archived":false,"fork":false,"pushed_at":"2022-10-12T22:09:04.000Z","size":3437,"stargazers_count":39,"open_issues_count":2,"forks_count":7,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-08T00:12:40.097Z","etag":null,"topics":["dialogue","dialogue-systems","fact-checking"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salesforce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null}},"created_at":"2021-10-09T05:03:27.000Z","updated_at":"2024-02-21T04:13:47.000Z","dependencies_parsed_at":"2023-02-23T03:15:21.545Z","dependency_job_id":null,"html_url":"https://github.com/salesforce/DialFact","commit_stats":{"total_commits":3,"total_committers":1,"mean_commits":3.0,"dds":0.0,"last_synced_commit":"d400b250147e45c106b18e52254b1060f7c1575d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2FDialFact","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2FDialFact/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2FDialFact/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2FDialFact/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salesforce","download_url":"https://codeload.github.com/salesforce/DialFact/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223690050,"owners_count":17186543,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dialogue","dialogue-systems","fact-checking"],"created_at":"2024-11-08T13:21:27.146Z","updated_at":"2024-11-08T13:21:29.796Z","avatar_url":"https://github.com/salesforce.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DialFact: A Benchmark for Fact-Checking in Dialogue\r\nAuthors: [Prakhar Gupta](https://prakharguptaz.github.io/), [Jason Wu](https://jasonwu0731.github.io/), Wenhao Liu and Caiming Xiong\r\n\r\nPaper link: https://arxiv.org/pdf/2110.08222\r\n\r\n\r\n## Abstract\r\nTo study the problem of Fact-Checking in Dialogue, we construct and introduce DIALFACT, a testing benchmark dataset crowd-annotated conversational claims, paired with pieces of evidence from Wikipedia. There are three sub-tasks in DIALFACT: 1) Verifiable claim detection task distinguishes whether a response carries verifiable factual information; 2) Evidence retrieval task retrieves the most relevant Wikipedia snippets as evidence; 3) Claim verification task predicts a dialogue response to be supported, refuted, or not enough information.\r\n\r\n\r\n\r\n------------\r\n\r\n## Dataset Details\r\n\r\nThe statistics for the Test and Validation sets are shown in the figure below. \r\n\r\n![Data stats](/images/stats.png?raw=true \"Data Stats\")\r\n\r\nThe test and validation are present in the data folder of this repo.\r\n\r\nThe synthetic training dataset for AugWoW model is present here: [data](https://drive.google.com/drive/folders/1CSciq9f3ZOvLuNk9m3aDElPVVLjpYfuv?usp=sharing)\r\n\r\n### Data format\r\nDescription of keys and values present in the dataset files:\r\n```json\r\n{\r\n    \"context_id\": \"Context ID\",\r\n    \"id\": \"Context ID --- ResponseID\",\r\n    \"data_type\": \"Type of response: generated or written\",\r\n    \"context\":\"List of utterances in dialogue history\",\r\n    \"response\": \"The claim or response\",\r\n    \"evidence_list\": \"List of evidences. Eack item in list is a list of following:\"\r\n        [\"Wikipedia page Title\",\"Wikipedia Link\",\"Test snippet shown.\",\"an index - not useful for the task\", \"optionally present value gt_evidence_added - indicates an evidence which belonged to the original utterance in WoW added for NEI claims.\" ],\r\n    \"response_label\": \"One of the three labels: SUPPORTS, REFUTES, NOT ENOUGH INFO\",\r\n    \"type_label\": \"If the response is factual (Verifiable) or personal (Non-Verifiable)\"\r\n}\r\n```\r\n\r\n------------\r\n\r\n\r\n## Results\r\n\r\nThe results for claim verification on test set. \r\n![Test Results](/images/testveri.png?raw=true \"Test Results\")\r\n\r\n\r\nThe results for claim verification on validation set. \r\n![Validation Results](/images/validationveri.png?raw=true \"Validation Results\")\r\n\r\n\r\n\r\n------------\r\n\r\n## Citation\r\n```\r\n@article{gupta2021dialfact,\r\n  title={DialFact: A Benchmark for Fact-Checking in Dialogue},\r\n  author={Gupta, Prakhar and Wu, Chien-Sheng and Liu, Wenhao and Xiong, Caiming},\r\n  journal={arXiv preprint arXiv:2110.08222},\r\n  year={2021}\r\n}\r\n```\r\n\r\n## Questions?\r\nFor any questions, feel free to open issues, or shoot emails to\r\n- Jason Wu (wu.jason@salesforce.com)\r\n- [Prakhar Gupta](https://prakharguptaz.github.io/) (CMU)\r\n\r\n## License\r\nThe code is released under BSD 3-Clause - see [LICENSE](LICENSE.txt) for details.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce%2Fdialfact","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalesforce%2Fdialfact","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce%2Fdialfact/lists"}