{"id":31126420,"url":"https://github.com/rucaibox/crslab","last_synced_at":"2025-09-17T22:48:59.520Z","repository":{"id":37702671,"uuid":"314949325","full_name":"RUCAIBox/CRSLab","owner":"RUCAIBox","description":"CRSLab is an open-source toolkit for building Conversational Recommender System (CRS).","archived":false,"fork":false,"pushed_at":"2024-04-12T07:13:48.000Z","size":667,"stargazers_count":539,"open_issues_count":12,"forks_count":116,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-09-06T16:29:17.562Z","etag":null,"topics":["conversation-system","conversational-recommendation","deep-learning","dialog-system","graph-neural-network","human-machine-interaction","knowledge-graph","pretrained-models","pytorch","recommendation","recommender-system","text-generation"],"latest_commit_sha":null,"homepage":"https://github.com/RUCAIBox/CRSLab","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RUCAIBox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-22T03:12:03.000Z","updated_at":"2025-08-19T05:19:24.000Z","dependencies_parsed_at":"2024-08-03T09:17:23.652Z","dependency_job_id":null,"html_url":"https://github.com/RUCAIBox/CRSLab","commit_stats":{"total_commits":181,"total_committers":14,"mean_commits":"12.928571428571429","dds":0.4033149171270718,"last_synced_commit":"eaf11c2874a93c8c747ee073ead7963ebc4cd74a"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/RUCAIBox/CRSLab","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCAIBox%2FCRSLab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCAIBox%2FCRSLab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCAIBox%2FCRSLab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCAIBox%2FCRSLab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RUCAIBox","download_url":"https://codeload.github.com/RUCAIBox/CRSLab/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCAIBox%2FCRSLab/sbom","scorecard":{"id":116988,"data":{"date":"2025-08-11","repo":{"name":"github.com/RUCAIBox/CRSLab","commit":"6497938919998b9cd7aa9c2f43effb39ad0c7088"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Code-Review","score":7,"reason":"Found 8/11 approved changesets -- score normalized to 7","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"36 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2021-356 / GHSA-2ww3-fxvq-293j","Warn: Project is vulnerable to: PYSEC-2024-167 / GHSA-cgvx-9447-vcch","Warn: Project is vulnerable to: PYSEC-2021-859 / GHSA-f8m6-h2c7-8h9x","Warn: Project is vulnerable to: PYSEC-2019-106 / GHSA-mr7p-25v2-35wr","Warn: Project is vulnerable to: PYSEC-2022-5 / GHSA-rqjh-jp2r-59cj","Warn: Project is vulnerable to: GHSA-6p56-wp2h-9hxr","Warn: Project is vulnerable to: GHSA-fpfv-jqm9-f5jm","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q","Warn: Project is vulnerable to: PYSEC-2024-110 / GHSA-jw8x-6495-233v","Warn: Project is vulnerable to: GHSA-jxfp-4rvq-9h9m","Warn: Project is vulnerable to: GHSA-g7vv-2v7x-gj9p","Warn: Project is vulnerable to: PYSEC-2023-299 / GHSA-282v-666c-3fvg","Warn: Project is vulnerable to: GHSA-37mw-44qp-f5jm","Warn: Project is vulnerable to: GHSA-37q5-v5qm-c9v8","Warn: Project is vulnerable to: PYSEC-2023-300 / GHSA-3863-2447-669p","Warn: Project is vulnerable to: GHSA-6rvg-6v2m-4j46","Warn: Project is vulnerable to: GHSA-9356-575x-2w9m","Warn: Project is vulnerable to: GHSA-fpwr-67px-3qhx","Warn: Project is vulnerable to: PYSEC-2024-229 / GHSA-hxxf-235m-72v3","Warn: Project is vulnerable to: GHSA-jjph-296x-mrcr","Warn: Project is vulnerable to: GHSA-phhr-52qp-3mj4","Warn: Project is vulnerable to: GHSA-q2wp-rjmx-x6x9","Warn: Project is vulnerable to: PYSEC-2025-40 / GHSA-qq3j-4f4f-9583","Warn: Project is vulnerable to: PYSEC-2024-227 / GHSA-qxrp-vhvm-j765","Warn: Project is vulnerable to: PYSEC-2023-301 / GHSA-v68g-wm8c-6x7j","Warn: Project is vulnerable to: PYSEC-2024-228 / GHSA-wrfc-pvp9-mr9g","Warn: Project is vulnerable to: GHSA-3749-ghw9-m3mg","Warn: Project is vulnerable to: PYSEC-2022-43015 / GHSA-47fc-vmwq-366v","Warn: Project is vulnerable to: PYSEC-2025-41 / GHSA-53q9-r3pm-6pq6","Warn: Project is vulnerable to: PYSEC-2024-252 / GHSA-5pcm-hx3q-hm94","Warn: Project is vulnerable to: GHSA-887c-mr87-cxwp","Warn: Project is vulnerable to: PYSEC-2024-251 / GHSA-pg7h-5qx3-wjr3","Warn: Project is vulnerable to: PYSEC-2024-250","Warn: Project is vulnerable to: PYSEC-2024-259"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 28 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-16T01:34:54.010Z","repository_id":37702671,"created_at":"2025-08-16T01:34:54.010Z","updated_at":"2025-08-16T01:34:54.010Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275680346,"owners_count":25508570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-17T02:00:09.119Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["conversation-system","conversational-recommendation","deep-learning","dialog-system","graph-neural-network","human-machine-interaction","knowledge-graph","pretrained-models","pytorch","recommendation","recommender-system","text-generation"],"created_at":"2025-09-17T22:48:57.011Z","updated_at":"2025-09-17T22:48:59.481Z","avatar_url":"https://github.com/RUCAIBox.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CRSLab\n\n[![Pypi Latest Version](https://img.shields.io/pypi/v/crslab)](https://pypi.org/project/crslab)\n[![Release](https://img.shields.io/github/v/release/rucaibox/crslab.svg)](https://github.com/rucaibox/crslab/releases)\n[![License](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)\n[![arXiv](https://img.shields.io/badge/arXiv-CRSLab-%23B21B1B)](https://arxiv.org/abs/2101.00939)\n[![Documentation Status](https://readthedocs.org/projects/crslab/badge/?version=latest)](https://crslab.readthedocs.io/en/latest/?badge=latest)\n\n[Paper](https://arxiv.org/pdf/2101.00939.pdf) | [Docs](https://crslab.readthedocs.io/en/latest/?badge=latest)\n| [中文版](./README_CN.md)\n\n**CRSLab** is an open-source toolkit for building Conversational Recommender System (CRS). It is developed based on\nPython and PyTorch. CRSLab has the following highlights:\n\n- **Comprehensive benchmark models and datasets**: We have integrated commonly-used 6 datasets and 18 models, including graph neural network and pre-training models such as R-GCN, BERT and GPT-2. We have preprocessed these datasets to support these models, and release for downloading.\n- **Extensive and standard evaluation protocols**: We support a series of widely-adopted evaluation protocols for testing and comparing different CRS.\n- **General and extensible structure**: We design a general and extensible structure to unify various conversational recommendation datasets and models, in which we integrate various built-in interfaces and functions for quickly development.\n- **Easy to get started**: We provide simple yet flexible configuration for new researchers to quickly start in our library. \n- **Human-machine interaction interfaces**: We provide flexible human-machine interaction interfaces for researchers to conduct qualitative analysis.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://i.loli.net/2020/12/30/6TPVG4pBg2rcDf9.png\" alt=\"RecBole v0.1 architecture\" width=\"400\"\u003e\n  \u003cbr\u003e\n  \u003cb\u003eFigure 1\u003c/b\u003e: The overall framework of CRSLab\n\u003c/p\u003e\n\n\n\n\n- [Installation](#Installation)\n- [Quick-Start](#Quick-Start)\n- [Models](#Models)\n- [Datasets](#Datasets)\n- [Performance](#Performance)\n- [Releases](#Releases)\n- [Contributions](#Contributions)\n- [Citing](#Citing)\n- [Team](#Team)\n- [License](#License)\n\n\n\n## Installation\n\nCRSLab works with the following operating systems：\n\n- Linux\n- Windows 10\n- macOS X\n\nCRSLab requires Python version 3.7 or later.\n\nCRSLab requires torch version 1.8. If you want to use CRSLab with GPU, please ensure that CUDA or CUDAToolkit version is 10.2 or later. Please use the combinations shown in this [Link](https://pytorch-geometric.com/whl/) to ensure the normal operation of PyTorch Geometric.\n\n\n\n### Install PyTorch\n\nUse PyTorch [Locally Installation](https://pytorch.org/get-started/locally/) or [Previous Versions Installation](https://pytorch.org/get-started/previous-versions/) commands to install PyTorch. For example, on Linux and Windows 10:\n\n```bash\n# CUDA 10.2\nconda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=10.2 -c pytorch\n\n# CUDA 11.1\nconda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge\n\n# CPU Only\nconda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cpuonly -c pytorch\n```\n\nIf you want to use CRSLab with GPU, make sure the following command prints `True` after installation:\n\n```bash\n$ python -c \"import torch; print(torch.cuda.is_available())\"\n\u003e\u003e\u003e True\n```\n\n\n\n### Install PyTorch Geometric\n\nEnsure that at least PyTorch 1.8.0 is installed:\n\n```bash\n$ python -c \"import torch; print(torch.__version__)\"\n\u003e\u003e\u003e 1.8.0\n```\n\nFind the CUDA version PyTorch was installed with:\n\n```bash\n$ python -c \"import torch; print(torch.version.cuda)\"\n\u003e\u003e\u003e 11.1\n```\n\nFor Linux:\n\nInstall the relevant packages:\n\n```\nconda install pyg -c pyg\n```\n\nFor others：\n\nCheck PyG [installation documents](https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html) to install the relevant packages.\n\n\n\n### Install CRSLab\n\nYou can install from pip:\n\n```bash\npip install crslab\n```\n\nOR install from source:\n\n```bash\ngit clone https://github.com/RUCAIBox/CRSLab \u0026\u0026 cd CRSLab\npip install -e .\n```\n\n\n\n## Quick-Start\n\nWith the source code, you can use the provided script for initial usage of our library with cpu by default:\n\n```bash\npython run_crslab.py --config config/crs/kgsf/redial.yaml\n```\n\nThe system will complete the data preprocessing, and training, validation, testing of each model in turn. Finally it will get the evaluation results of specified models.\n\nIf you want to save pre-processed datasets and training results of models, you can use the following command:\n\n```bash\npython run_crslab.py --config config/crs/kgsf/redial.yaml --save_data --save_system\n```\n\nIn summary, there are following arguments in `run_crslab.py`:\n\n- `--config` or `-c`: relative path for configuration file(yaml).\n- `--gpu` or `-g`: specify GPU id(s) to use, we now support multiple GPUs. Defaults to CPU(-1).\n- `--save_data` or `-sd`: save pre-processed dataset.\n- `--restore_data` or `-rd`: restore pre-processed dataset from file.\n- `--save_system` or `-ss`: save trained system.\n- `--restore_system` or `-rs`: restore trained system from file.\n- `--debug` or `-d`: use validation dataset to debug your system.\n- `--interact` or `-i`: interact with your system instead of training.\n- `--tensorboard` or `-tb`: enable tensorboard to monitor train performance.\n\n\n\n## Models\n\nIn CRSLab, we unify the task description of conversational recommendation into three sub-tasks, namely recommendation (recommend user-preferred items), conversation (generate proper responses) and policy (select proper interactive action). The recommendation and conversation sub-tasks are the core of a CRS and have been studied in most of works. The policy sub-task is needed by recent works, by which the CRS can interact with users through purposeful strategy.\nAs the first release version, we have implemented 18 models in the four categories of CRS model, Recommendation model, Conversation model and Policy model.\n\n|       Category       |                            Model                             |      Graph Neural Network?      |       Pre-training Model?       |\n| :------------------: | :----------------------------------------------------------: | :-----------------------------: | :-----------------------------: |\n|      CRS Model       | [ReDial](https://arxiv.org/abs/1812.07617)\u003cbr/\u003e[KBRD](https://arxiv.org/abs/1908.05391)\u003cbr/\u003e[KGSF](https://arxiv.org/abs/2007.04032)\u003cbr/\u003e[TG-ReDial](https://arxiv.org/abs/2010.04125)\u003cbr/\u003e[INSPIRED](https://www.aclweb.org/anthology/2020.emnlp-main.654.pdf) |       ×\u003cbr/\u003e√\u003cbr/\u003e√\u003cbr/\u003e×\u003cbr/\u003e×       |       ×\u003cbr/\u003e×\u003cbr/\u003e×\u003cbr/\u003e√\u003cbr/\u003e√       |\n| Recommendation model | Popularity\u003cbr/\u003e[GRU4Rec](https://arxiv.org/abs/1511.06939)\u003cbr/\u003e[SASRec](https://arxiv.org/abs/1808.09781)\u003cbr/\u003e[TextCNN](https://arxiv.org/abs/1408.5882)\u003cbr/\u003e[R-GCN](https://arxiv.org/abs/1703.06103)\u003cbr/\u003e[BERT](https://arxiv.org/abs/1810.04805) | ×\u003cbr/\u003e×\u003cbr/\u003e×\u003cbr/\u003e×\u003cbr/\u003e√\u003cbr/\u003e× | ×\u003cbr/\u003e×\u003cbr/\u003e×\u003cbr/\u003e×\u003cbr/\u003e×\u003cbr/\u003e√ |\n|  Conversation model  | [HERD](https://arxiv.org/abs/1507.04808)\u003cbr/\u003e[Transformer](https://arxiv.org/abs/1706.03762)\u003cbr/\u003e[GPT-2](http://www.persagen.com/files/misc/radford2019language.pdf) |          ×\u003cbr/\u003e×\u003cbr/\u003e×          |          ×\u003cbr/\u003e×\u003cbr/\u003e√          |\n|     Policy model     | PMI\u003cbr/\u003e[MGCG](https://arxiv.org/abs/2005.03954)\u003cbr/\u003e[Conv-BERT](https://arxiv.org/abs/2010.04125)\u003cbr/\u003e[Topic-BERT](https://arxiv.org/abs/2010.04125)\u003cbr/\u003e[Profile-BERT](https://arxiv.org/abs/2010.04125) |    ×\u003cbr/\u003e×\u003cbr/\u003e×\u003cbr/\u003e×\u003cbr/\u003e×    |    ×\u003cbr/\u003e×\u003cbr/\u003e√\u003cbr/\u003e√\u003cbr/\u003e√    |\n\nAmong them, the four CRS models integrate the recommendation model and the conversation model to improve each other, while others only specify an individual task.\n\nFor Recommendation model and Conversation model, we have respectively implemented the following commonly-used automatic evaluation metrics:\n\n|        Category        |                           Metrics                            |\n| :--------------------: | :----------------------------------------------------------: |\n| Recommendation Metrics |      Hit@{1, 10, 50}, MRR@{1, 10, 50}, NDCG@{1, 10, 50}      |\n|  Conversation Metrics  | PPL, BLEU-{1, 2, 3, 4}, Embedding Average/Extreme/Greedy, Distinct-{1, 2, 3, 4} |\n|     Policy Metrics     |        Accuracy, Hit@{1,3,5}           |\n\n\n\n## Datasets\n\nWe have collected and preprocessed 6 commonly-used human-annotated datasets, and each dataset was matched with proper KGs as shown below:\n\n|                           Dataset                            | Dialogs | Utterances |   Domains    | Task Definition | Entity KG  |  Word KG   |\n| :----------------------------------------------------------: | :-----: | :--------: | :----------: | :-------------: | :--------: | :--------: |\n|       [ReDial](https://redialdata.github.io/website/)        | 10,006  |  182,150   |    Movie     |       --        |  DBpedia   | ConceptNet |\n|      [TG-ReDial](https://github.com/RUCAIBox/TG-ReDial)      | 10,000  |  129,392   |    Movie     |   Topic Guide   | CN-DBpedia |   HowNet   |\n|        [GoRecDial](https://arxiv.org/abs/1909.03922)         |  9,125  |  170,904   |    Movie     |  Action Choice  |  DBpedia   | ConceptNet |\n|        [DuRecDial](https://arxiv.org/abs/2005.03954)         | 10,200  |  156,000   | Movie, Music |    Goal Plan    | CN-DBpedia |   HowNet   |\n|      [INSPIRED](https://github.com/sweetpeach/Inspired)      |  1,001  |   35,811   |    Movie     | Social Strategy |  DBpedia   | ConceptNet |\n| [OpenDialKG](https://github.com/facebookresearch/opendialkg) | 13,802  |   91,209   | Movie, Book  |  Path Generate  |  DBpedia   | ConceptNet |\n\n\n\n## Performance\n\nWe have trained and test the integrated models on the TG-Redial dataset, which is split into training, validation and test sets using a ratio of 8:1:1. For each conversation, we start from the first utterance, and generate reply utterances or recommendations in turn by our model. We perform the evaluation on the three sub-tasks.\n\n### Recommendation Task\n\n|   Model   |    Hit@1    |   Hit@10   |   Hit@50   |    MRR@1    |   MRR@10   |   MRR@50   |   NDCG@1    |  NDCG@10   |  NDCG@50   |\n| :-------: | :---------: | :--------: | :--------: | :---------: | :--------: | :--------: | :---------: | :--------: | :--------: |\n|  SASRec   |  0.000446   |  0.00134   |   0.0160   |   0.000446  |  0.000576  |  0.00114   |  0.000445   |  0.00075   |  0.00380   |\n|  TextCNN  |   0.00267   |   0.0103   |   0.0236   |   0.00267   |  0.00434   |  0.00493   |   0.00267   |  0.00570   |  0.00860   |\n|   BERT    |   0.00722   |  0.00490   |   0.0281   |   0.00722   |   0.0106   |   0.0124   |   0.00490   |   0.0147   |   0.0239   |\n|   KBRD    |   0.00401   |   0.0254   |   0.0588   |   0.00401   |  0.00891   |   0.0103   |   0.00401   |   0.0127   |   0.0198   |\n|   KGSF    |   0.00535   | **0.0285** | **0.0771** |   0.00535   |   0.0114   | **0.0135** |   0.00535   | **0.0154** | **0.0259** |\n| TG-ReDial | **0.00793** |   0.0251   |   0.0524   | **0.00793** | **0.0122** |   0.0134   | **0.00793** |   0.0152   |   0.0211   |\n\n\n### Conversation Task\n\n|    Model    |  BLEU@1   |  BLEU@2   |   BLEU@3   |   BLEU@4   |  Dist@1  |  Dist@2  |  Dist@3  |  Dist@4  |  Average  |  Extreme  |  Greedy   |   PPL    |\n| :---------: | :-------: | :-------: | :--------: | :--------: | :------: | :------: | :------: | :------: | :-------: | :-------: | :-------: | :------: |\n|    HERD     |   0.120   |  0.0141   |  0.00136   |  0.000350  |  0.181   |  0.369   |  0.847   |   1.30   |   0.697   |   0.382   |   0.639   |   472    |\n| Transformer |   0.266   |  0.0440   |   0.0145   |  0.00651   |  0.324   |  0.837   |   2.02   |   3.06   |   0.879   |   0.438   |   0.680   |   30.9   |\n|    GPT2     |  0.0858   |  0.0119   |  0.00377   |   0.0110   | **2.35** | **4.62** | **8.84** | **12.5** |   0.763   |   0.297   |   0.583   |   9.26   |\n|    KBRD     |   0.267   |  0.0458   |   0.0134   |  0.00579   |  0.469   |   1.50   |   3.40   |   4.90   |   0.863   |   0.398   |   0.710   |   52.5   |\n|    KGSF     | **0.383** | **0.115** | **0.0444** | **0.0200** |  0.340   |  0.910   |   3.50   |   6.20   | **0.888** | **0.477** | **0.767** |   50.1   |\n|  TG-ReDial  |   0.125   |  0.0204   |  0.00354   |  0.000803  |  0.881   |   1.75   |   7.00   |   12.0   |   0.810   |   0.332   |   0.598   | **7.41** |\n\n\n### Policy Task\n\n|   Model    |   Hit@1   |  Hit@10   |  Hit@50   |   MRR@1   |  MRR@10   |  MRR@50   |  NDCG@1   |  NDCG@10  |  NDCG@50  |\n| :--------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n|    MGCG    |   0.591   |   0.818   |   0.883   |   0.591   |   0.680   |   0.683   |   0.591   |   0.712   |   0.729   |\n| Conv-BERT  |   0.597   |   0.814   |   0.881   |   0.597   |   0.684   |   0.687   |   0.597   |   0.716   |   0.731   |\n| Topic-BERT |   0.598   |   0.828   |   0.885   |   0.598   |   0.690   |   0.693   |   0.598   |   0.724   |   0.737   |\n| TG-ReDial  | **0.600** | **0.830** | **0.893** | **0.600** | **0.693** | **0.696** | **0.600** | **0.727** | **0.741** |\n\nThe above results were obtained from our CRSLab in preliminary experiments. However, these algorithms were implemented and tuned based on our understanding and experiences, which may not achieve their optimal performance. If you could yield a better result for some specific algorithm, please kindly let us know. We will update this table after the results are verified.\n\n## Releases\n\n| Releases |     Date      |   Features   |\n| :------: | :-----------: | :----------: |\n|  v0.1.1  | 1 / 4 / 2021  | Basic CRSLab |\n|  v0.1.2  | 3 / 28 / 2021 |    CRSLab    |\n\n\n\n## Contributions\n\nPlease let us know if you encounter a bug or have any suggestions by [filing an issue](https://github.com/RUCAIBox/CRSLab/issues).\n\nWe welcome all contributions from bug fixes to new features and extensions.\n\nWe expect all contributions discussed in the issue tracker and going through PRs.\n\nWe thank the nice contributions through PRs from [@shubaoyu](https://github.com/shubaoyu), [@ToheartZhang](https://github.com/ToheartZhang).\n\n\n\n## Citing\n\nIf you find CRSLab useful for your research or development, please cite our [Paper](https://arxiv.org/pdf/2101.00939.pdf):\n\n```\n@article{crslab,\n    title={CRSLab: An Open-Source Toolkit for Building Conversational Recommender System},\n    author={Kun Zhou, Xiaolei Wang, Yuanhang Zhou, Chenzhan Shang, Yuan Cheng, Wayne Xin Zhao, Yaliang Li, Ji-Rong Wen},\n    year={2021},\n    journal={arXiv preprint arXiv:2101.00939}\n}\n```\n\n\n\n## Team\n\n**CRSLab** was developed and maintained by [AI Box](http://aibox.ruc.edu.cn/) group in RUC.\n\n\n\n## License\n\n**CRSLab** uses [MIT License](./LICENSE).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frucaibox%2Fcrslab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frucaibox%2Fcrslab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frucaibox%2Fcrslab/lists"}