{"id":13546899,"url":"https://github.com/yym6472/ConSERT","last_synced_at":"2025-04-02T19:32:00.652Z","repository":{"id":43551854,"uuid":"372439641","full_name":"yym6472/ConSERT","owner":"yym6472","description":"Code for our ACL 2021 paper - ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer","archived":false,"fork":false,"pushed_at":"2021-12-10T04:15:34.000Z","size":1177,"stargazers_count":539,"open_issues_count":23,"forks_count":81,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-03T15:38:16.005Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yym6472.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-31T08:42:59.000Z","updated_at":"2024-10-24T06:48:27.000Z","dependencies_parsed_at":"2022-09-13T19:42:30.173Z","dependency_job_id":null,"html_url":"https://github.com/yym6472/ConSERT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yym6472%2FConSERT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yym6472%2FConSERT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yym6472%2FConSERT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yym6472%2FConSERT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yym6472","download_url":"https://codeload.github.com/yym6472/ConSERT/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246880092,"owners_count":20848813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T12:00:47.468Z","updated_at":"2025-04-02T19:31:59.038Z","avatar_url":"https://github.com/yym6472.png","language":"Python","funding_links":[],"categories":["文本匹配 文本检索 文本相似度","🔁 Style Transfer"],"sub_categories":["其他_文本生成、文本对话"],"readme":"# ConSERT\n\nCode for our ACL 2021 paper - [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://aclanthology.org/2021.acl-long.393/)\n\n## Requirements\n\n```\ntorch==1.6.0\ncudatoolkit==10.0.103\ncudnn==7.6.5\nsentence-transformers==0.3.9\ntransformers==3.4.0\ntensorboardX==2.1\npandas==1.1.5\nsentencepiece==0.1.85\nmatplotlib==3.4.1\napex==0.1.0\n```\n\nTo install apex, run:\n```\ngit clone https://github.com/NVIDIA/apex\ncd apex\npip install -v --disable-pip-version-check --no-cache-dir ./\n```\n\n## Get Started\n\n### English STS Tasks\n\n1. Download pre-trained language model (e.g. bert-base-uncased) to folder `./bert-base-uncased` from [HuggingFace's Library](https://huggingface.co/bert-base-uncased)\n2. Download STS datasets to `./data` folder by running `cd data \u0026\u0026 bash get_transfer_data.bash`. The script is modified from [SentEval toolkit](https://github.com/facebookresearch/SentEval/blob/master/data/downstream/get_transfer_data.bash)\n3. Run the scripts in the folder `./scripts` to reproduce our experiments. For example, run the following script to train unsupervised consert-base:\n    ```bash\n    bash scripts/unsup-consert-base.sh\n    ```\n\n*Update*: Recently we update the support for RoBERTa model. Download the RoBERTa model to folder e.g. `./roberta-base` and use scripts in `./scripts/roberta` instead.\n\n### Chinese STS Tasks\n\n1. Download the pre-trained language model (e.g. chinese-roberta-wwm-ext) to folder `./chinese-roberta-wwm-ext` from [HuggingFace's Library](https://huggingface.co/hfl/chinese-roberta-wwm-ext)\n2. Download the chinese STS datasets to `./data` folder by running `cd data \u0026\u0026 bash get_chinese_sts_data.bash`\n3. Run the scripts in the folder `./scripts/chinese` to train models for chinese STS tasks. For example, run the following script to train *base* model for `atec_ccks` task:\n    ```bash\n    bash scripts/chinese/unsup-consert-base-atec_ccks.sh\n    ```\n   The `--chinese_dataset` option in `main.py` is used to select which chinese STS dataset to use.\n\n## Pre-trained Models \u0026 Results\n\n### English STS Tasks\n\n| ID | Model                                                         | STS12 | STS13 | STS14 | STS15 | STS16 | STSb | SICK-R | Avg. |\n|----|---------------------------------------------------------------|:-----:|:-----:|:-----:|:-----:|:-----:|:----:|:------:|:----:|\n| -  | bert-base-uncased (baseline) | 35.20 | 59.53 | 49.37 | 63.39 | 62.73 | 48.18 | 58.60 | 53.86 |\n| -  | bert-large-uncased (baseline) | 33.06 | 57.64 | 47.95 | 55.83 | 62.42 | 49.66 | 53.87 | 51.49 |\n| 1  | unsup-consert-base [\\[Google Drive\\]](https://drive.google.com/file/d/1KIbrhhIfhxO_4b0tbpdhLGkrjr_bhlaa/view?usp=sharing) [\\[百度云q571\\]](https://pan.baidu.com/s/1Nh_ypA-kP2cXIt3DE_0aXg)            |   64.64    |  78.49     |  69.07     | 79.72      |  75.95     |  73.97    |   67.31     |  72.74    |\n| 2  | unsup-consert-large [\\[Google Drive\\]](https://drive.google.com/file/d/1sYlJ_O43QHC1WYY9h3HW_bWLhxvb7lEH/view?usp=sharing) [\\[百度云9fm1\\]](https://pan.baidu.com/s/1XBonWQhDp68-TUfVV8NBBQ)           |  70.28     |   83.23    |  73.80     |  82.73     |    77.14   |  77.74    |  70.19      |  76.45    |\n| 3  | sup-sbert-base (re-impl.) [\\[Google Drive\\]](https://drive.google.com/file/d/1Q5wN6FRikBKEJ3jeJlPGPQ0WWEf0mkdi/view?usp=sharing) [\\[百度云msqy\\]](https://pan.baidu.com/s/1IDztb2LEdje4_aIhQHOhoQ)     |   69.93    |   76.00    |  72.15     |   78.59    | 73.53      |   76.10   |    73.01    |  74.19    |\n| 4  | sup-sbert-large (re-impl.) [\\[Google Drive\\]](https://drive.google.com/file/d/1JJjvxiyJEdS62GpH0mDz1qK0AApBPQ3Z/view?usp=sharing) [\\[百度云0oir\\]](https://pan.baidu.com/s/1yYlG2Us7CBVEt_kLIaazrw)    |   73.06    |  77.77     |   75.21    |   81.63    |   77.30    |   79.74   |   74.75     |   77.07   |\n| 5  | sup-consert-joint-base [\\[Google Drive\\]](https://drive.google.com/file/d/1uCNWUWPyjuTfyIqP4zsVNNxeGRNyXOrw/view?usp=sharing) [\\[百度云jks5\\]](https://pan.baidu.com/s/1R3QIA99_RTY5CeuJa8Q6UA)        |  70.92     |  79.98     |   74.88    |   81.76    |   76.46    |    78.99  |   78.15     |  77.31    |\n| 6  | sup-consert-joint-large [\\[Google Drive\\]](https://drive.google.com/file/d/1cE8wauMEHGKn52kyCpOcIKKeJ7p3-cTq/view?usp=sharing) [\\[百度云xua4\\]](https://pan.baidu.com/s/1nNrwJNoctWYRYaUjrz5qRw)       |  73.15     |  81.45     |   77.04    |   83.32    |  77.28     |    81.15  |   78.34     |  78.82    |\n| 7  | sup-consert-sup-unsup-base [\\[Google Drive\\]](https://drive.google.com/file/d/1w3SmCC6ibm8NSvh80ZB0ERDonpPvm8mz/view?usp=sharing) [\\[百度云5mc8\\]](https://pan.baidu.com/s/1Kgi14KXeby0eFCzraCYfDg)    |   73.02    |  84.86     |  77.32     |   82.70    |  78.20     |    81.34  |   75.00     |  78.92    |\n| 8  | sup-consert-sup-unsup-large [\\[Google Drive\\]](https://drive.google.com/file/d/15E8nAFprUFTv5KPPgxvedgigXXwcRVeT/view?usp=sharing) [\\[百度云tta1\\]](https://pan.baidu.com/s/1UeLwlUUWR3QogqTeUQZ0iw)   |  74.99     |   85.58    | 79.17      |   84.25    |   80.19    |    83.17  |   77.43     |  80.68    |\n| 9  | sup-consert-joint-unsup-base [\\[Google Drive\\]](https://drive.google.com/file/d/14jcb8NCDB3PGr0LZI_tZcmBATxkB4OEc/view?usp=sharing) [\\[百度云cf07\\]](https://pan.baidu.com/s/15fbohm8TdZLHhW9NEonl_Q)  |  74.46     |   84.19    |  77.08     |   83.77    |  78.55     |     81.37 |    77.01    |  79.49    |\n| 10 | sup-consert-joint-unsup-large [\\[Google Drive\\]](https://drive.google.com/file/d/1xo5QrlG_TJ6NetqSX1nYtaVLPzee2kpx/view?usp=sharing) [\\[百度云v5x5\\]](https://pan.baidu.com/s/1b53Tk0ZwTvlti_OSfHDhzg) |  76.93     |  85.20     |  78.69     |   85.44    |   79.34    |    82.93  |  76.71      |  80.75    |\n\nNote:\n1. All the *base* models are trained from `bert-base-uncased` and the *large* models are trained from `bert-large-uncased`.\n2. For the unsupervised transfer, we merge all unlabeled texts from 7 STS datasets (STS12-16, STSbenchmark and SICK-Relatedness) as the training data (total 89192 sentences), and use the STSbenchmark dev split (including 1500 human-annotated sentence pairs) to select the best checkpoint.\n3. The sentence representations are obtained by averaging the token embeddings at the last two layers of BERT.\n4. For model 2 to 10, we re-trained them on a single GeForce RTX 3090 with pytorch 1.8.1 and cuda 11.1 (rather than V100, pytorch 1.6.0 and cuda 10.0 in our initial experiments) and changed the `max_seq_length` from 64 to 40 to reduce the required GPU memory (only for *large* models). Consequently, the results shown here may be slightly different from those reported in our paper.\n\n### Chinese STS Tasks\n\n| ID | Model                              | atec_ccks | bq | lcqmc | pawsx | stsb |\n|----|------------------------------------|:-----:|:-----:|:-----:|:-----:|:-----:|\n| -  | chinese-roberta-wwm-ext (baseline) | 11.28 | 40.21 | 59.89 | 09.35 | 60.86 |\n| -  | chinese-roberta-wwm-ext-large (baseline) | 13.75 | 36.77 | 60.36 | 09.94 | 58.72 |\n| 1  | unsup-consert-base-atec_ccks [\\[百度云v593\\]](https://pan.baidu.com/s/10Elp2EWOquawa4pn3kDgzA)  | **27.39** | 47.61 | 60.70  | 08.17 | 64.74 |\n| 2  | unsup-consert-base-bq [\\[百度云cwnu\\]](https://pan.baidu.com/s/1gq8XZzoIiv_DIlOiMQQEwg)        | 11.55 | **47.20** | 61.47  | 08.47 | 64.44 |\n| 3  | unsup-consert-base-lcqmc [\\[百度云pdax\\]](https://pan.baidu.com/s/1OY-EMfON9P2GDg25ikWs-A)      | 04.57 | 38.15 | **67.34**  | 08.80 | 67.78 |\n| 4  | unsup-consert-base-pawsx [\\[百度云iko8\\]](https://pan.baidu.com/s/1W1KHtOE57AlIfRlA79MrjQ)      | 08.66 | 38.35 | 61.97  | **09.36** | 63.89 |\n| 5  | unsup-consert-base-stsb [\\[百度云7r7n\\]](https://pan.baidu.com/s/1I3nGw7VNUVXqV79dD3ZDCQ)       | 07.29 | 40.81 | 67.39  | 06.24  | **72.82** |\n| 6  | unsup-consert-large-atec_ccks [\\[百度云4zlz\\]](https://pan.baidu.com/s/1XzJ_6tAmP_wwAP89e2nsnQ) | **29.92** | 47.50 | 59.83 | 09.72 | 66.37 |\n| 7  | unsup-consert-large-bq [\\[百度云gnte\\]](https://pan.baidu.com/s/1RBCn5FnwCTJ8q6aTXfQuVQ)        | 15.47 | **47.08** | 60.10 | 10.04 | 66.94 |\n| 8  | unsup-consert-large-lcqmc [\\[百度云wetp\\]](https://pan.baidu.com/s/11Pvo5D4lghDmiWl-NDjmBQ)     | 11.40 | 37.79 | **66.45** | 10.81 | 67.68 |\n| 9  | unsup-consert-large-pawsx [\\[百度云u6t9\\]](https://pan.baidu.com/s/1QYxHx7LOEQ0bHwEKI--V1A)     | 11.25 | 36.25 | 65.15 | **11.38** | 67.45 |\n| 10 | unsup-consert-large-stsb [\\[百度云8tyz\\]](https://pan.baidu.com/s/17daGrOI8Or-O1WDR-cOmKA)      | 08.00 | 40.85 | 67.72 | 10.13 | **74.50** |\n\nNote:\n1. All the *base* models are trained from `hfl/chinese-roberta-wwm-ext` and the *large* models are trained from `hfl/chinese-roberta-wwm-ext-large`. \n2. For each model, we train it on *single* chinese STS dataset but evaluate it on all 5 datasets. **The bold numbers** indicate that the model is evaluated on the same dataset it trained on.\n3. The sentence representations are also obtained by averaging the token embeddings at the last two layers of BERT.\n4. For *base* models, we set `batch_size` to 96 and `max_seq_length` to 64, while for *large* models, we set `batch_size` to 32 and `max_seq_length` to 40 to reduce the required GPU memory.\n\n## Citation\n```\n@inproceedings{yan-etal-2021-consert,\n    title = \"{C}on{SERT}: A Contrastive Framework for Self-Supervised Sentence Representation Transfer\",\n    author = \"Yan, Yuanmeng  and\n      Li, Rumei  and\n      Wang, Sirui  and\n      Zhang, Fuzheng  and\n      Wu, Wei  and\n      Xu, Weiran\",\n    booktitle = \"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)\",\n    month = aug,\n    year = \"2021\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2021.acl-long.393\",\n    doi = \"10.18653/v1/2021.acl-long.393\",\n    pages = \"5065--5075\",\n    abstract = \"Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised SEntence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8{\\%} relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.\",\n}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyym6472%2FConSERT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyym6472%2FConSERT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyym6472%2FConSERT/lists"}