{"id":16039778,"url":"https://github.com/hpprc/simple-simcse","last_synced_at":"2025-03-17T16:31:17.399Z","repository":{"id":65236423,"uuid":"555223228","full_name":"hppRC/simple-simcse","owner":"hppRC","description":"A simple implementation of SimCSE","archived":false,"fork":false,"pushed_at":"2022-10-31T09:32:11.000Z","size":161,"stargazers_count":72,"open_issues_count":0,"forks_count":9,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-09T23:07:37.545Z","etag":null,"topics":["deep-learning","pytorch","sentence-embeddings"],"latest_commit_sha":null,"homepage":"https://aclanthology.org/2021.emnlp-main.552/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hppRC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-21T06:59:29.000Z","updated_at":"2024-08-15T03:07:21.000Z","dependencies_parsed_at":"2023-01-15T19:00:23.105Z","dependency_job_id":null,"html_url":"https://github.com/hppRC/simple-simcse","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hppRC%2Fsimple-simcse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hppRC%2Fsimple-simcse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hppRC%2Fsimple-simcse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hppRC%2Fsimple-simcse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hppRC","download_url":"https://codeload.github.com/hppRC/simple-simcse/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221696706,"owners_count":16865467,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","pytorch","sentence-embeddings"],"created_at":"2024-10-08T23:07:51.337Z","updated_at":"2024-10-27T15:17:11.640Z","avatar_url":"https://github.com/hppRC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple-SimCSE: A simple implementation of SimCSE\n\n## Introduction\n\n[SimCSE](https://aclanthology.org/2021.emnlp-main.552/) is one of the most exciting sentence embedding method using contrastive learning.\nSimCSE achived state of the art performance, and advance representation learning in NLP greatly.\n\nThe concept of SimCSE is very simple in itself (as the title suggests).\nHowever, [the official implementation of SimCSE](https://github.com/princeton-nlp/SimCSE) is abstracted to accommodate a variety of use cases, making the code a bit harder to read.\n\nOf course, the official implementation is great.\nHowever, a simpler implementation would be more helpful in understanding, in particular, it is important for those who are new to research about deep learning or who are just starting out with research about sentence embedding.\n\nTherefore, We implemented a simple version of SimCSE, with minimal abstraction and use of external libraries.\n\nUsing some basic features of [PyTorch](https://github.com/pytorch/pytorch) and [transformers](https://github.com/huggingface/transformers), we developed code to perform fine-tuning and evaluation of SimCSE from scratch.\n\n## Instllation \u0026 Training\n\nFor development, We used [poetry](https://python-poetry.org/), which is the dependency management and packaging tool for Python.\n\nIf you use poetry, you can install necessary packages by following command.\n\n```bash\npoetry install\n```\n\nOr, you can install them using `requiments.txt`.\n\n```bash\npip install -r requirements.txt\n```\n\nThe `requirements.txt` is output by following command.\n\n```bash\npoetry export -f requirements.txt --output requirements.txt\n```\n\nThen, you must execute `download.sh` to download training and evaluation datasets beforehand.\n`download.sh` will collect STS and training datasets used in the paper in parallel.\n\n```bash\nbash download.sh\n```\n\nFinaly, you can train your model as below.\n\n```bash\npoetry run python train.py\n\n# or\n# python train.py\n```\n\n\n## Evaluation (Unsup-SimCSE)\n\nIn doing this implementation, we investigated how well the Unsup-SimCSE model trained by this implementation would perform.\n\nWe performed fine-tuning of Unsup-SimCSE **50 times** with different random seeds ([0, 49]) with the same dataset and hyperparameters as described in the paper (see `train-original.sh` and `train-multiple.sh`).\nWe evaluated them on 7 STS tasks (STS12--16, STS Benchmark, SICK-R).\n\nWe evaluated models every 250 training steps on the development set of STS-B and keep the best checkpoint for the final evaluation (as with the original paper, described in Appendix A).\n\n### Overall\n\n\nThe table below shows the average performances on each STS task over 50 runs.\n\n|               | STS12     | STS13     | STS14     | STS15     | STS16     | STS-B     | SICK-R    | Avg.      |\n| ------------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| BERT-base     | **67.64** | 80.42     | 72.68     | 80.25     | 78.12     | 76.28     | 70.35     | 75.11     |\n| BERT-large    | 65.83     | 80.44     | 71.60     | 81.45     | 76.85     | 75.96     | **72.52** | 74.95     |\n| RoBERTa-base  | 64.92     | 80.06     | 71.52     | 79.92     | 78.73     | 78.67     | 68.39     | 74.60     |\n| RoBERTa-large | 66.46     | **81.93** | **73.93** | **82.71** | **80.58** | **80.57** | 70.82     | **76.71** |\n\nOverall, RoBERTa-large achieved the best average performance, however, the performance of all models compared poorly with the performance of the paper (see Table 5).\n\nThe reason is not clear, but the following two points are possible\n\n- There is something wrong with our implementation.\n- The hyperparameters used in the paper are tuned with a single random seed (see https://github.com/princeton-nlp/SimCSE/issues/63), so the hyperparameters are not good for our implementation.\n  - In that case, it would be very difficult to perfectly reproduce the performance of SimCSE with different implementations.\n\n\nWe also found that SimCSE is a bit sensitive to random seeds.\nThe following figures show histograms and KDE plots of average performance for each fine-tuning.\n\n![histplot](./.github/images/performances/overall-hist.png)![kdeplot](./.github/images/performances/overall-kde.png)\n\nBERT-base and BERT-large show a larger variance in performance compared to RoBERTa-base and RoBERTa-large.\n\nThere are many possible reasons for this difference, but the most likely cause is a difference in batch size.\n\nBERT-base and BERT-large are fine-tuned with batch size 64, however, RoBERTa-base and RoBERTa-large are fine-tuned with batch size 512.\n\nWe suspect that using smaller batch sizes is likely to result in more unstable performance.\n\n\n### Details\n\nTo encourage future research, we show the average, maximum, and minimum performance for each model for each STS task.\n\nThe results of all experiments for each model are in the `results` directory, so we hope they help you.\n\nIn addition, to investigate how the performance of each model changes during training, we show the performance transition the development set of STS Benchmark.\n\n\n#### BERT-base\n\n|      | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg.  |\n| ---- | ----- | ----- | ----- | ----- | ----- | ----- | ------ | ----- |\n| min  | 64.64 | 77.91 | 69.36 | 77.60 | 76.26 | 73.99 | 67.80  | 73.16 |\n| mean | 67.64 | 80.42 | 72.68 | 80.25 | 78.12 | 76.28 | 70.35  | 75.11 |\n| max  | 70.24 | 82.89 | 74.93 | 82.80 | 79.95 | 78.29 | 71.99  | 76.70 |\n\n\n![BERT-base](./.github/images/transitions/bert-base-uncased.png)\n\nThe performance of BERT-base is a bit unstable.\nThis is thought to be largely due to batch size.\n\nOn the other hand, it is also worth noting that BERT-base achieved the best performance on STS12.\n\n\n#### BERT-large\n\n|      | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg.  |\n| ---- | ----- | ----- | ----- | ----- | ----- | ----- | ------ | ----- |\n| min  | 61.00 | 73.62 | 65.20 | 76.37 | 69.35 | 69.26 | 67.58  | 70.31 |\n| mean | 65.83 | 80.44 | 71.60 | 81.45 | 76.85 | 75.96 | 72.52  | 74.95 |\n| max  | 69.12 | 84.39 | 75.18 | 83.63 | 79.34 | 79.10 | 75.15  | 77.28 |\n\n\n![BERT-large](./.github/images/transitions/bert-large-uncased.png)\n\n\nBERT-large has the most unstable performance of all models (min=70.31, max=77.28).\nHowever, the performance transition seems to be much more stable than BERT-base.\n\nFurthermore, BERT-large achieved the highest performance on SICK-R.\n\n\n#### RoBERTa-base\n\n|      | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg.  |\n| ---- | ----- | ----- | ----- | ----- | ----- | ----- | ------ | ----- |\n| min  | 63.64 | 79.28 | 70.92 | 79.09 | 78.22 | 78.04 | 67.17  | 74.17 |\n| mean | 64.92 | 80.06 | 71.52 | 79.92 | 78.73 | 78.67 | 68.39  | 74.60 |\n| max  | 66.17 | 81.18 | 72.32 | 80.60 | 79.40 | 79.41 | 69.15  | 75.14 |\n\n![RoBERTa-base](./.github/images/transitions/roberta-base.png)\n\nThe performance of RoBERTa-base is stable.\n\nThe large batch size should be at a slight disadvantage because of the small number of evaluation steps, but RoBERTa-base achieved relatively high performance.\n\n\n#### RoBERTa-large\n\n|      | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg.  |\n| ---- | ----- | ----- | ----- | ----- | ----- | ----- | ------ | ----- |\n| min  | 63.94 | 80.63 | 72.62 | 81.57 | 79.53 | 78.54 | 68.19  | 75.64 |\n| mean | 66.46 | 81.93 | 73.93 | 82.71 | 80.58 | 80.57 | 70.82  | 76.71 |\n| max  | 68.17 | 83.54 | 75.17 | 84.03 | 81.33 | 81.91 | 72.38  | 77.69 |\n\n\n![RoBERTa-large](./.github/images/transitions/roberta-large.png)\n\nRoBERTa-large achieved the best overall performance and performance changes were stable.\n\nAs can be seen from the performance transition between BERT-large and RoBERTa-large, there is a fairly significant difference between the two models.\n\nWhether this is due to hyperparameters or differences in pre-training data/tasks is a matter for further research.\n\n\n## In this work\n\nSimCSE has further advanced research of sentence embeddings.\nThe clarity of SimCSE is very impressive, and wide range of applications will be developed in the future.\n\nWe hope that this implementation will help in understanding and facilitate future research.\n\n\n## Citation\n\nAuthor: [Hayato Tsukagoshi](https://hpprc.dev)\n\nIf you would like to collaborate with me, please feel free to contact me at the email address below.\n\n[research.tsukagoshi.hayato@gmail.com](mailto:research.tsukagoshi.hayato@gmail.com)\n\n\n```bibtex\n@misc{\n  hayato-tsukagoshi-2022-simple-simcse,\n  author = {Hayato Tsukagoshi},\n  title = {Simple-SimCSE: A simple implementation of SimCSE},\n  year = {2022},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/hppRC/simple-simcse}}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhpprc%2Fsimple-simcse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhpprc%2Fsimple-simcse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhpprc%2Fsimple-simcse/lists"}