{"id":37077810,"url":"https://github.com/yxuansu/simctg","last_synced_at":"2026-01-14T09:01:48.160Z","repository":{"id":37837374,"uuid":"451958645","full_name":"yxuansu/SimCTG","owner":"yxuansu","description":"[NeurIPS'22 Spotlight] A Contrastive Framework for Neural Text Generation","archived":false,"fork":false,"pushed_at":"2024-03-07T04:38:08.000Z","size":7279,"stargazers_count":473,"open_issues_count":8,"forks_count":40,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-09-23T01:00:30.515Z","etag":null,"topics":["contrastive-learning","decode","deeplearning","language-modeling","languagemodel","nlp","textgeneration"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2202.06417","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yxuansu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-25T16:39:20.000Z","updated_at":"2025-08-20T04:00:28.000Z","dependencies_parsed_at":"2022-07-14T04:00:33.719Z","dependency_job_id":null,"html_url":"https://github.com/yxuansu/SimCTG","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/yxuansu/SimCTG","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yxuansu%2FSimCTG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yxuansu%2FSimCTG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yxuansu%2FSimCTG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yxuansu%2FSimCTG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yxuansu","download_url":"https://codeload.github.com/yxuansu/SimCTG/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yxuansu%2FSimCTG/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28414732,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T08:38:59.149Z","status":"ssl_error","status_checked_at":"2026-01-14T08:38:43.588Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["contrastive-learning","decode","deeplearning","language-modeling","languagemodel","nlp","textgeneration"],"created_at":"2026-01-14T09:01:47.306Z","updated_at":"2026-01-14T09:01:48.149Z","avatar_url":"https://github.com/yxuansu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A Contrastive Framework for Neural Text Generation\n**Authors**: Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier\n\nThis repository contains code, models, and other related resources of our paper [\"A Contrastive Framework for Neural Text Generation\"](https://arxiv.org/abs/2202.06417).\n\n:star2: Check out this great [[blog]](https://huggingface.co/blog/introducing-csearch) as well as this awesome [[demo]](https://huggingface.co/spaces/joaogante/contrastive_search_generation) that are generously supported by Huggingface ([@huggingface](https://github.com/huggingface) :hugs:) which compares contrastive search with other popular decoding methods. Many thanks to Huggingface :hugs:! \n\n**[Use contrastive search in Huggingface transformers]** In this \u003ca href='#contrastive_in_transformers'\u003etutorial\u003c/a\u003e, we demonstrate how to use contrastive search in Huggingface `transformers`.\n\n\n****\nIf you find our paper and resources useful, please kindly leave a star and cite our papers. Thanks!\n\n```bibtex\n@inproceedings{su2022a,\n  title={A Contrastive Framework for Neural Text Generation},\n  author={Yixuan Su and Tian Lan and Yan Wang and Dani Yogatama and Lingpeng Kong and Nigel Collier},\n  booktitle={Advances in Neural Information Processing Systems},\n  editor={Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho},\n  year={2022},\n  url={https://openreview.net/forum?id=V88BafmH9Pj}\n}\n\n@article{su2023contrastive,\n  title={Contrastive Search Is What You Need For Neural Text Generation},\n  author={Yixuan Su and Nigel Collier},\n  journal={Transactions on Machine Learning Research},\n  issn={2835-8856},\n  year={2023},\n  url={https://openreview.net/forum?id=GbkWw3jwL9}\n}\n```\n\n****\n\n### News:\n* [2022/11/22] Released a technical report that compares Contrastive Search with another recently proposed method, i.e. Contrastive Decoding. Check out our [[paper]](https://arxiv.org/abs/2211.10797) and [[code]](https://github.com/yxuansu/Contrastive_Search_versus_Contrastive_Decoding).\n* [2022/11/08] :fire: Contrastive Search has now officially supported by HuggingFace for both PyTorch and TensorFlow platforms! Check out this great [[blog]](https://huggingface.co/blog/introducing-csearch) as well as this awesome [[demo]](https://huggingface.co/spaces/joaogante/contrastive_search_generation) that are generously supported by Huggingface ([@huggingface](https://github.com/huggingface) :hugs:).\n* [2022/10/26] :fire: We have released a new manuscript [\"Contrastive Search Is What You Need For Neural Text Generation\"](https://arxiv.org/abs/2210.14140) which has two takeaways: (1) Autoregressive language models are naturally isotropic, therefore *SimCTG training may not be necessary*; (2) Contrastive search works exceptionally well on **off-the-shelf** language models across 16 languages. On 12 out of the 16 evaluated languages, it even performs comparably with human-written text! [Paper](https://arxiv.org/abs/2210.14140) and [code](https://github.com/yxuansu/Contrastive_Search_Is_What_You_Need) are all released. Check it out!\n* [2022/10/13] We have added a concise explanation on the implementations of contrastive search. Please find it [[here]](https://github.com/yxuansu/SimCTG/tree/main/contrastive_search_explanation).\n* [2022/09/14] :blush: SimCTG is accepted to NeurIPS 2022!\n* [2022/09/06] :fire: We have added supports for the newly released OPT models (see [\"OPT: Open Pre-trained Transformer Language Models\"](https://arxiv.org/abs/2205.01068)) by Meta. To see how to apply contrastive search on OPT models, check it [[here]](#contrastive_search_with_opt)!\n* [2022/06/03] :fire: We have released an easy-to-use library (i.e., simctg) which allows you to use SimCTG with a simple **pip install simctg** and **a few lines of code**. Check the comprehensive and huggingface-style tutorials \u003ca href='#tutorial'\u003e[here]\u003c/a\u003e and [[here]](https://github.com/yxuansu/SimCTG/tree/main/simctg)!\n* [2022/05/06] :star: We have released **_MAGIC_**, a follow up work of SimCTG, that is the SOTA method in zero-shot multi-modal text generation tasks (e.g., zero-shot image captioning and visually grounded story generation). Check it out! [[paper]](https://arxiv.org/abs/2205.02655) [[code]](https://github.com/yxuansu/MAGIC)\n* [2022/04/16] We have updated instructions on how to apply contrastive search on encoder-decoder models (e.g. BART and T5). More details can be found [[here]](https://github.com/yxuansu/SimCTG/tree/main/SimCTGEncDec).\n* [2022/04/07] SimCTG has been publicly deployed in the **AI sentence completion** module of [Tencent's Effidit platform (腾讯智能创作助手)](https://effidit.qq.com/). Check it out and have fun!\n* [2022/04/02] Add support on another benchmark (ROCStories) for the task of open-ended story generation.\n* [2022/03/06] Example of how to adapt our approach to open-ended story generation is released.\n* [2022/02/15] SimCTG is publicly released!\n\n****\n\n\u003cspan id='all_catelogue'/\u003e\n\n### Catalogue:\n* \u003ca href='#contrastive_in_transformers'\u003eApply Contrastive Search in Huggingface Transformers\u003c/a\u003e\n* \u003ca href='#introduction'\u003e1. Introduction\u003c/a\u003e\n* \u003ca href='#contrastive_search_with_LMs'\u003e2. Contrastive Search with GPT-2 and OPT :fire:\u003c/a\u003e\n    * \u003ca href='#install_simctg_package'\u003e2.1. Environment Setup\u003c/a\u003e\n    * \u003ca href='#contrastive_search_with_gpt2'\u003e2.2. Contrastive Search with GPT-2\u003c/a\u003e\n    * \u003ca href='#contrastive_search_with_opt'\u003e2.3. Contrastive Search with OPT\u003c/a\u003e\n* \u003ca href='#models'\u003e3. Huggingface Models\u003c/a\u003e\n* \u003ca href='#tutorial'\u003e4. Huggingface-Style Tutorials\u003c/a\u003e :star: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1ImvR-ldHf9rJyFzOCMJ_zjAGK8n1iTW7?usp=sharing)\n    * \u003ca href='#install_simctg'\u003e4.1. Install and Load SimCTG\u003c/a\u003e\n    * \u003ca href='#example_train_with_simctg'\u003e4.2. Example of Training Language Model with SimCTG\u003c/a\u003e\n        * \u003ca href='#init_simctg'\u003e4.2.1. Initialize Language Model\u003c/a\u003e\n        * \u003ca href='#init_loss_class'\u003e4.2.2. Initialize Loss Class\u003c/a\u003e\n        * \u003ca href='#init_training_data'\u003e4.2.3. Create Example Training Data\u003c/a\u003e\n        * \u003ca href='#compute_loss'\u003e4.2.4. Compute Loss\u003c/a\u003e\n    * \u003ca href='#contrastive_search_examples'\u003e4.3. Examples of Performing Generation with Contrastive Search\u003c/a\u003e\n        * \u003ca href='#example_document_generation'\u003e4.3.1. Open-Ended Document Generation\u003c/a\u003e\n        * \u003ca href='#example_dialogue_generation'\u003e4.3.2. Open-Domain Dialogue Generation\u003c/a\u003e\n    * \u003ca href='#example_off_the_shelf_generation'\u003e4.4. Contrastive Search with Off-the-shelf Language Models from Different Languages\u003c/a\u003e\n        * \u003ca href='#chinese_example_off_the_shelf_generation'\u003e4.4.1. Chinese Language Model\u003c/a\u003e\n        * \u003ca href='#japanese_example_off_the_shelf_generation'\u003e4.4.2. Japanese Language Model\u003c/a\u003e\n        * \u003ca href='#korean_example_off_the_shelf_generation'\u003e4.4.3. Korean Language Model\u003c/a\u003e\n    * \u003ca href='#training_tutorial'\u003e4.5. Detailed Tutorial of Training SimCTG on Wikitext-103\u003c/a\u003e :star:\n    * \u003ca href='#T5_tutorial'\u003e4.6. Apply SimCTG on T5\u003c/a\u003e :star:\n* \u003ca href='#environment_setup'\u003e5. Environment Setup\u003c/a\u003e\n* \u003ca href='#example_usage'\u003e6. Example Usage of Contrastive Search\u003c/a\u003e\n    * \u003ca href='#example_usage_english_simctg'\u003e6.1. Use SimCTG Pretrained on Wikipedia Corpus\u003c/a\u003e [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1MhK3cVHW9HQ1ArXu0M_sS_Po0_4N1xgQ?usp=sharing)\n    * \u003ca href='#example_usage_different_language_model'\u003e6.2. Use **Off-the-shelf** Language Models from Different Languages\u003c/a\u003e\n        * \u003ca href='#example_usage_chinese_gpt'\u003e6.2.1. Chinese Language Model\u003c/a\u003e [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1_55LEg2caLM-lYDVIhWjxgv75IWkEry6?usp=sharing)\n        * \u003ca href='#example_usage_japanese_gpt'\u003e6.2.2. Japanese Language Model\u003c/a\u003e [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1844kf-BttuPt1DaYhdgw-07-qz7V7pOd?usp=sharing)\n        * \u003ca href='#example_usage_korean_gpt'\u003e6.2.3. Korean Language Model\u003c/a\u003e [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1a8g1n86S-zmGe7Nb0PgVQnqWSAMfIR3D?usp=sharing)\n* \u003ca href='#wikitext103_tutorial'\u003e7. Document Generation\u003c/a\u003e [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1_zPZRlbJo5iw_Q7FUhP113udPnciUxVF?usp=sharing)\n* \u003ca href='#dialogue_tutorial'\u003e8. Open-domain Dialogue Generation\u003c/a\u003e [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](\nhttps://colab.research.google.com/drive/1_55LEg2caLM-lYDVIhWjxgv75IWkEry6?usp=sharing)\n* \u003ca href='#pretraining'\u003e9. Large-Scale Pre-training with SimCTG\u003c/a\u003e\n* \u003ca href='#story_generation'\u003e10. Open-Ended Story Generation\u003c/a\u003e\n* \u003ca href='#contrastive_for_encoder_decoder'\u003e11. Contrastive Search on Encoder-Decoder Models\u003c/a\u003e\n* \u003ca href='#contact'\u003e12. Contact\u003c/a\u003e\n* \u003ca href='#simctg_elsewhere'\u003e13. SimCTG Elsewhere\u003c/a\u003e\n\n\n****\n\n\u003cspan id='contrastive_in_transformers'/\u003e\n\n#### Apply Contrastive Search in Huggingface Transformers:\nHere, we demonstrate how to use contrastive search in Huggingface `transformers`.\n\n##### (1) Install Environment:\nFirst, to install the required packages, please run the following commands:\n```yaml\npip install torch\npip install \"transformers\u003e=4.24.0\"\n```\n\n##### (2) Generate Text with Contrastive Search:\nNext, we show how to reproduce the result as in \u003ca href='#contrastive_search_with_gpt2'\u003eSection 2.2\u003c/a\u003e\n\n```python\n# load the LMs\nimport torch\nfrom transformers import GPT2Tokenizer, GPT2LMHeadModel\nmodel_name = 'gpt2-large'\ntokenizer = GPT2Tokenizer.from_pretrained(model_name)\nmodel = GPT2LMHeadModel.from_pretrained(model_name)\nmodel.eval()\n\n# prepare the prefix\nprefix_text = r\"DeepMind Company is\"\ninput_ids = tokenizer(prefix_text, return_tensors='pt').input_ids\n\n# generate the result with contrastive search\noutput = model.generate(input_ids, penalty_alpha=0.6, top_k=4, max_length=512)\nprint(\"Output:\\n\" + 100 * '-')\nprint(tokenizer.decode(output[0], skip_special_tokens=True))\nprint(\"\" + 100 * '-')\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output: [click to expand]\u003c/b\u003e\u003c/summary\u003e\n  \n```\nOutput:\n----------------------------------------------------------------------------------------------------  \nDeepMind Company is a leader in artificial intelligence (AI). We have a long history of working\nwith companies such as Google, Facebook, Amazon, and Microsoft to build products that improve\npeople's lives, and today we are excited to announce that DeepMind's AlphaGo program has won the\ngame of Go, becoming the first program to defeat a professional Go player.\n\nThe victory is a testament to the power of deep learning, and to the incredible work of our\nresearch team, which has been at the forefront of AI research for the past five years. AlphaGo\nis one of the most advanced Go programs ever created, and its performance is an important step\ntowards the goal of human-level AI.\n\n\"This is the culmination of a decade of hard work,\" said Andy Ng, co-founder and CTO of DeepMind.\n\"We are thrilled to have achieved this milestone and look forward to continuing to develop AI that\ncan be used in a wide range of applications and to help people live better lives.\"\n\nDeepMind's work on Go began in 2010, when it began to train a neural network to play Go using\nmillions of games played by top Go players around the world. Since then, the team has refined the\nalgorithm, adding more and more layers of reinforcement learning to make it better at recognizing\npatterns and making decisions based on those patterns. In the past year and a half, the team has\nmade significant progress in the game, winning a record-tying 13 games in a row to move into the\ntop four of the world rankings.\n\n\"The game of Go is a complex game in which players have to be very careful not to overextend their\nterritory, and this is something that we have been able to improve over and over again,\" said\nDr. Demis Hassabis, co-founder and Chief Scientific Officer of DeepMind. \"We are very proud of our\nteam's work, and we hope that it will inspire others to take the next step in their research and\napply the same techniques to other problems.\"\n\nIn addition to the win in Go, DeepMind has also developed an AI system that can learn to play a\nnumber of different games, including poker, Go, and chess. This AI system, called Tarsier, was\ndeveloped in partnership with Carnegie Mellon University and the University of California, \nBerkeley, and is being used to teach computer vision and machine learning to identify objects in\nimages and recognize speech in natural language. Tarsier has been trained to play the game of Go\nand other games on a number of different platforms...\n----------------------------------------------------------------------------------------------------\n```\n\u003c/details\u003e\n\n##### (3) Huggingface Demo:\n\nAlso check out this awesome [[demo]](https://huggingface.co/spaces/joaogante/contrastive_search_generation) generously supported by Huggingface ([@huggingface](https://github.com/huggingface) :hugs:) which compares contrastive search with other popular decoding methods. Many thanks to Huggingface!\n\n****\n\n\u003cspan id='introduction'/\u003e\n\n#### 1. Introduction: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nText generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions---the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method---contrastive search---to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach outperforms state-of-the-art text generation methods as evaluated by both human and automatic metrics.\n****\n\n\n\u003cspan id='contrastive_search_with_LMs'/\u003e\n\n#### 2. Contrastive Search with GPT-2 and OPT: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nIn this section, we illustrate how to apply contrastive search on GPT-2 models and the [OPT](https://arxiv.org/abs/2205.01068) models released by Meta.\n\n\u003cspan id='install_simctg_package'/\u003e\n\n##### 2.1. Environment Setup:\nTo install our simctg package, simply using the command below. More details of the simctg package can be found \u003ca href='#tutorial'\u003e[here]\u003c/a\u003e and [[here]](https://github.com/yxuansu/SimCTG/tree/main/simctg).\n```yaml\npip install simctg --upgrade\n```\n\n\u003cspan id='contrastive_search_with_gpt2'/\u003e\n\n##### 2.2. Contrastive Search with GPT-2:\nLet's see how to produce text with contrastive search using GPT-2 models. More details can be found [here]((https://github.com/yxuansu/SimCTG/tree/main/simctg)).\n\n(i) First, we load the GPT-2 model as\n```python\nimport torch\nfrom simctg.simctggpt import SimCTGGPT\nmodel_name = r'gpt2-large'\nmodel = SimCTGGPT(model_name)\nmodel.eval()\ntokenizer = model.tokenizer\neos_token_id = tokenizer.eos_token_id\n```\n\n(ii) Then, we prepare the prefix text as\n```python\nprefix_text = r\"DeepMind Company is\"\ntokens = tokenizer.tokenize(prefix_text)\ninput_ids = tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n```\n\n(iii) Last, we generate the text with contrastive search as\n```python\nbeam_width, alpha, decoding_len = 4, 0.6, 256\noutput = model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width, \n                                       alpha=alpha, decoding_len=decoding_len,\n                                      end_of_sequence_token_id = eos_token_id, early_stop = True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(tokenizer.decode(output))\nprint(\"\" + 100 * '-')\n```\n\u003cdetails open\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e\n  \n```\nOutput:\n----------------------------------------------------------------------------------------------------\nDeepMind Company is a leader in artificial intelligence (AI). We have a long history of working with \ncompanies such as Google, Facebook, Amazon, and Microsoft to build products that improve people's \nlives, and today we are excited to announce that DeepMind's AlphaGo program has won the game of Go,\nbecoming the first program to defeat a professional Go player.\n\nThe victory is a testament to the power of deep learning, and to the incredible work of our research\nteam, which has been at the forefront of AI research for the past five years. AlphaGo is one of the\nmost advanced Go programs ever created, and its performance is an important step towards the goal of\nhuman-level AI.\n\n\"This is the culmination of a decade of hard work,\" said Andy Ng, co-founder and CTO of DeepMind. \n\"We are thrilled to have achieved this milestone and look forward to continuing to develop AI that can\nbe used in a wide range of applications and to help people live better lives.\"\n\nDeepMind's work on Go began in 2010, when it began to train a neural network to play Go using millions\nof games played by top Go players around the world. Since then, the team has refined the algorithm,\nadding more and more layers of reinforcement learning to make it...\n----------------------------------------------------------------------------------------------------\n```\n\u003c/details\u003e\n\nFor comparison, let's see the result generated by **greedy search**.\n```python\ndecoding_len = 256\noutput = model.greedy_search(input_ids=input_ids, decoding_len=decoding_len,\n                                       end_of_sequence_token_id = eos_token_id, early_stop = True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(tokenizer.decode(output))\nprint(\"\" + 100 * '-')\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e\n  \n```\nOutput:\n----------------------------------------------------------------------------------------------------\nDeepMind Company is a leading AI research company, with a focus on deep learning and deep learning-based systems.\n\nThe company's research is focused on the development of deep learning-based systems that can learn from large \namounts of data, and that can be used to solve real-world problems.\n\nDeepMind's research is also used by the UK government to develop new technologies for the UK's National Health Service.\n\nDeepMind's research is also used by the UK government to develop new technologies for the UK's National Health Service.\n\nDeepMind's research is also used by the UK government to develop new technologies for the UK's National Health Service.\n\nDeepMind's research is also used by the UK government to develop new technologies for the UK's National Health Service.\n\nDeepMind's research is also used by the UK government to develop new technologies for the UK's National Health Service.\n\nDeepMind's research is also used by the UK government to develop new technologies for the UK's National Health Service.\n\nDeepMind's research is also used by the UK government to develop new technologies for the UK's National Health Service.\n\nDeepMind's research is also used by the UK government to develop new technologies for the UK's National Health Service...\n----------------------------------------------------------------------------------------------------\n```\n\u003c/details\u003e\n\nWe can also see the result generated by **nucleus sampling** (p=0.95).\n```python\ndecoding_len = 256\noutput = model.nucleus_sampling(input_ids=input_ids, decoding_len=decoding_len, nucleus_p=0.95,\n                                       end_of_sequence_token_id = eos_token_id, early_stop = True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(tokenizer.decode(output[1:]))\nprint(\"\" + 100 * '-')\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e\n  \n```\nOutput:\n----------------------------------------------------------------------------------------------------\nDeepMind Company is a Cardiff-based start-up with an exclusive mission to build the world’s largest ever\ndeep-learning system to analyse the world’s digital content and in particular, super-sized image content.\n  \nThe system, the largest in the world with no previous expertise in image or digital content detection, \nwill have previously relied on a mixture of machine learning, artificial neural networks, and storage,\nprocessing and retrieval techniques.\n  \nThe AI system, called ImageNet, will take new approach to our challenge of data science and machine\nlearning, significantly improving efficiency, natural language processing and full understanding of\ncomplex, high-dimensional images, with an Eye of the Tiger framework for extracting techniques to\nensure correct detection of particular images in complex scenes.\n  \nDr. Mark Ward, Dr. Alex Kudle, Dr. Ralph Pinchbeck and CTO, DeepMind Dr. Alex Kudle\n  \nCase Study: Derpy’s Most Wanted: Fighting Cybersecurity, building a robot-aided smuggling network\n  \nInfoSec News, 06/07/2017\n  \nDimitrios Papadimitriou (left) and Chris Bardy (right) at G+ XE, July 2017\n  \nHow to model an industrial malware botn...\n----------------------------------------------------------------------------------------------------\n```\n\u003c/details\u003e\n\n\n\u003cspan id='contrastive_search_with_opt'/\u003e\n\n##### 2.3. Contrastive Search with OPT:\nLet's see how to produce text with contrastive search using OPT models. More details can be found [here]((https://github.com/yxuansu/SimCTG/tree/main/simctg)).\n\n(i) First, we load the OPT model as\n```python\nimport torch\nfrom simctg.simctgopt import SimCTGOPT\nmodel_name = 'facebook/opt-6.7b'\nmodel = SimCTGOPT(model_name)\ntokenizer = model.tokenizer\nmodel.eval()\nbos_token_id = tokenizer.bos_token_id\neos_token_id = tokenizer.eos_token_id\n```\n\n(ii) Then, we use the same example from the original [paper](https://arxiv.org/abs/2205.01068) (see Figure 9 in the Appendix E) to show how to generate text with contrastive search. The prefix text is provided as\n```python\nprefix_text = r\"\"\"A chat between a curious human and the Statue of Liberty.\n\nHuman: What is your name?\nStatue: I am the Statue of Liberty.\nHuman: Where do you live?\nStatue: New York City.\nHuman: How long have you lived there?\"\"\"\n```\n\n(iii) We prepare the input ids as\n\n**[Important Tip]** As the authors suggested in their [[tutorial]](https://huggingface.co/docs/transformers/model_doc/opt), in contrastive to GPT2, OPT adds the EOS token \u003c/s\u003e to the beginning of every prompt. So make sure the special token is added at the front of the prompt.\n\n```python\ntokens = tokenizer.tokenize(prefix_text)\ninput_ids = [bos_token_id] + tokenizer.convert_tokens_to_ids(tokens) # adds \u003c/s\u003e to the beginning of every prompt\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n```\n\n(iv) Last, we generate the text with contrastive search as\n```python\nbeam_width, alpha, decoding_len = 5, 0.6, 256\noutput = model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width, \n                                       alpha=alpha, decoding_len=decoding_len,\n                                       end_of_sequence_token_id = eos_token_id, early_stop = True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(tokenizer.decode(output[1:]))\nprint(\"\" + 100 * '-')\n```\n\n\u003cdetails open\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\nOutput:\n----------------------------------------------------------------------------------------------------\nA chat between a curious human and the Statue of Liberty.\n\nHuman: What is your name?\nStatue: I am the Statue of Liberty.\nHuman: Where do you live?\nStatue: New York City.\nHuman: How long have you lived there?\nStatue: Since 1884.\nHuman: Why did you come to America?\nStatue: I was given to the United States by France as a gift for helping the French during the Franco-Prussian War.\nHuman: What do you think of America?\nStatue: I love it. It is the greatest country in the world.\nHuman: What’s the weather like in New York?\nStatue: It is cold.\nHuman: Is it safe to walk around at night?\nStatue: Yes. There are policemen everywhere.\nHuman: Do you have any children?\nStatue: Not yet. My pedestal is empty.\nHuman: What would you like to say to people who want to immigrate to America?\nStatue: Come on over. You will be happy here. We have everything you need.\n----------------------------------------------------------------------------------------------------\n```\n\u003c/details\u003e\n\nFor comparison, let's see the result generated by **greedy search**.\n```python\ndecoding_len = 256\noutput = model.greedy_search(input_ids=input_ids, decoding_len=decoding_len,\n                                       end_of_sequence_token_id = eos_token_id, early_stop = True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(tokenizer.decode(output[1:]))\nprint(\"\" + 100 * '-')\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e\n  \n```\nOutput:\n----------------------------------------------------------------------------------------------------\nA chat between a curious human and the Statue of Liberty.\n\nHuman: What is your name?\nStatue: I am the Statue of Liberty.\nHuman: Where do you live?\nStatue: New York City.\nHuman: How long have you lived there?\nStatue: I have lived here for over 100 years.\nHuman: What do you do?\nStatue: I welcome people from all over the world to come to America.\nHuman: What do you think of America?\nStatue: I love America.\nHuman: What do you think of immigrants?\nStatue: I love immigrants.\nHuman: What do you think of America?\nStatue: I love America.\nHuman: What do you think of immigrants?\nStatue: I love immigrants.\nHuman: What do you think of America?\nStatue: I love America.\nHuman: What do you think of immigrants?\nStatue: I love immigrants.\nHuman: What do you think of America?\nStatue: I love America.\nHuman: What do you think of immigrants?\nStatue: I love immigrants.\nHuman: What do you think of America?\nStatue: I love America.\nHuman: What do you think of immigrants?\nStatue: I love immigrants.\nHuman: What do you think of America?\nStatue: I love America.\nHuman: What do you think of immigrants?\nStatue: I love immigrants.\nHuman...\n----------------------------------------------------------------------------------------------------\n```\n\u003c/details\u003e\n\nWe can also see the result generated by **nucleus sampling** (p=0.95).\n```python\ndecoding_len = 256\noutput = model.nucleus_sampling(input_ids=input_ids, decoding_len=decoding_len, nucleus_p=0.95,\n                                       end_of_sequence_token_id = eos_token_id, early_stop = True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(tokenizer.decode(output[1:]))\nprint(\"\" + 100 * '-')\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e\n  \n```\nOutput:\n----------------------------------------------------------------------------------------------------\nA chat between a curious human and the Statue of Liberty.\n\nHuman: What is your name?\nStatue: I am the Statue of Liberty.\nHuman: Where do you live?\nStatue: New York City.\nHuman: How long have you lived there?\nStatue: Since 1876.\nHuman: Why is the Statue of Liberty guarded?\nStatue: Because there are many people trying to steal her.\n\na comparison about an unexpressed thought\n\nI would also share the story of “A Humble Fear.” At a conference in New York the Dalai Lama gave a \nspeech to the International Thinkers Congress in New York. The whole thing was recorded, and the \nvideo is quite interesting. (on a side note, I love the fact that there were some people who laughed\nwhen he described himself as a humble being… I think the video is hilarious, there is a reason why\nI put up the video. Because if you cannot find the humor in this you’re sadly lacking…)\n\nIn the speech, the Dalai Lama compares the search for truth to searching for treasure. He says: \n“However there is a huge difference between being a thief and a collector. A thief simply takes things, \nwhereas a collector looks for the beauty, even if it is just a single object.”\n\nThe above quote is perhaps the most cliched Buddhist philosophy of our times. However the comparison\nbetween a collector and a thief is quite interesting. I like to think that the Buddha...\n----------------------------------------------------------------------------------------------------\n```\n\u003c/details\u003e\n\n\n\n****\n\n\u003cspan id='models'/\u003e\n\n#### 3. Huggingface Models: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\n\n|Model Name|Task|Language|Training Corpus (Size)|Model Size|Model Address|\n|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n|cambridgeltl/simctg_wikitext103|Document Generation|English|Wikitext-103 (529MB)|117M|[[link]](https://huggingface.co/cambridgeltl/simctg_wikitext103/)|\n|cambridgeltl/simctg_lccc_dialogue|Open-domain Dialogue Generation|Chinese|LCCC (708MB)|117M|[[link]](https://huggingface.co/cambridgeltl/simctg_lccc_dialogue/)|\n|cambridgeltl/simctg_english_wikipedia|General Domain Pre-training|English|Wikipedia (14.11GB)|117M|[[link]](https://huggingface.co/cambridgeltl/simctg_english_wikipedia/)|\n|cambridgeltl/simctg_writingprompts|Open-Ended Story Generation|English|WritingPrompts (865MB)|117M|[[link]](https://huggingface.co/cambridgeltl/simctg_writingprompts/)|\n|cambridgeltl/simctg_rocstories|Open-Ended Story Generation|English|ROCStories (12MB)|117M|[[link]](https://huggingface.co/cambridgeltl/simctg_rocstories/)|\n\n\n****\n\n\u003cspan id='tutorial'/\u003e\n\n#### 4. Huggingface-Style Tutorials: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nWe have encapsulated our work as an easy-to-use library (i.e., package). In the following, we provide huggingface-style tutorials on how to use SimCTG and contrastive search with just **a few lines of code**! \n\n:star: **[Documentation]** We have provided detailed documentation of the (i) **source code** of the package and (ii) **instructions** on how to use it. Please refer to [[here]](https://github.com/yxuansu/SimCTG/tree/main/simctg).\n\n:star: **[Google Colab]** We provide a Google Colab for the easy reproductivity of our tutorial. [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1ImvR-ldHf9rJyFzOCMJ_zjAGK8n1iTW7?usp=sharing)\n\n\u003cspan id='install_simctg'/\u003e\n\n##### 4.1. Install and Load SimCTG:\nTo use our package, we recommand you to use Python with version \u003e= 3.6. The SimCTG can be installed and loaded with the commands below.\n\n(1) Install SimCTG with pip.\n```yaml\npip install simctg --upgrade\n```\n\n(2) Load SimCTG package with Python.\n```python\nimport torch\n# load SimCTG language model\nfrom simctg.simctggpt import SimCTGGPT\n# load SimCTG loss class\nfrom simctg.lossfunction import SimCTGLoss\n```\n\n\n\u003cspan id='example_train_with_simctg'/\u003e\n\n##### 4.2. Example of Training Language Model with SimCTG:\n\n\u003cspan id='init_simctg'/\u003e\n\n###### 4.2.1. Initialize Language Model:\n```python\nmodel_name = r'gpt2'\n# initialize the language model with a vanilla GPT-2\nmodel = SimCTGGPT(model_name)\ntokenizer = model.tokenizer\n```\n\n:bell: The detailed description of SimCTGGPT can be found [[here]](https://github.com/yxuansu/SimCTG/blob/main/simctg/README.md#3-simctggpt-class).\n\n\u003cspan id='init_loss_class'/\u003e\n\n###### 4.2.2. Initialize Loss Class:\n```python\nmargin = 0.5\nvocab_size = len(tokenizer)\npad_token_id = tokenizer.bos_token_id\nsimctgloss = SimCTGLoss(margin=margin, vocab_size=vocab_size, pad_token_id=pad_token_id)\n```\n\n:bell: The detailed description of SimCTGLoss can be found [[here]](https://github.com/yxuansu/SimCTG/blob/main/simctg/README.md#2-simctgloss-class).\n\n**[Note]** If the margin is set as 0.0, then the SimCTG loss is equivalent to the MLE loss. \n\n\u003cspan id='init_training_data'/\u003e\n\n###### 4.2.3. Create Example Training Data:\n```python\nfrom torch.nn.utils import rnn\ntext_list = ['Pandas are so cute!', 'The weather in Cambridge today is very good!']\n# transform batch of texts to batch of token ids\ntokens_list = [tokenizer.tokenize(text) for text in text_list]\nbatch_id_list = [tokenizer.convert_tokens_to_ids(item) for item in tokens_list]\nbatch_id_list = [torch.LongTensor(item) for item in batch_id_list]\n# pad the batch of token ids\nbatch_tensor = rnn.pad_sequence(batch_id_list, batch_first=True, padding_value=pad_token_id)\n# get batch input ids and batch label ids\nbatch_inputs = batch_tensor[:, :-1].clone()\nbatch_labels = batch_tensor[:, 1:].clone()\n# by setting pad token ids as -100, we stop the gradient update on these padded positions\nbatch_labels[batch_labels[:, :] == pad_token_id] = -100\n```\n\n\u003cspan id='compute_loss'/\u003e\n\n###### 4.2.4. Compute Loss:\n```python\n# forward computation\nlast_hidden_states, logits = model(input_ids=batch_inputs, labels=batch_labels)\n# loss computation\nmle_loss, cl_loss = simctgloss(last_hidden_states=last_hidden_states, logits=logits, \n                               input_ids=batch_inputs, labels=batch_labels)\nsimctg_loss = mle_loss + cl_loss\n```\n\n**[Note]** If the margin in SimCTG loss is set as 0.0, the returned cl_loss will always be 0.0 and the SimCTG loss is equivalent to the MLE loss.\n\n\u003cspan id='contrastive_search_examples'/\u003e\n\n##### 4.3. Examples of Performing Generation with Contrastive Search:\n\n\u003cspan id='example_document_generation'/\u003e\n\n###### 4.3.1. Open-Ended Document Generation:\n\nWe show how to reproduce our result in the case study (i.e., Table 4) of our paper.\n```python\nimport torch\n# load SimCTG language model\nfrom simctg.simctggpt import SimCTGGPT\nmodel_name = r'cambridgeltl/simctg_wikitext103'\nmodel = SimCTGGPT(model_name)\nmodel.eval()\ntokenizer = model.tokenizer\n\n# prepare input\nprefix_text = r\"Butt criticized Donald 's controls in certain situations in the game , as well as the difficulty of some levels and puzzles . Buchanan also criticized the controls , calling\"\nprint ('Prefix is: {}'.format(prefix_text))\ntokens = tokenizer.tokenize(prefix_text)\ninput_ids = tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\n# generate result\nbeam_width, alpha, decoding_len = 8, 0.6, 128\noutput = model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width, \n                                       alpha=alpha, decoding_len=decoding_len) \nprint(\"Output:\\n\" + 100 * '-')\nprint(tokenizer.decode(output))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\nOutput:\n----------------------------------------------------------------------------------------------------\nButt criticized Donald's controls in certain situations in the game, as well as the difficulty of some\nlevels and puzzles. Buchanan also criticized the controls, calling them \" unimpressive \" and a \" nightmare \"\nof an experience to play with players unfamiliar with Tetris. On the other hand, his opinion was shared by\nother reviewers, and some were critical of the game's technical design for the Wii version of Tetris.\nIn addition, Tintin's review included a quote from Roger Ebert, who said that Tetris was better than the\noriginal game due to its simplicity and ease of play. Ebert's comments were included in the game's DVD\ncommentary, released on March 22, 2010. It is unclear if any of the video commentary was taken from the DVD...\n```\n\u003c/details\u003e\n\n\u003cspan id='example_dialogue_generation'/\u003e\n\n###### 4.3.2. Open-Domain Dialogue Generation:\nWe show how to reproduce our result in the case study (i.e., Table 7) of our paper.\n```python\nimport torch\n# load SimCTG language model\nfrom simctg.simctggpt import SimCTGGPT\nmodel_name = r'cambridgeltl/simctg_lccc_dialogue'\nmodel = SimCTGGPT(model_name)\nmodel.eval()\ntokenizer = model.tokenizer\neos_token = '[SEP]'\neos_token_id = tokenizer.convert_tokens_to_ids([eos_token])[0]\n\n# prepare input\ncontext_list = ['刺猬很可爱！以前别人送了只没养，味儿太大！', '是很可爱但是非常臭', '是啊，没办法养', '那个怎么养哦不会扎手吗']\nprefix_text = eos_token.join(context_list).strip(eos_token) + eos_token\nprint ('Prefix is: {}'.format(prefix_text))\ntokens = tokenizer.tokenize(prefix_text)\ninput_ids = tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\n# generate result\nbeam_width, alpha, decoding_len = 5, 0.6, 64\noutput = model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width, alpha=alpha, \n                                       decoding_len=decoding_len, end_of_sequence_token_id=eos_token_id,\n                                       early_stop=True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(''.join(tokenizer.decode(output).split()))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\nOutput:\n----------------------------------------------------------------------------------------------------\n刺猬很可爱！以前别人送了只没养，味儿太大！[SEP]是很可爱但是非常臭[SEP]是啊，没办法养[SEP]那个怎么养哦不会扎手吗[SEP]我觉得还好，就是有点臭\n```\n\u003c/details\u003e\n\n\u003cspan id='example_off_the_shelf_generation'/\u003e\n\n##### 4.4. Contrastive Search with Off-the-shelf Language Models from Different Languages:\nIn the following, we show how to apply contrastive search on off-the-shelf language models of different languages.\n\n\u003cspan id='chinese_example_off_the_shelf_generation'/\u003e\n\n###### 4.4.1. Chinese Language Model:\n```python\nimport torch\n# load SimCTG language model\nfrom simctg.simctggpt import SimCTGGPT\nmodel_name = r'uer/gpt2-chinese-cluecorpussmall'\nmodel = SimCTGGPT(model_name)\nmodel.eval()\ntokenizer = model.tokenizer\neos_token = '[SEP]'\neos_token_id = tokenizer.convert_tokens_to_ids([eos_token])[0]\n\n# Example 1\nprefix_text = '苹果公司'\nprint ('Prefix is: {}'.format(prefix_text))\ntokens = tokenizer.tokenize(prefix_text)\ninput_ids = tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\nbeam_width, alpha, decoding_len = 3, 0.6, 128\noutput = model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width, alpha=alpha, \n                                       decoding_len=decoding_len, end_of_sequence_token_id=eos_token_id,\n                                       early_stop=True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(''.join(tokenizer.decode(output).split()))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\nOutput:\n----------------------------------------------------------------------------------------------------\n苹果公司在中国市场推出的iphone7，不仅在外观设计上有所改变，在配置上也进行了升级。苹果还宣布，新一代iphone将采用5.7英寸\n屏幕，分辨率达到2560×1440像素，显示效果非常出色。此外，该机还支持指纹识别功能，可实现手指快速扫描、人脸识别等功能。\n```\n\u003c/details\u003e\n\n```python\n# Example 2\nprefix_text = '百节年为首，春节是中华民族最隆重的传统佳节。它不仅集中体现了中华'\nprint ('Prefix is: {}'.format(prefix_text))\ntokens = tokenizer.tokenize(prefix_text)\ninput_ids = tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\nbeam_width, alpha, decoding_len = 3, 0.6, 128\noutput = model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width, alpha=alpha, \n                                       decoding_len=decoding_len, end_of_sequence_token_id=eos_token_id,\n                                       early_stop=True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(''.join(tokenizer.decode(output).split()))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\nOutput:\n----------------------------------------------------------------------------------------------------\n百节年为首，春节是中华民族最隆重的传统佳节。它不仅集中体现了中华文化精髓，也表现了人民群众生活水平的提高和对美好生活的向往。\n```\n\u003c/details\u003e\n\n\u003cspan id='japanese_example_off_the_shelf_generation'/\u003e\n\n###### 4.4.2. Japanese Language Model:\n```python\nimport torch\n# load SimCTG language model\nfrom simctg.simctggpt import SimCTGGPT\nmodel_name = r'colorfulscoop/gpt2-small-ja'\nmodel = SimCTGGPT(model_name)\nmodel.eval()\ntokenizer = model.tokenizer\neos_token = tokenizer.eos_token\neos_token_id = tokenizer.convert_tokens_to_ids([eos_token])[0]\n\n# prepare input\nprefix_text = r'臥龍桜（がりゅうざくら）は、岐阜県高山市一之宮町にある一本桜。龍が地'\nprint ('Prefix is: {}'.format(prefix_text))\ntokens = tokenizer.tokenize(prefix_text)\ninput_ids = tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\n# generate result\nbeam_width, alpha, decoding_len = 5, 0.6, 128\noutput = model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width, alpha=alpha, \n                                       decoding_len=decoding_len, end_of_sequence_token_id=eos_token_id,\n                                       early_stop=True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(tokenizer.decode(output))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\nOutput:\n----------------------------------------------------------------------------------------------------\n臥龍桜(がりゅうざくら)は、岐阜県高山市一之宮町にある一本桜。龍が地中に染みつく様子を図案化したもので、樹齢400年を越す日本\nさくら名所100選に選定されている。一之宮町指定天然記念物。岐阜県飛騨地方(東濃地方)の山間地に生育し、約1万年前に絶滅したと\n考えられている。「花の本」とも称され、開花期は5月上旬から下旬までで、桜の枝張りは濃緑色である。花は直径約10cmの花弁を咲か\nせる八重咲きで、花弁の色は紅紫色で、雄しべは4本、雌しべは1本ある。雄しべの先\n```\n\u003c/details\u003e\n\n\u003cspan id='korean_example_off_the_shelf_generation'/\u003e\n\n###### 4.4.3. Korean Language Model:\n```python\nimport torch\n# load SimCTG language model\nfrom simctg.simctggpt import SimCTGGPT\nmodel_name = r'skt/ko-gpt-trinity-1.2B-v0.5'\nmodel = SimCTGGPT(model_name)\nmodel.eval()\ntokenizer = model.tokenizer\neos_token = tokenizer.eos_token\neos_token_id = tokenizer.convert_tokens_to_ids([eos_token])[0]\n\n# prepare input\nprefix_text = r'인간처럼 생각하고, 행동하는 \\'지능\\'을 통해 인류가 이제까지 풀지 못했던'\nprint ('Prefix is: {}'.format(prefix_text))\ntokens = tokenizer.tokenize(prefix_text)\ninput_ids = tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\n# generate result\nbeam_width, alpha, decoding_len = 5, 0.6, 64\noutput = model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width, alpha=alpha, \n                                       decoding_len=decoding_len, end_of_sequence_token_id=eos_token_id,\n                                       early_stop=True) \nprint(\"Output:\\n\" + 100 * '-')\nprint(tokenizer.decode(output))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\nOutput:\n----------------------------------------------------------------------------------------------------\n인간처럼 생각하고, 행동하는 \\'지능\\'을 통해 인류가 이제까지 풀지 못했던 난제를 해결하려 한다. 이 책의 제목이기도 한 '슈퍼인텔리전스'는 \n인공지능(AI)의 등장으로 야기된 사회 변화를 일컫는 말로, 이 책을 관통하는 키워드이기도 하다. 저자는 \"기술과 인간 사이의 경계가 무너지고 \n있다\"고 지적한다. AI가 인간의 사고방식과 행동을 모방할 뿐만\n```\n\u003c/details\u003e\n\n\n\u003cspan id='training_tutorial'/\u003e\n\n##### 4.5. Detailed Tutorial of Training SimCTG on Wikitext-103: \nWe also provide a comprehensive tutorial on how to reproduce our experiments on Wikitext-103 using the released package. Check it [[here]](https://github.com/yxuansu/SimCTG/tree/main/training_tutorial_on_wikitext103)!\n\n\n\u003cspan id='T5_tutorial'/\u003e\n\n##### 4.6. Apply SimCTG on T5:\nWe also provide detailed tutorials on how to apply SimCTG and contrastive search on T5 model. For more details, please refer to [[here]](https://github.com/yxuansu/SimCTG/tree/main/simctg#4-simctgt5-class-back-to-top) and [[here]](https://github.com/yxuansu/SimCTG/blob/main/SimCTGEncDec/README.md#2-t5-back-to-top).\n\n\n****\n\n\u003cspan id='environment_setup'/\u003e\n\n#### 5. Environment Setup: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\n```yaml\npython version \u003e= 3.6\npip3 install -r requirements.txt\n```\n\n****\n\n:exclamation::exclamation::exclamation: **[Note]** The following instructions were originally used in the experiments of our paper. Now we have provided an easy-to-use library which helps you to implement SimCTG with just a few lines of code (**Of course, the original code still works!**). Check it [[here]](#tutorial)!\n\n****\n\u003cspan id='example_usage'/\u003e\n\n#### 6. Example Usage of Contrastive Search: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\n\n\u003cspan id='example_usage_english_simctg'/\u003e\n\n##### 6.1. Use SimCTG Pretrained on Wikipedia Corpus:\nHere, we show how to use contrastive search to generate the result.\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1MhK3cVHW9HQ1ArXu0M_sS_Po0_4N1xgQ?usp=sharing)\n\n```python\nimport torch\nimport sys\nsys.path.append(r'./pretraining')\nfrom simctg import SimCTGPretraining\n# load SimCTG model pretrained on the large-scale Wikipedia corpus\nmodel_path = r'cambridgeltl/simctg_english_wikipedia'\nmodel = SimCTGPretraining(model_path)\nmodel.eval()\n\n# we randomly select a prefix from the dev set of Wikipedia pre-training corpus and prepare the text prefix input\ntext = r'Insect farming is the practice of raising and breeding insects as livestock, also referred to as minilivestock or micro stock. Insects may be farmed for the commodities'\ntokens = model.tokenizer.tokenize(text)\ninput_ids = model.tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\n# use contrastive search to generate the result\nbeam_width, alpha, decoding_len = 5, 0.6, 128\neos_token = '\u003c|endoftext|\u003e'\nprint (model.fast_contrastive_search(input_ids, beam_width, alpha, decoding_len, eos_token))\n```\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e\n  \n```yaml\nInsect farming is the practice of raising and breeding insects as livestock, also referred to as minilivestock\nor micro stock. Insects may be farmed for the  commodities they produce, such as honey, corn, sorghum, and \nother crops. In some cases, the production of insects is a way to increase income for the owner or his family. \nThis type of farming has been described as \"an economic system that benefits all people regardless of race, sex, \nor social status\" (p.\\xa09). A large number of farmers in North America, Europe, and South America have used the \nmethod of farming for food production in order to feed their families and livestock. The most common method of \nfarming is by hand-cropping, which consists of cutting a hole in the ground and using a saw\n```\n\u003c/details\u003e\n\n\nMore details on how to pre-train SimCTG on large-scale corpus and the details of the argument setup in contrastive search can be found [[here]](https://github.com/yxuansu/SimCTG/tree/main/pretraining).\n\n\n\u003cspan id='example_usage_different_language_model'/\u003e\n\n##### 6.2. Use Off-the-shelf Language Models from Different Languages:\nImportantly, we found that contrastive search can be directly applied to **off-the-shelf** language models even **without** contrastive training. The only condition is that \u003cins\u003ethe corresponding language should be naturally tokenized by character units\u003c/ins\u003e. Some examples include Chinese, Japanese, and Korean. In the following, we showcase how to use contrastive search with off-the-shelf Chinese, Japanese, and Korean language models. More analysis of why contrastive search works well on vanilla language models can be found in the Appendix C of our paper. \n\n\n\u003cspan id='example_usage_chinese_gpt'/\u003e\n\n###### 6.2.1. Chinese Language Model:\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1_55LEg2caLM-lYDVIhWjxgv75IWkEry6?usp=sharing)\n\n```python\nimport torch\nimport sys\nsys.path.append(r'./pretraining')\nfrom simctg import SimCTGPretraining\n# load an off-the-shelf Chinese GPT (https://huggingface.co/uer/gpt2-chinese-cluecorpussmall)\nmodel_path = r'uer/gpt2-chinese-cluecorpussmall'\nmodel = SimCTGPretraining(model_path)\nmodel.eval()\n\n# prepare text prefix input\ntext = r'苹果公司'\ntokens = model.tokenizer.tokenize(text)\ninput_ids = model.tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\n# (1) use contrastive search to generate the result\nbeam_width, alpha, decoding_len = 3, 0.6, 128\neos_token = '[SEP]'\nprint (model.fast_contrastive_search(input_ids, beam_width, alpha, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n苹果公司在中国市场推出的iphone7，不仅在外观设计上有所改变，在配置上也进行了升级。苹果还宣布，新一代iphone将采用\n5.7英寸屏幕，分辨率达到2560×1440像素，显示效果非常出色。此外，该机还支持指纹识别功能，可实现手指快速扫描、人脸识\n别等功能。\n```\n\u003c/details\u003e\n\n```python \n# (2) use nucleus sampling to generate the result\nnucleus_p, decoding_len = 0.95, 128\neos_token = '[SEP]'\nprint (model.nucleus_sampling(input_ids, nucleus_p, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n苹果公司的设计套件。2.不同的颜色设计有不同的热塑性材质。热塑性材质中的ca34bc是真正能够让人感觉舒适的材质。3.比利\n时家具建筑师埃莉诺特·夏格和大家举一些非常实用又非常普遍的例子在这里艾格的设计师们会简单介绍一下为什么美国家具是比利\n时建筑的一个分支或一个分支，他们\n```\n\u003c/details\u003e\n\n```python \n# (3) use greedy search to generate the result\ndecoding_len = 128\neos_token = '[SEP]'\nprint (model.greedy_search(input_ids, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n苹果公司的一个重要客户，他们的产品在全球范围内都有着非常高的知名度。[UNK]我们的产品在全球范围内都有着非常高的知名度，\n我们的产品在全球范围内都有着非常高的知名度。[UNK]在这样的背景下，苹果公司的产品在全球范围内都有着非常高的知名度。[UNK]\n我们的产品在全球范围内都有着非常高的知\n```\n\u003c/details\u003e\n\n```python \n# (4) use beam search to generate the result\nbeam_width, decoding_len = 10, 128\neos_token = '[SEP]'\nprint (model.beam_search(input_ids, 10, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n苹果公司总裁兼首席执行官蒂姆·库克（timcook）表示：[UNK]苹果公司是全球最大的智能手机制造商之一，苹果公司是全球最大的\n智能手机制造商之一，苹果公司是全球最大的智能手机制造商之一，苹果公司是全球最大的智能手机制造商之一，苹果公司是全球最大\n的智能手机制造商之一，苹果公司是全球\n```\n\u003c/details\u003e\n\n\n```python\n# ------------------------------------------ Another Example --------------------------------------------- #\n# prepare text prefix input\ntext = r'百节年为首，春节是中华民族最隆重的传统佳节。它不仅集中体现了中华'\ntokens = model.tokenizer.tokenize(text)\ninput_ids = model.tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\n# (1) use contrastive search to generate the result\nbeam_width, alpha, decoding_len = 3, 0.6, 128\neos_token = '[SEP]'\nprint (model.fast_contrastive_search(input_ids, beam_width, alpha, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n百节年为首，春节是中华民族最隆重的传统佳节。它不仅集中体现了中华文化精髓，也表现了人民群众生活水平的提高和对美好生活的向往。\n```\n\u003c/details\u003e\n\n```python\n# (2) use nucleus sampling to generate the result\nnucleus_p, decoding_len = 0.95, 128\neos_token = '[SEP]'\nprint (model.nucleus_sampling(input_ids, nucleus_p, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n百节年为首，春节是中华民族最隆重的传统佳节。它不仅集中体现了中华传统文化，更是经济、政治、文化上的一个精神机能的全面发展。\n人们在生活中不仅能够充分认识到这个民族的非物质文化遗产，而且能够在此基础上追求书面化的概念。中国历史上有许多著名的「人物」\n，他们深深地扎根于中国历史的传统历史文化中，热爱中华文化，热爱中华文化的传承\n```\n\u003c/details\u003e\n\n```python\n# (3) use greedy search to generate the result\ndecoding_len = 128\neos_token = '[SEP]'\nprint (model.greedy_search(input_ids, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n百节年为首，春节是中华民族最隆重的传统佳节。它不仅集中体现了中华民族的传统美德，也体现了中华民族的传统文化。[UNK]中华民族\n的传统美德，是中华民族的传统美德。[UNK]中华民族的传统美德，是中华民族的传统美德。[UNK]中华民族的传统美德，是中华民族的传\n统美德。[UNK]中华民族的传统美德，是中华民族的传统美德。[UNK]中华民族的传统美德，是中华民族的传\n```\n\u003c/details\u003e\n\n```python\n# (4) use beam search to generate the result\nbeam_width, decoding_len = 10, 128\neos_token = '[SEP]'\nprint (model.beam_search(input_ids, 10, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n百节年为首，春节是中华民族最隆重的传统佳节。它不仅集中体现了中华民族伟大复兴的历史使命，也体现了中华民族伟大复兴的历史使命。\n中华民族伟大复兴的历史使命，不仅体现了中华民族伟大复兴的历史使命，也体现了中华民族伟大复兴的历史使命。中华民族伟大复兴的历\n史使命，不仅体现了中华民族伟大复兴的历史使命，也体现了中华民族伟大复兴的历\n```\n\u003c/details\u003e\n\nMore details on how to use different decoding methods to generate the result can be found [[here]](https://github.com/yxuansu/SimCTG/tree/main/pretraining).\n\n\n\u003cspan id='example_usage_japanese_gpt'/\u003e\n\n###### 6.2.2. Japanese Language Model:\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1844kf-BttuPt1DaYhdgw-07-qz7V7pOd?usp=sharing)\n\n```python\nimport torch\nimport sys\nsys.path.append(r'./pretraining')\nfrom simctg import SimCTGPretraining\n# load an off-the-shelf Japanese GPT (https://huggingface.co/colorfulscoop/gpt2-small-ja)\nmodel_path = r'colorfulscoop/gpt2-small-ja'\nmodel = SimCTGPretraining(model_path)\nmodel.eval()\n\n'''\n   Prepare text prefix input. The prefix is copied from a random Japanese Wikipedia \n   page here (https://ja.wikipedia.org/wiki/%E8%87%A5%E9%BE%8D%E6%A1%9C).\n'''\ntext = r'臥龍桜（がりゅうざくら）は、岐阜県高山市一之宮町にある一本桜。龍が地'\ntokens = model.tokenizer.tokenize(text)\ninput_ids = model.tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\n# (1) use contrastive search to generate the result\nbeam_width, alpha, decoding_len = 5, 0.6, 128\neos_token = model.tokenizer.eos_token\nprint (model.fast_contrastive_search(input_ids, beam_width, alpha, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n臥龍桜(がりゅうざくら)は、岐阜県高山市一之宮町にある一本桜。龍が地中に染みつく様子を図案化したもので、樹齢400年\nを越す日本さくら名所100選に選定されている。一之宮町指定天然記念物。岐阜県飛騨地方(東濃地方)の山間地に生育し、約\n1万年前に絶滅したと考えられている。「花の本」とも称され、開花期は5月上旬から下旬までで、桜の枝張りは濃緑色である。\n花は直径約10cmの花弁を咲かせる八重咲きで、花弁の色は紅紫色で、雄しべは4本、雌しべは1本ある。雄しべの先\n```\n\u003c/details\u003e\n\n\n\n```python\n# (2) use nucleus sampling to generate the result\nnucleus_p, decoding_len = 0.95, 128\neos_token = model.tokenizer.eos_token\nprint (model.nucleus_sampling(input_ids, nucleus_p, decoding_len, eos_token))\n```\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n臥龍桜(がりゅうざくら)は、岐阜県高山市一之宮町にある一本桜。龍が地中に棲む奇岩に由来する。毎年5月上旬には多くの花見\n客が訪れている。かつて、雪見の藩お抱え家臣、雲口である長久城主長久竜泰が祭っている「月輪寺」には手水鉢が2つあり、長\n久氏の勢力が強まると同時に関連する寺もあり、山を挟むように吉野側の赤峰山から北へ順に樹齢250年を越してきたが、江戸時\n代に廃材が搬出されてから薪が取れなくなっている。古い株は毎年12月の初午に燃えつき風雨が吹き荒れて朽ち果てる。根は分枝\n```\n\u003c/details\u003e\n\n\n```python\n# (3) use greedy search to generate the result\ndecoding_len = 128\neos_token = model.tokenizer.eos_token\nprint (model.greedy_search(input_ids, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n臥龍桜(がりゅうざくら)は、岐阜県高山市一之宮町にある一本桜。龍が地中に棲む龍の棲むとされる桜で、樹齢は1000年以上。樹\n高は10mほどで、幹周りは8mほどになる。樹齢は300年ほどで、樹高は20mほどになる。樹形が整っており、枝張りも良く、樹勢も\n旺盛である。樹形は、樹高が1mほどで、幹周りは4mほどになる。枝張りはよく発達し、樹勢は旺盛である。冬になると、幹周りの\n樹冠が紅葉する。また、紅葉の時期には、樹冠が赤く紅葉する。樹\n```\n\u003c/details\u003e\n\n\n```python\n# (4) use beam search to generate the result\nbeam_width, decoding_len = 10, 128\neos_token = model.tokenizer.eos_token\nprint (model.beam_search(input_ids, 10, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n臥龍桜(がりゅうざくら)は、岐阜県高山市一之宮町にある一本桜。龍が地中深くに咲く桜で、岐阜県の天然記念物に指定されている。\n岐阜県高山市一之宮町一之宮(いちのみやちょういちのみや)は、岐阜県高山市一之宮町一之宮にある一本桜である。龍が地中深くに\n咲く桜で、岐阜県の天然記念物に指定されている。岐阜県高山市一之宮町一之宮(いちのみやちょういちのみや)は、岐阜県高山市一\n之宮町一之宮(いちのみやちょういちのみや)は、岐阜県高山市一之宮町一之宮(いちのみやちょういちのみや)は、岐阜県高山\n```\n\u003c/details\u003e\n\n**[Note]** Sadly, I do not speak Japanese (I wish I do!), so I can only judge the quality of the generated text using [Google translate](https://translate.google.com/). It would be great if anyone could tell me whether the generated text is good or not. Thank you in advance!\n\n****\n\n\u003cspan id='example_usage_korean_gpt'/\u003e\n\n###### 6.2.3. Korean Language Model:\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1a8g1n86S-zmGe7Nb0PgVQnqWSAMfIR3D?usp=sharing)\n\n```python\nimport torch\nimport sys\nsys.path.append(r'./pretraining')\nfrom simctg import SimCTGPretraining\n# load an off-the-shelf Korean GPT (https://huggingface.co/skt/ko-gpt-trinity-1.2B-v0.5)\nmodel_path = r'skt/ko-gpt-trinity-1.2B-v0.5'\nmodel = SimCTGPretraining(model_path)\nmodel.eval()\n\n'''\n   Prepare text prefix input.\n'''\ntext = r'인간처럼 생각하고, 행동하는 \\'지능\\'을 통해 인류가 이제까지 풀지 못했던'\ntokens = model.tokenizer.tokenize(text)\ninput_ids = model.tokenizer.convert_tokens_to_ids(tokens)\ninput_ids = torch.LongTensor(input_ids).view(1,-1)\n\n# (1) use contrastive search to generate the result\nbeam_width, alpha, decoding_len = 5, 0.6, 64 \n# because this model is pretty large, so we set the generation length (decoding_len) as 64\neos_token = model.tokenizer.eos_token\nprint (model.fast_contrastive_search(input_ids, beam_width, alpha, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n인간처럼생각하고,행동하는\\'지능\\'을통해인류가이제까지풀지못했던난제를해결하려한다.이책의제목이기도한'슈퍼인텔리전스'는인공지능\n(AI)의등장으로야기된사회변화를일컫는말로,이책을관통하는키워드이기도하다.저자는\"기술과인간사이의경계가무너지고있다\"고지적한다.\nAI가인간의사고방식과행동을모방할뿐만\n```\n\u003c/details\u003e\n\n```python\n# (2) use nucleus sampling to generate the result\nnucleus_p, decoding_len = 0.95, 64\neos_token = model.tokenizer.eos_token\nprint (model.nucleus_sampling(input_ids, nucleus_p, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n인간처럼생각하고,행동하는\\'지능\\'을통해인류가이제까지풀지못했던큰수수께끼를풀수있다.'지능\\'은인공두뇌그자체이기도하지만그공간의\n반영이라는해석도가능하다.예를들면시간부등호처럼복잡한수식을쉽게떠올릴수있다는이야기다.마치구글에검색창에'Quick'이라는단어를입력하\n면자동으로'중력'은일정한법칙에따라\n```\n\u003c/details\u003e\n\n```python\n# (3) use greedy search to generate the result\ndecoding_len = 64\neos_token = model.tokenizer.eos_token\nprint (model.greedy_search(input_ids, decoding_len, eos_token))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel Output:\u003c/b\u003e\u003c/summary\u003e \n\n```\n인간처럼생각하고,행동하는\\'지능\\'을통해인류가이제까지풀지못했던문제를해결할수있다고주장한다.이지능은\\'지능\\'그자체라기보다\\'지능\\'\n그자체를구성하는\\'지능\\'그자체라고할수있다.이지능은\\'지능\\'그자체라기보다\\'지능\\'그자체를구성하는\\'지능\\'그자체라고\n```\n\u003c/details\u003e\n\n\n```python\n# (4) use beam search to generate the result\n# We do not print the result, because beam search stops generation immediately.\n```\n\n**[Note]** Sadly, I am not a Korean speaker either, so I can only judge the quality of the generated text using [Google translate](https://translate.google.com/) as well. It would be great if anyone could tell me whether the generated text is good or not. Thank you!\n\n****\n\n\u003cspan id='wikitext103_tutorial'/\u003e\n\n:exclamation::exclamation::exclamation: **[Note]** The following instructions were originally used in the experiments of our paper. Now we have provided an easy-to-use library which helps you to implement SimCTG with just a few lines of code (**Of course, the original code still works!**). Check it \u003ca href='#tutorial'\u003e[here]\u003c/a\u003e!\n\n#### 7. Document Generation: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nThe detailed tutorial of experiment on document generation is provided [[here]](https://github.com/yxuansu/SimCTG/tree/main/document_generation).\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1_zPZRlbJo5iw_Q7FUhP113udPnciUxVF?usp=sharing)\n\n****\n\n\u003cspan id='dialogue_tutorial'/\u003e\n\n:exclamation::exclamation::exclamation: **[Note]** The following instructions were originally used in the experiments of our paper. Now we have provided an easy-to-use library which helps you to implement SimCTG with just a few lines of code (**Of course, the original code still works!**). Check it \u003ca href='#tutorial'\u003e[here]\u003c/a\u003e!  \n\n#### 8. Open-domain Dialogue Generation: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nThe detailed tutorial of experiment on open-domain dialogue generation provided [[here]](https://github.com/yxuansu/SimCTG/tree/main/dialogue_generation).\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](\nhttps://colab.research.google.com/drive/1_55LEg2caLM-lYDVIhWjxgv75IWkEry6?usp=sharing)\n\n****\n\n\u003cspan id='pretraining'/\u003e\n\n#### 9. Large-Scale Pre-training with SimCTG: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nIn addition to fine-tuning on downstream tasks (e.g. document generation and open-domain dialogue generation), we can also use a large-scale general domain corpus (i.e. Wikipedia) to pre-train a SimCTG model. [Here](https://github.com/yxuansu/SimCTG/tree/main/pretraining), we show the details of how to pre-train SimCTG using a large-scale English Wikipedia corpus.\n\n\n****\n\n\u003cspan id='story_generation'/\u003e\n\n#### 10. Open-Ended Story Generation: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nWe also show how to adapt our approach to open-ended story generation task. The details are provided [[here]](https://github.com/yxuansu/SimCTG/tree/main/story_generation).\n\n\n****\n\n\u003cspan id='contrastive_for_encoder_decoder'/\u003e\n\n#### 11. Contrastive Search on Encoder-Decoder Models: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\n\nDetails on how to apply contrastive search on encoder-decoder models (e.g. BART and T5) can be found [[here]](https://github.com/yxuansu/SimCTG/tree/main/SimCTGEncDec).\n\n****\n\n\u003cspan id='contact'/\u003e\n\n#### 12. Contact: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nIf you have any questions, feel free to contact me via (ys484 at cam.ac.uk).\n\n****\n\n\u003cspan id='simctg_elsewhere'/\u003e\n\n#### 13. SimCTG Elsewhere: \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\n\nWe thank the community's effort for extending SimCTG!\n\n- [Zenn](https://zenn.dev/) has provided [a tutorial and implementation of contrastive search based on T5](https://zenn.dev/kwashizzz/articles/ml-simctg-contrastive-framework). \n- [Tencent's Effidit platform (腾讯智能创作助手)](https://effidit.qq.com/) has integrated SimCTG into its AI sentence completion module. Check it out and have fun! \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyxuansu%2Fsimctg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyxuansu%2Fsimctg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyxuansu%2Fsimctg/lists"}