{"id":13753030,"url":"https://github.com/Sleepychord/CogLTX","last_synced_at":"2025-05-09T20:34:42.058Z","repository":{"id":40262358,"uuid":"301370512","full_name":"Sleepychord/CogLTX","owner":"Sleepychord","description":"The source code of NeurIPS 2020 paper \"CogLTX: Applying BERT to Long Texts\"","archived":false,"fork":false,"pushed_at":"2022-05-17T14:26:34.000Z","size":74,"stargazers_count":268,"open_issues_count":17,"forks_count":54,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-16T05:32:30.258Z","etag":null,"topics":["bert","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Sleepychord.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-05T10:26:21.000Z","updated_at":"2024-08-28T12:18:04.000Z","dependencies_parsed_at":"2022-08-17T19:35:09.641Z","dependency_job_id":null,"html_url":"https://github.com/Sleepychord/CogLTX","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sleepychord%2FCogLTX","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sleepychord%2FCogLTX/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sleepychord%2FCogLTX/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sleepychord%2FCogLTX/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Sleepychord","download_url":"https://codeload.github.com/Sleepychord/CogLTX/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253321813,"owners_count":21890471,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","pytorch"],"created_at":"2024-08-03T09:01:14.890Z","updated_at":"2025-05-09T20:34:39.958Z","avatar_url":"https://github.com/Sleepychord.png","language":"Python","funding_links":[],"categories":["BERT优化"],"sub_categories":["大语言对话模型及数据"],"readme":"# CogLTX\n\nCogLTX is a framework to apply current BERT-like pretrained language models to long texts. CogLTX does not need new Transformer structures or pretraining, but want to put forward a solution in finetuning and inference. See the paper (http://keg.cs.tsinghua.edu.cn/jietang/publications/NIPS20-Ding-et-al-CogLTX.pdf) for details. \n\nThis repo is a preview version, we will work out a more easy-to-use version in the future. You need to wait for some time due to another project for the first author, and be assured that we are not forgetting!\n\n## Environment\nThe script `setup_env.sh` describe how to set up an environment by conda. The most important part is \n```\npip install torch==1.3.1 torchvision==0.4.2 transformers==2.4.1 pytorch-lightning==0.6 gensim ujson fuzzywuzzy\n``` \nNote that the backward compatiblity of pytorch-lightning is not good, and some bugs in the 0.6 version are fixed manually in the code. We will change the framework in the later version of CogLTX.\n\n## Preprocess\nThe preprocess is different for different types of data, but in general can be split into 3 phases:\n1. Read and tokenize the data.\n2. Split the long text into blocks by `Buffer.split_document_into_blocks(document, tokenizer, cnt=0, hard=True, properties=None)`. The `hard` parameter switches the mode, i.e. whether to use dynamic programming to decide the separation. `properties` are associated information with the document, usually labels.\n3. Save them as a list of samples by pickle.\n\nSee the folders (`newsqa  hotpotqa  20news`) for details.\n## Running\nTo train CogLTX on a specific task, we need write an entry like `run_newsqa.py, run_hotpotqa.py, run_20news.py`, which includes the configuration, postprocess and evaluation. \n\n## Others\nThe data of NewsQA, hotpotQA and 20news can be found in the original dataset paper, but we do not release the codes and data about Alibaba due to commercial reasons. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSleepychord%2FCogLTX","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSleepychord%2FCogLTX","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSleepychord%2FCogLTX/lists"}