{"id":23317486,"url":"https://github.com/iwasakiyuuki/bert-abstractive-text-summarization","last_synced_at":"2025-08-10T03:32:59.487Z","repository":{"id":39854172,"uuid":"212520222","full_name":"IwasakiYuuki/Bert-abstractive-text-summarization","owner":"IwasakiYuuki","description":"Japanese Sentence Summarization with BERT","archived":false,"fork":false,"pushed_at":"2023-05-09T02:46:02.000Z","size":77,"stargazers_count":49,"open_issues_count":0,"forks_count":15,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-02T16:50:18.670Z","etag":null,"topics":["bert","deep-learning","japanese-language","pytorch","summarization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IwasakiYuuki.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-03T07:26:17.000Z","updated_at":"2024-09-18T18:27:44.000Z","dependencies_parsed_at":"2022-08-27T19:21:26.763Z","dependency_job_id":null,"html_url":"https://github.com/IwasakiYuuki/Bert-abstractive-text-summarization","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/IwasakiYuuki/Bert-abstractive-text-summarization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IwasakiYuuki%2FBert-abstractive-text-summarization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IwasakiYuuki%2FBert-abstractive-text-summarization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IwasakiYuuki%2FBert-abstractive-text-summarization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IwasakiYuuki%2FBert-abstractive-text-summarization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IwasakiYuuki","download_url":"https://codeload.github.com/IwasakiYuuki/Bert-abstractive-text-summarization/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IwasakiYuuki%2FBert-abstractive-text-summarization/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269672015,"owners_count":24457111,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","deep-learning","japanese-language","pytorch","summarization"],"created_at":"2024-12-20T16:30:40.517Z","updated_at":"2025-08-10T03:32:59.359Z","avatar_url":"https://github.com/IwasakiYuuki.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Abstractive text summarization using BERT\nThis is the models using BERT (refer the paper [Pretraining-Based Natural Language Generation for Text Summarization\n](https://arxiv.org/abs/1902.09243) ) for one of the NLP(Natural Language Processing) task, abstractive text summarization.\n \n## Requirements\n- Python 3.6.5+\n- Pytorch 0.4.1+\n- Tensorflow\n- Pandas\n- tqdm\n- Numpy\n- MeCab\n- Tensorboard X and others...\n\nAll packages used here can be installed by pip as follow:\n\n~~~\npip install -r requirement.txt\n~~~\n\n## Docker\nIf you train the model with GPU, it is easy to use [Pytorch docker images](https://hub.docker.com/r/pytorch/pytorch) in DockerHub.\n \nIn this study, pytorch/pytorch:0.4.1-cuda9-cudnn7-devel(2.62GB) has been used.\n\n## Before using\nWhen you use this, please follow the steps below. \n1. Make a repository named \"/data/checkpoint\" under root. \nAnd put bert_model, vocabulary file and config file for bert. \nThese files can be download [here](http://nlp.ist.i.kyoto-u.ac.jp/index.php?BERT%E6%97%A5%E6%9C%AC%E8%AA%9EPretrained%E3%83%A2%E3%83%87%E3%83%AB).\n\n2. Put data file for training and validate under /workspace/data/. The format is as follow:\n\n```preprocess.py\ndata = {\n    'settings': opt,\n    'dict': {\n        'src': text2token,\n        'tgt': text2token},\n    'train': {\n        'src': content[:100000],\n        'tgt': summary[:100000]},\n    'valid': {\n        'src': content[100000:],\n        'tgt': summary[100000:]}}\ntorch.save(data, opt.save_data)\n```\n\noverall directory structure is as follow:\n```\n`-- data                        # under workspace \n    |-- checkpoint\n    |   |-- bert_config.json    # BERT config file\n    |   |-- pytorch_model.bin   # BERT model file\n    |   `-- vocab.txt           # vocabulary file\n    `-- preprocessed_data.data  # train and valid data file\n```\n## Setting\n|Name   |Value   |\n|---|---|\n|Encoder   |BERT   |\n|Decoder   |Transformer (Only Decoder)   |\n|Embed dimension   |768   |\n|Hidden dimension   |3072   |\n|Encoder layers   |12   |\n|Decoder layers   |8   |\n|Optimizer   |Adam   |\n|Learning rate   |init=0.0001   |\n|Wormup step  |4000   |\n|Input max length   |512   |\n|Batch size   |4   |\n\n## Usage\n### Train the model\n```\npython train.py -data data/preprocessed_data.data -bert_path data/checkpoint/ -proj_share_weight -label_smoothing -batch_size 4 -epoch 10 -save_model trained -save_mode best\n```\n### Generate summarization with trained model\n```\npython summarize.py -model data/checkpoint/trained/trained.chkpt -src data/preprocessed_data.data -vocab data/checkpoint/vocab.txt -output pred.txt\n```\n\n## Resut\n### Tensorboard X image\n![image](https://user-images.githubusercontent.com/24263438/66286505-cd044800-e90c-11e9-8bb8-659173def48d.png)\n\n\n## TODO\n- Eval the model with score such as ROUGE-N\n- Make some examples\n\n## Acknowledge\n- This repository structure and many codes are borrowed from [jadore801120/attention-is-all-you-need-pytorch](https://github.com/jadore801120/attention-is-all-you-need-pytorch).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiwasakiyuuki%2Fbert-abstractive-text-summarization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiwasakiyuuki%2Fbert-abstractive-text-summarization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiwasakiyuuki%2Fbert-abstractive-text-summarization/lists"}