{"id":14987642,"url":"https://github.com/kyzhouhzau/nlpgnn","last_synced_at":"2025-04-05T17:09:27.585Z","repository":{"id":37438762,"uuid":"242922531","full_name":"kyzhouhzau/NLPGNN","owner":"kyzhouhzau","description":"1. Use BERT, ALBERT and GPT2 as  tensorflow2.0's layer.  2. Implement GCN, GAN, GIN and GraphSAGE based on message passing.","archived":false,"fork":false,"pushed_at":"2024-08-14T14:26:03.000Z","size":210,"stargazers_count":335,"open_issues_count":12,"forks_count":66,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-05T17:09:23.108Z","etag":null,"topics":["albert","albert-ner","bert","bert-cls","bert-ner","bilstm-attention","gan","gcn","gin","gnn","gpt2","graph-classfication","graph-convolutional-networks","graphsage","message-passing","nlp","tensorflow2","textcnn","textgcn","tf2"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyzhouhzau.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-25T05:59:23.000Z","updated_at":"2025-03-25T18:01:56.000Z","dependencies_parsed_at":"2024-09-30T01:00:33.941Z","dependency_job_id":"4046f602-fa37-4eb6-9c7a-c312f0a2c203","html_url":"https://github.com/kyzhouhzau/NLPGNN","commit_stats":{"total_commits":6,"total_committers":2,"mean_commits":3.0,"dds":"0.16666666666666663","last_synced_commit":"b9ecec2c6df1b3e40a54511366dcb6085cf90c34"},"previous_names":["kyzhouhzau/fennlp"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyzhouhzau%2FNLPGNN","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyzhouhzau%2FNLPGNN/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyzhouhzau%2FNLPGNN/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyzhouhzau%2FNLPGNN/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyzhouhzau","download_url":"https://codeload.github.com/kyzhouhzau/NLPGNN/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247369953,"owners_count":20927928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["albert","albert-ner","bert","bert-cls","bert-ner","bilstm-attention","gan","gcn","gin","gnn","gpt2","graph-classfication","graph-convolutional-networks","graphsage","message-passing","nlp","tensorflow2","textcnn","textgcn","tf2"],"created_at":"2024-09-24T14:15:06.046Z","updated_at":"2025-04-05T17:09:27.555Z","avatar_url":"https://github.com/kyzhouhzau.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nlpgnn\n\n[![Build Status](https://travis-ci.org/kyzhouhzau/fennlp.svg?branch=master)](https://travis-ci.org/kyzhouhzau/fennlp/branches)\n[![PyPI version](https://badge.fury.io/py/fennlp.svg)](https://badge.fury.io/py/fenlp)\n[![GitHub version](https://badge.fury.io/gh/kyzhouhzau%2Ffennlp.svg)](https://badge.fury.io/gh/kyzhouhzau%2Ffennlp)\n[![Maintainability](https://api.codeclimate.com/v1/badges/d587092245542684c80b/maintainability)](https://codeclimate.com/github/kyzhouhzau/fennlp/maintainability)\n[![License](https://img.shields.io/github/license/kyzhouhzau/fennlp)](https://github.com/kyzhouhzau/fennlp/blob/master/LICENSE)\n[![Coverage Status](https://coveralls.io/repos/github/kyzhouhzau/fennlp/badge.svg)](https://coveralls.io/github/kyzhouhzau/fennlp)\n\n# Package description\nThe field of natural language processing is currently undergoing tremendous changes, and many excellent models have been proposed in recent years, including BERT, GPT, etc.  \nAt the same time, graph neural network as an exquisite design is constantly being used in the field of natural language processing, such as TextGCN and Tensor-TextGCN.  \nThis toolbox is dedicated to natural language processing and expects to implement models in the simplest way.  \nKeywords: NLP; GNN\n\nModels:\n* BERT\n* ALBERT\n* GPT2\n* TextCNN\n* Bilstm+Attention\n* GCN, GAN \n* GIN, GraphSAGE\n* TextGCN, TextSAGE\n\nExamples (See tests for more details):\n\n* BERT-NER (Chinese and English Version)\n* BERT-CRF-NER (Chinese and English Version)\n* BERT-CLS (Chinese and English Version)\n* ALBERT-NER (Chinese and English Version)\n* ALBERT-CLS (Chinese and English Version)\n* GPT2-generation (English Version)\n* Bilstm+Attention (Chinese and English Version)\n* TextCNN(Chinese and English Version)\n* GCN, GAN, GIN, GraphSAGE  (Base on message passing)\n* TextGCN and TextSAGE for text classification\n\n\nAll the above experiments were tested on GTX 1080 GPU with memory 8000MiB.\n\n# Status\n2020/5/--: convert the project name to NLPGNN from fennlp.\n\n2020/5/17: try to convert sentence to graph based on bert attention matrix, but failed.\nThis section provides a solution to visualize the BERT attention matrix.\nFor more detail, you can check dictionary \"BERT-GCN\". \n\n2020/5/11: add TextGCN and TextSAGE for text classification.\n\n2020/5/5: add GIN, GraphSAGE for graph classfication.\n\n2020/4/25: add GAN, GIN model, based on message passing methods.\n\n2020/4/23: add GCN model, based on message passing methods.\n\n2020/4/16：currently focusing on models of GNN in nlp, and trying to integrate some GNN models into fennlp.\n\n2020/4/2: add GPT2 model, could used parameters released by OpenAI (base,medium,large). \nMore detail reference dictionary \"TG/EN/interactive.py\"\n\n2020/3/26: add Bilstm+Attention example for classification\n\n2020/3/23: add RAdam optimizer.\n\n2020/3/19: add test example \"albert_ner_train.py\" \"albert_ner_test.py\"\n\n2020/3/16: add model for training sub word embedding based on bpe methods.\nThe trained embedding is used in TextCNN model for improve it's improvement.\nSee \"tran_bpe_embeding.py\" for more details.\n\n2020/3/8: add test example \"run_tucker.py\" for train TuckER on WN18.\n\n2020/3/3: add test example \"tran_text_cnn.py\" for train TextCNN model. \n\n2020/3/2: add test example \"train_bert_classification.py\" for text classification based on bert.\n\n# Requirement\n* tensorflow-gpu\u003e=2.0\n* typeguard\n* gensim\n* tqdm\n* sentencepiece\n\n# Usage\n\n1. clone source\n```\ngit clone https://github.com/kyzhouhzau/NLPGNN.git\n```\n2. install package\n```\npython setup.py install \n```\n\n3. run model\n```\npython bert_ner_train.py\n```\n\n# For NER：\n\n## Input\n* put train, valid and test file in \"Input\" dictionary.\n* data format: reference data in  \"tests\\NER\\Input\\train\"\n\n    e.g. \"拮 抗 RANKL 对 破 骨 细 胞 的 作 用 。\tO O O O B-Anatomy I-Anatomy I-Anatomy E-Anatomy O O O O\"\n    \n    For each line in train contains two parts, the first part \"拮 抗 RANKL 对 破 骨 细 胞 的 作 用 。\" is a sentence.\n    The second part \"O O O O B-Anatomy I-Anatomy I-Anatomy E-Anatomy O O O O\" is the tag for each word in the sentence.\n    Both of them use '\\t' to concatenate.\n\n### 1、bert (base, large)\n\n```python\nfrom nlpgnn.models import bert\nbert = bert.BERT()\n```\n\n```\npython bert_ner_train.py\n```\n\n```\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\nbert (BERT)                  multiple                  101677056 \n_________________________________________________________________\ndense (Dense)                multiple                  35374     \n=================================================================\nTotal params: 101,712,430\nTrainable params: 101,712,430\nNon-trainable params: 0\n_________________________________________________________________\n```\n\n### 2、bert + crf\n```python\nfrom nlpgnn.models import bert\nfrom nlpgnn.metrics.crf import CrfLogLikelihood\nbert = bert.BERT()\ncrf = CrfLogLikelihood()\n```\n\n```\npython bert_ner_crf_train.py\n```\n```\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\nbert (BERT)                  multiple                  101677056 \n_________________________________________________________________\ndense (Dense)                multiple                  35374     \n_________________________________________________________________\ncrf (CrfLogLikelihood)       multiple                  2116      \n=================================================================\nTotal params: 101,714,546\nTrainable params: 101,714,546\nNon-trainable params: 0\n_________________________________________________________________\n```\n\n### 3、albert (base, large, xlage, xxlage)\n```python\nfrom nlpgnn.models import albert\nbert = albert.ALBERT()\n```\n\n```\npython albert_ner_train.py \n```\n```\nlarge\nModel: \"albert_ner\"\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\nalbert (ALBERT)              multiple                  11092992  \n_________________________________________________________________\ndense (Dense)                multiple                  6921      \n=================================================================\nTotal params: 11,099,913\nTrainable params: 11,099,913\nNon-trainable params: 0\n_________________________________________________________________\n\n```\n\nUsing the default parameters, we get the following results on \"中文糖尿病标注数据集\" and \"CoNLL-2003\" valid data.\n\n\n|model    | macro-F1| macro-P| macro-R|  lr    |epoch   |maxlen  |batch_size| data|\n| ------- |  -------| -------| -------| ------- |------- |------- |------- |-------|\n|   bert+base  | 0.7005 | 0.7244 | 0.7031 | 2e-5  |3      |   128  |  6 |中文糖尿病标注数据集|\n| bert+base+crf| 0.7009 | 0.7237 | 0.7041 | 2e-5(bert),2e-3(crf)|3 |    128   |  6 |中文糖尿病标注数据集|\n|   bert+base  | 0.9128  | 0.9208 | 0.9227 | 2e-5 |5      |   128  |  8 |  CoNLL-2003    |\n|  albert+base | 0.8512  | 0.8678 | 0.8589 | 1e-4 |8      |   128  |  16|  CoNLL-2003    |\n|  albert+large| 0.8670  | 0.8778 | 0.8731 | 2e-5 |10     |   128  |  4 |  CoNLL-2003    |\n\n# For Sentence Classfication\n                 \n## Input\n* put train, valid and test file in \"Input\" dictionary.\n* data format: reference data in \"\\tests\\CLS\\BERT( or ALBERT)\\Input\".\n\n    e.g. \"作 为 地 球 上 曾 经 最 强 的 拳 王 之 一 ， 小 克 里 琴 科 谈 自 己 是 否 会 复 出    2\"\n    \n    For each line in train(test,valid) contains two parts, the first part \"作 为 地 球 上 曾 经 最 强 的 拳 王 之 一 ，\n    小 克 里 琴 科 谈 自 己 是 否 会 复 出\" is the sentence, and second part \"2\" is the label.\n\n### 1、bert (base, large)\n\n```python\nfrom nlpgnn.models import bert\nbert = bert.BERT()\n```\n\n``` \npython train_bert_classification.py\n```\n```\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\nbert (BERT)                  multiple                  102267648 \n_________________________________________________________________\ndense (Dense)                multiple                  11535     \n=================================================================\nTotal params: 102,279,183\nTrainable params: 102,279,183\nNon-trainable params: 0\n_________________________________________________________________\n\n```\n\n\n### 2、TextCNN\n\n```python\nfrom nlpgnn.models import TextCNN\nmodel = TextCNN.TextCNN()\n```\n\n``` \npython train_text_cnn.py\n```\n\n```\nUse \"WordPiece embedding\" to Initialize word embedding. Train your embeddings.\npython train_bpe_embedding.py\n```\nFor more detail reference [WordPiece](https://mp.weixin.qq.com/s/Y6mCTqlf1yaTu99iKl-82Q) \n\n\nUsing the default parameters, we get the following results on \"新闻标题短文本分类\"  and SST-2 valid data.\n\n|model    | ACC     |  lr    |epoch   |maxlen  |batch_size|data|\n| ------- | ------- |------- |------- |------- |-------   |-------   |\n|bert+base|0.8899   |1e-5     |    5   |  50    |    32    |新闻标题短文本分类|\n|bert+base| 0.9266|2e-5  |  3     | 128    |  8    |SST-2|\n|albert+base| 0.9186|1e-5  |  3     | 128    |  16    |SST-2|\n|albert+large| 0.9461|1e-6  |  3     | 128    |  4    |SST-2|\n|Bilstm+attention| 0.8269|0.01  |  3     | 128    |  64    |SST-2|\n|TextCNN| 0.8233|0.01  |  3     | 128    |  64    |SST-2|\n\n# For Text Generation\n### 1、GPT2 \n\n```python\nfrom nlpgnn.models import gpt2\nbert = gpt2.GPT2()\n```\n\n``` \npython interactive.py\n```\n\n```\nModel: \"gen_gp_t2\" base\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\ngpt2 (GPT2)                  multiple                  124439808 \n=================================================================\nTotal params: 124,439,808\nTrainable params: 124,439,808\nNon-trainable params: 0\n_________________________________________________________________\n```\n\nExample:\n```\nInput \u003e\u003e\u003e Stocks continue to fall this week\n\nOutput \u003e\u003e\u003e as stocks fall for the second consecutive week as investors flee for safe havens.\n\n\"The market is off the charts,\" said John Schmieding, senior vice president, market strategy at RBC Capital Markets. \n\"We don't know what the Fed's intent is on, what direction it's going in. We don't know where they plan to go. \nWe don't know what direction they're going to move into.\"\n\n```\n\n# TensorBoard\ntensorboard can help you visualize losses and evaluate indicators:\n\nuseage:\n```\n tensorboard --port 6006 --logdir=\"./tensorboard\"\n```\n![loss](picture/loss1.png) ![acc](picture/acc1.png)\n\n\n# GNN\n## 1、GCN, GAN, GIN, GraphSAGE (Based on message passing)\n\nSame data split and parameters setting as proposed in this [paper](https://arxiv.org/abs/1609.02907) \n\n* Nodes Classfication\n\n|model    | Cora     |  Pubmed    |Citeseer  |\n| ------- | -------  |-------     |-------   |\n|GCN      |81.80   |79.50    |  71.20    |  \n|GAN      |83.00   | 79.00   |  72.30    |\n|GAAE     |82.40   |79.60    |  71.70   |  \n\n* Graph Classfication\n\n|model    | MUTAG     |  PROTEINS    |NCI1  |\n| ------- | -------  |-------     |-------   |\n|GIN     |87.62±8.76\u003csup\u003e#\u003c/sup\u003e  | 73.05±1.85\u003csup\u003e#\u003c/sup\u003e  | 73.13±5.57\u003csup\u003e#\u003c/sup\u003e  |  \n|GraphSAGE|86.06±8.26| 75.11±2.87 | 76.91±3.45 | \n\nNote: The # sign indicates that the current result is less than the paper result. \nIn the paper the author use [this method to evaluate models](https://github.com/weihua916/powerful-gnns/issues/8).\nThis method is time expensive. So I did not do it like that here.\n\n* Text Classfication\n\n|model    | R8     |  R52    |\n| ------- | -------  |------- |\n|TextSAGE|  96.68±0.42 |92.80±0.32|\n|TextGCN2019|  97.108±0.243 |  92.512±0.249 |\n\n# Parameter Settings\n1、For English tasks, you need to set the parameter \"cased\" (in fennlp.datas.checkpoint.LoadCheckpoint) to be consistent with your preprocessed input data to ensure that the tokenizer can correctly distinguish case.\n\n2、When you use bert or albert, the following parameters is necessary:\n```\nparam.maxlen\nparam.label_size\nparam.batch_size\n```\nif you don't know the count of label_size, the script will tell you when you first run the train codes.\n\n3、Learning rate and batch_size will determine model convergence, see [Link](https://mp.weixin.qq.com/s/C-MMNaQdafK3JNh1ZGOVJA)  for more detail.\n\n4、If you are not familiar with the optimizer in bert and albert, it does not matter.\nThe most important thing you need to remember is that the parameters \"learning_rate\" and \"decay_steps\" (in fennlp.optimizers.optim.AdamWarmup)\nis important. You can set the \"learning rate\" to a relatively small value, and let \"decay_steps\" equal to samples*epoch/batch_size or little higher.\n\n5、If you find that the code runs slower, you can try to use @ tf.function and set the appropriate model writing and evaluation frequency.\n\n6、Any other problem you can concat me by \"zhoukaiyinhzau@gmail.com\" or ask questions in issue.   \n\n# Reference\n\n[1] [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805)  \n[2] [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942)  \n[3] [Language Models are Unsupervised Multitask Learners](https://openai.com/blog/better-language-models/)  \n[4] [Neural Message Passing for Quantum Chemistry](https://arxiv.org/pdf/1704.01212.pdf)  \n[5] [Semi-Supervised Classification with Graph Convolutional Networks](https://arxiv.org/abs/1609.02907)  \n[6] [Graph Attention Networks](https://arxiv.org/abs/1710.10903)  \n[7] [How Powerful are Graph Neural Networks?](https://arxiv.org/abs/1810.00826)  \n[8] [GraphSAGE: Inductive Representation Learning on Large Graphs](http://snap.stanford.edu/graphsage/)  \n[9] [Diffusion Improves Graph Learning](https://arxiv.org/abs/1911.05485)  \n[10] [Benchmarking Graph Neural Networks](https://arxiv.org/abs/2003.00982)  \n[11] [Text Level Graph Neural Network for Text Classification](https://www.aclweb.org/anthology/D19-1345/)  \n[12] [Graph Convolutional Networks for Text Classification](https://arxiv.org/abs/1809.05679)  \n[13] [Tensor Graph Convolutional Networks for Text Classification](https://arxiv.org/abs/2001.05313)  \n[14] [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://arxiv.org/pdf/1801.07606.pdf)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyzhouhzau%2Fnlpgnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyzhouhzau%2Fnlpgnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyzhouhzau%2Fnlpgnn/lists"}