{"id":18439561,"url":"https://github.com/idiap/g2g-transformer","last_synced_at":"2025-10-31T14:30:49.138Z","repository":{"id":144962020,"uuid":"306273402","full_name":"idiap/g2g-transformer","owner":"idiap","description":"Pytorch implementation of “Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement”","archived":false,"fork":false,"pushed_at":"2021-03-23T14:35:50.000Z","size":227,"stargazers_count":62,"open_issues_count":4,"forks_count":9,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-23T01:03:27.427Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/idiap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-22T08:34:37.000Z","updated_at":"2024-12-25T03:41:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"5027ba0b-c024-48d4-9e92-7adbfe55878c","html_url":"https://github.com/idiap/g2g-transformer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fg2g-transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fg2g-transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fg2g-transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fg2g-transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/idiap","download_url":"https://codeload.github.com/idiap/g2g-transformer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247732727,"owners_count":20986912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T06:25:25.564Z","updated_at":"2025-10-31T14:30:49.133Z","avatar_url":"https://github.com/idiap.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Graph-to-Graph Transformers\n=================\n\nSelf-attention models, such as Transformer, have been hugely successful in a wide\nrange of natural language processing (NLP) tasks, especially when combined with\nlanguage-model pre-training, such as BERT.\n\nWe propose [\"Graph-to-Graph Transformer\"](https://www.aclweb.org/anthology/2020.findings-emnlp.294/) and \n[\"Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement\"](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00358/97778/Recursive-Non-Autoregressive-Graph-to-Graph)(accepted to TACL)\nto generalize vanilla Transformer to encode graph structure, and builds the desired\noutput graph.\n\n**Note** : To use G2GTr model for transition-based dependency parsing, please refer to [G2GTr](https://github.com/alirezamshi/G2GTr) repository.\n\nContents\n---------------\n\n- [Installation](#installation)\n- [Quick Start](#othertasks)\n- [Data Pre-processing and Initial Parser](#datapreprocessingandinitialparser)\n- [Training](#training)\n- [Evaluation](#evaluation)\n- [Predict Raw Sentences](#predictraw)\n- [Citations](#citations)\n\n\u003ca name=\"installation\"/\u003e  \n\nInstallation\n--------------  \nFollowing packages should be included in your environment:\n\n- Python \u003e= 3.7\n- PyTorch \u003e= 1.4.0\n- Transformers(huggingface) = 2.4.1\n  \nThe easier way is to run the following command:  \n\n``` python \nconda env create -f environment.yml\nconda activate rngtr\n```\n\n\u003ca name=\"othertasks\"/\u003e  \n\nQuick Start\n-------------\n\nGraph-to-Graph Transformer architecture is general and can be applied to\nany NLP tasks which interacts with graphs. To use our implementation in your\ntask, you just need to add `BertGraphModel` class to your code to encode\nboth token-level and graph-level information. Here is a sample usage:\n\n```python\n#Loading BertGraphModel and initialize it with available BERT models.\nimport torch\nfrom parser.utils.graph import initialize_bertgraph,BertGraphModel\n# inputing unlabelled graph with label size 5, and Layer Normalization of key\n# you can load other BERT pre-trained models too.\nencoder = initialize_bertgraph('bert-base-cased',layernorm_key=True,layernorm_value=False,\n             input_label_graph=False,input_unlabel_graph=True,label_size=5)\n\n#sample input\ninput = torch.tensor([[1,2],[3,4]])\ngraph = torch.tensor([ [[1,0],[0,1]],[[0,1],[1,0]] ])\ngraph_rel = torch.tensor([[0,1],[3,4]])\noutput = encoder(input_ids=input,graph_arc=graph,graph_rel=graph_rel)\nprint(output[0].shape)\n## torch.Size([2, 2, 768])\n\n# inputting labelled graph\nencoder = initialize_bertgraph('bert-base-cased',layernorm_key=True,layernorm_value=False,\n             input_label_graph=True,input_unlabel_graph=False,label_size=5)\n\n#sample input\ninput = torch.tensor([[1,2],[3,4]])\ngraph = torch.tensor([ [[2,0],[0,3]],[[0,1],[4,0]] ])\noutput = encoder(input_ids=input,graph_arc=graph,)\nprint(output[0].shape)\n## torch.Size([2, 2, 768])\n```\nIf you just want to use `BertGraphModel` in your research, you can just import it\nfrom our repository:\n```python\nfrom parser.utils.graph import BertGraphModel,BertGraphConfig\nconfig = BertGraphConfig(YOUR-CONFIG)\nconfig.add_graph_par(GRAPH-CONFIG)\nencoder = BertGraphModel(config)\n```\n\n\u003ca name=\"datapreprocessingandinitialparser\"/\u003e  \n\nData Pre-processing and Initial Parser \n-----------------  \n\n### Dataset Preparation  \n\nWe evaluated our model on [UD Treebanks](https://universaldependencies.org/), English \nand Chinese [Penn Treebanks](https://catalog.ldc.upenn.edu/LDC99T42), \nand [CoNLL 2009 Shared Task](https://www.aclweb.org/anthology/W09-1201/). \nIn following sections, we prepare datasets and their evaluation scripts.\n\n#### Penn Treebanks\nEnglish Penn Treebank can be downloaded from [english](https://catalog.ldc.upenn.edu/LDC99T42) and\n[chinese](https://catalog.ldc.upenn.edu/LDC2005T01) under LDC license. For English \nPenn Treebank, replace gold POS tags with Stanford POS tagger with following command in \n[this repository](https://github.com/shuoyangd/hoolock):  \n```\nbash scripts/postag.sh ${data_dir}/ptb3-wsj-[train|dev|dev.proj|test].conllx\n```\n\n#### CoNLL 2009 Treebanks\nYou can download Treebanks from [here](https://catalog.ldc.upenn.edu/LDC2012T03) under\nLDC license. We use predicted POS tags provided by organizers.\n\n#### UD Treebanks\nYou can find required Treebanks from [here](https://universaldependencies.org/).\n(use version 2.3)\n\n### Initial Parser\nAs mentioned in our paper, you can use any initial parser to produce dependency graph. \nHere we use [Biaffine Parser](https://arxiv.org/abs/1611.01734) for Penn Treebanks, and German Corpus. We also apply our\nmodel to ouput prediction of [UDify parser](https://arxiv.org/abs/1904.02099) for UD Treebanks.  \n**Biaffine Parser**: To prepare biaffine initial parser, we use [this repository](https://github.com/yzhangcs/parser) \nto produce output predictions.  \n**UDify Parser**: For UD Treebanks, we use [UDify repository](https://github.com/Hyperparticle/udify)\nto produce required initial dependency graph.  \nAlternatively, you can easily run the following command file to produce all required outputs:  \n```\nbash job_scripts/udify_dataset.bash\n```\n\n\u003ca name=\"training\"/\u003e  \n\nTraining\n-------------\n\nTo train your own model, you can easily fill out the script in `job_scripts` directory, \nand run it. Here is the list of sample scripts:  \n\nModel | Script \n--- | ---  \nSyntactic Transformer | `baseline.bash` | \nAny initial parser+RNGTr | `rngtr.bash` |\nEmpty+RNGTr | `empty_rngtr.bash` |\n\n\u003ca name=\"evaluation\"/\u003e  \n\nEvaluation\n-------------\n\nFirst you should download official scripts from [UD](https://universaldependencies.org/conll18/evaluation.html), \n[Penn Treebaks](https://depparse.uvt.nl/), and [German](https://ufal.mff.cuni.cz/conll2009-st/eval-data.html). Then,\nrun the following command:  \n```\nbash job_scripts/predict.bash\n```\n\nTo replicate `refinement analysis` and `error analysis` results, you should use \n[MaltEval](http://www.maltparser.org/malteval.html) tools.\n\n\u003ca name=\"predictraw\"/\u003e  \n\nPredict Raw Sentences\n--------------------- \n\nYou can also predict dependency graphs of raw texts with a pre-trained model by modifying ```predict.bash``` file. Just set ```input_type``` to ```raw```. Then, put all your sentences in a .txt file, and the output will be in CoNNL format.\n\nCitations\n-------------\n\n\u003ca name=\"citations\"/\u003e  \n\nIf you use this code for your research, please cite these works as:\n```\n@misc{mohammadshahi2020recursive,\n      title={Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement}, \n      author={Alireza Mohammadshahi and James Henderson},\n      year={2020},\n      eprint={2003.13118},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n```\n@inproceedings{mohammadshahi-henderson-2020-graph,\n    title = \"Graph-to-Graph Transformer for Transition-based Dependency Parsing\",\n    author = \"Mohammadshahi, Alireza  and\n      Henderson, James\",\n    booktitle = \"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings\",\n    month = nov,\n    year = \"2020\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://www.aclweb.org/anthology/2020.findings-emnlp.294\",\n    pages = \"3278--3289\",\n    abstract = \"We propose the Graph2Graph Transformer architecture for conditioning on and predicting arbitrary graphs, and apply it to the challenging task of transition-based dependency parsing. After proposing two novel Transformer models of transition-based dependency parsing as strong baselines, we show that adding the proposed mechanisms for conditioning on and predicting graphs of Graph2Graph Transformer results in significant improvements, both with and without BERT pre-training. The novel baselines and their integration with Graph2Graph Transformer significantly outperform the state-of-the-art in traditional transition-based dependency parsing on both English Penn Treebank, and 13 languages of Universal Dependencies Treebanks. Graph2Graph Transformer can be integrated with many previous structured prediction methods, making it easy to apply to a wide range of NLP tasks.\",\n}\n```\nHave a question not listed here? Open [a GitHub Issue](https://github.com/idiap/g2g-transformer/issues) or \nsend us an [email](alireza.mohammadshahi@idiap.ch).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2Fg2g-transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidiap%2Fg2g-transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2Fg2g-transformer/lists"}