{"id":13512740,"url":"https://github.com/DFKI-NLP/TRE","last_synced_at":"2025-03-31T00:30:40.099Z","repository":{"id":97960080,"uuid":"180105960","full_name":"DFKI-NLP/TRE","owner":"DFKI-NLP","description":"[AKBC 19] Improving Relation Extraction by Pre-trained Language Representations","archived":false,"fork":false,"pushed_at":"2021-10-01T09:27:30.000Z","size":45,"stargazers_count":108,"open_issues_count":3,"forks_count":12,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-01T15:37:23.836Z","etag":null,"topics":["information-extraction","machine-learning","multi-task-learning","nlp","relation-extraction","transformer"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1906.03088","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DFKI-NLP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-04-08T08:38:16.000Z","updated_at":"2024-03-18T09:38:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"c6e5bd86-553a-4b7b-872c-8f4644851dc9","html_url":"https://github.com/DFKI-NLP/TRE","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DFKI-NLP%2FTRE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DFKI-NLP%2FTRE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DFKI-NLP%2FTRE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DFKI-NLP%2FTRE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DFKI-NLP","download_url":"https://codeload.github.com/DFKI-NLP/TRE/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246399816,"owners_count":20770907,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["information-extraction","machine-learning","multi-task-learning","nlp","relation-extraction","transformer"],"created_at":"2024-08-01T04:00:29.910Z","updated_at":"2025-03-31T00:30:39.810Z","avatar_url":"https://github.com/DFKI-NLP.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["Language Models"],"readme":"# Improving Relation Extraction by Pre-trained Language Representations\r\n\r\nThis repository contains the code of our paper:  \r\n[Improving Relation Extraction by Pre-trained Language Representations.](https://openreview.net/forum?id=BJgrxbqp67)  \r\nChristoph Alt*, Marc Hübner*, Leonhard Hennig\r\n\r\nWe fine-tune the pre-trained OpenAI GPT [1] to the task of relation extraction and show that it achieves state-of-the-art results on SemEval 2010 Task 8 and TACRED relation extraction datasets.\r\n\r\nOur code depends on huggingface's PyTorch reimplementation of the OpenAI GPT [2] - so thanks to them.\r\n\r\n## Installation\r\n\r\nFirst, clone the repository to your machine and install the requirements with the following command:\r\n\r\n```bash\r\npip install -r requirements.txt\r\n```\r\n\r\nWe also need the weights of the pre-trained Transformer, which can be downloaded with the following command:\r\n```\r\n./download-model.sh\r\n```\r\n\r\nThe english spacy model is required for sentence segmentation:\r\n```\r\npython -m spacy download en\r\n```\r\n\r\n## Prepare the data\r\n\r\nWe evaluate our model on [SemEval 2010 Task 8](https://drive.google.com/file/d/0B_jQiLugGTAkMDQ5ZjZiMTUtMzQ1Yy00YWNmLWJlZDYtOWY1ZDMwY2U4YjFk) and [TACRED](https://catalog.ldc.upenn.edu/LDC2018T24), which is available through LDC.\r\n\r\nOur model expects the input dataset to be in JSONL format. To convert a dataset run the following command:\r\n```bash\r\npython dataset_converter.py \u003cDATASET DIR\u003e \u003cCONVERTED DATASET DIR\u003e --dataset=\u003cDATASET NAME\u003e\r\n```\r\n\r\n## Training\r\nE.g. for training on the TACRED dataset, run the following command:\r\n\r\n```bash\r\nCUDA_VISIBLE_DEVICES=0 python relation_extraction.py train \\\r\n  --write-model True \\\r\n  --masking-mode grammar_and_ner \\\r\n  --batch-size 8 \\\r\n  --max-epochs 3 \\\r\n  --lm-coef 0.5 \\\r\n  --learning-rate 5.25e-5 \\\r\n  --learning-rate-warmup 0.002 \\\r\n  --clf-pdrop 0.1 \\\r\n  --attn-pdrop 0.1 \\\r\n  --word-pdrop 0.0 \\\r\n  --dataset tacred \\\r\n  --data-dir \u003cCONVERTED DATASET DIR\u003e \\\r\n  --seed=0 \\\r\n  --log-dir ./logs/\r\n```\r\n\r\n## Evaluation\r\n```bash\r\nCUDA_VISIBLE_DEVICES=0 python relation_extraction.py evaluate \\\r\n  --dataset tacred \\\r\n  --masking_mode grammar_and_ner \\\r\n  --test_file ./data/tacred/test.jsonl \\\r\n  --save_dir ./logs/ \\\r\n  --model_file \u003cMODEL FILE (e.g. model_epoch...)\u003e \\\r\n  --batch_size 8 \\\r\n  --log_dir ./logs/\r\n```\r\n\r\n## Trained Models\r\n\r\nThe models we trained on SemEval and TACRED to produce our paper results can be found here:\r\n\r\n| Dataset  | Masking Mode    | P    | R    | F1   | Download                                                                    |\r\n| -------- | --------------- | ---- | ---- | ---- | --------------------------------------------------------------------------- |\r\n| TACRED   | grammar_and_ner | 70.0 | 65.0 | 67.4 | [Link](https://dfkide-my.sharepoint.com/:u:/g/personal/lehe02_dfki_de/EQ0zv4QicbVMoW6nBhnOkA8BV4Yzt7agM0hAfX07VQez4w?Web=0\u0026Download=1) |\r\n| SemEval  | None            | 87.6 | 86.8 | 87.1 | [Link](https://dfkide-my.sharepoint.com/:u:/g/personal/lehe02_dfki_de/EZBQNU99Uz9MiOuP-RHAdaYB7w8jIW0mADM4f05tGbhwZg?Web=0\u0026Download=1) |\r\n\r\n### Download and extract model files\r\n\r\nFirst, download the archive corresponding to the model you want to evaluate (links in the table above).\r\n\r\n```bash\r\nwget --content-disposition \u003cDOWNLOAD URL\u003e\r\n```\r\n\r\nExtract the model archive containing model.pt, text_encoder.pkl, and label_encoder.pkl.\r\n\r\n```bash\r\ntar -xvzf \u003cMODEL ARCHIVE\u003e\r\n```\r\n\r\n### Run evaluation\r\n\r\n- `dataset`: dataset to evaluate, can be one of \"semeval\" or \"tacred\".\r\n- `test-file`: path to the JSONL test file used during evaluation\r\n- `log-dir`: directory to store the evaluation results and predictions\r\n- `save-dir`: directory containing the downloaded model files (model.pt, text_encoder.pkl, and label_encoder.pkl)\r\n- `masking-mode`: masking mode to use during evaluation, can be one of \"None\", \"grammar_and_ner\", \"grammar\", \"ner\" or \"unk\" (**caution:** must match the mode for training)\r\n\r\nFor example, to evaluate the TACRED model with \"grammar_and_ner\" masking, run the following command:\r\n\r\n```bash\r\nCUDA_VISIBLE_DEVICES=0 python relation_extraction.py evaluate \\\r\n      --dataset tacred \\\r\n      --test-file ./\u003cCONVERTED DATASET DIR\u003e/test.jsonl \\\r\n      --log-dir \u003cRESULTS DIR\u003e \\\r\n      --save-dir \u003cMODEL DIR\u003e \\\r\n      --masking_mode grammar_and_ner\r\n```\r\n\r\n## Citations\r\nIf you use our code in your research or find our repository useful, please consider citing our work.\r\n\r\n```\r\n@InProceedings{alt_improving_2019,\r\n  author = {Alt, Christoph and H\\\"{u}bner, Marc and Hennig, Leonhard},\r\n  title = {Improving Relation Extraction by Pre-trained Language Representations},\r\n  booktitle = {Proceedings of AKBC 2019},\r\n  year = {2019},\r\n  url = {https://openreview.net/forum?id=BJgrxbqp67},\r\n}\r\n```\r\n\r\n## License\r\nlm-transformer-re is released under the MIT license. See [LICENSE](LICENSE) for additional details.\r\n\r\n## References\r\n1. [Improving language understanding by generative pre-training](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf). Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.\r\n2. [PyTorch implementation of OpenAI's Finetuned Transformer Language Model](https://github.com/huggingface/pytorch-openai-transformer-lm)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDFKI-NLP%2FTRE","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDFKI-NLP%2FTRE","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDFKI-NLP%2FTRE/lists"}