{"id":26590075,"url":"https://github.com/rajatasusual/information_extractor","last_synced_at":"2026-04-18T15:31:57.511Z","repository":{"id":283684647,"uuid":"952553037","full_name":"rajatasusual/information_extractor","owner":"rajatasusual","description":"information_extractor is a tool that leverages spaCy for coreference resolution and SpanBERT for relation extraction. This project integrates named entity recognition (NER) with relation extraction to identify and analyze relationships between entities in text.  ","archived":false,"fork":false,"pushed_at":"2025-04-14T23:06:51.000Z","size":173,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-11-27T18:28:23.052Z","etag":null,"topics":["bert","coreference-resolution","entity-relationship","entity-resolution","spacy","spanbert"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rajatasusual.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-21T13:24:37.000Z","updated_at":"2025-04-14T23:06:54.000Z","dependencies_parsed_at":"2025-04-13T20:29:52.193Z","dependency_job_id":"582e7af2-8b7f-41cb-a59c-34638735819e","html_url":"https://github.com/rajatasusual/information_extractor","commit_stats":null,"previous_names":["rajatasusual/spacy_coref_rel","rajatasusual/information_extractor"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/rajatasusual/information_extractor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajatasusual%2Finformation_extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajatasusual%2Finformation_extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajatasusual%2Finformation_extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajatasusual%2Finformation_extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rajatasusual","download_url":"https://codeload.github.com/rajatasusual/information_extractor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajatasusual%2Finformation_extractor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31974279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","coreference-resolution","entity-relationship","entity-resolution","spacy","spanbert"],"created_at":"2025-03-23T13:20:39.059Z","updated_at":"2026-04-18T15:31:52.502Z","avatar_url":"https://github.com/rajatasusual.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# information_extractor  \n\n## Overview  \n[![CI](https://github.com/rajatasusual/information_extractor/actions/workflows/ci.yml/badge.svg)](https://github.com/rajatasusual/information_extractor/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n**information_extractor** is a tool that leverages **spaCy** for coreference resolution and **SpanBERT** for relation extraction. This project integrates named entity recognition (NER) with relation extraction to identify and analyze relationships between entities in text.  \n\n## Features\n\n### SpanBERT Model\n- Pre-trained model for relation extraction between entities\n- Supports multiple entity types (PERSON, ORGANIZATION, LOCATION, etc.)\n- Handles special token markers for subject and object entities\n- Uses BERT architecture for sequence classification\n- GPU acceleration support when available\n- Configurable batch size and sequence length\n\n### Entity Processing\n- Maps between spaCy and SpanBERT entity labels\n- Supports common entity types:\n    - Organizations (ORG)\n    - Persons (PERSON)\n    - Locations (GPE, LOC)\n    - Dates (DATE)\n    - And more\n\n### Relation Extraction\n- Creates entity pairs from spaCy sentences\n- Handles bidirectional relationships\n- Configurable confidence threshold\n- Deduplicates relations with confidence scoring\n- Returns structured relation tuples\n- Detailed logging for debugging\n\n### Pretrained Models\n\nThe `assets` directory contains the following pretrained models:\n\n- **pretrained_spanbert/** finetuned for TARCED use cases.\n- **corefereee_model_en** from stanford research\n- **en_core_web_md-3.50** from spaCy\n\n## Installation  \n\nTo install and set up the project, run the following commands:  \n\n```bash\nGIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/rajatasusual/information_extractor.git\ncd information_extractor\npip3 install -r requirements.txt\ngit lfs pull --include \"assets/pretrained_spanbert/pytorch_model.bin\"\n```\n\nEnsure that you have **Git LFS** installed to handle large model files.  \n\n## Usage  \n\nTo extract relations using **spaCy** and **SpanBERT**, you can run the provided example script:  \n\n```bash\npython code/information_extraction.py\n```\n\n### Example (Inside `code/information_extraction.py`)  \n\n```python\nimport spacy\nfrom spanbert_module import SpanBERT  # Import SpanBERT model\n\n# Load spaCy NLP model\nnlp = spacy.load(\"en_core_web_md\")\n\n# Sample text\ntext = \"Bill Gates founded Microsoft. Microsoft is headquartered in Redmond.\"\n\n# Process text with spaCy\ndoc = nlp(text)\n\n# Load SpanBERT\npretrained_dir = \"assets/pretrained_spanbert\"\nspanbert = SpanBERT(pretrained_dir=pretrained_dir)\n\n# Extract relations\nrelations = spanbert.extract_relations(doc)\nprint(relations)\n```\n\n## Acknowledgments  \n\nThis project integrates **SpanBERT** from **Facebook Research**. If you use this project, please cite:  \n\n```\n@article{joshi2019spanbert,\n    title={{SpanBERT}: Improving Pre-training by Representing and Predicting Spans},\n    author={Mandar Joshi and Danqi Chen and Yinhan Liu and Daniel S. Weld and Luke Zettlemoyer and Omer Levy},\n    journal={arXiv preprint arXiv:1907.10529},\n    year={2019}\n}\n```\n\n## License \u0026 Disclaimer  \n\nThis project is intended for research and educational purposes. The SpanBERT model belongs to **Facebook Research**, and its use must comply with their licensing terms. We are not affiliated with Facebook Research.  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frajatasusual%2Finformation_extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frajatasusual%2Finformation_extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frajatasusual%2Finformation_extractor/lists"}