{"id":19932156,"url":"https://github.com/amazon-science/refined","last_synced_at":"2025-04-13T02:16:57.482Z","repository":{"id":45711399,"uuid":"488248659","full_name":"amazon-science/ReFinED","owner":"amazon-science","description":"ReFinED is an efficient and accurate entity linking (EL) system.","archived":false,"fork":false,"pushed_at":"2024-12-13T16:46:44.000Z","size":443,"stargazers_count":212,"open_issues_count":20,"forks_count":45,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-04-13T02:16:53.247Z","etag":null,"topics":["entity-extraction","entity-linking","entity-resolution","nlp","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amazon-science.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-03T14:44:56.000Z","updated_at":"2025-04-04T19:43:06.000Z","dependencies_parsed_at":"2023-02-14T17:45:39.664Z","dependency_job_id":"92718769-2044-44b1-8448-682cb2cc482d","html_url":"https://github.com/amazon-science/ReFinED","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2FReFinED","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2FReFinED/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2FReFinED/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2FReFinED/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amazon-science","download_url":"https://codeload.github.com/amazon-science/ReFinED/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248654104,"owners_count":21140237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["entity-extraction","entity-linking","entity-resolution","nlp","pytorch"],"created_at":"2024-11-12T23:09:16.191Z","updated_at":"2025-04-13T02:16:57.461Z","avatar_url":"https://github.com/amazon-science.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ReFinED\n## Quickstart\n```commandline\npip install https://github.com/amazon-science/ReFinED/archive/refs/tags/V1.zip\n```\n```python\nfrom refined.inference.processor import Refined\nrefined = Refined.from_pretrained(model_name='wikipedia_model_with_numbers',\n                                  entity_set=\"wikipedia\")\nspans = refined.process_text(\"\u003cadd_text_here\u003e\")\n```\n\n## Overview\nReFinED is an entity linking (EL) system which links entity mentions in documents to their corresponding entities in Wikipedia or Wikidata (over 30M entities).\nThe combination of accuracy, speed, and scalability of ReFinED means the system is capable of being deployed to extract entities from web-scale datasets with higher accuracy and an order of magnitude lower cost than existing approaches.\n\n### News\n- **(November 2022)**\n  - Code refactoring 🔨\n  - Increased inference speed by 2x (replicates results from our paper) 💨\n  - Released `aida_model` (trained on news articles) and `questions_model` (trained on questions) to replicate the results from our paper ✅\n  - New features 🚀\n    - Entity linking evaluation code \n    - Fine-tuning script (allows use of custom datasets)\n    - Training script\n    - Data generation script (includes adding additional entities).\n\n\n### Hardware Requirements\nReFinED has a low hardware requirement. For fast inference speed, a GPU should be used, but this is not a strict requirement. \n\n\n### Model Architecture\nIn summary, ReFinED uses a Transformer model to perform mention detection, entity typing, and entity disambiguation for all mentions in a document in a single forward pass. The model is trained on a dataset we generated dataset using Wikipedia hyperlinks, which consists of over 150M entity mentions. The model uses entity descriptions and fine-grained entity types to perform linking. Therefore, new entities can be added to the system without retraining.\n\n#### ReFinED Paper\nThe ReFinED model architecture is described in the paper below (https://arxiv.org/abs/2207.04108):\n```bibtex\n@inproceedings{ayoola-etal-2022-refined,\n    title = \"{R}e{F}in{ED}: An Efficient Zero-shot-capable Approach to End-to-End Entity Linking\",\n    author = \"Tom Ayoola, Shubhi Tyagi, Joseph Fisher, Christos Christodoulopoulos, Andrea Pierleoni\",\n    booktitle = \"NAACL\",\n    year = \"2022\"\n}\n```\n \n\n#### Incorporating Knowledge Base Information Paper\nThe following paper is an extension of ReFinED which incorporates Knowledge Base (KB) information into the ED model in a fully differentiable and scalable manner (https://arxiv.org/abs/2207.04106):\n```bibtex\n@inproceedings{ayoola-etal-2022-improving,\n    title = \"Improving Entity Disambiguation by Reasoning over a Knowledge Base\",\n    author = \"Tom Ayoola, Joseph Fisher, Andrea Pierleoni\",\n    booktitle = \"NAACL\",\n    year = \"2022\"\n}\n```\n\n### Examples\nWhile classical NER systems, such as widely used spaCy, classify entities to high-level classes (e.g. PERSON, LOCATION, NUMBER, ...; 26 in total for spaCy), ReFinED supports over 1k low-level classes (e.g. Human, Football Team, Politician, Screenwriter, Association Football Player, Guitarist, ...). As an example, for the sentence \"England qualified for the 1970 FIFA World Cup in Mexico as reigning champions.\", ReFinED predicts \"England\" → {national football team} and \"Mexico\" → {country}; while spaCy maps both \"England\" and \"Mexico\" → {GPE - country}. Using fine-grained classes, the model is able to probabilistically narrow-down the set of possible candidates for \"England\" leading to correct disambiguation of the entity. Additionally, ReFinED uses textual descriptions of entities to perform disambiguation.\n\n## Library\n\n### Getting Started\nThe setup for ReFinED is very simple because the data files and datasets are downloaded automatically.\n1. Install the dependencies using the command below:\n```commandline\npip install -r requirments.txt\n```\nIf the command above fails (which currently, happens on a Mac), run the commands below instead:\n```commandline\nconda create -n refined38 -y python=3.8 \u0026\u0026 conda activate refined38\nconda install -c conda-forge python-lmdb -y\npip install -r requirments.txt\n```\n\n2. Add the `src` folder to your Python path. One way to do this is by running this command:\n```commandline\nexport PYTHONPATH=$PYTHONPATH:src\n```\n3. Now you can use ReFinED is your code as follows:\n```python\nfrom refined.inference.processor import Refined\nrefined = Refined.from_pretrained(...)\n```\n\n### Importing ReFinED as a library\nTo import the ReFinED model into your existing code run the commands below (note that the conda commands are only needed on a Mac):\n```commandline\npip install https://github.com/amazon-science/ReFinED/archive/refs/tags/V1.zip\n```\n\nAlternatively, if the command above does not work, try the commands below which will install some dependencies using conda.\n```commandline\nconda create -n refined38 -y python=3.8 \u0026\u0026 conda activate refined38\nconda install -c conda-forge python-lmdb -y\ngit clone https://github.com/amazon-science/ReFinED.git\ncd ReFinED\npython setup.py bdist_wheel --universal\npip install dist/ReFinED-1.0-py2.py3-none-any.whl\ncd ..\n```\n\n### Inference - performing EL with a trained model\nWe have released several trained models that are ready to use. See the code below or `example_scripts/refined_demo.py` for a working example. Inference speed can be improved by setting use_precomputed_descriptions=True which increases disk usage.\n\n```python\nfrom refined.inference.processor import Refined\n\n\nrefined = Refined.from_pretrained(model_name='wikipedia_model_with_numbers',\n                                  entity_set=\"wikipedia\")\n\nspans = refined.process_text(\"England won the FIFA World Cup in 1966.\")\n\nprint(spans)\n```\nExpected output:\n```text\n[['England', Entity(wikidata_entity_id=Q47762, wikipedia_entity_title=England national football team), 'ORG'], ['FIFA World Cup', Entity(wikidata_entity_id=Q19317, wikipedia_entity_title=FIFA World Cup), 'EVENT'], ['1966', Entity(...), 'DATE']]\n```\nNote that str(span) only returns a few fields of the returned object for readability. Many other fields, such as top-k predictions and predicted fine-grained entity types, are also accessible from the returned `Span`.\n\n#### Parameters\n***model_name***: We provide four pretrained models\n1. 'wikipedia_model': This is the model which matches the setup described in the paper\n2. 'wikipedia_model_with_numbers': This model extends the above model, to also include detection of SpaCy numerical data types in \nthe mention detection layer (\"DATE\", \"CARDINAL\", \"MONEY\", \"PERCENT\", \"TIME\", \"ORDINAL\", \"QUANTITY\"). The detected types are \navailable at ``span.coarse_type``. If the coarse_type is detected as \"DATE\", the date will be normalised to a standard \nformat available at ``span.date``. All non-numerical types will have a coarse_type of \"MENTION\", and will be passed through \nthe entity disambiguation layer to attempt to resolve them to a wikidata entity.\n3. 'aida_model': This is the model which matches the setup described in the paper for fine-tuning the model on AIDA for entity linking. Note that this model is different to the model fine-tuned on AIDA for entity disambiguation only, which is also described in the paper.\n4. 'questions_model': This model is fine-tuned on short question text (lowercase text). The model was fine-tuned on the WebQSP EL dataset and the setup is described in our paper.\n\n***entity_set***: Set to \"wikidata\" to resolve against all ~33M (after some filtering) entities in wikidata (requires more memory) or to \"wikipedia\" to \nlimit to resolving against the ~6M entities which have a wikipedia page.\n\n***data_dir*** (optional): The local directory where the data/model files will be downloaded to/loaded from (defaults to ~/.cache/refined/).\n\n***download_files*** (optional): Set to True the first time the code is run, to automatically download the data/model files from S3 to your \nlocal directory. Files will not be downloaded if they already exist but network calls will still be made to compare timestamps.\n\n***use_precomputed_descriptions*** (optional): Set to True to use precomputed embeddings of all descriptions of entities \nin the knowledge base (speeds up inference).\n\n***device*** (optional): The device to load the model/run inference on.\n\n\n### Evaluation\n\n#### Entity disambiguation\nWe provide the script `replicate_results.py` which replicates the results reported in our paper.\n\nEntity disambiguation evaluation is run using the ``eval_all`` function:\n\n```python\nfrom refined.inference.processor import Refined\nfrom refined.evaluation.evaluation import eval_all\n\nrefined = Refined.from_pretrained(model_name='wikipedia_model',\n                                  entity_set=\"wikipedia\")\n\nresults_numbers = eval_all(refined=refined, el=False)\n```\n\nThe script will automatically download the test dataset splits to `~/.cache/refined/`. Please ensure you have the \npermission to use each dataset for your use case as defined by their independent licenses. \n\n**Expected results:**\n\nWe show the expected results from the evaluation scripts below. The numbers for \"wikipedia_model\" with entity set \"wikipedia\" most closely match \nthe numbers in the paper (they differ marginally as we have updated to a newer version of Wikipedia). For both models, \nperformance on Wikidata entities is slightly lower, as all entities in the datasets are linked to Wikipedia entities (so \nadding Wikidata entities just adds a large quantity of entities that will never appear in the gold labels). \n\nThe performance of \"wikipedia_model_with_numbers\" is\nslightly lower, which is expected as the model is also trained to identify numerical types.\n\n| model_name     | entity_set |  AIDA | MSNBC | AQUAINT | ACE2004 | CWEB | WIKI |\n| ----------- | ----------- | ----------- |-------|---------| ----------- | ----------- |------| \n| wikipedia_model      | wikipedia       |  87.4 | 94.5  | 91.9    | 91.4 | 77.7 | 88.7 |\n| wikipedia_model      | wikidata       |  85.6 | 92.8  | 90.4    | 91.1 | 76.3 | 88.2 |\n| wikipedia_model_with_numbers   | wikipedia       | 85.1  | 93.5  | 90.3    | 91.7 | 76.4 | 89.4 |\n| wikipedia_model_with_numbers   | wikidata        |  84.9 | 93.6  | 90.0    | 91.2 | 75.8 | 88.9 |\n\n#### Entity linking\nEntity linking evaluation is run using the ``eval_all`` function with `el=True`:\n\n```python\nfrom refined.inference.processor import Refined\nfrom refined.evaluation.evaluation import eval_all\n\nrefined = Refined.from_pretrained(model_name='aida_model',\n                                  entity_set=\"wikipedia\")\n\nresults_numbers = eval_all(refined=refined, el=True)\n```\n\nThe results below slightly differ from the ones reported in our paper (which were produced by the Gerbil framework using an older version of Wikipedia). The `wikipedia_model` is not trained on Wikipedia hyperlinks only. Whereas, the `aida_model` is fine-tuned on the AIDA training dataset (with the weights initialised from the `wikipedia_model`). \n\n| model_name                              | entity_set | AIDA | MSNBC | \n|-----------------------------------------| ----------- |------|-------| \n| aida_model          | wikipedia       | 85.0 | 75.1  |\n| wikipedia_model  | wikipedia       | 78.3 | 73.4  |\n\nWe observe that most EL errors on the AIDA dataset are actually dataset annotation errors.\nThe AIDA dataset does not provide the entity label for every mention that can be linked to Wikipedia.\nInstead, many mentions are incorrectly labelled as NIL mentions, meaning no corresponding Wikipedia page was found for\nthe mention (during annotation). This means that EL model predictions for these mentions will be unfairly considered as incorrect.\nTo measure the impact, we added an option to filter out model predictions which **exactly** align with\nNIL mentions in the dataset:\n```python\neval_all(refined=refined, el=True, filter_nil_spans=True)\n```\nWe report 90.2 F1 on the AIDA dataset when we set `filter_nil_spans=True`, when using our \"aida_model\".\n\n### Inference speed\nWe run the model over the AIDA dataset development split using our script, `benchmark_on_aida.py`.\n\n| Hardware | Time taken to run EL on AIDA test dataset (231 news articles) | \n|----------|---------------------------------------------------------------| \n| V100 GPU | 6.5s                                                           \n| T4 GPU   | 7.4s                                                          |\n| CPU      | 29.7s                                                         |\n\nThe first time the model is loaded it will take longer because the data files need to be downloaded to disk.\n\n### Fine-tuning\nSee [FINE_TUNING.md](FINE_TUNING.md) for instructions on how to fine-tune the model on standard and custom datasets.\n\n### Training\nSee [TRAINING.md](TRAINING.md) for instructions on how to train the model on our Wikipedia hyperlinks dataset.\n\n### Generating and updating the data files and training dataset\nTo regenerate **all** data files run the `preprocess_all.py` script. This script downloads the most recent Wikipedia and Wikidata dump and generates the data files and Wikipedia training dataset.\nNote that ReFinED is capable of zero-shot entity linking, which means the data files (which will include recently added entities) can be updated without having retraining the model.\n\n### Adding additional/custom entities\nAdditional entities (which are not in Wikidata) can be added to the entity set considered by ReFinED by running `preprocess_all.py` script with the argument `--additional_entities_file \u003cpath_to_file\u003e`.\nThe file must be a jsonlines file where each row is the JSON string for an `AdditionalEntity`. Ideally, the entity types provided should be Wikidata classes such as \"Q5\" for human.\n\n### Built With\n\n* [PyTorch](https://pytorch.org/) - PyTorch is an open source machine learning library based on the Torch library.\n* [Transformers](https://pytorch.org/hub/huggingface_pytorch-transformers/) - Implementations of Transformer models.\n* Works with Python 3.8.10.\n\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis library is licensed under the Apache 2.0 License.\n\n## Contact us\nIf you have questions please open Github issues instead of sending us emails, as some of the listed email addresses are no longer active.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Frefined","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famazon-science%2Frefined","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Frefined/lists"}