{"id":15014120,"url":"https://github.com/tomaarsen/spanmarkerner","last_synced_at":"2025-05-14T23:07:52.783Z","repository":{"id":149229483,"uuid":"620225616","full_name":"tomaarsen/SpanMarkerNER","owner":"tomaarsen","description":"SpanMarker for Named Entity Recognition","archived":false,"fork":false,"pushed_at":"2025-01-08T11:37:50.000Z","size":90751,"stargazers_count":430,"open_issues_count":26,"forks_count":30,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-05-14T23:07:41.102Z","etag":null,"topics":["huggingface","ner","nlp","spacy","spacy-extension","transformers"],"latest_commit_sha":null,"homepage":"https://tomaarsen.github.io/SpanMarkerNER/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tomaarsen.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-03-28T09:14:44.000Z","updated_at":"2025-05-12T11:25:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"69d03baa-c6b3-4b37-bc93-4b3df24f53fd","html_url":"https://github.com/tomaarsen/SpanMarkerNER","commit_stats":{"total_commits":272,"total_committers":3,"mean_commits":90.66666666666667,"dds":0.007352941176470562,"last_synced_commit":"c360ff050a0ad0c21d8eb33a4f197f9d3727b598"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomaarsen%2FSpanMarkerNER","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomaarsen%2FSpanMarkerNER/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomaarsen%2FSpanMarkerNER/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomaarsen%2FSpanMarkerNER/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tomaarsen","download_url":"https://codeload.github.com/tomaarsen/SpanMarkerNER/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254243363,"owners_count":22038046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["huggingface","ner","nlp","spacy","spacy-extension","transformers"],"created_at":"2024-09-24T19:45:13.113Z","updated_at":"2025-05-14T23:07:47.771Z","avatar_url":"https://github.com/tomaarsen.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003e\nSpanMarker for Named Entity Recognition\n\u003c/h1\u003e\n\u003ca href=\"https://huggingface.co/tomaarsen/span-marker-roberta-large-ontonotes5\" target=\"_blank\"\u003e\n    \u003cimg src=\"https://github.com/tomaarsen/SpanMarkerNER/assets/37621491/c76d6393-bb0b-44c3-9412-fd9c8313dcc1\"\u003e\n\u003c/a\u003e\n\n[🤗 Models](https://huggingface.co/models?library=span-marker) |\n[🛠️ Getting Started In Google Colab](https://colab.research.google.com/github/tomaarsen/SpanMarkerNER/blob/main/notebooks/getting_started.ipynb) |\n[📄 Documentation](https://tomaarsen.github.io/SpanMarkerNER) | 📊 [Thesis](https://raw.githubusercontent.com/tomaarsen/SpanMarkerNER/main/thesis.pdf)\n\u003c/div\u003e\n\nSpanMarker is a framework for training powerful Named Entity Recognition models using familiar encoders such as BERT, RoBERTa and ELECTRA.\nBuilt on top of the familiar [🤗 Transformers](https://github.com/huggingface/transformers) library, SpanMarker inherits a wide range of powerful functionalities, such as easily loading and saving models, hyperparameter optimization, automatic logging in various tools, checkpointing, callbacks, mixed precision training, 8-bit inference, and more.\n\n\u003c!--Tightly implemented on top of the [🤗 Transformers](https://github.com/huggingface/transformers/) library, SpanMarker can take advantage of its valuable functionality.--\u003e\n\u003c!-- like performance dashboard integration, automatic mixed precision, 8-bit inference--\u003e\n\nBased on the [PL-Marker](https://arxiv.org/pdf/2109.06067.pdf) paper, SpanMarker breaks the mold through its accessibility and ease of use. Crucially, SpanMarker works out of the box with many common encoders such as `bert-base-cased`, `roberta-large` and `bert-base-multilingual-cased`, and automatically works with datasets using the `IOB`, `IOB2`, `BIOES`, `BILOU` or no label annotation scheme.\n\nAdditionally, the SpanMarker library has been integrated with the Hugging Face Hub and the Hugging Face Inference API. See the SpanMarker documentation on [Hugging Face](https://huggingface.co/docs/hub/span_marker) or see [all SpanMarker models on the Hugging Face Hub](https://huggingface.co/models?library=span-marker).\nThrough the Inference API integration, users can test any SpanMarker model on the Hugging Face Hub for free using a widget on the [model page](https://huggingface.co/tomaarsen/span-marker-bert-base-fewnerd-fine-super). Furthermore, each public SpanMarker model offers a free API for fast prototyping and can be deployed to production using Hugging Face Inference Endpoints.\n\n| Inference API Widget (on a model page) | Free Inference API (`Deploy` \u003e `Inference API` on a model page) |\n| ------------- | ------------- |\n|  ![image](https://github.com/tomaarsen/SpanMarkerNER/assets/37621491/234078b7-22c8-491c-8686-faccd394f683) |  ![image](https://github.com/tomaarsen/SpanMarkerNER/assets/37621491/410e5191-9354-4e27-b718-2d69af678eb7) |\n\n## Documentation\nFeel free to have a look at the [documentation](https://tomaarsen.github.io/SpanMarkerNER).\n\n## Installation\nYou may install the [`span_marker`](https://pypi.org/project/span-marker) Python module via `pip` like so:\n```\npip install span_marker\n```\n\n## Quick Start\n### Training\nPlease have a look at our [Getting Started](notebooks/getting_started.ipynb) notebook for details on how SpanMarker is commonly used. It explains the following snippet in more detail. Alternatively, have a look at the [training scripts](training_scripts) that have been successfully used in the past.\n\n| Colab                                                                                                                                                                                                         | Kaggle                                                                                                                                                                                                             | Gradient                                                                                                                                                                                         | Studio Lab                                                                                                                                                                                                             |\n|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tomaarsen/SpanMarkerNER/blob/main/notebooks/getting_started.ipynb)                       | [![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/tomaarsen/SpanMarkerNER/blob/main/notebooks/getting_started.ipynb)                       | [![Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://console.paperspace.com/github/tomaarsen/SpanMarkerNER/blob/main/notebooks/getting_started.ipynb)                       | [![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/tomaarsen/SpanMarkerNER/blob/main/notebooks/getting_started.ipynb)                       |\n\n```python\nfrom pathlib import Path\nfrom datasets import load_dataset\nfrom transformers import TrainingArguments\nfrom span_marker import SpanMarkerModel, Trainer, SpanMarkerModelCardData\n\n\ndef main() -\u003e None:\n    # Load the dataset, ensure \"tokens\" and \"ner_tags\" columns, and get a list of labels\n    dataset_id = \"DFKI-SLT/few-nerd\"\n    dataset_name = \"FewNERD\"\n    dataset = load_dataset(dataset_id, \"supervised\")\n    dataset = dataset.remove_columns(\"ner_tags\")\n    dataset = dataset.rename_column(\"fine_ner_tags\", \"ner_tags\")\n    labels = dataset[\"train\"].features[\"ner_tags\"].feature.names\n    # ['O', 'art-broadcastprogram', 'art-film', 'art-music', 'art-other', ...\n\n    # Initialize a SpanMarker model using a pretrained BERT-style encoder\n    encoder_id = \"bert-base-cased\"\n    model_id = f\"tomaarsen/span-marker-{encoder_id}-fewnerd-fine-super\"\n    model = SpanMarkerModel.from_pretrained(\n        encoder_id,\n        labels=labels,\n        # SpanMarker hyperparameters:\n        model_max_length=256,\n        marker_max_length=128,\n        entity_max_length=8,\n        # Model card arguments\n        model_card_data=SpanMarkerModelCardData(\n            model_id=model_id,\n            encoder_id=encoder_id,\n            dataset_name=dataset_name,\n            dataset_id=dataset_id,\n            license=\"cc-by-sa-4.0\",\n            language=\"en\",\n        ),\n    )\n\n    # Prepare the 🤗 transformers training arguments\n    output_dir = Path(\"models\") / model_id\n    args = TrainingArguments(\n        output_dir=output_dir,\n        # Training Hyperparameters:\n        learning_rate=5e-5,\n        per_device_train_batch_size=32,\n        per_device_eval_batch_size=32,\n        num_train_epochs=3,\n        weight_decay=0.01,\n        warmup_ratio=0.1,\n        bf16=True,  # Replace `bf16` with `fp16` if your hardware can't use bf16.\n        # Other Training parameters\n        logging_first_step=True,\n        logging_steps=50,\n        evaluation_strategy=\"steps\",\n        save_strategy=\"steps\",\n        eval_steps=3000,\n        save_total_limit=2,\n        dataloader_num_workers=2,\n    )\n\n    # Initialize the trainer using our model, training args \u0026 dataset, and train\n    trainer = Trainer(\n        model=model,\n        args=args,\n        train_dataset=dataset[\"train\"],\n        eval_dataset=dataset[\"validation\"],\n    )\n    trainer.train()\n\n    # Compute \u0026 save the metrics on the test set\n    metrics = trainer.evaluate(dataset[\"test\"], metric_key_prefix=\"test\")\n    trainer.save_metrics(\"test\", metrics)\n\n    # Save the final checkpoint\n    trainer.save_model(output_dir / \"checkpoint-final\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\n### Inference\n```python\nfrom span_marker import SpanMarkerModel\n\n# Download from the 🤗 Hub\nmodel = SpanMarkerModel.from_pretrained(\"tomaarsen/span-marker-bert-base-fewnerd-fine-super\")\n# Run inference\nentities = model.predict(\"Amelia Earhart flew her single engine Lockheed Vega 5B across the Atlantic to Paris.\")\n[{'span': 'Amelia Earhart', 'label': 'person-other', 'score': 0.7659597396850586, 'char_start_index': 0, 'char_end_index': 14},\n {'span': 'Lockheed Vega 5B', 'label': 'product-airplane', 'score': 0.9725785851478577, 'char_start_index': 38, 'char_end_index': 54},\n {'span': 'Atlantic', 'label': 'location-bodiesofwater', 'score': 0.7587679028511047, 'char_start_index': 66, 'char_end_index': 74},\n {'span': 'Paris', 'label': 'location-GPE', 'score': 0.9892390966415405, 'char_start_index': 78, 'char_end_index': 83}]\n```\n\n## Pretrained Models\n\nAll models in this list contain `train.py` files that show the training scripts used to generate them. Additionally, all training scripts used are stored in the [training_scripts](training_scripts) directory.\nThese trained models have Hosted Inference API widgets that you can use to experiment with the models on their Hugging Face model pages. Additionally, Hugging Face provides each model with a free API (`Deploy` \u003e `Inference API` on the model page).\n\nThese models are further elaborated on in my [thesis](https://raw.githubusercontent.com/tomaarsen/SpanMarkerNER/main/thesis.pdf).\n\n### FewNERD\n* [`tomaarsen/span-marker-bert-base-fewnerd-fine-super`](https://huggingface.co/tomaarsen/span-marker-bert-base-fewnerd-fine-super) is a model that I have trained in 2 hours on the finegrained, supervised [Few-NERD dataset](https://huggingface.co/datasets/DFKI-SLT/few-nerd). It reached a 70.53 Test F1, competitive in the all-time [Few-NERD leaderboard](https://paperswithcode.com/sota/named-entity-recognition-on-few-nerd-sup) using `bert-base`. My training script resembles the one that you can see above.\n\n* [`tomaarsen/span-marker-roberta-large-fewnerd-fine-super`](https://huggingface.co/tomaarsen/span-marker-roberta-large-fewnerd-fine-super) was trained in 6 hours on the finegrained, supervised [Few-NERD dataset](https://huggingface.co/datasets/DFKI-SLT/few-nerd) using `roberta-large`. It reached a 71.03 Test F1, reaching a new state of the art in the all-time [Few-NERD leaderboard](https://paperswithcode.com/sota/named-entity-recognition-on-few-nerd-sup).\n* [`tomaarsen/span-marker-xlm-roberta-base-fewnerd-fine-super`](https://huggingface.co/tomaarsen/span-marker-xlm-roberta-base-fewnerd-fine-super) is a multilingual model that I have trained in 1.5 hours on the finegrained, supervised [Few-NERD dataset](https://huggingface.co/datasets/DFKI-SLT/few-nerd). It reached a 68.6 Test F1 on English, and works well on other languages like Spanish, French, German, Russian, Dutch, Polish, Icelandic, Greek and many more.\n\n### OntoNotes v5.0\n* [`tomaarsen/span-marker-roberta-large-ontonotes5`](https://huggingface.co/tomaarsen/span-marker-roberta-large-ontonotes5) was trained in 3 hours on the OntoNotes v5.0 dataset, reaching a performance of 91.54 F1. For reference, the current strongest spaCy model (`en_core_web_trf`) reaches 89.8 F1. This SpanMarker model uses a `roberta-large` encoder under the hood.\n\n### CoNLL03\n* [`tomaarsen/span-marker-xlm-roberta-large-conll03`](https://huggingface.co/tomaarsen/span-marker-xlm-roberta-large-conll03) is a SpanMarker model using `xlm-roberta-large` that was trained in 45 minutes. It reaches a state of the art 93.1 F1 on CoNLL03 without using document-level context. For reference, the current strongest spaCy model (`en_core_web_trf`) reaches 91.6.\n* [`tomaarsen/span-marker-xlm-roberta-large-conll03-doc-context`](https://huggingface.co/tomaarsen/span-marker-xlm-roberta-large-conll03-doc-context) is another SpanMarker model using the `xlm-roberta-large` encoder. It uses [document-level context](https://tomaarsen.github.io/SpanMarkerNER/notebooks/document_level_context.html) to reach a state of the art 94.4 F1. For the best performance, inference should be performed using document-level context ([docs](https://tomaarsen.github.io/SpanMarkerNER/notebooks/document_level_context.html#Inference)). This model was trained in 1 hour.\n\n### CoNLL++\n* [`tomaarsen/span-marker-xlm-roberta-large-conllpp-doc-context`](https://huggingface.co/tomaarsen/span-marker-xlm-roberta-large-conllpp-doc-context) was trained in an hour using the `xlm-roberta-large` encoder on the CoNLL++ dataset. Using [document-level context](https://tomaarsen.github.io/SpanMarkerNER/notebooks/document_level_context.html), it reaches a very competitive 95.5 F1. For the best performance, inference should be performed using document-level context ([docs](https://tomaarsen.github.io/SpanMarkerNER/notebooks/document_level_context.html#Inference)).\n\n### MultiNERD\n* [`tomaarsen/span-marker-xlm-roberta-base-multinerd`](https://huggingface.co/tomaarsen/span-marker-xlm-roberta-base-multinerd) is a multilingual SpanMarker model using the `xlm-roberta-large` encoder trained on the huge [MultiNERD](https://huggingface.co/datasets/Babelscape/multinerd) dataset. It reaches a 91.31 F1 on all 10 training languages and 94.55 F1 on English only. The model can predict between 15 classes. For best performance, separate punctuation from your words as described [here](https://huggingface.co/tomaarsen/span-marker-xlm-roberta-base-multinerd#limitations). Note that [`tomaarsen/span-marker-mbert-base-multinerd`](https://huggingface.co/tomaarsen/span-marker-mbert-base-multinerd) does not have this limitation and performs better.\n\n* [`tomaarsen/span-marker-mbert-base-multinerd`](https://huggingface.co/tomaarsen/span-marker-mbert-base-multinerd) is the successor of [`tomaarsen/span-marker-xlm-roberta-base-multinerd`](https://huggingface.co/tomaarsen/span-marker-xlm-roberta-base-multinerd). It's a multilingual SpanMarker model using `bert-base-multilingual-cased` trained on the [MultiNERD](https://huggingface.co/datasets/Babelscape/multinerd) dataset. It reaches a state-of-the-art 92.48 F1 on all 10 training languages and 95.18 F1 on English only. This model generalizes well to languages using the Latin and Cyrillic script.\n\n## Using pretrained SpanMarker models with spaCy\nAll [SpanMarker models on the Hugging Face Hub](https://huggingface.co/models?library=span-marker) can also be easily used in spaCy. It's as simple as including 1 line to add the `span_marker` pipeline. See the [Documentation](https://tomaarsen.github.io/SpanMarkerNER/notebooks/spacy_integration.html) or [API Reference](https://tomaarsen.github.io/SpanMarkerNER/api/span_marker.spacy_integration.html) for more information.\n```python\nimport spacy\n\n# Load the spaCy model with the span_marker pipeline component\nnlp = spacy.load(\"en_core_web_sm\", exclude=[\"ner\"])\nnlp.add_pipe(\"span_marker\", config={\"model\": \"tomaarsen/span-marker-roberta-large-ontonotes5\"})\n\n# Feed some text through the model to get a spacy Doc\ntext = \"\"\"Cleopatra VII, also known as Cleopatra the Great, was the last active ruler of the \\\nPtolemaic Kingdom of Egypt. She was born in 69 BCE and ruled Egypt from 51 BCE until her \\\ndeath in 30 BCE.\"\"\"\ndoc = nlp(text)\n\n# And look at the entities\nprint([(entity, entity.label_) for entity in doc.ents])\n\"\"\"\n[(Cleopatra VII, \"PERSON\"), (Cleopatra the Great, \"PERSON\"), (the Ptolemaic Kingdom of Egypt, \"GPE\"),\n(69 BCE, \"DATE\"), (Egypt, \"GPE\"), (51 BCE, \"DATE\"), (30 BCE, \"DATE\")]\n\"\"\"\n```\n![image](https://user-images.githubusercontent.com/37621491/246170623-6351cb7e-bbb0-4472-af16-9a351a253da9.png)\n\n## Context\n\u003ch1 align=\"center\"\u003e\n    \u003ca href=\"https://github.com/argilla-io/argilla\"\u003e\n    \u003cimg src=\"https://github.com/dvsrepo/imgs/raw/main/rg.svg\" alt=\"Argilla\" width=\"150\"\u003e\n    \u003c/a\u003e\n\u003c/h1\u003e\n\nI have developed this library as a part of my thesis work at [Argilla](https://github.com/argilla-io/argilla). Feel free to read my finished thesis [here](https://raw.githubusercontent.com/tomaarsen/SpanMarkerNER/main/thesis.pdf) in this repository!\n\n## Changelog\nSee [CHANGELOG.md](CHANGELOG.md) for news on all SpanMarker versions.\n\n## License\nSee [LICENSE](LICENSE) for the current license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomaarsen%2Fspanmarkerner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftomaarsen%2Fspanmarkerner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomaarsen%2Fspanmarkerner/lists"}