{"id":15887815,"url":"https://github.com/stefan-it/co-funer","last_synced_at":"2025-04-02T16:43:13.016Z","repository":{"id":230254480,"uuid":"777654821","full_name":"stefan-it/co-funer","owner":"stefan-it","description":"Experiments on CO-Fun NER Dataset","archived":false,"fork":false,"pushed_at":"2024-03-28T15:31:44.000Z","size":66,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-08T07:23:43.256Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stefan-it.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-26T09:11:48.000Z","updated_at":"2024-03-28T15:32:28.000Z","dependencies_parsed_at":"2024-03-28T17:49:35.101Z","dependency_job_id":"94e239ac-14b9-46fa-a24a-9f0741600a13","html_url":"https://github.com/stefan-it/co-funer","commit_stats":null,"previous_names":["stefan-it/co-funer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefan-it%2Fco-funer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefan-it%2Fco-funer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefan-it%2Fco-funer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefan-it%2Fco-funer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stefan-it","download_url":"https://codeload.github.com/stefan-it/co-funer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246852952,"owners_count":20844494,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-06T06:04:58.603Z","updated_at":"2025-04-02T16:43:12.997Z","avatar_url":"https://github.com/stefan-it.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CO-Funer\n\nThis repository shows how to fine-tune Flair models on the [CO-Fun](https://arxiv.org/abs/2403.15322) NER dataset.\n\n## Dataset\n\nThe [Company Outsourcing in Fund Prospectuses (CO-Fun) dataset](https://arxiv.org/abs/2403.15322) consists of\n948 sentences with 5,969 named entity annotations, including 2,340 Outsourced Services, 2,024 Companies, 1,594 Locations\nand 11 Software annotations.\n\nOverall, the following named entities are annotated:\n\n* `Auslagerung` (engl. outsourcing)\n* `Unternehmen` (engl. company)\n* `Ort` (engl. location)\n* `Software`\n\nThe CO-Fun NER dataset from the [Model Hub](https://huggingface.co/datasets/stefan-it/co-funer) is used for fine-tuning\nFlair models.\n\n## Fine-Tuning\n\nThe main fine-tuning is done in [`experiment.py`](experiment.py).\n\nFine-tuning can be started by calling the `run_experiment()` method and passing a so called `ExperimentConfiguration`.\nIn `ExperimentConfiguration` all necessary hyper-parameters and fine-tuning options are stored. The interface looks\nlike:\n\n```python\nclass ExperimentConfiguration:\n    batch_size: int\n    learning_rate: float\n    epoch: int\n    context_size: int\n    seed: int\n    base_model: str\n    base_model_short: str\n    layers: str = \"-1\"\n    subtoken_pooling: str = \"first\"\n    use_crf: bool = False\n    use_tensorboard: bool = True\n```\n\nA hyper-parameter search grid is defined in [`script.py`](script.py). This file is used to start the fine-tuning process.\nAdditionally, the script upload the Flair model to the Model Hub. The following environment variables should be set:\n\n| Environment Variable | Description                                                                                              |\n| -------------------- | -------------------------------------------------------------------------------------------------------- |\n| `CONFIG`             | Should point to a configuration file in the `configs` folder, e.g. `configs/german_dbmdz_bert_base.json` |\n| `HF_TOKEN`           | HF Access Token, which can be found [here](https://huggingface.co/settings/tokens)                       |\n| `HUB_ORG_NAME`       | Should point to user-name or organization where the model should be uploaded to                          |\n| `HF_UPLOAD`          | If this variable is set, fine-tuned Flair model won't be uploaded to the Model Hub                       |\n\n\n## Hyper-Parameter Search\n\nIn this example the following hyper-parameter search grid is used:\n\n* Batch Sizes = `[8, 16]`\n* Learning Rates = `[3e-05, 5e-05]`\n* Seeds = `[1, 2, 3, 4, 5]`\n\nThis means 20 models will be fine-tuned in total (2 x 2 x 5 = 20).\n\n## Model Upload\n\nAfter each model is fine-tuned, it will automatically be uploaded to the Hugging Model Hub. The following files are uploaded:\n\n* `pytorch-model.bin`: Flair internally tracks the best model as `best-model.pt` over all epochs. To be compatible with the Model Hub the `best-model.pt`, is renamed automatically to `pytorch_model.bin`\n* `training.log`: Flair stores the training log in `training.log`. This file is later needed to parse the best F1-score on development set\n* `./runs`: In this folder the TensorBoard logs are stored. This enables a nice display of metrics on the Model Hub\n\n## Model Card\n\nAdditionally, this repository shows how to automatically generate model cards for all uploaded models. This includes\nalso a results overview table with linked models.\n\nThe [`Example.ipynb`](Example.ipynb) notebook gives a detailed overview of all necessary steps.\n\n## Results\n\nWe perform experiment for three BERT-based German language models:\n\n* [German BERT](https://huggingface.co/google-bert/bert-base-german-cased)\n* [German DBMDZ BERT](https://huggingface.co/dbmdz/bert-base-german-cased)\n* [GBERT](https://huggingface.co/deepset/gbert-base)\n\nThe following overview table shows the best configuration (batch size, learning rate) for each model on the development\ndataset:\n\n| Model Name        | Configuration      | Seed 1       | Seed 2       | Seed 3       | Seed 4       | Seed 5          | Average             |\n| ----------------- |--------------------|--------------|--------------|--------------|--------------|-----------------|---------------------|\n| German BERT       | `bs8-e10-lr5e-05`  | [0.9346][1]  | [0.9388][2]  | [0.9301][3]  | [0.9291][4]  | [0.9346][5]     | 0.9334 ± 0.0039     |\n| German DBMDZ BERT | `bs8-e10-lr5e-05`  | [0.9378][11] | [0.928][12]  | [0.9383][13] | [0.9374][14] | [0.9364][15]    | 0.9356 ± 0.0043     |\n| GBERT             | `bs8-e10-lr5e-05`  | [0.9477][21] | [0.935][22]  | [0.9517][23] | [0.9443][24] | [0.9342][25]    | **0.9426** ± 0.0077 |\n\nIt can be seen, that GBERT has strongest performance and achieves an average F1-Score of 94.26% on the development set.\n\nNow, we retrieve the F1-Score for the best configuration of each model on the test set:\n\n\n| Model Name        | Configuration      | Seed 1       | Seed 2       | Seed 3       | Seed 4       | Seed 5          | Average             |\n| ----------------- |--------------------|--------------|--------------|--------------|--------------|-----------------|---------------------|\n| German BERT       | `bs8-e10-lr5e-05`  | [0.9141][1]  | [0.9159][2]  | [0.9121][3]  | [0.9062][4]  | [0.9105][5]     | 0.9118 ± 0.0033     |\n| German DBMDZ BERT | `bs8-e10-lr5e-05`  | [0.9134][11] | [0.9076][12] | [0.9070][13] | [0.8821][14] | [0.9091][15]    | 0.9038 ± 0.0111     |\n| GBERT             | `bs8-e10-lr5e-05`  | [0.9180][21] | [0.9117][22] | [0.9163][23] | [0.9155][24] | [0.9110][25]    | **0.9145** ± 0.0027 |\n\nThe GBERT model has strongest performance again. With 91.45% our result is close to the reported 92.2% (see Table 1 in\nthe CO-Fun paper).\n\n[1]: https://hf.co/stefan-it/flair-co-funer-german_bert_base-bs8-e10-lr5e-05-1\n[2]: https://hf.co/stefan-it/flair-co-funer-german_bert_base-bs8-e10-lr5e-05-2\n[3]: https://hf.co/stefan-it/flair-co-funer-german_bert_base-bs8-e10-lr5e-05-3\n[4]: https://hf.co/stefan-it/flair-co-funer-german_bert_base-bs8-e10-lr5e-05-4\n[5]: https://hf.co/stefan-it/flair-co-funer-german_bert_base-bs8-e10-lr5e-05-5\n[11]: https://hf.co/stefan-it/flair-co-funer-german_dbmdz_bert_base-bs8-e10-lr5e-05-1\n[12]: https://hf.co/stefan-it/flair-co-funer-german_dbmdz_bert_base-bs8-e10-lr5e-05-2\n[13]: https://hf.co/stefan-it/flair-co-funer-german_dbmdz_bert_base-bs8-e10-lr5e-05-3\n[14]: https://hf.co/stefan-it/flair-co-funer-german_dbmdz_bert_base-bs8-e10-lr5e-05-4\n[15]: https://hf.co/stefan-it/flair-co-funer-german_dbmdz_bert_base-bs8-e10-lr5e-05-5\n[21]: https://hf.co/stefan-it/flair-co-funer-gbert_base-bs8-e10-lr5e-05-1\n[22]: https://hf.co/stefan-it/flair-co-funer-gbert_base-bs8-e10-lr5e-05-2\n[23]: https://hf.co/stefan-it/flair-co-funer-gbert_base-bs8-e10-lr5e-05-3\n[24]: https://hf.co/stefan-it/flair-co-funer-gbert_base-bs8-e10-lr5e-05-4\n[25]: https://hf.co/stefan-it/flair-co-funer-gbert_base-bs8-e10-lr5e-05-5\n\n## Release of Fine-tuned Models\n\nAll fine-tuned models for this repository are available on the Hugging Face Model Hub incl. a working inference widget\nthat allows to perform NER:\n\n![Inference Widget](images/inference-widget.png)\n\nAll fine-tuned models can be found [here](https://huggingface.co/models?search=flair-co-funer). The best models can be\nfound in\n[this collection](https://huggingface.co/collections/stefan-it/fine-tuned-co-funer-models-66058539530368090082214f).\n\n# Changelog\n\n* 28.03.2024: Initial version of this repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefan-it%2Fco-funer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefan-it%2Fco-funer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefan-it%2Fco-funer/lists"}