{"id":15674666,"url":"https://github.com/stefan-it/italian-bertelectra","last_synced_at":"2025-05-06T23:24:28.231Z","repository":{"id":110973992,"uuid":"280541306","full_name":"stefan-it/italian-bertelectra","owner":"stefan-it","description":"🇮🇹 Italian BERT and ELECTRA models (incl. evaluation)","archived":false,"fork":false,"pushed_at":"2022-10-20T05:36:24.000Z","size":547,"stargazers_count":18,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-31T04:01:41.177Z","etag":null,"topics":["bert","bert-model","electra","italian"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stefan-it.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-17T23:05:25.000Z","updated_at":"2025-01-27T06:49:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"9ce515bc-1cfb-4b3c-b082-70b6cb35067c","html_url":"https://github.com/stefan-it/italian-bertelectra","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefan-it%2Fitalian-bertelectra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefan-it%2Fitalian-bertelectra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefan-it%2Fitalian-bertelectra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefan-it%2Fitalian-bertelectra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stefan-it","download_url":"https://codeload.github.com/stefan-it/italian-bertelectra/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252784374,"owners_count":21803665,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","bert-model","electra","italian"],"created_at":"2024-10-03T15:49:05.466Z","updated_at":"2025-05-06T23:24:28.199Z","avatar_url":"https://github.com/stefan-it.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🇮🇹 Italian BERTELECTRA models\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4263142.svg)](https://doi.org/10.5281/zenodo.4263142)\n\nIn this repository we monitor all experiments for our trained [DBMDZ models](https://github.com/dbmdz/berts)\nfor Italian. It includes both BERT and ELECTRA models and we use 🤗 Transformers library to fine-tune models.\n\nMade with 🤗 and ❤️ from Munich.\n\n# Changelog\n\n* 08.11.2020: Add DOI/Zenodo information, including citation section.\n* 02.11.2020: Initial version and public release of Italian XXL ELECTRA model.\n\n# Training\n\nThe source data for the Italian BERT model consists of a recent Wikipedia dump and\nvarious texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final\ntraining corpus has a size of 13GB and 2,050,057,573 tokens.\n\nFor sentence splitting, we use NLTK (faster compared to spacy).\nOur cased and uncased models are training with an initial sequence length of 512\nsubwords for ~2-3M steps.\n\nFor the XXL Italian models, we use the same training data from OPUS and extend\nit with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).\nThus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.\n\nNote: Unfortunately, a wrong vocab size was used when training the XXL models.\nThis explains the mismatch of the \"real\" vocab size of 31,102, compared to the\nvocab size specified in `config.json`. However, the model is working and all\nevaluations were done under those circumstances.\nSee [this issue](https://github.com/dbmdz/berts/issues/7) for more information.\n\nThe Italian ELECTRA model was trained on the \"XXL\" corpus for 1M steps in total using a batch\nsize of 128. We pretty much following the ELECTRA training procedure as used for\n[BERTurk](https://github.com/stefan-it/turkish-bert/tree/master/electra). The ELECTRA model uses\nthe same vocab as the XXL BERT model; but this time we use the correct vocab size in the `config.json`\nfile ;)\n\n# Experiments\n\nWe use the awesome 🤗 Transformers library for all fine-tuning experiments.\n\nPlease star and watch [Transformers](https://github.com/huggingface/transformers) on GitHub!\n\nAll JSON-based configuration files for our experiments can be found in the\n[configuration](https://github.com/stefan-it/italian-bertelectra/tree/main/configs) folder\nin this repository. To replicate the results, just clone the latest version of Transforms, `cd`\ninto the `examples/token-classification` folder and run `python3 run_ner.py \u003cconfiguration.json\u003e`.\n\n## PoS Tagging\n\n### Italian-ISDT\n\nDescription:\n\n\u003e The Italian corpus annotated according to the UD annotation scheme was obtained by conversion\n\u003e from ISDT (Italian Stanford Dependency Treebank), released for the dependency parsing shared\n\u003e task of Evalita-2014 (Bosco et al. 2014).\n\nDetails:\n\n* [Italian-ISDT Repository](https://github.com/UniversalDependencies/UD_Italian-ISDT)\n* Commit: `f20fa2b`\n\nResults (Development set)\n\n| Model                                         | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Avg.\n| --------------------------------------------- | ----- | ----- | ----- | ----- | ----- | -------------- |\n| `dbmdz/bert-base-italian-cased`               | 98.56 | 98.64 | 98.65 | 98.74 | 98.62 | 98.64 ± 0.06\n| `dbmdz/bert-base-italian-uncased`             | 98.17 | 98.25 | 98.23 | 98.21 | 98.24 | 98.22 ± 0.03\n| `dbmdz/bert-base-italian-xxl-cased`           | 98.52 | 98.63 | 98.76 | 98.70 | 98.63 | 98.65 ± 0.08\n| `dbmdz/bert-base-italian-xxl-uncased`         | 98.38 | 98.33 | 98.35 | 98.41 | 98.30 | 98.35 ± 0.04\n| `bert-base-multilingual-cased`                | 98.50 | 98.54 | 98.49 | 98.54 | 98.43 | 98.50 ± 0.04\n| `bert-base-multilingual-uncased`              | 98.24 | 98.17 | 98.22 | 98.27 | 98.23 | 98.23 ± 0.03\n| `xlm-roberta-base`                            | 98.63 | 98.61 | 98.66 | 98.60 | 98.62 | 98.62 ± 0.02\n| `electra-base-italian-xxl-cased` (1M)         | 98.72 | 98.75 | 98.78 | 98.68 | 98.76 | **98.74** ± 0.03\n\nResults (Test set)\n\n| Model                                         | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Avg.\n| --------------------------------------------- | ----- | ----- | ----- | ----- | ----- | --------------- |\n| `dbmdz/bert-base-italian-cased`               | 98.81 | 98.88 | 98.93 | 98.77 | 98.79 | 98.84 ± 0.06\n| `dbmdz/bert-base-italian-uncased`             | 98.65 | 98.54 | 98.66 | 98.62 | 98.61 | 98.62 ± 0.04\n| `dbmdz/bert-base-italian-xxl-cased`           | 98.93 | 98.85 | 98.92 | 98.89 | 98.89 | **98.90** ± 0.03\n| `dbmdz/bert-base-italian-xxl-uncased`         | 98.72 | 98.65 | 98.73 | 98.75 | 98.78 | 98.73 ± 0.04\n| `bert-base-multilingual-cased`                | 98.79 | 98.71 | 98.60 | 98.72 | 98.64 | 98.69 ± 0.07\n| `bert-base-multilingual-uncased`              | 98.57 | 98.51 | 98.49 | 98.47 | 98.44 | 98.50 ± 0.04\n| `xlm-roberta-base`                            | 98.86 | 98.79 | 98.78 | 98.77 | 98.76 | 98.79 ± 0.04\n| `electra-base-italian-xxl-cased` (1M)         | 98.84 | 98.93 | 98.82 | 98.88 | 98.92 | 98.88 ± 0.04\n\nPlots for results on both development and test set:\n\n![UD Italian-ISDT (Development)](figures/plot_Italian-ISDT_dev.png)\n\n![UD Italian-ISDT (Test)](figures/plot_Italian-ISDT_test.png)\n\n### Italian-PoSTWITA\n\nDescription:\n\n\u003e PoSTWITA-UD has been created by enriching the dataset used for the EVALITA 2016 task of Part-of-Speech tagging\n\u003e of Social Media (see (Bosco et al. 2016)). The original corpus consists of 6,438 tweets of the development set\n\u003e (114,967 tokens) and 300 tweets of the test set (4,759 tokens), annotated at PoS level only. The conversion and\n\u003e syntactic annotation process was carried out through alternating steps of automatic scripting and manual revision,\n\u003e and finally with some out-of-domain parsing experiments. Parsing results also underwent a manual revision by two\n\u003e independent annotators.\n\nDetails:\n\n* [Italian-PoSTWITA Repository](https://github.com/UniversalDependencies/UD_Italian-PoSTWITA)\n* Commit: `662b235`\n\nResults (Development set)\n\n| Model                                         | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Avg.\n| --------------------------------------------- | ----- | ----- | ----- | ----- | ----- | -------------- |\n| `dbmdz/bert-base-italian-cased`               | 96.83 | 96.80 | 96.93 | 96.87 | 96.89 | 96.86 ± 0.05\n| `dbmdz/bert-base-italian-uncased`             | 96.82 | 96.80 | 96.69 | 96.76 | 96.69 | 96.75 ± 0.05\n| `dbmdz/bert-base-italian-xxl-cased`           | 97.34 | 97.37 | 97.35 | 97.31 | 97.22 | 97.32 ± 0.05\n| `dbmdz/bert-base-italian-xxl-uncased`         | 96.96 | 96.96 | 97.12 | 96.98 | 97.02 | 97.01 ± 0.06\n| `bert-base-multilingual-cased`                | 95.96 | 95.92 | 95.98 | 95.94 | 95.92 | 95.94 ± 0.02\n| `bert-base-multilingual-uncased`              | 96.06 | 96.17 | 96.28 | 96.23 | 96.25 | 96.20 ± 0.08\n| `xlm-roberta-base`                            | 96.81 | 96.73 | 96.87 | 96.80 | 96.74 | 96.79 ± 0.05\n| `electra-base-italian-xxl-cased` (1M)         | 97.35 | 97.26 | 97.33 | 97.36 | 97.47 | **97.35** ± 0.07\n\nResults (Test set)\n\n| Model                                         | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Avg.\n| --------------------------------------------- | ----- | ----- | ----- | ----- | ----- | -------------- |\n| `dbmdz/bert-base-italian-cased`               | 96.93 | 97.03 | 97.05 | 97.12 | 97.11 | 97.05 ± 0.07\n| `dbmdz/bert-base-italian-uncased`             | 96.73 | 96.71 | 96.58 | 96.60 | 96.58 | 96.64 ± 0.07\n| `dbmdz/bert-base-italian-xxl-cased`           | 97.15 | 97.17 | 97.34 | 97.05 | 97.12 | 97.17 ± 0.10\n| `dbmdz/bert-base-italian-xxl-uncased`         | 96.95 | 97.12 | 96.96 | 97.02 | 97.01 | 97.01 ± 0.06\n| `bert-base-multilingual-cased`                | 96.17 | 96.15 | 96.09 | 96.15 | 96.07 | 96.13 ± 0.04\n| `bert-base-multilingual-uncased`              | 96.23 | 96.06 | 96.07 | 96.15 | 96.09 | 96.12 ± 0.06\n| `xlm-roberta-base`                            | 96.94 | 96.82 | 97.05 | 97.06 | 96.99 | 96.97 ± 0.09\n| `electra-base-italian-xxl-cased` (1M)         | 97.27 | 97.39 | 97.23 | 97.22 | 97.23 | **97.27** ± 0.06\n\nPlots for results on both development and test set:\n\n![UD Italian-PoSTWITA (Development)](figures/plot_Italian-PoSTWITA_dev.png)\n\n![UD Italian-PoSTWITA (Test)](figures/plot_Italian-PoSTWITA_test.png)\n\n## NER\n\n### EVALITA 2009\n\n\u003e In the Named Entity Recognition subtask, systems are required to recognize only the Named Entities occurring\n\u003e in a text; more specifically Person, Organization, Location and Geo-Political Entities (see the annotation\n\u003e report for more details). As in the previous edition of EVALITA, the evaluation will be based on the Italian\n\u003e Content Annotation Bank (I-CAB) where Named Entities are annotated in the IOB format (where \"B-begin\" and\n\u003e \"I-inside\" denote the tokens belonging to Named Entities and \"O-outside\" is used for all other tokens).\n\u003e The dataset that has been used for the NER task at EVALITA 2007 (525 news stories), will be distributed as\n\u003e development set, while the testset will consist of completely new data.\n\nUnfortunately, some part of the EVALITA 2007 development set is part of the 2009 train set, so we don't report\ndev scores here (they were above 99.9%).\n\nResults (Test set)\n\n| Model                                         | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Avg.\n| --------------------------------------------- | ----- | ----- | ----- | ----- | ----- | -------------- |\n| `dbmdz/bert-base-italian-cased`               | 86.12 | 86.23 | 86.01 | 85.89 | 85.55 | 85.96 ± 0.23\n| `dbmdz/bert-base-italian-uncased`             | 85.63 | 85.51 | 84.98 | 85.21 | 84.80 | 85.23 ± 0.31\n| `dbmdz/bert-base-italian-xxl-cased`           | 88.33 | 88.29 | 88.45 | 88.24 | 88.27 | 88.32 ± 0.07\n| `dbmdz/bert-base-italian-xxl-uncased`         | 87.62 | 88.20 | 88.40 | 88.52 | 87.76 | 88.10 ± 0.35\n| `bert-base-multilingual-cased`                | 84.46 | 83.86 | 84.87 | 84.84 | 85.41 | 84.69 ± 0.51\n| `bert-base-multilingual-uncased`              | 83.36 | 83.46 | 83.96 | 83.88 | 83.71 | 83.67 ± 0.23\n| `xlm-roberta-base`                            | 84.23 | 84.83 | 84.41 | 84.51 | 83.81 | 84.36 ± 0.34\n| `electra-base-italian-xxl-cased` (1M)         | 87.50 | 88.10 | 88.56 | 88.22 | 88.31 | **88.14** ± 0.35\n\nPlots for results on test set:\n\n![UD Italian-EVALITA (Test)](figures/plot_Italian-EVALITA_test.png)\n\n# Model usage\n\nAll trained models can be used from the [DBMDZ](https://github.com/dbmdz) Hugging Face [model hub page](https://huggingface.co/dbmdz)\nusing their model name. The following models are available:\n\n* Cased and uncased BERT models (medium size corpus): `dbmdz/bert-base-italian-cased` and `dbmdz/bert-base-italian-uncased`\n* Cased and uncased BERT models (XXL size corpus): `dbmdz/bert-base-italian-xxl-cased` and `dbmdz/bert-base-italian-xxl-cased`\n* *ELECTRA* model (discriminator and generator on XXL size corpus): `dbmdz/electra-base-italian-xxl-cased-discriminator` and `dbmdz/electra-base-italian-xxl-cased-generator`\n\nExample usage with 🤗/Transformers:\n\n```python\nfrom transformers import AutoModel, AutoTokenizer\n\nmodel_name = \"dbmdz/bert-base-italian-xxl-cased\"\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\nmodel = AutoModel.from_pretrained(model_name)\n```\n\nThis loads the Italian XXL BERT cased model. The recently introduced ELEC**TR**A base model can be loaded with:\n\n```python\nfrom transformers import AutoModel, AutoTokenizer\n\nmodel_name = \"dbmdz/electra-base-italian-xxl-cased-discriminator\"\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\nmodel = AutoModelWithLMHead.from_pretrained(model_name)\n```\n\n# License\n\nAll models are licensed under [MIT](LICENSE).\n\n# Contact (Bugs, Feedback, Contribution and more)\n\nFor questions about our BERT models just open an issue\n[in the DBMDZ BERT repo](https://github.com/dbmdz/berts/issues/new) or in\n[this repo](https://github.com/stefan-it/italian-bertelectra/issues/new) 🤗\n\n# Citation\n\nYou can use the following BibTeX entry for citation:\n\n```bibtex\n@software{stefan_schweter_2020_4263142,\n  author       = {Stefan Schweter},\n  title        = {Italian BERT and ELECTRA models},\n  month        = nov,\n  year         = 2020,\n  publisher    = {Zenodo},\n  version      = {1.0.1},\n  doi          = {10.5281/zenodo.4263142},\n  url          = {https://doi.org/10.5281/zenodo.4263142}\n}\n```\n\n# Acknowledgments\n\nResearch supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).\nThanks for providing access to the TFRC ❤️\n\nThanks to the generous support from the [Hugging Face](https://huggingface.co/) team,\nit is possible to download both cased and uncased models from their S3 storage 🤗\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefan-it%2Fitalian-bertelectra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefan-it%2Fitalian-bertelectra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefan-it%2Fitalian-bertelectra/lists"}