{"id":24480834,"url":"https://github.com/dsfsi/puoberta","last_synced_at":"2026-02-04T22:35:59.214Z","repository":{"id":200461478,"uuid":"703987068","full_name":"dsfsi/PuoBERTa","owner":"dsfsi","description":"A Roberta-based language model specially designed for Setswana, using the new PuoData dataset.","archived":false,"fork":false,"pushed_at":"2025-12-30T18:25:14.000Z","size":3755,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-01-03T15:21:20.429Z","etag":null,"topics":["african-languages","africannlp","dsfsi-datasets","nlproc","setswana","tn","tsn"],"latest_commit_sha":null,"homepage":"https://huggingface.co/dsfsi/PuoBERTa","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dsfsi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-12T10:01:10.000Z","updated_at":"2025-12-30T18:25:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"4f500500-770a-4173-a1a3-251d35ce8900","html_url":"https://github.com/dsfsi/PuoBERTa","commit_stats":null,"previous_names":["dsfsi/puoberta"],"tags_count":0,"template":false,"template_full_name":"dsfsi/dsfsi-project-starter","purl":"pkg:github/dsfsi/PuoBERTa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsfsi%2FPuoBERTa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsfsi%2FPuoBERTa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsfsi%2FPuoBERTa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsfsi%2FPuoBERTa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dsfsi","download_url":"https://codeload.github.com/dsfsi/PuoBERTa/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsfsi%2FPuoBERTa/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29098239,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-04T21:05:08.033Z","status":"ssl_error","status_checked_at":"2026-02-04T21:04:53.031Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["african-languages","africannlp","dsfsi-datasets","nlproc","setswana","tn","tsn"],"created_at":"2025-01-21T11:17:30.615Z","updated_at":"2026-02-04T22:35:59.207Z","avatar_url":"https://github.com/dsfsi.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PuoBERTa: A Curated Setswana Language Model\n\n[![Zenodo doi badge](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.8434795-blue.svg)](https://doi.org/10.5281/zenodo.8434795) [![arXiv](https://img.shields.io/badge/arXiv-2310.09141-b31b1b.svg)](https://arxiv.org/abs/2310.09141) [![HuggingFace](https://img.shields.io/badge/%F0%9F%A4%97-Model-yellow)](https://huggingface.co/dsfsi/PuoBERTa) [![HuggingFace Space](https://img.shields.io/badge/%F0%9F%A4%97-Demo-orange)](https://huggingface.co/spaces/dsfsi/PuoBERTaSpace)\n\nA RoBERTa-based language model specially designed for Setswana, trained on the PuoData dataset for accurate and culturally relevant NLP applications.\n\n**Try it now:** [Interactive Demo](https://huggingface.co/spaces/dsfsi/PuoBERTaSpace) | [Model on HuggingFace](https://huggingface.co/dsfsi/PuoBERTa) | [Paper](https://arxiv.org/abs/2310.09141)\n\nGive Feedback 📑: [DSFSI Resource Feedback Form](https://docs.google.com/forms/d/e/1FAIpQLSf7S36dyAUPx2egmXbFpnTBuzoRulhL5Elu-N1eoMhaO7v10w/formResponse)\n\n---\n\n## Table of Contents\n- [Quick Start](#quick-start)\n- [Model Details](#model-details)\n- [Installation](#installation)\n- [Usage Examples](#usage-examples)\n- [Downstream Performance](#downstream-performance)\n- [Pre-Training Dataset](#pre-training-dataset)\n- [Citation](#citation-information)\n- [Contributing](#contributing)\n- [Contact](#model-card-contact)\n\n---\n\n## Quick Start\n\n### Try Online (No Installation Required)\n\nVisit our [**Interactive Demo**](https://huggingface.co/spaces/dsfsi/PuoBERTaSpace) to try all PuoBERTa models in your browser:\n- **Fill-Mask**: Predict masked words in Setswana text\n- **News Classification**: Categorize Setswana news articles\n- **Named Entity Recognition (NER)**: Extract entities from text\n- **Part-of-Speech (POS) Tagging**: Identify grammatical roles of words\n\n### Quick Start with Code\n\nGet started with PuoBERTa in just a few lines of code:\n\n```python\nfrom transformers import pipeline\n\n# Use the fill-mask pipeline\nfill_mask = pipeline('fill-mask', model='dsfsi/PuoBERTa')\nresult = fill_mask(\"Setswana ke puo ya \u003cmask\u003e.\")\nprint(result)\n```\n\nFor more detailed examples, check out the [examples directory](examples/) with ready-to-run scripts for various use cases.\n\n---\n\n## Model Details\n\n\n### Model Description\n\nThis is a masked language model trained on Setswana corpora, making it a valuable tool for a range of downstream applications from translation to content creation. It's powered by the PuoData dataset to ensure accuracy and cultural relevance.\n\n- **Developed by:** Vukosi Marivate ([@vukosi](https://huggingface.co/@vukosi)), Moseli Mots'Oehli ([@MoseliMotsoehli](https://huggingface.co/@MoseliMotsoehli)) , Valencia Wagner, Richard Lastrucci and Isheanesu Dzingirai\n- **Model type:** RoBERTa Model\n- **Language(s) (NLP):** Setswana (BCP-47: `tn`)\n- **License:** CC BY 4.0\n- **Training Dataset:** [PuoData](https://github.com/dsfsi/PuoData)\n\n---\n\n## Installation\n\nInstall the required dependencies:\n\n```bash\npip install transformers torch\n```\n\nFor fine-tuning and advanced usage:\n```bash\npip install transformers torch datasets accelerate\n```\n\n---\n\n## Usage Examples\n\n### 1. Masked Language Modeling (Fill-Mask)\n\nUse PuoBERTa to predict masked words in Setswana text:\n\n```python\nfrom transformers import pipeline\n\n# Create a fill-mask pipeline\nfill_mask = pipeline('fill-mask', model='dsfsi/PuoBERTa')\n\n# Predict masked tokens\ntext = \"Setswana ke puo ya \u003cmask\u003e.\"\nresults = fill_mask(text)\n\nfor result in results:\n    print(f\"Token: {result['token_str']}, Score: {result['score']:.4f}\")\n```\n\n### 2. Getting Text Embeddings\n\nExtract contextual embeddings for Setswana text:\n\n```python\nfrom transformers import RobertaTokenizer, RobertaModel\nimport torch\n\n# Load model and tokenizer\ntokenizer = RobertaTokenizer.from_pretrained('dsfsi/PuoBERTa')\nmodel = RobertaModel.from_pretrained('dsfsi/PuoBERTa')\n\n# Encode text\ntext = \"Dumela! Ke a leboga.\"\ninputs = tokenizer(text, return_tensors=\"pt\")\n\n# Get embeddings\nwith torch.no_grad():\n    outputs = model(**inputs)\n    embeddings = outputs.last_hidden_state\n\nprint(f\"Embeddings shape: {embeddings.shape}\")\n```\n\n### 3. Fine-tuning for Text Classification\n\nFine-tune PuoBERTa for downstream tasks like news categorization:\n\n```python\nfrom transformers import RobertaForSequenceClassification, Trainer, TrainingArguments\nfrom transformers import RobertaTokenizer\n\n# Load model for classification (e.g., 10 news categories)\nmodel = RobertaForSequenceClassification.from_pretrained(\n    'dsfsi/PuoBERTa',\n    num_labels=10\n)\ntokenizer = RobertaTokenizer.from_pretrained('dsfsi/PuoBERTa')\n\n# Prepare your dataset\n# train_dataset, eval_dataset = ...\n\n# Define training arguments\ntraining_args = TrainingArguments(\n    output_dir='./results',\n    num_train_epochs=3,\n    per_device_train_batch_size=16,\n    per_device_eval_batch_size=64,\n    warmup_steps=500,\n    weight_decay=0.01,\n    logging_dir='./logs',\n)\n\n# Create trainer\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    eval_dataset=eval_dataset,\n)\n\n# Train the model\ntrainer.train()\n```\n\n### 4. Using Pre-trained Downstream Models\n\nWe provide ready-to-use models for specific tasks:\n\n```python\nfrom transformers import pipeline\n\n# News categorization (10 categories)\nnews_classifier = pipeline('text-classification', model='dsfsi/PuoBERTa-News')\nresult = news_classifier(\"Palamente e ne e kopana gompieno go tlotla melao e mesha.\")\nprint(f\"Category: {result[0]['label']}, Score: {result[0]['score']:.4f}\")\n\n# Named Entity Recognition (PER, LOC, ORG, DATE)\nner = pipeline('ner', model='dsfsi/PuoBERTa-NER', aggregation_strategy=\"simple\")\nentities = ner(\"Vukosi Marivate o tswa kwa University of Pretoria.\")\nfor entity in entities:\n    print(f\"{entity['word']}: {entity['entity_group']} ({entity['score']:.4f})\")\n\n# Part-of-Speech Tagging\npos_tagger = pipeline('token-classification', model='dsfsi/PuoBERTa-POS', aggregation_strategy=\"simple\")\npos_tags = pos_tagger(\"Ke rata go bala dibuka.\")\nfor tag in pos_tags:\n    print(f\"{tag['word']}: {tag['entity_group']}\")\n```\n\n### Downstream Models\n\n- **News Categorization:** [dsfsi/PuoBERTa-News](https://huggingface.co/dsfsi/PuoBERTa-News)\n- **Named Entity Recognition:** [dsfsi/PuoBERTa-NER](https://huggingface.co/dsfsi/PuoBERTa-NER)\n- **Part-of-Speech Tagging:** [dsfsi/PuoBERTa-POS](https://huggingface.co/dsfsi/PuoBERTa-POS)\n\n--- \n\n## Downstream Performance\n\nPuoBERTa has been evaluated on multiple downstream tasks and shows competitive performance against multilingual models while being specifically optimized for Setswana.\n\n### Daily News Dikgang (News Categorization)\n\nPerformance on the Setswana news categorization task using the Daily News Dikgang dataset. Learn more about the dataset in the [Dataset Folder](daily-news-dikgang).\n\n| **Model**                   | **5-fold Cross Validation F1**       | **Test F1**       |\n|-----------------------------|--------------------------------------|-------------------|\n| Logistic Regression + TFIDF | 60.1                                 | 56.2              |\n| NCHLT TSN RoBERTa           | 64.7                                 | 60.3              |\n| PuoBERTa                    | **63.8**                             | **62.9**          |\n| PuoBERTaJW300               | 66.2                                 | 65.4              |\n\n**Pre-trained model:** [dsfsi/PuoBERTa-News](https://huggingface.co/dsfsi/PuoBERTa-News)\n\n### MasakhaPOS (Part-of-Speech Tagging)\n\nPerformance on the MasakhaPOS downstream task for Setswana.\n\n| Model | Test Performance |\n|---|---|\n| **Multilingual Models** |  |\n| AfroLM | 83.8 |\n| AfriBERTa | 82.5 |\n| AfroXLMR-base | 82.7 |\n| AfroXLMR-large | 83.0 |\n| **Monolingual Models** |  |\n| NCHLT TSN RoBERTa | 82.3 |\n| PuoBERTa | **83.4** |\n| PuoBERTa+JW300 | 84.1 |\n\n**Pre-trained model:** [dsfsi/PuoBERTa-POS](https://huggingface.co/dsfsi/PuoBERTa-POS)\n\n### MasakhaNER (Named Entity Recognition)\n\nPerformance on the MasakhaNER downstream task for Setswana.\n\n| Model | Test Performance (f1 score) |\n|---|---|\n| **Multilingual Models** |  |\n| AfriBERTa | 83.2 |\n| AfroXLMR-base | 87.7 |\n| AfroXLMR-large | 89.4 |\n| **Monolingual Models** |  |\n| NCHLT TSN RoBERTa | 74.2 |\n| PuoBERTa | **78.2** |\n| PuoBERTa+JW300 | 80.2 |\n\n**Pre-trained model:** [dsfsi/PuoBERTa-NER](https://huggingface.co/dsfsi/PuoBERTa-NER)\n\n---\n\n## Pre-Training Dataset\n\nPuoBERTa was trained on **PuoData**, a large, curated corpus of Setswana text designed to ensure the model is well-trained and culturally attuned to the language.\n\n**Access the dataset:**\n- [GitHub Repository](https://github.com/dsfsi/PuoData)\n- [HuggingFace Dataset](https://huggingface.co/datasets/dsfsi/PuoData)\n- [Research Paper](https://arxiv.org/abs/2310.09141)\n\nThe dataset includes diverse sources of Setswana text to provide comprehensive language coverage for robust model training.\n\n---\n\n## Citation Information\n\nBibtex Reference\n\n```\n@inproceedings{marivate2023puoberta,\n  title   = {PuoBERTa: Training and evaluation of a curated language model for Setswana},\n  author  = {Vukosi Marivate and Moseli Mots'Oehli and Valencia Wagner and Richard Lastrucci and Isheanesu Dzingirai},\n  year    = {2023},\n  booktitle= {Artificial Intelligence Research. SACAIR 2023. Communications in Computer and Information Science},\n  url= {https://link.springer.com/chapter/10.1007/978-3-031-49002-6_17},\n  keywords = {NLP},\n  preprint_url = {https://arxiv.org/abs/2310.09141},\n  dataset_url = {https://github.com/dsfsi/PuoBERTa},\n  software_url = {https://huggingface.co/dsfsi/PuoBERTa}\n}\n```\n\n## Contributing\n\nWe welcome contributions from the community! Whether you want to:\n- Add new examples or improve documentation\n- Report bugs or suggest features\n- Share your fine-tuned models\n- Contribute datasets or use cases\n\nPlease see our [Contributing Guidelines](CONTRIBUTING.md) for detailed information on how to get started.\n\nQuick links:\n- [Open an issue](https://github.com/dsfsi/PuoBERTa/issues)\n- [Submit a pull request](https://github.com/dsfsi/PuoBERTa/pulls)\n- [Give feedback](https://docs.google.com/forms/d/e/1FAIpQLSf7S36dyAUPx2egmXbFpnTBuzoRulhL5Elu-N1eoMhaO7v10w/formResponse)\n\n## Model Card Authors\n\nVukosi Marivate\n\n## Model Card Contact\n\nFor more details, reach out or check our [website](https://dsfsi.github.io/).\n\nEmail: vukosi.marivate@cs.up.ac.za\n\n**Enjoy exploring Setswana through AI!**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdsfsi%2Fpuoberta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdsfsi%2Fpuoberta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdsfsi%2Fpuoberta/lists"}