{"id":25284648,"url":"https://github.com/cahya-wirawan/indonesian-language-models","last_synced_at":"2025-10-27T19:31:26.189Z","repository":{"id":41288591,"uuid":"145279863","full_name":"cahya-wirawan/indonesian-language-models","owner":"cahya-wirawan","description":"Indonesian Language Models and its Usage","archived":false,"fork":false,"pushed_at":"2023-05-22T11:21:46.000Z","size":63947,"stargazers_count":149,"open_issues_count":1,"forks_count":28,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-04-24T05:11:09.837Z","etag":null,"topics":["deep-learning","fastai","huggingface-transformers","language-model","machine-learning","nlp","pytorch","transformer"],"latest_commit_sha":null,"homepage":"https://cahya-wirawan.github.io/indonesian-language-models","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cahya-wirawan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null}},"created_at":"2018-08-19T06:43:45.000Z","updated_at":"2024-03-31T07:58:35.000Z","dependencies_parsed_at":"2022-09-24T01:20:25.552Z","dependency_job_id":null,"html_url":"https://github.com/cahya-wirawan/indonesian-language-models","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cahya-wirawan%2Findonesian-language-models","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cahya-wirawan%2Findonesian-language-models/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cahya-wirawan%2Findonesian-language-models/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cahya-wirawan%2Findonesian-language-models/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cahya-wirawan","download_url":"https://codeload.github.com/cahya-wirawan/indonesian-language-models/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238550178,"owners_count":19490836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","fastai","huggingface-transformers","language-model","machine-learning","nlp","pytorch","transformer"],"created_at":"2025-02-12T20:52:12.725Z","updated_at":"2025-10-27T19:31:23.203Z","avatar_url":"https://github.com/cahya-wirawan.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Indonesian Language Models\n\nThe language model is a probability distribution over word sequences used to predict the next word based on previous \nsentences. This ability makes the language model the core component of modern natural language processing. We use it \nfor many different tasks, such as speech recognition, conversational AI, information retrieval, sentiment analysis, \nor text summarization.\n\nFor this reason, many big companies are competing to build large and larger language models, such as Google BERT, \nFacebook RoBERTa, or OpenAI GPT3, with its massive number of parameters. Most of the time, they built only \nlanguage models in English and some other European languages. Other countries with low resource languages have big \nchallenges to catch up on this technology race.\n\nTherefore the author tries to build some language models for Indonesian, started with ULMFiT in 2018. The first \nlanguage model has been only trained with  Indonesian Wikipedia, which is very small compared to other datasets used \nto train the English language model.\n\n\n## Universal Language Model Fine-tuning (ULMFiT)\nJeremy Howard and Sebastian Ruder proposed [ULMFiT](https://arxiv.org/abs/1801.06146) in early 2018 as a novel method for \nfine-tuning language models for inductive transfer learning. The language model [ULMFiT for Indonesian](https://github.com/cahya-wirawan/indonesian-language-models/tree/master/ULMFiT) \nhas been trained as part of the author's project while learning [FastAI](https://www.fast.ai). It achieved a perplexity \nof **27.67** on Indonesian Wikipedia.\n\n## Transformers\nAshish Vaswani et al. proposed Transfomer in the paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762). \nIt is a novel architecture that aims to solve sequence-to-sequence tasks while handling long-range dependencies \nwith ease.\n\nAt the time of writing (March 2021), there are already more than 50 different types of transformer-based language \nmodels (according to the model list at huggingface), such as BERT, GPT2, Longformer, or MT5, built by companies and \nindividual contributors. The author built also several \n[Indonesian transformer-based language models](https://github.com/cahya-wirawan/indonesian-language-models/tree/master/Transformers) \nusing [Huggingface Transformers Library](https://github.com/huggingface/transformers) and hosted them in the \n[Huggingfaces model hub](https://huggingface.co/cahya).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcahya-wirawan%2Findonesian-language-models","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcahya-wirawan%2Findonesian-language-models","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcahya-wirawan%2Findonesian-language-models/lists"}