https://github.com/sanjibnarzary/bodo-tokenizers

Pre tokenized models for Bodo. This repositoryincludes all the tokenized models to be used in the Neural Machine Translation. The models include pre tokenized models trained using ByteLevelBPETokenizer, BPETokenizer, SentencePieceBPETokenizer, BertWordPieceTokenizer

bodo indian-language less-resource-languages natural-language-processing nlp nlp-bodo tokenizer

Last synced: 01 Mar 2025