{"id":21638114,"url":"https://github.com/purecloudlabs/roberta-tokenizer","last_synced_at":"2025-04-11T16:42:32.797Z","repository":{"id":65507453,"uuid":"560857333","full_name":"purecloudlabs/roberta-tokenizer","owner":"purecloudlabs","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-03T06:34:24.000Z","size":93,"stargazers_count":18,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-07T09:12:17.066Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/purecloudlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-02T12:28:42.000Z","updated_at":"2025-04-03T20:47:02.000Z","dependencies_parsed_at":"2025-01-03T07:26:39.091Z","dependency_job_id":"38dd6641-c472-4642-9572-feaa05692a33","html_url":"https://github.com/purecloudlabs/roberta-tokenizer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purecloudlabs%2Froberta-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purecloudlabs%2Froberta-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purecloudlabs%2Froberta-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purecloudlabs%2Froberta-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/purecloudlabs","download_url":"https://codeload.github.com/purecloudlabs/roberta-tokenizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248442326,"owners_count":21104158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T04:08:14.441Z","updated_at":"2025-04-11T16:42:32.776Z","avatar_url":"https://github.com/purecloudlabs.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RoBERTa Java Tokenizer #\n\n\n## About\n\n---\nThis repo contains a Java tokenizer used by RoBERTa model. The implementation is mainly according to HuggingFace Python\nRoBERTa Tokenizer, but also we took references from other implementations as mentioned in the code and below:\n\n* https://huggingface.co/docs/transformers/model_doc/roberta#transformers.RobertaTokenizer\n\n* https://github.com/huggingface/tflite-android-transformers/blob/master/gpt2/src/main/java/co/huggingface/android_transformers/gpt2/tokenization/GPT2Tokenizer.kt\n\n* https://github.com/hyunwoongko/gpt2-tokenizer-java/blob/master/src/main/java/ai/tunib/tokenizer/GPT2Tokenizer.java\n\nThe algorithm used is a byte-level Byte Pair Encoding.\n\nhttps://huggingface.co/docs/transformers/tokenizer_summary#bytelevel-bpe\n## How do I get set up? ###\n\n---\n\n* Clone the repo for explicit usage.\n* Add the Maven dependency to your `pom.xml` for usage in your project:\n\n```\n\u003cdependency\u003e\n    \u003cgroupId\u003ecloud.genesys\u003c/groupId\u003e\n    \u003cartifactId\u003eroberta-tokenizer\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.7\u003c/version\u003e\n\u003c/dependency\u003e\n\n\u003cdistributionManagement\u003e\n    \u003crepository\u003e\n      \u003cid\u003eossrh\u003c/id\u003e\n      \u003curl\u003ehttps://s01.oss.sonatype.org/service/local/staging/deploy/maven2/\u003c/url\u003e\n    \u003c/repository\u003e\n    ...\n\u003c/distributionManagement\u003e\n```\n\n\n### Tests ###\n\n---\n\n* Unit tests - Run on local machine.\n\n### File Dependencies ###\n\n---\n\nSince we want efficiency when initializing the tokenizer, we use a factory to create the relevant resources\nfiles and create it \"lazily\".\n\nFor this tokenizer we need 3 data files:\n\n* `base_vocabulary.json` -  map of numbers ([0,255]) to symbols (UniCode Characters). Only those symbols will be known by the\n  algorithm. e.g., given _s_ as input it iterates over the bytes of the String _s_ and replaces each given byte with the mapped symbol.\n  This way we assure what characters are passed.\n\n* `vocabulary.json` - Is a file that holds all the words(sub-words) and their token according to training.\n\n* `merges.txt` - describes the merge rules of words. The algorithm splits the given word into two subwords, afterwards\n  it decides the best split according to the rank of the sub words. The higher those words are, the higher the rank.\n\n__Please note__:\n\n1. All three files must be under the same directory.\n\n2. They must be named like mentioned above.\n\n3. The result of the tokenization depends on the vocabulary and merges files.\n\n### Example ###\n\n---\n\n```\n\nString baseDirPath = \"base/dir/path\";\nRobertaTokenizerResources robertaResources = new RobertaTokenizerResources(baseDirPath);\nTokenizer robertaTokenizer = new RobertaTokenizer(robertaResources);\n...\nString sentence = \"this must be the place\";\nlong[] tokenizedSentence = robertaTokenizer.tokenize(sentence);\nSystem.out.println(tokenizedSentence);\n\n```\n\nAn example output would be: `[0, 9226, 531, 28, 5, 317, 2]` - Depends on the given vocabulary and merges files.\n\n### Contribution guidelines\n\n---\n\n* Use temporary branches for every issue/task.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpurecloudlabs%2Froberta-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpurecloudlabs%2Froberta-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpurecloudlabs%2Froberta-tokenizer/lists"}