{"id":18674046,"url":"https://github.com/jmaczan/bpe-tokenizer","last_synced_at":"2025-09-18T23:44:59.213Z","repository":{"id":241610874,"uuid":"806991682","full_name":"jmaczan/bpe-tokenizer","owner":"jmaczan","description":"Byte-Pair Encoding tokenizer for training large language models on huge datasets","archived":false,"fork":false,"pushed_at":"2024-06-04T11:22:32.000Z","size":111,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-30T22:32:02.944Z","etag":null,"topics":["bpe","bpe-tokenizer","byte-pair-encoding","chunking","deep-learning","from-scratch","large-language-models","llm","machine-learning","python","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmaczan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-05-28T09:31:37.000Z","updated_at":"2024-11-11T04:10:42.000Z","dependencies_parsed_at":"2025-08-30T22:31:14.715Z","dependency_job_id":"f1daa3ce-0171-4528-ae28-b835e687fa2d","html_url":"https://github.com/jmaczan/bpe-tokenizer","commit_stats":null,"previous_names":["jmaczan/bpe-tokenizer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jmaczan/bpe-tokenizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmaczan%2Fbpe-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmaczan%2Fbpe-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmaczan%2Fbpe-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmaczan%2Fbpe-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmaczan","download_url":"https://codeload.github.com/jmaczan/bpe-tokenizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmaczan%2Fbpe-tokenizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275852425,"owners_count":25540137,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-18T02:00:09.552Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bpe","bpe-tokenizer","byte-pair-encoding","chunking","deep-learning","from-scratch","large-language-models","llm","machine-learning","python","tokenizer"],"created_at":"2024-11-07T09:17:20.219Z","updated_at":"2025-09-18T23:44:59.144Z","avatar_url":"https://github.com/jmaczan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bpe-tokenizer\n\nByte-Pair Encoding tokenizer for large language models that can be trained on arbitrarily huge datasets\n\n\u003cfigure\u003e\n\u003cdiv align=\"center\"\u003e\n\u003ca href=\"https://anitamaczan.pl/#materia\" target=\"_blank\"\u003e\n\u003cimg src=\"https://anitamaczan.pl/materia.jpg\" width=\"250\" alt=\"'Materia' by Anita Maczan, Acrylic on canvas, 40x50, 2024\"\u003e\n\u003c/a\u003e\n\u003c/div\u003e\n\u003cfigcaption\u003e\u003cdiv align=\"center\" style=\"font-family: monospace; font-size: 0.75rem\"\u003e\"Materia\" by Anita Maczan, Acrylic on canvas, 40x50, 2024\u003c/div\u003e\u003c/figcaption\u003e\n\u003c/p\u003e\n\u003c/figure\u003e\n\nThis implementation is suitable for working with huge datasets, because it processes data in chunks, both during tokenization and training\n\n## Training\n\n```py\nfrom bpe_tokenizer import BPETokenizer\n\nBPETokenizer().train(dataset_path=\"./path/to/dataset.txt\")\n```\n\n### CLI\n\n```sh\npython bpe_tokenizer.py train --training_dataset path_to_your_dataset.txt --vocabulary_size 5000 --training_output path_to_output_tokenizer.json\n```\n\ndefaults:\n\n- training_dataset = \"training.txt\"\n- vocabulary_size = 50257\n- training_output = \"tokenizer.json\"\n\n## Tokenize\n\n```py\nfrom bpe_tokenizer import BPETokenizer\n\nBPETokenizer().tokenize(text_to_be_tokenized)\n```\n\n### CLI\n\n```sh\npython bpe_tokenizer.py tokenize --tokenizer_data path_to_tokenizer_data.json --run_data tokenize.json\n```\n\ndefaults:\n\n- tokenizer_data = \"tokenizer.json\"\n- run_data = \"tokenize.txt\"\n\nrun_data file structure:\n\n```json\n{\n  \"data\": \"Study hard what interests you the most in the most undisciplined, irreverent and original manner possible - Richard Feynmann\"\n}\n```\n\n## Detokenize\n\n```py\nfrom bpe_tokenizer import BPETokenizer\n\nBPETokenizer().detokenize(array_of_tokens_to_be_parsed_to_text)\n```\n\n### CLI\n\n```sh\npython bpe_tokenizer.py detokenize --tokenizer_data path_to_tokenizer_data.json --run_data detokenize.json\n```\n\ndefaults:\n\n- tokenizer_data = \"tokenizer.json\"\n- run_data = \"detokenize.txt\"\n\nrun_data file structure:\n\n```json\n{\n  \"data\": [\n    275, 265, 282, 32, 111, 108, 100, 32, 98, 97, 115, 116, 97, 114, 100, 284,\n    268, 265, 282, 32, 282, 110, 292, 98, 97, 115, 116, 97, 114, 100, 33\n  ]\n}\n```\n\n## License\n\nGPL v3\n\nJędrzej Maczan, 2024\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmaczan%2Fbpe-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmaczan%2Fbpe-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmaczan%2Fbpe-tokenizer/lists"}