Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jmaczan/bpe-tokenizer
Byte-Pair Encoding tokenizer for training large language models on huge datasets
https://github.com/jmaczan/bpe-tokenizer
bpe bpe-tokenizer byte-pair-encoding chunking deep-learning from-scratch large-language-models llm machine-learning python tokenizer
Last synced: 8 days ago
JSON representation
Byte-Pair Encoding tokenizer for training large language models on huge datasets
- Host: GitHub
- URL: https://github.com/jmaczan/bpe-tokenizer
- Owner: jmaczan
- License: gpl-3.0
- Created: 2024-05-28T09:31:37.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2024-06-04T11:22:32.000Z (5 months ago)
- Last Synced: 2024-06-05T10:34:12.956Z (5 months ago)
- Topics: bpe, bpe-tokenizer, byte-pair-encoding, chunking, deep-learning, from-scratch, large-language-models, llm, machine-learning, python, tokenizer
- Language: Python
- Homepage:
- Size: 108 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# bpe-tokenizer
Byte-Pair Encoding tokenizer for large language models that can be trained on arbitrarily huge datasets
"Materia" by Anita Maczan, Acrylic on canvas, 40x50, 2024
This implementation is suitable for working with huge datasets, because it processes data in chunks, both during tokenization and training
## Training
```py
from bpe_tokenizer import BPETokenizer
BPETokenizer().train(dataset_path="./path/to/dataset.txt")
```
### CLI
```sh
python bpe_tokenizer.py train --training_dataset path_to_your_dataset.txt --vocabulary_size 5000 --training_output path_to_output_tokenizer.json
```
defaults:
- training_dataset = "training.txt"
- vocabulary_size = 50257
- training_output = "tokenizer.json"
## Tokenize
```py
from bpe_tokenizer import BPETokenizer
BPETokenizer().tokenize(text_to_be_tokenized)
```
### CLI
```sh
python bpe_tokenizer.py tokenize --tokenizer_data path_to_tokenizer_data.json --run_data tokenize.json
```
defaults:
- tokenizer_data = "tokenizer.json"
- run_data = "tokenize.txt"
run_data file structure:
```json
{
"data": "Study hard what interests you the most in the most undisciplined, irreverent and original manner possible - Richard Feynmann"
}
```
## Detokenize
```py
from bpe_tokenizer import BPETokenizer
BPETokenizer().detokenize(array_of_tokens_to_be_parsed_to_text)
```
### CLI
```sh
python bpe_tokenizer.py detokenize --tokenizer_data path_to_tokenizer_data.json --run_data detokenize.json
```
defaults:
- tokenizer_data = "tokenizer.json"
- run_data = "detokenize.txt"
run_data file structure:
```json
{
"data": [
275, 265, 282, 32, 111, 108, 100, 32, 98, 97, 115, 116, 97, 114, 100, 284,
268, 265, 282, 32, 282, 110, 292, 98, 97, 115, 116, 97, 114, 100, 33
]
}
```
## License
GPL v3
Jędrzej Maczan, 2024