https://github.com/mlverse/tok
Tokenizers from HuggingFace
https://github.com/mlverse/tok
Last synced: about 1 year ago
JSON representation
Tokenizers from HuggingFace
- Host: GitHub
- URL: https://github.com/mlverse/tok
- Owner: mlverse
- License: other
- Created: 2023-04-20T20:25:27.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-10-23T18:10:22.000Z (over 1 year ago)
- Last Synced: 2025-04-09T15:09:37.690Z (about 1 year ago)
- Language: R
- Homepage:
- Size: 136 MB
- Stars: 42
- Watchers: 3
- Forks: 3
- Open Issues: 2
-
Metadata Files:
- Readme: README.Rmd
- Changelog: NEWS.md
- License: LICENSE
Awesome Lists containing this project
README
---
output: github_document
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# tok
[](https://github.com/mlverse/tok/actions)
[](https://CRAN.R-project.org/package=tok)
[](https://cran.r-project.org/package=tok)
tok provides bindings to the [🤗tokenizers](https://huggingface.co/docs/tokenizers/v0.13.3/en/index) library. It uses the same Rust libraries that powers the Python implementation.
We still don't provide the full API of tokenizers. Please open a issue if there's
a feature you are missing.
## Installation
You can install tok from CRAN using:
```
install.packages("tok")
```
Installing tok from source requires working Rust toolchain. We recommend using [rustup.](https://rustup.rs/)
On Windows, you'll also have to add the `i686-pc-windows-gnu` and `x86_64-pc-windows-gnu` targets:
rustup target add x86_64-pc-windows-gnu
rustup target add i686-pc-windows-gnu
Once Rust is working, you can install this package via:
``` r
remotes::install_github("dfalbel/tok")
```
## Features
We still don't have complete support for the 🤗tokenizers API. Please open an issue
if you need a feature that is currently not implemented.
## Loading tokenizers
`tok` can be used to load and use tokenizers that have been previously serialized.
For example, HuggingFace model weights are usually accompanied by a 'tokenizer.json'
file that can be loaded with this library.
To load a pre-trained tokenizer from a json file, use:
```{r}
path <- testthat::test_path("assets/tokenizer.json")
tok <- tok::tokenizer$from_file(path)
```
Use the `encode` method to tokenize sentendes and `decode` to transform them back.
```{r}
enc <- tok$encode("hello world")
tok$decode(enc$ids)
```
## Using pre-trained tokenizers
You can also load any tokenizer available in HuggingFace hub by using the `from_pretrained`
static method. For example, let's load the GPT2 tokenizer with:
```{r}
tok <- tok::tokenizer$from_pretrained("gpt2")
enc <- tok$encode("hello world")
tok$decode(enc$ids)
```