https://github.com/yuniko-software/qwen3-tokenizer

Multi-language BPE tokenizer implementation for Qwen3 models. Lightweight byte-pair encoding for C#/.NET, Java, Rust
https://github.com/yuniko-software/qwen3-tokenizer

bpe-tokenizer csharp dotnet embedding-models huggingface inference java machine-learning onnx qwen rust vector-database

Last synced: 8 months ago
JSON representation

Multi-language BPE tokenizer implementation for Qwen3 models. Lightweight byte-pair encoding for C#/.NET, Java, Rust

Host: GitHub
URL: https://github.com/yuniko-software/qwen3-tokenizer
Owner: yuniko-software
License: apache-2.0
Created: 2025-10-23T20:28:46.000Z (8 months ago)
Default Branch: main
Last Pushed: 2025-10-30T22:58:33.000Z (8 months ago)
Last Synced: 2025-10-30T23:21:04.347Z (8 months ago)
Topics: bpe-tokenizer, csharp, dotnet, embedding-models, huggingface, inference, java, machine-learning, onnx, qwen, rust, vector-database
Language: C#
Homepage:
Size: 90.8 KB
Stars: 0
Watchers: 0
Forks: 0
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE

README

# Qwen3 Tokenizer

Multi-language tokenizer implementations for Qwen3 models.

## Status

⚠️ **This project is currently in progress and not intended for production use.**

## Languages

- **C# / .NET** - Qwen3 tokenizer implementation

## License

See [LICENSE](LICENSE) file for details.