https://github.com/infinitensor/tokeneer

another tokenizer crate
https://github.com/infinitensor/tokeneer

Last synced: 9 months ago
JSON representation

another tokenizer crate

• Host: GitHub
• URL: https://github.com/infinitensor/tokeneer
• Owner: InfiniTensor
• License: mit
• Created: 2024-08-06T03:47:31.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2024-09-19T02:21:33.000Z (almost 2 years ago)
• Last Synced: 2024-12-30T16:19:25.290Z (over 1 year ago)
• Language: Rust
• Size: 40 KB
• Stars: 1
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Tokeneer

[![CI](https://github.com/InfiniTensor/tokeneer/actions/workflows/build.yml/badge.svg?branch=main)](https://github.com/InfiniTensor/tokeneer/actions)

[![Latest version](https://img.shields.io/crates/v/tokeneer.svg)](https://crates.io/crates/tokeneer)

[![Documentation](https://docs.rs/tokeneer/badge.svg)](https://docs.rs/tokeneer)

[![license](https://img.shields.io/github/license/InfiniTensor/tokeneer)](https://mit-license.org/)

![GitHub repo size](https://img.shields.io/github/repo-size/InfiniTensor/tokeneer)

![GitHub code size in bytes](https://img.shields.io/github/languages/code-size/InfiniTensor/tokeneer)

[![GitHub Issues](https://img.shields.io/github/issues/InfiniTensor/tokeneer)](https://github.com/InfiniTensor/tokeneer/issues)

[![GitHub Pull Requests](https://img.shields.io/github/issues-pr/InfiniTensor/tokeneer)](https://github.com/InfiniTensor/tokeneer/pulls)

![GitHub contributors](https://img.shields.io/github/contributors/InfiniTensor/tokeneer)

![GitHub commit activity](https://img.shields.io/github/commit-activity/m/InfiniTensor/tokeneer)

Another tokenizer crate.