https://github.com/jiauzhang/textok

Text Tokenizer in C++
https://github.com/jiauzhang/textok

byte-pair-encoding language-model llm nlp tokenizer

Last synced: 4 months ago
JSON representation

Text Tokenizer in C++

• Host: GitHub
• URL: https://github.com/jiauzhang/textok
• Owner: JiauZhang
• License: gpl-2.0
• Created: 2025-01-10T20:12:10.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-01-10T20:49:47.000Z (over 1 year ago)
• Last Synced: 2025-02-26T09:17:41.626Z (over 1 year ago)
• Topics: byte-pair-encoding, language-model, llm, nlp, tokenizer
• Language: Python
• Homepage:
• Size: 14.6 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

README

          
### Install

```shell

pip install textok

```

### Sponsor

    

        

            公众号

        

    

    

        

            

        

    

    

        

            AliPay

            WeChatPay