https://github.com/vforteli/wordpiecetokenizer

WordPiece tokenizer for dotnet (eg with ML.Net)
https://github.com/vforteli/wordpiecetokenizer

bert dotnet llm mlnet tokenizer

Last synced: 4 months ago
JSON representation

WordPiece tokenizer for dotnet (eg with ML.Net)

• Host: GitHub
• URL: https://github.com/vforteli/wordpiecetokenizer
• Owner: vforteli
• License: mit
• Created: 2024-04-19T18:43:50.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-11-17T10:40:09.000Z (11 months ago)
• Last Synced: 2025-05-07T14:06:28.504Z (6 months ago)
• Topics: bert, dotnet, llm, mlnet, tokenizer
• Language: C#
• Homepage:
• Size: 920 KB
• Stars: 1
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

          
WordPieceTokenizer for dotnet