https://github.com/vforteli/wordpiecetokenizer

WordPiece tokenizer for dotnet (eg with ML.Net)
https://github.com/vforteli/wordpiecetokenizer

bert dotnet llm mlnet tokenizer

Last synced: about 13 hours ago
JSON representation

WordPiece tokenizer for dotnet (eg with ML.Net)

• Host: GitHub
• URL: https://github.com/vforteli/wordpiecetokenizer
• Owner: vforteli
• License: mit
• Created: 2024-04-19T18:43:50.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2024-11-16T14:22:08.000Z (2 days ago)
• Last Synced: 2024-11-16T14:36:11.305Z (2 days ago)
• Topics: bert, dotnet, llm, mlnet, tokenizer
• Language: C#
• Homepage:
• Size: 910 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
WordPieceTokenizer for dotnet