Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

Projects in Awesome Lists tagged with bytepairencoding

A curated list of projects in awesome lists tagged with bytepairencoding .

https://github.com/dbtreasure/zig-bpe

Byte Pair Encoding (BPE) in the Zig programming language (0.13.0)

bytepairencoding tiktoken zig

Last synced: 01 Nov 2024

https://github.com/shivendrra/tokenizers

self made byte-pair-encoding tokenizer

bpe-tokenizer bytepairencoding llm tokenization tokenizer

Last synced: 26 Oct 2024

https://github.com/reshiadavan/thoth

An Industry Standard Tokenizer, purposed for large-scale language models like OpenAI's GPT Series.

bytepairencoding gpt-2 gpt-4 llama2 natural-language-processing python rust sentencepiece tiktoken tokenizer

Last synced: 13 Nov 2024

https://github.com/madhu102938/bpe-cbow

implementation of BPE algorithm and training of the tokens generated

bytepairencoding cbow tokenizer-nlp word2vec

Last synced: 15 Nov 2024