Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
Projects in Awesome Lists tagged with tokenizers
A curated list of projects in awesome lists tagged with tokenizers .
https://github.com/Prismadic/magnet
the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly
apple-silicon claude distributed-computing distributed-systems embeddings fine-tuning finetuning-llms gemini huggingface inference-api langchain llm-training milvus mistral mlx nats nats-messaging nats-streaming sentence-splitting tokenizers
Last synced: 20 Aug 2024
https://github.com/megagonlabs/ginza-transformers
Use custom tokenizers in spacy-transformers
ginza natural-language-processing nlp spacy spacy-transformers sudachitra tokenizers transformers
Last synced: 24 Sep 2024
https://github.com/sappho192/tokenizers.dotnet
[Unofficial] Simple .NET wrapper of HuggingFace Tokenizers library
csharp dotnet huggingface library nuget rust tokenizers
Last synced: 24 Sep 2024
