https://github.com/beomi/megatronlm_dataset_autotokenizer
Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.
https://github.com/beomi/megatronlm_dataset_autotokenizer
gpt-neox megatron-lm tokenizers transformers
Last synced: 5 months ago
JSON representation
Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.
- Host: GitHub
- URL: https://github.com/beomi/megatronlm_dataset_autotokenizer
- Owner: Beomi
- License: apache-2.0
- Created: 2023-11-16T07:23:11.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-11-16T13:54:45.000Z (almost 2 years ago)
- Last Synced: 2025-05-07T06:55:23.217Z (5 months ago)
- Topics: gpt-neox, megatron-lm, tokenizers, transformers
- Language: Python
- Homepage:
- Size: 498 KB
- Stars: 6
- Watchers: 2
- Forks: 1
- Open Issues: 0