https://github.com/languagemachines/ucto

Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --
https://github.com/languagemachines/ucto

computational-linguistics folia language natural-language-processing nlp punctuation tokeniser

Last synced: 7 months ago
JSON representation

Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --

• Host: GitHub
• URL: https://github.com/languagemachines/ucto
• Owner: LanguageMachines
• License: gpl-3.0
• Created: 2013-03-26T11:16:57.000Z (over 12 years ago)
• Default Branch: master
• Last Pushed: 2025-02-08T13:38:25.000Z (8 months ago)
• Last Synced: 2025-03-30T00:05:54.162Z (7 months ago)
• Topics: computational-linguistics, folia, language, natural-language-processing, nlp, punctuation, tokeniser
• Language: C++
• Homepage: https://languagemachines.github.io/ucto
• Size: 6.17 MB
• Stars: 68
• Watchers: 12
• Forks: 14
• Open Issues: 12
• Metadata Files:
  • Readme: README
  • Changelog: NEWS
  • License: COPYING
  • Authors: AUTHORS
  • Codemeta: codemeta.json

Awesome Lists containing this project

README

          
Please see README.md for more information