Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/LanguageMachines/ucto

Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --
https://github.com/LanguageMachines/ucto

computational-linguistics folia language natural-language-processing nlp punctuation tokeniser

Last synced: 3 months ago
JSON representation

Host: GitHub
URL: https://github.com/LanguageMachines/ucto
Owner: LanguageMachines
License: gpl-3.0
Created: 2013-03-26T11:16:57.000Z (almost 12 years ago)
Default Branch: master
Last Pushed: 2024-05-17T12:34:58.000Z (9 months ago)
Last Synced: 2024-07-27T07:15:49.687Z (6 months ago)
Topics: computational-linguistics, folia, language, natural-language-processing, nlp, punctuation, tokeniser
Language: C++
Homepage: https://languagemachines.github.io/ucto
Size: 6.04 MB
Stars: 65
Watchers: 13
Forks: 13
Open Issues: 12
Metadata Files:
- Readme: README
- License: COPYING
- Authors: AUTHORS
- Codemeta: codemeta.json

Awesome Lists containing this project

awesome-nlp - ucto - 適用於各種語言的基於 Unicode 的常規表達式標記生成器。工具和 C++函式庫。支持 FoLiA 格式。 (函式庫 / 書籍)

README

Please see README.md for more information