https://github.com/haskell/tokenize

Simple tokenizer for English text
https://github.com/haskell/tokenize

Last synced: 5 months ago
JSON representation

Simple tokenizer for English text

• Host: GitHub
• URL: https://github.com/haskell/tokenize
• Owner: haskell
• License: other
• Created: 2024-04-08T17:21:01.000Z (almost 2 years ago)
• Default Branch: master
• Last Pushed: 2025-05-11T05:52:19.000Z (11 months ago)
• Last Synced: 2025-08-18T21:28:01.213Z (7 months ago)
• Language: Haskell
• Homepage: https://hackage.haskell.org/package/tokenize
• Size: 18.6 KB
• Stars: 5
• Watchers: 7
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

          
[![Hackage version](https://img.shields.io/hackage/v/tokenize.svg?label=Hackage&color=informational)](http://hackage.haskell.org/package/tokenize)

[![tokenize on Stackage Nightly](https://stackage.org/package/tokenize/badge/nightly)](https://stackage.org/nightly/package/tokenize)

[![Stackage LTS version](https://www.stackage.org/package/tokenize/badge/lts?label=Stackage)](https://www.stackage.org/package/tokenize)

[![Cabal build](https://github.com/haskell/tokenize/workflows/Haskell-CI/badge.svg)](https://github.com/haskell/tokenize/actions)

The `tokenize` Package

======================

Simple tokenizer for English text.

Used by `hackage-server`.

See [`tokenize` on Hackage](http://hackage.haskell.org/package/tokenize) for more information.