https://github.com/xp-framework/tokenize

Tokenize text
https://github.com/xp-framework/tokenize

php stream tokenizer xp-framework

Last synced: 6 months ago
JSON representation

Tokenize text

• Host: GitHub
• URL: https://github.com/xp-framework/tokenize
• Owner: xp-framework
• Created: 2015-12-08T22:02:29.000Z (about 10 years ago)
• Default Branch: master
• Last Pushed: 2024-10-05T11:13:28.000Z (about 1 year ago)
• Last Synced: 2025-06-26T14:52:54.419Z (6 months ago)
• Topics: php, stream, tokenizer, xp-framework
• Language: PHP
• Size: 36.1 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: ChangeLog.md

Awesome Lists containing this project

README

          
Tokenize

========

[![Build status on GitHub](https://github.com/xp-framework/tokenize/workflows/Tests/badge.svg)](https://github.com/xp-framework/tokenize/actions)

[![XP Framework Module](https://raw.githubusercontent.com/xp-framework/web/master/static/xp-framework-badge.png)](https://github.com/xp-framework/core)

[![BSD Licence](https://raw.githubusercontent.com/xp-framework/web/master/static/licence-bsd.png)](https://github.com/xp-framework/core/blob/master/LICENCE.md)

[![Requires PHP 7.0+](https://raw.githubusercontent.com/xp-framework/web/master/static/php-7_0plus.svg)](http://php.net/)

[![Supports PHP 8.0+](https://raw.githubusercontent.com/xp-framework/web/master/static/php-8_0plus.svg)](http://php.net/)

[![Latest Stable Version](https://poser.pugx.org/xp-framework/tokenize/version.svg)](https://packagist.org/packages/xp-framework/tokenize)

Tokenizing text

```php

use text\{StringTokenizer, StreamTokenizer};

use io\File;

// Supports strings and streams

$tokens= new StringTokenizer('He asked: Can you parse this?', ' .?!,;:', true);

$tokens= new StreamTokenizer((new File('parse-me.txt'))->in(), ' .?!,;:', true);

// Can iterate using foreach...

foreach ($tokens as $token) {

  Console::writeLine($token);

}

// ...or with an iterator API

while ($tokens->hasMoreTokens()) {

  Console::writeLine($tokens->nextToken());

}

// Returns: ["He", " ", "asked", ":", " ", "Can", " ", "you", " ", "parse", " ", "this", "?"]

```