https://github.com/meniny/tokenizer
🧜🏻♂️ A tiny human language tokenizer framework in Swift.
https://github.com/meniny/tokenizer
cocoa ios swift tokenizer
Last synced: about 1 year ago
JSON representation
🧜🏻♂️ A tiny human language tokenizer framework in Swift.
- Host: GitHub
- URL: https://github.com/meniny/tokenizer
- Owner: Meniny
- License: mit
- Created: 2017-12-27T07:07:52.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2019-04-06T06:52:47.000Z (about 7 years ago)
- Last Synced: 2025-05-18T12:39:00.277Z (about 1 year ago)
- Topics: cocoa, ios, swift, tokenizer
- Language: Swift
- Homepage: https://meniny.cn/
- Size: 181 KB
- Stars: 5
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Introduction
## What's this?
`Tokenizer` is a tiny human language tokenizer framework in Swift..
## Preview
## Installation
#### CocoaPods
```ruby
pod 'Tokenizer'
```
## Contribution
You are welcome to fork and submit pull requests.
## License
`Tokenizer` is open-sourced software, licensed under the `MIT` license.
## Usage
```swift
import Tokenizer
let stackoverflow = "Stack Overflow is the largest, most trusted online community for developers to learn, share their programming knowledge, and build their careers."
let baidu = "全球最大的中文搜索引擎、最大的中文网站。1999年底,身在美国硅谷的李彦宏看到了中国互联网及中文搜索引擎服务的巨大发展潜力，抱着技术改变世界的梦想，他毅然辞掉硅谷的高薪工作，携搜索引擎专利技术，于 2000年1月1日在中关村创建了百度公司。"
dump(stackoverflow.tokenize())
dump(Tokenizer.tokenize(text: baidu))
```