https://github.com/wso2/analytics-solutions
https://github.com/wso2/analytics-solutions
common
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/wso2/analytics-solutions
- Owner: wso2
- License: apache-2.0
- Created: 2015-09-03T04:03:13.000Z (almost 11 years ago)
- Default Branch: master
- Last Pushed: 2023-04-16T18:08:25.000Z (about 3 years ago)
- Last Synced: 2025-03-29T02:06:16.117Z (about 1 year ago)
- Topics: common
- Language: JavaScript
- Size: 7.38 MB
- Stars: 42
- Watchers: 136
- Forks: 60
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# siddhi-execution-tokenizer
## Entity
### tokenize *(Stream Processor)*
This splits a string into words
Syntax
```
entity:tokenize( entity.element)
```
QUERY PARAMETERS
Name
Description
Default Value
Possible Data Types
Optional
Dynamic
entity.element
The input entity element which should be split.
STRING
No
No
Examples
EXAMPLE 1
```
define stream inputStream (hashtags string);
@info(name = 'query1')
from inputStream#entity:tokenize(hashtags)
select hashtag
insert into outputStream;
```
This query performs tokenization for the given entity element.
## Text
### tokenize *(Stream Processor)*
This splits a string into words
Syntax
```
text:tokenize( text)
```
QUERY PARAMETERS
Name
Description
Default Value
Possible Data Types
Optional
Dynamic
text
The input text which should be split.
STRING
No
No
Examples
EXAMPLE 1
```
define stream inputStream (text string);
@info(name = 'query1')
from inputStream#text:tokenize(text)
select text
insert into outputStream;
```
This query performs tokenization for the given string.