{"id":18966033,"url":"https://github.com/wso2/analytics-solutions","last_synced_at":"2025-04-05T03:10:17.728Z","repository":{"id":4881610,"uuid":"41841233","full_name":"wso2/analytics-solutions","owner":"wso2","description":null,"archived":false,"fork":false,"pushed_at":"2023-04-16T18:08:25.000Z","size":7738,"stargazers_count":42,"open_issues_count":15,"forks_count":60,"subscribers_count":136,"default_branch":"master","last_synced_at":"2025-03-29T02:06:16.117Z","etag":null,"topics":["common"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wso2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-09-03T04:03:13.000Z","updated_at":"2024-11-27T12:33:15.000Z","dependencies_parsed_at":"2023-07-06T01:54:25.808Z","dependency_job_id":null,"html_url":"https://github.com/wso2/analytics-solutions","commit_stats":null,"previous_names":[],"tags_count":65,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wso2%2Fanalytics-solutions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wso2%2Fanalytics-solutions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wso2%2Fanalytics-solutions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wso2%2Fanalytics-solutions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wso2","download_url":"https://codeload.github.com/wso2/analytics-solutions/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247280272,"owners_count":20912967,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["common"],"created_at":"2024-11-08T14:35:11.804Z","updated_at":"2025-04-05T03:10:17.711Z","avatar_url":"https://github.com/wso2.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# siddhi-execution-tokenizer\n\n## Entity\n\n### tokenize *\u003ca target=\"_blank\" href=\"https://wso2.github.io/siddhi/documentation/siddhi-4.0/#stream-processor\"\u003e(Stream Processor)\u003c/a\u003e*\n\n\u003cp style=\"word-wrap: break-word\"\u003eThis splits a string into words\u003c/p\u003e\n\n\u003cspan id=\"syntax\" class=\"md-typeset\" style=\"display: block; font-weight: bold;\"\u003eSyntax\u003c/span\u003e\n```\nentity:tokenize(\u003cSTRING\u003e entity.element)\n```\n\n\u003cspan id=\"query-parameters\" class=\"md-typeset\" style=\"display: block; color: rgba(0, 0, 0, 0.54); font-size: 12.8px; font-weight: bold;\"\u003eQUERY PARAMETERS\u003c/span\u003e\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003cth\u003eName\u003c/th\u003e\n        \u003cth style=\"min-width: 20em\"\u003eDescription\u003c/th\u003e\n        \u003cth\u003eDefault Value\u003c/th\u003e\n        \u003cth\u003ePossible Data Types\u003c/th\u003e\n        \u003cth\u003eOptional\u003c/th\u003e\n        \u003cth\u003eDynamic\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd style=\"vertical-align: top\"\u003eentity.element\u003c/td\u003e\n        \u003ctd style=\"vertical-align: top; word-wrap: break-word\"\u003eThe input entity element which should be split.\u003c/td\u003e\n        \u003ctd style=\"vertical-align: top\"\u003e\u003c/td\u003e\n        \u003ctd style=\"vertical-align: top\"\u003eSTRING\u003c/td\u003e\n        \u003ctd style=\"vertical-align: top\"\u003eNo\u003c/td\u003e\n        \u003ctd style=\"vertical-align: top\"\u003eNo\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003cspan id=\"examples\" class=\"md-typeset\" style=\"display: block; font-weight: bold;\"\u003eExamples\u003c/span\u003e\n\u003cspan id=\"example-1\" class=\"md-typeset\" style=\"display: block; color: rgba(0, 0, 0, 0.54); font-size: 12.8px; font-weight: bold;\"\u003eEXAMPLE 1\u003c/span\u003e\n```\ndefine stream inputStream (hashtags string);\n@info(name = 'query1')\nfrom inputStream#entity:tokenize(hashtags)\nselect hashtag\ninsert into outputStream;\n```\n\u003cp style=\"word-wrap: break-word\"\u003eThis query performs tokenization for the given entity element.\u003c/p\u003e\n\n## Text\n\n### tokenize *\u003ca target=\"_blank\" href=\"https://wso2.github.io/siddhi/documentation/siddhi-4.0/#stream-processor\"\u003e(Stream Processor)\u003c/a\u003e*\n\n\u003cp style=\"word-wrap: break-word\"\u003eThis splits a string into words\u003c/p\u003e\n\n\u003cspan id=\"syntax\" class=\"md-typeset\" style=\"display: block; font-weight: bold;\"\u003eSyntax\u003c/span\u003e\n```\ntext:tokenize(\u003cSTRING\u003e text)\n```\n\n\u003cspan id=\"query-parameters\" class=\"md-typeset\" style=\"display: block; color: rgba(0, 0, 0, 0.54); font-size: 12.8px; font-weight: bold;\"\u003eQUERY PARAMETERS\u003c/span\u003e\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003cth\u003eName\u003c/th\u003e\n        \u003cth style=\"min-width: 20em\"\u003eDescription\u003c/th\u003e\n        \u003cth\u003eDefault Value\u003c/th\u003e\n        \u003cth\u003ePossible Data Types\u003c/th\u003e\n        \u003cth\u003eOptional\u003c/th\u003e\n        \u003cth\u003eDynamic\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd style=\"vertical-align: top\"\u003etext\u003c/td\u003e\n        \u003ctd style=\"vertical-align: top; word-wrap: break-word\"\u003eThe input text which should be split.\u003c/td\u003e\n        \u003ctd style=\"vertical-align: top\"\u003e\u003c/td\u003e\n        \u003ctd style=\"vertical-align: top\"\u003eSTRING\u003c/td\u003e\n        \u003ctd style=\"vertical-align: top\"\u003eNo\u003c/td\u003e\n        \u003ctd style=\"vertical-align: top\"\u003eNo\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003cspan id=\"examples\" class=\"md-typeset\" style=\"display: block; font-weight: bold;\"\u003eExamples\u003c/span\u003e\n\u003cspan id=\"example-1\" class=\"md-typeset\" style=\"display: block; color: rgba(0, 0, 0, 0.54); font-size: 12.8px; font-weight: bold;\"\u003eEXAMPLE 1\u003c/span\u003e\n```\ndefine stream inputStream (text string);\n@info(name = 'query1')\nfrom inputStream#text:tokenize(text)\nselect text\ninsert into outputStream;\n```\n\u003cp style=\"word-wrap: break-word\"\u003eThis query performs tokenization for the given string.\u003c/p\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwso2%2Fanalytics-solutions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwso2%2Fanalytics-solutions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwso2%2Fanalytics-solutions/lists"}