{"id":23036886,"url":"https://github.com/clitetailor/tokenize-monster","last_synced_at":"2025-04-02T23:17:24.483Z","repository":{"id":57377768,"uuid":"90770208","full_name":"clitetailor/tokenize-monster","owner":"clitetailor","description":"Yet another powerful tokenizer in js.","archived":false,"fork":false,"pushed_at":"2017-06-30T11:11:54.000Z","size":103,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-15T03:33:23.913Z","etag":null,"topics":["functional-programming","tokenize","tokenizer"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clitetailor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-09T16:54:57.000Z","updated_at":"2017-06-25T05:03:03.000Z","dependencies_parsed_at":"2022-09-19T03:31:04.784Z","dependency_job_id":null,"html_url":"https://github.com/clitetailor/tokenize-monster","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clitetailor%2Ftokenize-monster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clitetailor%2Ftokenize-monster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clitetailor%2Ftokenize-monster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clitetailor%2Ftokenize-monster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clitetailor","download_url":"https://codeload.github.com/clitetailor/tokenize-monster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246905868,"owners_count":20852820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["functional-programming","tokenize","tokenizer"],"created_at":"2024-12-15T17:28:01.866Z","updated_at":"2025-04-02T23:17:24.465Z","avatar_url":"https://github.com/clitetailor.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"Tokenize Monster\r\n================\r\n\r\n\u003e Yet another powerful tokenizer in **JS**.\r\n\r\nInstallation\r\n------------\r\n\r\n```bash\r\nnpm install --save tokenize-monster\r\n```\r\n\r\nGetting Started\r\n---------------\r\n\r\n```javascript\r\nconst tokenizer = require('tokenize-monster');\r\n\r\nconst tokens = tokenizer.matchAndMap(\r\n\t'i\\'d like to \u003cdrink\u003e some \u003cmilk\u003e',\r\n\t/\u003c[^\u003e\u003c]+\u003e/,\r\n\ttext =\u003e [[text]],\r\n\ttext =\u003e [text]\r\n);\r\n\r\nconsole.log(tokens);\r\n// =\u003e [ 'i\\'d like to ', [ '\u003cdrink\u003e' ], ' some ', [ '\u003cmilk\u003e' ], '' ]\r\n\r\nconst nTokens = tokenizer.matchAndMap(\r\n\t'i\\'d like to \u003cdrink\u003e some \u003cmilk\u003e',\r\n\t/\u003c[^\u003e\u003c]+\u003e/,\r\n\ttext =\u003e [text],\r\n\ttext =\u003e []\r\n);\r\n\r\nconsole.log(nTokens);\r\n// =\u003e [ '\u003cdrink\u003e', '\u003cmilk\u003e' ]\r\n```\r\n\r\nExample\r\n-------\r\n\r\n- [Simple XML Parser](https://github.com/clitetailor/simple-xml-parser)\r\n\r\nDocumentation\r\n-------------\r\n\r\nOur docs are available at [Tokenize Monster Wiki](https://github.com/clitetailor/tokenize-monster/wiki).\r\n\r\nChangelog\r\n---------\r\n\r\nVisit [Github Releases](https://github.com/clitetailor/tokenize-monster/releases) page for more information.\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclitetailor%2Ftokenize-monster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclitetailor%2Ftokenize-monster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclitetailor%2Ftokenize-monster/lists"}