{"id":19573927,"url":"https://github.com/hexresearch/fuzzy-parse","last_synced_at":"2025-10-19T05:13:49.207Z","repository":{"id":62435995,"uuid":"123128136","full_name":"hexresearch/fuzzy-parse","owner":"hexresearch","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-23T20:01:09.000Z","size":111,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-04T23:49:26.457Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hexresearch.png","metadata":{"files":{"readme":"README.markdown","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-27T12:47:18.000Z","updated_at":"2025-01-23T20:01:13.000Z","dependencies_parsed_at":"2024-11-11T06:37:10.851Z","dependency_job_id":"f3b24eed-793a-4b08-80a8-4cfe7fdd286c","html_url":"https://github.com/hexresearch/fuzzy-parse","commit_stats":{"total_commits":38,"total_committers":1,"mean_commits":38.0,"dds":0.0,"last_synced_commit":"94fe623cdf011eaf96a645e95ebc263e6ab3fe6e"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/hexresearch/fuzzy-parse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hexresearch%2Ffuzzy-parse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hexresearch%2Ffuzzy-parse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hexresearch%2Ffuzzy-parse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hexresearch%2Ffuzzy-parse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hexresearch","download_url":"https://codeload.github.com/hexresearch/fuzzy-parse/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hexresearch%2Ffuzzy-parse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279705398,"owners_count":26213613,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-19T02:00:07.647Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T06:37:01.512Z","updated_at":"2025-10-19T05:13:49.154Z","avatar_url":"https://github.com/hexresearch.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\n\n# Data.Text.Fuzzy.Tokenize\n\nThe lightweight and multi-functional text tokenizer allowing different types of text tokenization\ndepending on it's settings.\n\nIt may be used in different sutiations, for DSL, text markups or even for parsing simple grammars\neasier and sometimes faster than in case of usage mainstream parsing combinators or parser\ngenerators.\n\nThe primary goal of this package  is to parse unstructured text data, however it may be  used for\nparsing  such data formats as CSV with ease.\n\nCurrently it supports the following types of entities: atoms, string literals (currently with the\nminimal set of escaped characters), punctuation characters and delimeters.\n\n## Examples\n\n### Simple CSV-like tokenization\n\n```haskell\ntokenize (delims \":\") \"aaa : bebeb : qqq ::::\" :: [Text]\n\n[\"aaa \",\" bebeb \",\" qqq \"]\n```\n\n```haskell\ntokenize (delims \":\"\u003c\u003esq\u003c\u003eemptyFields ) \"aaa : bebeb : qqq ::::\" :: [Text]\n\n[\"aaa \",\" bebeb \",\" qqq \",\"\",\"\",\"\",\"\"]\n```\n\n```haskell\ntokenize (delims \":\"\u003c\u003esq\u003c\u003eemptyFields ) \"aaa : bebeb : qqq ::::\" :: [Maybe Text]\n\n[Just \"aaa \",Just \" bebeb \",Just \" qqq \",Nothing,Nothing,Nothing,Nothing]\n```\n\n```haskell\ntokenize (delims \":\"\u003c\u003esq\u003c\u003eemptyFields ) \"aaa : 'bebeb:colon inside' : qqq ::::\" :: [Maybe Text]\n\n[Just \"aaa \",Just \" \",Just \"bebeb:colon inside\",Just \" \",Just \" qqq \",Nothing,Nothing,Nothing,Nothing]\n```\n\n```haskell\nlet spec = sl\u003c\u003edelims \":\"\u003c\u003esq\u003c\u003eemptyFields\u003c\u003enoslits\ntokenize spec \"   aaa :   'bebeb:colon inside' : qqq ::::\" :: [Maybe Text]\n\n[Just \"aaa \",Just \"bebeb:colon inside \",Just \"qqq \",Nothing,Nothing,Nothing,Nothing]\n```\n\n```haskell\nlet spec = delims \":\"\u003c\u003esq\u003c\u003eemptyFields\u003c\u003euw\u003c\u003enoslits\ntokenize spec \"  a  b  c  : 'bebeb:colon inside' : qqq ::::\"  :: [Maybe Text]\n\n[Just \"a b c\",Just \"bebeb:colon inside\",Just \"qqq\",Nothing,Nothing,Nothing,Nothing]\n```\n\n### Primitive lisp-like language\n\n```haskell\n{-# LANGUAGE QuasiQuotes, ExtendedDefaultRules #-}\n\nimport Text.InterpolatedString.Perl6 (q)\nimport Data.Text.Fuzzy.Tokenize\n\ndata TTok = TChar Char\n          | TSChar Char\n          | TPunct Char\n          | TText Text\n          | TStrLit Text\n          | TKeyword Text\n          | TEmpty\n          deriving(Eq,Ord,Show)\n\ninstance IsToken TTok where\n mkChar = TChar\n mkSChar = TSChar\n mkPunct = TPunct\n mkText = TText\n mkStrLit = TStrLit\n mkKeyword = TKeyword\n mkEmpty = TEmpty\n\nmain = do\n\n   let spec = delims \" \\n\\t\" \u003c\u003e comment \";\"\n                             \u003c\u003e punct \"{}()[]\u003c\u003e\"\n                             \u003c\u003e sq \u003c\u003e sqq\n                             \u003c\u003e uw\n                             \u003c\u003e keywords [\"define\",\"apply\",\"+\"]\n     let code = [q|\n       (define add (a b ) ; define simple function\n         (+ a b) )\n       (define r (add 10 20))\n|]\n     let toks = tokenize spec code :: [TTok]\n\n   print toks\n```\n\n## Notes\n\n### About the delimeter tokens\nThis type of tokens appears during a \"delimited\" formats processing and disappears in results.\nCurrenly you will never see it unless normalization is turned off by 'nn' option.\n\nThe delimeters make sense in case of processing the CSV-like formats, but in this case you probably\nneed only values in results.\n\nThis behavior may be changed later. But right now delimeters seem pointless in results. If you\nprocess some sort of grammar where delimeter character is important, you may use punctuation\ninstead, i.e:\n\n```haskell\nlet spec = delims \" \\t\"\u003c\u003epunct \",;()\" \u003c\u003eemptyFields\u003c\u003esq\ntokenize spec \"( delimeters , are , important, 'spaces are not');\" :: [Text]\n\n[\"(\",\"delimeters\",\",\",\"are\",\",\",\"important\",\",\",\"spaces are not\",\")\",\";\"]\n```\n\n### Other\nFor CSV-like formats it makes sense to split text to lines first, otherwise newline characters may\ncause to weird results\n\n\n# Authors\n\nThis library is written and maintained by Dmitry Zuikov, dzuikov@gmail.com\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhexresearch%2Ffuzzy-parse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhexresearch%2Ffuzzy-parse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhexresearch%2Ffuzzy-parse/lists"}