{"id":2044,"url":"https://github.com/mathewsanders/Mustard","last_synced_at":"2025-08-02T23:31:19.805Z","repository":{"id":84489434,"uuid":"77703960","full_name":"mathewsanders/Mustard","owner":"mathewsanders","description":"🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.","archived":false,"fork":false,"pushed_at":"2018-05-14T20:24:17.000Z","size":140,"stargazers_count":689,"open_issues_count":0,"forks_count":18,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-10-30T15:27:49.797Z","etag":null,"topics":["substrings","swift","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"Swift","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mathewsanders.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-12-30T18:42:45.000Z","updated_at":"2024-10-22T12:42:05.000Z","dependencies_parsed_at":"2023-03-02T03:45:33.215Z","dependency_job_id":null,"html_url":"https://github.com/mathewsanders/Mustard","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mathewsanders%2FMustard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mathewsanders%2FMustard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mathewsanders%2FMustard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mathewsanders%2FMustard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mathewsanders","download_url":"https://codeload.github.com/mathewsanders/Mustard/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228503090,"owners_count":17930509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["substrings","swift","tokenizer"],"created_at":"2024-01-05T20:16:02.015Z","updated_at":"2024-12-06T17:30:31.300Z","avatar_url":"https://github.com/mathewsanders.png","language":"Swift","funding_links":[],"categories":["Text"],"sub_categories":["Other free courses","Other Testing","Keychain"],"readme":"# Mustard 🌭\n\n[![GitHub license](https://img.shields.io/badge/license-MIT-lightgrey.svg?style=flat)](https://github.com/mathewsanders/Mustard/blob/master/LICENSE) [![Carthage compatible](https://img.shields.io/badge/Carthage-compatible-4BC51D.svg?style=flat)](https://github.com/Carthage/Carthage) [![Swift Package Manager compatible](https://img.shields.io/badge/Swift%20Package%20Manager-compatible-EF5138%20.svg?style=flat)](https://swift.org/package-manager/)\n\nMustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.\n\n## Quick start using character sets\n\nFoundation includes the `String` method [`components(separatedBy:)`](https://developer.apple.com/documentation/foundation/nsstring/1413214-components) that allows us to get substrings divided up by certain characters:\n\n````Swift\nlet sentence = \"hello 2017 year\"\nlet words = sentence.components(separatedBy: .whitespaces)\n// words.count -\u003e 3\n// words = [\"hello\", \"2017\", \"year\"]\n````  \n\nMustard provides a similar feature, but with the opposite approach, where instead of matching by separators you can match by one or more character sets, which is useful if separators simply don't exist:\n\n````Swift\nimport Mustard\n\nlet sentence = \"hello2017year\"\nlet words = sentence.components(matchedWith: .letters, .decimalDigits)\n// words.count -\u003e 3\n// words = [\"hello\", \"2017\", \"year\"]\n````  \n\nIf you want more than just the substrings, you can use the `tokens(matchedWith: CharacterSet...)` method which will return an array of `TokenType`.\n\nAs a minimum, `TokenType` requires properties for text (the substring matched), and range (the range of the substring in the original string). When using CharacterSets as a tokenizer, the more specific type `CharacterSetToken` is returned, which includes the property `set` which contains the instance of CharacterSet that was used to create the match.\n\n````Swift\nimport Mustard\n\nlet tokens = \"123Hello world\u0026^45.67\".tokens(matchedWith: .decimalDigits, .letters)\n// tokens: [CharacterSet.Token]\n// tokens.count -\u003e 5 (characters '\u0026', '^', and '.' are ignored)\n//\n// second token..\n// token[1].text -\u003e \"Hello\"\n// token[1].range -\u003e Range\u003cString.Index\u003e(3..\u003c8)\n// token[1].set -\u003e CharacterSet.letters\n//\n// last token..\n// tokens[4].text -\u003e \"67\"\n// tokens[4].range -\u003e Range\u003cString.Index\u003e(19..\u003c21)\n// tokens[4].set -\u003e CharacterSet.decimalDigits\n````\n\n## Advanced matching with custom tokenizers\n\nMustard can do more than match from character sets. You can create your own tokenizers with more\nsophisticated matching behavior by implementing the `TokenizerType` and `TokenType` protocols.\n\nHere's an example of using `DateTokenizer` ([see example for implementation](Documentation/Template%20tokenizer.md)) that finds substrings that match a `MM/dd/yy` format.\n\n`DateTokenizer` returns tokens with the type `DateToken`. Along with the substring text and range, `DateToken` includes a `Date` object corresponding to the date in the substring:\n\n````Swift\nimport Mustard\n\nlet text = \"Serial: #YF 1942-b 12/01/17 (Scanned) 12/03/17 (Arrived) ref: 99/99/99\"\n\nlet tokens = text.tokens(matchedWith: DateTokenizer())\n// tokens: [DateTokenizer.Token]\n// tokens.count -\u003e 2\n// ('99/99/99' is *not* matched by `DateTokenizer` because it's not a valid date)\n//\n// first date\n// tokens[0].text -\u003e \"12/01/17\"\n// tokens[0].date -\u003e Date(2017-12-01 05:00:00 +0000)\n//\n// last date\n// tokens[1].text -\u003e \"12/03/17\"\n// tokens[1].date -\u003e Date(2017-12-03 05:00:00 +0000)\n````\n\n## Documentation \u0026 Examples\n\n- [Greedy tokens and tokenizer order](Documentation/Greedy%20tokens%20and%20tokenizer%20order.md)\n- [Token types and AnyToken](Documentation/Token%20types%20and%20AnyToken.md)\n- [TokenizerType: implementing your own tokenizer](Documentation/TokenizerType%20protocol.md)\n- [EmojiTokenizer: matching emoji substrings](Documentation/Matching%20emoji.md)\n- [LiteralTokenizer: matching specific substrings](Documentation/Literal%20tokenizer.md)\n- [DateTokenizer: tokenizer based on template match](Documentation/Template%20tokenizer.md)\n- [Alternatives to using Mustard](Documentation/Alternatives%20to%20using%20Mustard.md)\n- [Performance comparisons](Documentation/Performance%20Comparisons.md)\n\n## Roadmap\n- [x] Include detailed examples and documentation\n- [x] Ability to skip/ignore characters within match\n- [x] Include more advanced pattern matching for matching tokens\n- [x] Make project logo 🌭\n- [x] Performance testing / benchmarking against Scanner\n- [ ] Include interface for working with Character tokenizers\n\n## Requirements\n\n- Swift 4.1\n\n## Author\n\nMade with :heart: by [@permakittens](http://twitter.com/permakittens)\n\n## Contributing\n\nFeedback, or contributions for bug fixing or improvements are welcome. Feel free to submit a pull request or open an issue.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmathewsanders%2FMustard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmathewsanders%2FMustard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmathewsanders%2FMustard/lists"}