{"id":20683172,"url":"https://github.com/chimehq/flexer","last_synced_at":"2025-04-22T12:21:37.957Z","repository":{"id":63906874,"uuid":"258594849","full_name":"ChimeHQ/Flexer","owner":"ChimeHQ","description":"Lexing library for Swift","archived":false,"fork":false,"pushed_at":"2025-04-06T10:28:02.000Z","size":94,"stargazers_count":35,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-06T11:19:22.411Z","etag":null,"topics":["lexer","protocol-oriented","swift"],"latest_commit_sha":null,"homepage":null,"language":"Swift","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ChimeHQ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["mattmassicotte"]}},"created_at":"2020-04-24T18:40:56.000Z","updated_at":"2025-04-06T10:35:45.000Z","dependencies_parsed_at":"2024-09-18T14:15:08.319Z","dependency_job_id":null,"html_url":"https://github.com/ChimeHQ/Flexer","commit_stats":{"total_commits":22,"total_committers":1,"mean_commits":22.0,"dds":0.0,"last_synced_commit":"6a3ab33a479bf9e3a057b29e3708569c1dc9460f"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChimeHQ%2FFlexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChimeHQ%2FFlexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChimeHQ%2FFlexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChimeHQ%2FFlexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ChimeHQ","download_url":"https://codeload.github.com/ChimeHQ/Flexer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250237863,"owners_count":21397405,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lexer","protocol-oriented","swift"],"created_at":"2024-11-16T22:15:39.267Z","updated_at":"2025-04-22T12:21:37.929Z","avatar_url":"https://github.com/ChimeHQ.png","language":"Swift","funding_links":["https://github.com/sponsors/mattmassicotte"],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n[![Build Status][build status badge]][build status]\n[![Platforms][platforms badge]][platforms]\n[![Matrix][matrix badge]][matrix]\n\n\u003c/div\u003e\n\n# Flexer\n\nFlexer is a small library for building lexers in Swift. It is compatible with all Apple platforms.\n\n- API tailored for hand-written parsing\n- Fully Swift `String`-compatible\n- Based around `Sequence` and `IteratorProtocol` procotols\n\nIt turns out that Swift's `Sequence` and `Iterator` concepts work pretty well for processing tokens. They make for a familiar API that also offers a surprising amount of power. Flexer builds on these concepts with some new protocols that are made specifically for lexing, but are generally applicable to all `Sequence` types.\n\n## Integration\n\n```swift\ndependencies: [\n    .package(url: \"https://github.com/ChimeHQ/Flexer\")\n]\n```\n\n## Look-Ahead\n\nCore to lexing is the ability to look ahead at future tokens without advancing. Flexer implements look-ahead with a protocol called `LookAheadIteratorProtocol`. The whole implementation is inspired by the `lazy` property of `Sequence`, and works very similarly.\n\n```swift\nlet lookAheadSequence = anySequence.lookAhead\n\nlet next = lookAheadSequence.peek()\n```\n\nThe main work of building your lexer is then defining a Sequence type of tokens. All of the lexing facilities you might need can then be exposed with a `typealias`.\n\n```swift\ntypealias MyLexer = LookAheadSequence\u003cMyTokenSequence\u003e\n  \nlet tokenSequence = MyLexer(string: myString)\n\nlet nextToken = lexer.next()\nlet futureToken = lexer.peek()\nlet tabToken = lexer.nextUntil({ $0.kind == .tab })\n```\n\n## Token Sequences\n\nYour custom token sequence can be built by creating a struct that conforms to `Sequence`. To make this easier, Flexer includes a type that can be used as a foundation for creating more complex token streams, called `BasicTextCharacterSequence`. It is a sequence of `BasicTextCharacter` elements. It breaks up a string into commonly-needed tokens, catagorized by kind and range within the source string. This approach uses the `Token` type, which stores a kind and a range within the source string.\n\nIt is usually much easier to build up more complex lexing functionality with the convenience of Swift switch pattern matching, instead of having to worry about the underlying characters and ranges themselves. You can do this by wrapping up a `BasicTextCharacterSequence` in your own custom sequence.\n\nHere's a fully-functioning example that produces four different token types. It shows off some of the scanning and look-ahead facilities that can be handy both for constructing and also using your lexer.\n\n```swift\nenum ExampleTokenKind {\n    case word\n    case number\n    case symbol\n    case whitespace\n}\n\ntypealias ExampleToken = Flexer.Token\u003cExampleTokenKind\u003e\n\nstruct ExampleTokenSequence: Sequence, IteratorProtocol, StringInitializable {\n    public typealias Element = ExampleToken\n\n    private var lexer: BasicTextCharacterLexer\n\n    public init(string: String) {\n        self.lexer = BasicTextCharacterLexer(string: string)\n    }\n\n    public mutating func next() -\u003e Element? {\n        guard let token = lexer.peek() else {\n            return nil\n        }\n\n        switch token.kind {\n        case .lowercaseLetter, .uppercaseLetter, .underscore:\n            guard let endingToken = lexer.nextUntil(notIn: [.lowercaseLetter, .uppercaseLetter, .underscore, .digit]) else {\n                return nil\n            }\n\n            return ExampleToken(kind: .word, range: token.startIndex..\u003cendingToken.endIndex)\n        case .digit:\n            guard let endingToken = lexer.nextUntil({ $0.kind != .digit }) else {\n                return nil\n            }\n\n            return ExampleToken(kind: .number, range: token.startIndex..\u003cendingToken.endIndex)\n        case .newline, .tab, .space:\n            guard let endingToken = lexer.nextUntil(notIn: [.newline, .tab, .space]) else {\n                return nil\n            }\n\n            return ExampleToken(kind: .whitespace, range: token.startIndex..\u003cendingToken.endIndex)\n        default:\n            break\n        }\n\n        guard let endingToken = lexer.nextUntil(in: [.newline, .tab, .space, .lowercaseLetter, .uppercaseLetter, .underscore, .digit]) else {\n            return nil\n        }\n\n        return ExampleToken(kind: .symbol, range: token.startIndex..\u003cendingToken.endIndex)\n    }\n}\n\ntypealias ExampleTokenLexer = LookAheadSequence\u003cExampleTokenSequence\u003e\n```\n\n## Contributing and Collaboration\n\nI would love to hear from you! Issues or pull requests work great. Both a [Matrix space][matrix] and [Discord][discord] are available for live help, but I have a strong bias towards answering in the form of documentation. You can also find me on [mastodon](https://mastodon.social/@mattiem).\n\nI prefer collaboration, and would love to find ways to work together if you have a similar project.\n\nI prefer indentation with tabs for improved accessibility. But, I'd rather you use the system you want and make a PR than hesitate because of whitespace.\n\nBy participating in this project you agree to abide by the [Contributor Code of Conduct](CODE_OF_CONDUCT.md).\n\n[build status]: https://github.com/ChimeHQ/Flexer/actions\n[build status badge]: https://github.com/ChimeHQ/Flexer/workflows/CI/badge.svg\n[platforms]: https://swiftpackageindex.com/ChimeHQ/Flexer\n[platforms badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fswiftpackageindex.com%2Fapi%2Fpackages%2FChimeHQ%2FFlexer%2Fbadge%3Ftype%3Dplatforms\n[matrix]: https://matrix.to/#/%23chimehq%3Amatrix.org\n[matrix badge]: https://img.shields.io/matrix/chimehq%3Amatrix.org?label=Matrix\n[discord]: https://discord.gg/esFpX6sErJ\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchimehq%2Fflexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchimehq%2Fflexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchimehq%2Fflexer/lists"}