{"id":13995301,"url":"https://github.com/yassram/SwiLex","last_synced_at":"2025-07-22T21:32:31.623Z","repository":{"id":56073162,"uuid":"295320712","full_name":"yassram/SwiLex","owner":"yassram","description":"A universal lexer library in Swift.","archived":false,"fork":false,"pushed_at":"2020-11-27T05:03:55.000Z","size":38,"stargazers_count":30,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-08-10T14:19:00.589Z","etag":null,"topics":["lexer","lexical-analysis","swift-package-manager","syntax-analysis","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"Swift","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yassram.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-14T06:11:17.000Z","updated_at":"2024-05-27T18:28:43.000Z","dependencies_parsed_at":"2022-08-15T12:40:26.778Z","dependency_job_id":null,"html_url":"https://github.com/yassram/SwiLex","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yassram%2FSwiLex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yassram%2FSwiLex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yassram%2FSwiLex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yassram%2FSwiLex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yassram","download_url":"https://codeload.github.com/yassram/SwiLex/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227177894,"owners_count":17743194,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lexer","lexical-analysis","swift-package-manager","syntax-analysis","tokenizer"],"created_at":"2024-08-09T14:03:20.484Z","updated_at":"2024-11-29T17:31:27.012Z","avatar_url":"https://github.com/yassram.png","language":"Swift","funding_links":[],"categories":["Swift"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003cimg src=\"SwiLex.png\" width=\"45%\" alt=“SwiLex”/\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Swift-5.2-orange.svg\"/\u003e\n    \u003cimg src=\"https://img.shields.io/badge/swiftpm-compatible-brightgreen.svg\"/\u003e\n    \u003cimg src=\"https://img.shields.io/badge/platforms-iOS%20|%20macOS%20|%20Linux-brightgreen.svg\"/\u003e\n    \u003ca href=\"https://github.com/yassram/SwiLex/actions\"\u003e\n        \u003cimg src=\"https://github.com/yassram/SwiLex/workflows/TestSuite/badge.svg\"/\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://twitter.com/ramsserio\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/twitter-@ramsserio-blue.svg\"/\u003e\n    \u003c/a\u003e\n    \u003cbr\u003e\n\u003cstrong\u003eA universal lexer library in Swift.\u003c/strong\u003e\n\u003c/p\u003e\n\n- [Introduction](#introduction)\n- [Installation](#installation)\n- [Usage](#usage)\n    - [Simple Lexer example](#simple-lexer-example)\n    - [Conditianal tokens Lexer example](#conditianal-tokens-lexer-example)\n- [Documentation](#documentation)\n- [Features](#features)\n- [Contributing](#contributing)\n- [Author](#author)\n- [License](#license)\n\n\n## Introduction\nIn computer science a Lexer or a tokenizer is a program that converts a sequence of characters (raw string) into a sequence of tokens. \n\nSwiLex is a universal lexer which means that you can use it to build *any* lexer only by defining your tokens in a Swift enum (in few lines of code). \n\n*Combined with **SwiParse** it allows to build a full Parser. ([learn more](https://github.com/yassram/SwiParse/))*\n\n## Installation\nSwiLex is distributed as a Swift Package using [SPM](https://swift.org/package-manager/).\n\nTo install SwiLex, add the following line to the `dependencies` list in your `Package.swift`:\n\n```swift\n.package(url: \"https://github.com/yassram/SwiLex.git\", .upToNextMinor(from: \"1.1.0\")),\n```\n\n*SwiLex will support other dependency managers soon.*\n\n## Usage\n\nImport SwiLex to use it:\n\n```swift\nimport SwiLex\n```\n\n### Simple Lexer example\n\nDefine a SwiLaxable enum to list your possible tokens with their corresponding regular expressions:\n\n```swift \nenum WordNumberTokens: String, SwiLexable {\n    static var separators: Set\u003cCharacter\u003e = [\" \"]\n    \n    case text = \"[A-Za-z]*\"\n    case number = \"[0-9]*\"\n    \n    case eof\n    case none\n}\n```\n\nThen create an instance of SwiLex and call the lex function:\n\n```swift\nvar lexer = SwiLex\u003cWordNumberTokens\u003e()\nlet tokenList = try lexer.lex(input: \"  H3Ll0   W0r 1d  1234 aBcD \")\n\n// returns [text[H], number[3], text[Ll], number[0], text[W], number[0], \n//          text[r], number[1], text[d], number[1234], text[aBcD]]\n```\n\nThis will return a list of tokens with the type of the token and its value (the matched string).\n\n### Conditianal tokens Lexer example\n\nSwiLex provides a mechanism for conditionally activating tokens. This tokens are only available for specific `Mode`.\n\nIn the following example we want to make the `.quotedString` token available only when the `.quote` mode is active.\n\nStart by defining the possible `Mode`s.\n\n```swift \nenum Modes {\n    case normal\n    case quote\n}\n```\n\nThen define a SwiLaxable enum to list the possible tokens with their corresponding regular expressions (like for a simple Lexer):\n\n```swift \nenum QuotedStringTextNumberTokens: String, SwiLexable {\n    static var separators: Set\u003cCharacter\u003e = [\" \"]\n    \n    case text = \"[A-Za-z]*\"\n    case number = \"[0-9]*\"\n    case doubleQuote = \"\\\"\"\n    case quotedString = #\"[^\"]*\"#\n    \n    case eof\n    case none\n}\n```\n\nNext, define the tokens' availability for the possible modes by implementing the `activeForModes` property:\n\n```swift\nextension QuotedStringTextNumberTokens {\nvar activeForModes: Set\u003cModes\u003e {\n    switch self {\n        case .doubleQuote:\n            return [.quote, .normal]\n        case .quotedString:\n            return [.quote]\n        default:\n            return [.normal]\n        }\n    }\n}\n```\n\u003e - `.doubleQuote` is available for both `.normal` and `.quote` modes allowing to switch between them (by defining the end and the begining of a quote).\u003cbr\u003e\n\u003e - `.quotedString` is only available for the `.quote` mode.\u003cbr\u003e\n\u003e - All the other tokens are only availbale for the `.normal` mode. \n\nThe last step is to tell when to change the curent mode. \n\nFor that implement the function `changeMode(current mode: Modes?) -\u003e Modes?`:\n\n```swift\nextension QuotedStringTextNumberTokens {\n    func changeMode(current mode: Modes?) -\u003e Modes? {\n        switch self {\n        case .doubleQuote:\n            return mode == .normal ? .quote : .normal\n        default:\n            return mode\n        }\n    }\n}\n```\n\nFinally create an instance of SwiLex and call the lex function with the initial mode:\n\n```swift\nlet input = #\"4lb3rt Einstein said \"If you can't explain it to a 6 year old, you don't understand it yourself.\"\"#\n\nvar lexer = SwiLex\u003cQuotedStringTextNumberTokens\u003e()\nlet tokenList = try lexer.lex(input: input, initialMode: .normal)\n\n// [number[4], text[lb], number[3], text[rt], text[Einstein], text[said], doubleQuote[\"], \n//  quotedString[If you can't explain it to a 6 year old, you don't understand it yourself.], \n//  doubleQuote[\"]]\n\n```\n\nThis will return a list of tokens with the type of the token and its value (the matched string).\n\n### Documentation\nA documentation with more examples is available [here](https://github.com/yassram/SwiLex/wiki)\n\n\n## Features\n- [x] Tokens defined using only a simple `SwiLexable` enum.\n- [x] Support conditional tokens with custom modes.\n- [x] Support actions on each match by implementing the `onLex(raw: Substring)` function.\n- [x] Errors with line number and the issue's substring.\n- [ ] Add detailed documentation with more examples.\n- [ ] Support Cocoapods and Carthage.\n- [x] 🔥 **SwiParse**, a general-purpose parser generator  that can be linked to **SwiLex** to generate a full parser. *([released here](https://github.com/yassram/SwiParse))* 🔥\n\n\n## Contributing\nThis is an open source project, so feel free to contribute. How?\n- Open an \u003ca href=\"https://github.com/yassram/SwiLex/issues/new\"\u003e issue\u003c/a\u003e.\n- Send feedback via \u003ca href=\"mailto:ramsserio@gmail.com\"\u003eemail\u003c/a\u003e.\n- Propose your own fixes, suggestions and open a pull request with the changes.\n\n## Author\nYassir Ramdani\n\n## License\n\n```\nMIT License\n\nCopyright (c) 2020 yassir RAMDANI\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyassram%2FSwiLex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyassram%2FSwiLex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyassram%2FSwiLex/lists"}