{"id":16789648,"url":"https://github.com/ryangjchandler/lexical","last_synced_at":"2025-04-07T13:07:53.252Z","repository":{"id":173422549,"uuid":"650725467","full_name":"ryangjchandler/lexical","owner":"ryangjchandler","description":"Quickly build lexers in PHP.","archived":false,"fork":false,"pushed_at":"2025-01-27T10:13:50.000Z","size":45,"stargazers_count":51,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-31T11:08:00.796Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ryangjchandler.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"ryangjchandler"}},"created_at":"2023-06-07T17:12:29.000Z","updated_at":"2025-01-26T05:32:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"b4e3be94-b24d-4f53-bf52-fe04319c5f69","html_url":"https://github.com/ryangjchandler/lexical","commit_stats":{"total_commits":22,"total_committers":2,"mean_commits":11.0,"dds":"0.045454545454545414","last_synced_commit":"bdb6224c11f057c9f1bdf81951fab05bd0ce014a"},"previous_names":["ryangjchandler/lexical"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryangjchandler%2Flexical","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryangjchandler%2Flexical/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryangjchandler%2Flexical/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryangjchandler%2Flexical/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ryangjchandler","download_url":"https://codeload.github.com/ryangjchandler/lexical/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247657281,"owners_count":20974345,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T08:27:56.257Z","updated_at":"2025-04-07T13:07:53.235Z","avatar_url":"https://github.com/ryangjchandler.png","language":"PHP","funding_links":["https://github.com/sponsors/ryangjchandler"],"categories":[],"sub_categories":[],"readme":"# Lexical\n\n[![Latest Version on Packagist](https://img.shields.io/packagist/v/ryangjchandler/lexical.svg?style=flat-square)](https://packagist.org/packages/ryangjchandler/lexical)\n[![Tests](https://img.shields.io/github/actions/workflow/status/ryangjchandler/lexical/run-tests.yml?branch=main\u0026label=tests\u0026style=flat-square)](https://github.com/ryangjchandler/lexical/actions/workflows/run-tests.yml)\n[![Total Downloads](https://img.shields.io/packagist/dt/ryangjchandler/lexical.svg?style=flat-square)](https://packagist.org/packages/ryangjchandler/lexical)\n\n## Installation\n\nYou can install the package via Composer:\n\n```bash\ncomposer require ryangjchandler/lexical\n```\n\n## Usage\n\nLet's write a simple lexer for mathematical expressions. The expressions can contain numbers (only integers) and a handful of operators (`+`, `-`, `*`, `/`).\n\nBegin by creating a new enumeration that describes the token types.\n\n```php\nenum TokenType\n{\n    case Number;\n    case Add;\n    case Subtract;\n    case Multiply;\n    case Divide;\n}\n```\n\nLexical provides a set of attributes that can be added to each case in an enumeration:\n* `Regex` - accepts a single regular expression.\n* `Literal` - accepts a string of continuous characters.\n* `Error` - designates a specific enumeration case as the \"error\" type.\n\nUsing those attributes with `TokenType` looks like this.\n\n```php\nenum TokenType\n{\n    #[Regex(\"[0-9]+\")]\n    case Number;\n    \n    #[Literal(\"+\")]\n    case Add;\n    \n    #[Literal(\"-\")]\n    case Subtract;\n    \n    #[Literal(\"*\")]\n    case Multiply;\n\n    #[Literal(\"/\")]\n    case Divide;\n}\n```\n\nWith the attributes in place, we can start to build a lexer using the `LexicalBuilder`.\n\n```php\n$lexer = (new LexicalBuilder)\n    -\u003ereadTokenTypesFrom(TokenType::class)\n    -\u003ebuild();\n```\n\nThe `readTokenTypesFrom()` method is used to tell the builder where we should look for the various tokenising attributes. The `build()` method will take those attributes and return an object that implements `LexerInterface`, configured to look for the specified token types.\n\nThen it's just a case of calling the `tokenise()` method on the lexer object to retrieve an array of tokens.\n\n```php\n$tokens = $lexer-\u003etokenise('1+2'); // -\u003e [[TokenType::Number, '1', Span(0, 1)], [TokenType::Add, '+', Span(1, 2)], [TokenType::Number, '2', Span(2, 3)]]\n```\n\nThe `tokenise()` method returns a list of tuples, where the first item is the \"type\" (`TokenType` in this example), the second item is the \"literal\" (a string containing the matched characters) and the third item is the \"span\" of the token (the start and end positions in the original string).\n\n### Skipping whitespace and other patterns\n\nContinuing with the example of a mathematical expression, the lexer currently understands `1+2` but it would fail to tokenise `1 + 2` (added whitespace). This is because by default it expects each and every possible character to fall into a pattern.\n\nThe whitespace is insignificant in this case, so can be skipped safely. To do this, we need to add a new `Lexer` attribute to the `TokenType` enumeration and pass through a regular expression that matches the characters we want to skip.\n\n```php\n#[Lexer(skip: \"[ \\t\\n\\f]+\")]\nenum TokenType\n{\n    // ...\n}\n```\n\nNow the lexer will skip over any whitespace characters and successfully tokenise `1 + 2`.\n\n### Error handling\n\nWhen a lexer encounters an unexpected character, it will throw an `UnexpectedCharacterException`.\n\n```php\ntry {\n    $tokens = $lexer-\u003etokenise();\n} catch (UnexpectedCharacterException $e) {\n    dd($e-\u003echaracter, $e-\u003eposition);\n}\n```\n\nAs mentioned above, there is an `Error` attribute that can be used to mark an enum case as the \"error\" type.\n\n```php\nenum TokenType\n{\n    // ...\n\n    #[Error]\n    case Error;\n}\n```\n\nNow when the input is tokenised, the unrecognised character will be consumed like other tokens and will have a type of `TokenType::Error`.\n\n```php\n$tokens = $lexer-\u003etokenise('1 % 2'); // -\u003e [[TokenType::Number, '1'], [TokenType::Error, '%'], [TokenType::Number, '2']]\n```\n\n### Custom `Token` objects\n\nIf you prefer to work with dedicated objects instead of Lexical's default tuple values for each token, you can provide a custom callback to map the matched token type and literal into a custom object.\n\n```php\nclass Token\n{\n    public function __construct(\n        public readonly TokenType $type,\n        public readonly string $literal,\n        public readonly Span $span,\n    ) {}\n}\n\n$lexer = (new LexicalBuilder)\n    -\u003ereadTokenTypesFrom(TokenType::class)\n    -\u003eproduceTokenUsing(fn (TokenType $type, string $literal, Span $span) =\u003e new Token($type, $literal, $span))\n    -\u003ebuild();\n\n$lexer-\u003etokenise('1 + 2'); // -\u003e [Token { type: TokenType::Number, literal: \"1\" }, ...]\n```\n\n### Token Producers\n\nRegular expressions and literal tokens can get you quite far when it comes to tokenisation, but there are scenarios where it would be easier to write \"real\" code to tokenise your input.\n\nLexical makes this possible by providing a Token Producer API. Token producers are regular PHP objects that implement the `RyanChandler\\Lexical\\Contracts\\TokenProviderInterface` or `RyanChandler\\Lexical\\Contracts\\TolerantTokenProviderInterface` interfaces.\n\nThey are attached to your token types using the `RyanChandler\\Lexical\\Attributes\\Custom` attribute, passing through the fully-qualified name of the token producer class.\n\n```php\nuse RyanChandler\\Lexical\\Attributes\\Custom;\nuse RyanChandler\\Lexical\\InputSource;\n\nenum Literals\n{\n    #[Custom(StringTokenProducer::class)]\n    case String;\n}\n\nclass StringTokenProducer implements TokenProducerInterface\n{\n    public function produce(InputSource $source): ?string\n    {\n        // \n    }\n}\n```\n\nThe `InputSource` object provided to the `produce()` method can be used to determine whether or not a token can be produced at the current offset. It comes with a range of utility methods such as `current()`, `peek()` and `match()`.\n\nIf your token type has an `Error` case defined, your token producer will need to implement the `TolerantTokenProducerInterface` instead. This interface has an additional method, `canProduce()`, which is used to determine whether or not the token can be seen anywhere in the remaining input.\n\nHere's an example token producer that tokenises double-quoted strings.\n\n```php\nclass StringTokenProducer implements TolerantTokenProducerInterface\n{\n    public function canProduce(InputSource $source): int|false\n    {\n        $matches = $source-\u003ematch('/\"/', PREG_OFFSET_CAPTURE);\n\n        if (! $matches) {\n            return false;\n        }\n\n        return $matches[0][1];\n    }\n\n    public function produce(InputSource $source): ?string\n    {\n        // If we're not looking at a double quote, return since we can't produce a token here.\n        if ($source-\u003ecurrent() !== '\"') {\n            return null;\n        }\n\n        // Place an offset marker in case we need to rewind at any point.\n        $source-\u003emark();\n\n        // Consume the \" character.\n        $token = $source-\u003econsume();\n\n        while ($source-\u003ecurrent() !== '\"') {\n            // If we reach the end of the file before we find a closing double-quote,\n            // we can rewind to the marker and return early.\n            if ($source-\u003eisEof()) {\n                $source-\u003erewind();\n\n                return null;\n            }\n\n            // Consume the current character.\n            $token .= $source-\u003econsume();\n        }\n\n        // If we reach this point, we must be at a double-quote character since the\n        // loop above has finished and we haven't returned yet.\n        $token .= $source-\u003econsume();\n\n        // Return the consumed text and let the Lexer handle the rest.\n        return $token;\n    }\n}\n```\n\n## Testing\n\n```bash\ncomposer test\n```\n\n## Changelog\n\nPlease see [CHANGELOG](CHANGELOG.md) for more information on what has changed recently.\n\n## Contributing\n\nPlease see [CONTRIBUTING](https://github.com/spatie/.github/blob/main/CONTRIBUTING.md) for details.\n\n## Security Vulnerabilities\n\nPlease review [our security policy](../../security/policy) on how to report security vulnerabilities.\n\n## Credits\n\n- [Ryan Chandler](https://github.com/ryangjchandler)\n- [All Contributors](../../contributors)\n\n## License\n\nThe MIT License (MIT). Please see [License File](LICENSE.md) for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryangjchandler%2Flexical","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fryangjchandler%2Flexical","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryangjchandler%2Flexical/lists"}