{"id":22092113,"url":"https://github.com/krakphp/lex","last_synced_at":"2025-08-03T19:07:52.027Z","repository":{"id":57009022,"uuid":"57080634","full_name":"krakphp/lex","owner":"krakphp","description":"Functional Lexing Library","archived":false,"fork":false,"pushed_at":"2020-05-30T20:18:26.000Z","size":8,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-19T06:40:48.302Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/krakphp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-04-25T22:30:47.000Z","updated_at":"2020-09-11T23:47:57.000Z","dependencies_parsed_at":"2022-08-21T12:30:09.888Z","dependency_job_id":null,"html_url":"https://github.com/krakphp/lex","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/krakphp/lex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krakphp%2Flex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krakphp%2Flex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krakphp%2Flex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krakphp%2Flex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/krakphp","download_url":"https://codeload.github.com/krakphp/lex/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krakphp%2Flex/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268596676,"owners_count":24275930,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-03T02:00:12.545Z","response_time":2577,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T03:08:22.158Z","updated_at":"2025-08-03T19:07:51.972Z","avatar_url":"https://github.com/krakphp.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lex\n\nLex is a library for lexical analysis in PHP. Currently, only simple regular expression lexers are available; but considering that you shouldn't be lexing anything complex in php, this should be fine :).\n\n## Installation\n\n```\ncomposer require krak/lex\n```\n\n## Usage\n\n```php\n\u003c?php\n\nrequire_once __DIR__ . '/vendor/autoload.php';\n\nuse function Krak\\Lex\\lexer,\n    Krak\\Lex\\skipLexer;\n\nconst TOK_INT = 'int';\nconst TOK_PLUS = 'plus';\nconst TOK_MINUS = 'minus';\nconst TOK_WS = 'whitespace';\n\n// creates a lexer that will use these RE's to match input\n// the A (anchor flag) is required\n$lex = lexer([\n    '/\\d+/A' =\u003e TOK_INT,\n    '/\\+/A' =\u003e TOK_PLUS,\n    '/\\-/A' =\u003e TOK_MINUS,\n    '/\\s+/A' =\u003e TOK_WS\n]);\n\n// decorator for skipping tokens, in this case, just throw away the whitespace tokens\n$lex = skipLexer($lex, [TOK_WS]);\n\n// lex the input and return an iterator of tokens\n$toks = $lex('1 + 2 - 3');\n\nforeach ($toks as $matched_tok) {\n    printf(\n        \"Matched token '%s' with input '%s' at offset %d\\n\",\n        $matched_tok-\u003etoken,\n        $matched_tok-\u003ematch,\n        $matched_tok-\u003eoffset\n    );\n}\n```\n\nThe following program would output\n\n```\nMatched token 'int' with input '1' at offset 0\nMatched token 'plus' with input '+' at offset 2\nMatched token 'int' with input '2' at offset 4\nMatched token 'minus' with input '-' at offset 6\nMatched token 'int' with input '3' at offset 8\n```\n\n## TokenStream\n\nA token stream is a simple interface for consuming one token at a time. This is very useful for [Recursive Decent Parsers](https://en.wikipedia.org/wiki/Recursive_descent_parser)\n\n```php\n\u003c?php\n\nuse function Krak\\Lex\\lexer,\n    Krak\\Lex\\tokenStreamLexer;\n\n$lex = lexer(['/a/A' =\u003e 'a', '/b/A' =\u003e 'b']);\n$lex = tokenStreamLexer($lex);\n$stream = $lex('aba');\n\nassert($stream-\u003epeek() == 'a');\nassert($stream-\u003egetToken() == 'a');\nassert($stream-\u003egetToken() == 'b');\nassert($stream-\u003egetToken() == 'a');\nassert($stream-\u003eisEmpty());\n```\n\n## API\n\n### Lexers\n\nEach lexer will accept a string input and then return an iterable of `MatchedToken`\n\n#### lexer($token_map, $throw = true)\n\nMain lexer which lexes the strings based off of the `$token_map`. `$throw` determines whether or not the lexer should throw an exception on unrecognized input.\n\n#### skipLexer($lex, $tokens)\n\nLexer decorator which will skip any Matched Tokens in the set of the `$tokens` passed in.\n\n#### tokenStreamLexer($lex)\n\nLexer decorator that will convert the output of the `$lex` into a `TokenStream`\n\n#### mockLexer($tokens)\n\nReturns `$tokens` as is.\n\n### class MatchedToken\n\n#### $match\n\nReturns the text that was matched.\n\n#### $token\n\nReturns the token name that was matched.\n\n#### $offset\n\nReturns the string offset at which the match started.\n\n### interface TokenStream extends \\\\IteratorAggregate\n\n#### getToken()\n\nReturns the current token and advances the internal pointer up by one.\n\n#### peek()\n\nReturns the current token but does *not* advance the internal pointer\n\n#### isEmpty()\n\nreturns true if the token stream is empty, false if not.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrakphp%2Flex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkrakphp%2Flex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrakphp%2Flex/lists"}