{"id":21470328,"url":"https://github.com/unboundedsystems/relexer","last_synced_at":"2026-01-05T05:04:37.007Z","repository":{"id":48034643,"uuid":"113495903","full_name":"unboundedsystems/relexer","owner":"unboundedsystems","description":"A regular expression based stream lexing library for javascript","archived":false,"fork":false,"pushed_at":"2024-08-09T19:20:00.000Z","size":54,"stargazers_count":1,"open_issues_count":8,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-08-10T13:34:25.600Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/unboundedsystems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-07T20:35:43.000Z","updated_at":"2024-08-10T13:34:25.601Z","dependencies_parsed_at":"2024-04-09T06:24:35.167Z","dependency_job_id":"a202375b-73e0-4102-9df5-b8375d67fb55","html_url":"https://github.com/unboundedsystems/relexer","commit_stats":{"total_commits":33,"total_committers":4,"mean_commits":8.25,"dds":0.2727272727272727,"last_synced_commit":"431e754c62460d1c6f0a27bf64e2894c12c31c84"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unboundedsystems%2Frelexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unboundedsystems%2Frelexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unboundedsystems%2Frelexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unboundedsystems%2Frelexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/unboundedsystems","download_url":"https://codeload.github.com/unboundedsystems/relexer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226027244,"owners_count":17562132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T09:26:59.658Z","updated_at":"2026-01-05T05:04:37.000Z","avatar_url":"https://github.com/unboundedsystems.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Relexer\n[![Build Status](https://travis-ci.org/unboundedsystems/relexer.svg?branch=master)](https://travis-ci.org/unboundedsystems/relexer)\n\nRegular Expression based lexing of Node.js streams\n\n[Relexer on GitHub](https://github.com/unboundedsystems/relexer)\n\nRelexer will lex a Node.js readable stream with user-controllable\nbuffering.  This allows large files to be analyzed without buffering\nthe entire file in memory.\n\n## Quick Start\n\nBelow is a typescript example lexer to generate tokens that are words\nseparated by whitespace\n\n```typescript\n//example.ts\nimport * as relexer from 'relexer';\n\nlet tokens: string[] = [];\n\nconst rules: relexer.Rules = [\n    //Match one-or-more non-whitepsace characters\n    { re: '[^\\\\s]+', action: async (match, pos) =\u003e { tokens.push(match); } },  \n    //Ignore whitespace\n    { re: '\\\\s+', action: async (match, pos) =\u003e { }; } \n];\n\nconst lexer = relexer.create(rules);\n\nlexer.lex(process.stdin).then(() =\u003e {\n  console.log(tokens);\n});\n```\n\n### Rules\n\nIn the example above, rules are an array of regular expression (`re`)\nstrings and actions (`action`) objects. (Note that RegExp objects will\nnot work since RegExp.toString() encloses the RegExp in slashes, e.g.,\n/xxx/))\n\nThe action objects are functions that return void promises that are\nresolved when the action is complete.  relexer will not process the next\ntoken until the returned promise is fulfilled.  In the meantime the\ninput stream is paused.  An action rejecting the promise will cause\nlex (see below) to reject its promise with the same error.  In the\nexample, async lambda functions serve as a concise way to write\nactions.\n\n### Lex\n\nThe `lex` method begins lexing based on the rules passed to\n`relexer.create`.  It returns a promise that is resolved when the\nstream ends and all tokens are processed.  The promise is rejected if\nno rules match, there is an I/O error, or an action rejects its\npromise.\n\nWhen this program is run with\n```shell\n$ echo \"Hello, how are you today?\" | node example.js\n```\nit should produce\n```javascript\n[ 'Hello,', 'how', 'are', 'you', 'today?']\n```\n\n## Buffering\n\nRelexer works by constructing a large regular expression and using the\nnative javascript RegExp package to process it.  Therefore, relexer\nhas to work within the restrictions of that library.  In particular,\nthere is no way to tell if a prefix of a string can never match a\nregular expression.  Thus, perfect matching may require buffering the\nentire stream contents, which is clearly not acceptable in many cases\ninvolving large streams.\n\nAs a workaround, relexer limits the size of buffer it will aggregate\nin order to match a token to the rule set.  By default relexer will\nbuffer 1024 bytes.  But this can be changed using the `aggregateUntil`\noption to `lex`.  E.g.,\n\n```typescript\nlexer.lex(process.stdin, { aggregateUntil: 10 });\n```\n\nIn this case, no more than 10 bytes will be buffered, and\nso any tokens longer than 10 characters may not come back as a match.\nHowever, because the stream can return chunks larger than the\n`aggregateUntil` limit, there may be cases were relexer can return\nlarger tokens.\n\nTo continue the quick start example with an `aggregateUntil` limit of\n10, if the stream returns data 1 byte at a time, the lexer will not\nmatch the token `01234567890` because it is 11 characters long.\nInstead, `lex` will reject its promise with a `No rules matched` exception.\nHowever, if the stream returns the token in an 11 or more byte chunk, relexer\nwill match `01234567890`.\n\nGenerally you should set `aggregateUntil` to a value large enough for the\nlargest expected token size, but small enough that no issues will\narise if relexer has to fill a buffer of that size.\n\nThe default `aggregateUntil` limit is 1024 bytes which should suffice\nfor most applications.\n\n## More information\n\nIf you have additional documentation to contribute, please submit a\npull request.  As time and contributions permit more documentation\nwill be added.  In the meantime, the unit tests in the\n[test](http:/github.com/unboundedsystems/relexer/tree/master/test)\ndirectory in the GitHub repository has more examples.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funboundedsystems%2Frelexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funboundedsystems%2Frelexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funboundedsystems%2Frelexer/lists"}