{"id":20348987,"url":"https://github.com/here-be/snapdragon-scanner","last_synced_at":"2025-04-12T01:22:29.774Z","repository":{"id":66000905,"uuid":"158237996","full_name":"here-be/snapdragon-scanner","owner":"here-be","description":"Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.","archived":false,"fork":false,"pushed_at":"2018-11-19T16:17:07.000Z","size":13,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-05-09T22:04:09.237Z","etag":null,"topics":["lex","lexer","parse","scan","scanner","snapdragon","string","token","tokenize","tokenizer"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/here-be.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-19T14:34:04.000Z","updated_at":"2023-06-08T04:45:49.000Z","dependencies_parsed_at":"2023-06-06T02:45:12.249Z","dependency_job_id":null,"html_url":"https://github.com/here-be/snapdragon-scanner","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/here-be%2Fsnapdragon-scanner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/here-be%2Fsnapdragon-scanner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/here-be%2Fsnapdragon-scanner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/here-be%2Fsnapdragon-scanner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/here-be","download_url":"https://codeload.github.com/here-be/snapdragon-scanner/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248502055,"owners_count":21114729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lex","lexer","parse","scan","scanner","snapdragon","string","token","tokenize","tokenizer"],"created_at":"2024-11-14T22:23:32.931Z","updated_at":"2025-04-12T01:22:29.764Z","avatar_url":"https://github.com/here-be.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# snapdragon-scanner [![NPM version](https://img.shields.io/npm/v/snapdragon-scanner.svg?style=flat)](https://www.npmjs.com/package/snapdragon-scanner) [![NPM monthly downloads](https://img.shields.io/npm/dm/snapdragon-scanner.svg?style=flat)](https://npmjs.org/package/snapdragon-scanner) [![NPM total downloads](https://img.shields.io/npm/dt/snapdragon-scanner.svg?style=flat)](https://npmjs.org/package/snapdragon-scanner) [![Linux Build Status](https://img.shields.io/travis/here-be/snapdragon-scanner.svg?style=flat\u0026label=Travis)](https://travis-ci.org/here-be/snapdragon-scanner)\n\n\u003e Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.\n\nPlease consider following this project's author, [Jon Schlinkert](https://github.com/jonschlinkert), and consider starring the project to show your :heart: and support.\n\n## Install\n\nInstall with [npm](https://www.npmjs.com/):\n\n```sh\n$ npm install --save snapdragon-scanner\n```\n\n## What is this?\n\nThis is a simple Lexical Scanner that takes an object of regex patterns, and uses those patterns to process an input string into an array of tokens.\n\n**What is the difference between this and [snapdragon-lexer](https://github.com/here-be/snapdragon-lexer)?**\n\nsnapdragon-lexer uses registered _handler functions_ to capture and handle tokens, snapdragon-scanner simply iterates over an object of regular expression patterns to create tokens. You can think of snapdragon-scanner as the \"lite\" version of snapdragon-lexer.\n\n## Usage\n\n```js\nconst Scanner = require('snapdragon-scanner');\n```\n\n## API\n\n### [Scanner](index.js#L27)\n\nCreate a new Scanner with the given `str` and optional `rules`.\n\n**Params**\n\n* `input` **{String}**: Input string to scan.\n* `options` **{Object}**: (optional) Pass an object of regex patterns on `options.rules`, or use `.addRules()` or `.addRule()` after instantiating.\n\n**Example**\n\n```js\nconst Scanner = require('snapdragon-scanner');\nconst scanner = new Scanner('var foo = \"bar\";', {\n  rules: {\n    space: /^ +/,\n    tab: /^\\t+/,\n    newline: /^\\n+/,\n    text: /^\\w+/,\n    equal: /^=/,\n    quote: /^[\"']/,\n    semi: /^;/,\n    dot: /^\\./\n  }\n});\n```\n\n### [.addRule](index.js#L59)\n\nAdd a rule to the scanner.\n\n**Params**\n\n* `rule` **{String}**\n* `match` **{RegExp}**: Match array from `RegExp.exec()`.\n\n**Example**\n\n```js\nconsole.log(scanner.token('text', ['foo']);\n//=\u003e { rule: 'text', value: 'foo', match: [foo] };\n```\n\n### [.addRule](index.js#L80)\n\nAdd a rule to the scanner.\n\n**Params**\n\n* `rule` **{String}**\n* `regex` **{RegExp}**: Regular expression to use when [scanning](#scan).\n\n**Example**\n\n```js\nscanner.addRule(rule, regex);\n// example\nscanner.addRule('text', /^\\w+/);\n```\n\n### [.addRules](index.js#L104)\n\nAdd an object of rules to the scanner.\n\n**Params**\n\n* `rules` **{Object}**\n\n**Example**\n\n```js\nscanner.addRules({\n  text: /^\\w+/,\n  slash: /^\\//,\n  dot: /^\\./\n});\n```\n\n### [.match](index.js#L133)\n\nAttempts to match `scanner.string` with the given regex. Also validates the regex to ensure that it starts with `^` since matching should always be against the beginning of the string, and throws if the regex matches an empty string, to avoid catastrophic backtracking.\n\n**Params**\n\n* `regex` **{RegExp}**: (required)\n* `returns` **{Array|null}**: Returns the match array or null from `RegExp.exec`.\n\n**Example**\n\n```js\nconst scanner = new Scanner('foo/bar', { text: /^\\w+/ });\nconst match = scanner.match(scanner.rules.get('text'));\nconsole.log(match);\n//=\u003e [ 'foo', index: 0, input: 'foo/bar', groups: undefined ]\n```\n\n### [.consume](index.js#L164)\n\nRemove the given length of substring from `scanner.string`.\n\n**Params**\n\n* `len` **{Number}**\n* `value` **{String}**: Optionally pass the value being consumed for minor performance improvement.\n* `returns` **{String}**: Returns the consumed value\n\n**Example**\n\n```js\nscanner.consume(1);\nscanner.consume(1, '*');\n```\n\n### [.enqueue](index.js#L185)\n\nPush a token onto the `scanner.queue` array.\n\n**Params**\n\n* `token` **{object}**\n* `returns` **{Object}**: Returns the token.\n\n**Example**\n\n```js\nconsole.log(scanner.queue.length); // 0\nscanner.enqueue({ rule: 'foo' });\nconsole.log(scanner.queue.length); // 1\n```\n\n### [.dequeue](index.js#L205)\n\nShift a token from `scanner.queue`.\n\n* `returns` **{Object}**: Returns the first token in the `scanner.queue`.\n\n**Example**\n\n```js\nconsole.log(scanner.queue.length); // 0\nscanner.enqueue({ rule: 'foo' });\nconsole.log(scanner.queue.length); // 1\nscanner.dequeue();\nconsole.log(scanner.queue.length); // 0\n```\n\n### [.advance](index.js#L222)\n\nIterates over the registered regex patterns until a match is found, then returns a token from the match and regex `rule`.\n\n* `returns` **{Object}**: Returns a token with `rule`, `value` and `match` properties.\n\n**Example**\n\n```js\nconst token = scanner.advance();\nconsole.log(token) // { rule: 'text', value: 'foo' }\n```\n\n### [.lookahead](index.js#L257)\n\nLookahead `n` tokens and return the last token. Pushes any intermediate tokens onto `scanner.tokens.` To lookahead a single token, use [.peek()](#peek).\n\n**Params**\n\n* `n` **{number}**\n* `returns` **{Object}**\n\n**Example**\n\n```js\nconst token = scanner.lookahead(2);\n```\n\n### [.peek](index.js#L276)\n\nReturns a token representing the next match, but without consuming the matched substring (e.g. the cursor position is not advanced).\n\n* `returns` **{Object|undefined}**: Returns a token, or undefined if no match was found.\n\n**Example**\n\n```js\nconst token = scanner.peek();\n```\n\n### [.peek](index.js#L292)\n\nReturns a token representing the next match, but without consuming the matched substring (e.g. the cursor position is not advanced).\n\n* `returns` **{Object|undefined}**: Returns a token, or undefined if no match was found.\n\n**Example**\n\n```js\nconst token = scanner.peek();\n```\n\n### [.scan](index.js#L307)\n\nReturns the next token and advances the cursor position.\n\n* `returns` **{Object|undefined}**: Returns a token, or undefined if no match was found.\n\n**Example**\n\n```js\nconst token = scanner.scan();\n```\n\n### [.scanWhile](index.js#L327)\n\nScan until the given `fn` does not return true.\n\n**Params**\n\n* `fn` **{Function}**: Must return true to continue scanning.\n* `returns` **{Array}**: Returns an array if scanned tokens.\n\n**Example**\n\n```js\nscanner.scanWhile(tok =\u003e tok.rule !== 'space');\n```\n\n### [.bos](index.js#L341)\n\nReturns true if the scanner has not consumed any of the input string.\n\n* `returns` **{Boolean}**\n\n### [.eos](index.js#L353)\n\nReturns true if `scanner.string` and `scanner.queue` are empty.\n\n* `returns` **{Boolean}**\n\n## Token objects\n\nScanner tokens are plain JavaScript objects with the following properties:\n\n```js\n{\n  type: String;\n  value: String\n  match: Array\n}\n```\n\n### Token properties\n\n* `type` **{String}** - The name of the regex that matched the substring.\n* `value` **{String}** - The substring that was captured by the regex.\n* `match` **{Array}** - The match array from [RegExp.exec()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/exec)\n\n## Release history\n\nSee [the changelog](CHANGELOG.md).\n\n## About\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eContributing\u003c/strong\u003e\u003c/summary\u003e\n\nPull requests and stars are always welcome. For bugs and feature requests, [please create an issue](../../issues/new).\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eRunning Tests\u003c/strong\u003e\u003c/summary\u003e\n\nRunning and reviewing unit tests is a great way to get familiarized with a library and its API. You can install dependencies and run tests with the following command:\n\n```sh\n$ npm install \u0026\u0026 npm test\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eBuilding docs\u003c/strong\u003e\u003c/summary\u003e\n\n_(This project's readme.md is generated by [verb](https://github.com/verbose/verb-generate-readme), please don't edit the readme directly. Any changes to the readme must be made in the [.verb.md](.verb.md) readme template.)_\n\nTo generate the readme, run the following command:\n\n```sh\n$ npm install -g verbose/verb#dev verb-generate-readme \u0026\u0026 verb\n```\n\n\u003c/details\u003e\n\n### Related projects\n\nYou might also be interested in these projects:\n\n* [snapdragon-lexer](https://www.npmjs.com/package/snapdragon-lexer): Converts a string into an array of tokens, with useful methods for looking ahead and… [more](https://github.com/here-be/snapdragon-lexer) | [homepage](https://github.com/here-be/snapdragon-lexer \"Converts a string into an array of tokens, with useful methods for looking ahead and behind, capturing, matching, et cetera.\")\n* [snapdragon-node](https://www.npmjs.com/package/snapdragon-node): Snapdragon utility for creating a new AST node in custom code, such as plugins. | [homepage](https://github.com/jonschlinkert/snapdragon-node \"Snapdragon utility for creating a new AST node in custom code, such as plugins.\")\n* [snapdragon-token](https://www.npmjs.com/package/snapdragon-token): Create a snapdragon token. Used by the snapdragon lexer, but can also be used by… [more](https://github.com/here-be/snapdragon-token) | [homepage](https://github.com/here-be/snapdragon-token \"Create a snapdragon token. Used by the snapdragon lexer, but can also be used by plugins.\")\n\n### Author\n\n**Jon Schlinkert**\n\n* [GitHub Profile](https://github.com/jonschlinkert)\n* [Twitter Profile](https://twitter.com/jonschlinkert)\n* [LinkedIn Profile](https://linkedin.com/in/jonschlinkert)\n\n### License\n\nCopyright © 2018, [Jon Schlinkert](https://github.com/jonschlinkert).\nReleased under the MIT License.\n\n***\n\n_This file was generated by [verb-generate-readme](https://github.com/verbose/verb-generate-readme), v0.8.0, on November 19, 2018._","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhere-be%2Fsnapdragon-scanner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhere-be%2Fsnapdragon-scanner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhere-be%2Fsnapdragon-scanner/lists"}