{"id":18256024,"url":"https://github.com/ull-esit-pl/moo-ignore","last_synced_at":"2025-04-04T17:31:45.731Z","repository":{"id":41450926,"uuid":"369531045","full_name":"ULL-ESIT-PL/moo-ignore","owner":"ULL-ESIT-PL","description":"A wrapper around the moo lexer generator that provides a nearley.js compatible lexer with the capacity to ignore specified tokens","archived":false,"fork":false,"pushed_at":"2023-11-24T08:27:08.000Z","size":69,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-24T14:29:39.784Z","etag":null,"topics":["lexer","lexical-analysis","moo","nearley","nearleyjs","ull"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ULL-ESIT-PL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-21T12:39:03.000Z","updated_at":"2022-10-25T11:53:33.000Z","dependencies_parsed_at":"2022-09-19T00:51:35.524Z","dependency_job_id":null,"html_url":"https://github.com/ULL-ESIT-PL/moo-ignore","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ULL-ESIT-PL%2Fmoo-ignore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ULL-ESIT-PL%2Fmoo-ignore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ULL-ESIT-PL%2Fmoo-ignore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ULL-ESIT-PL%2Fmoo-ignore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ULL-ESIT-PL","download_url":"https://codeload.github.com/ULL-ESIT-PL/moo-ignore/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223151220,"owners_count":17096070,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lexer","lexical-analysis","moo","nearley","nearleyjs","ull"],"created_at":"2024-11-05T10:19:27.915Z","updated_at":"2024-11-05T10:19:28.561Z","avatar_url":"https://github.com/ULL-ESIT-PL.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![npm version](https://badge.fury.io/js/moo-ignore.svg)](https://badge.fury.io/js/moo-ignore)\n[![Test](https://github.com/ULL-ESIT-PL/moo-ignore/actions/workflows/node.yml/badge.svg?branch=main)](https://github.com/ULL-ESIT-PL/moo-ignore/actions/workflows/node.yml)\n\n# Moo-ignore\n\nMoo-ignore (🐄) is a wrapper around the [moo](https://www.npmjs.com/package/moo) tokenizer/lexer generator that provides a [nearley.js](https://github.com/hardmath123/nearley) compatible lexer with the capacity to ignore specified tokens.\n\n\n## Usage\n\nInstall it: \n\n```\n$ npm install moo-ignore\n``` \n\n## Exports\n\nThis module exports an object having the `makeLexer` constructor and the `moo` object (as in `const moo = require(\"moo\")`):\n\n```js\nconst { makeLexer, moo } = require(\"moo-ignore\");\n```\n\n## Ignoring tokens\n\nThen you can use it in your Nearley.js program and ignore some tokens like white spaces and comments:\n\n\n```js\n@{%\nconst tokens = require(\"./tokens\");\nconst { makeLexer } = require(\"moo-ignore\");\n\nlet lexer = makeLexer(tokens);\nlexer.ignore(\"ws\", \"comment\");\n\nconst getType = ([t]) =\u003e t.type;\n%}\n\n@lexer lexer\n\nS -\u003e FUN LP name COMMA name COMMA name RP \n      DO \n        DO  END SEMICOLON \n        DO END \n      END\n     END\n\nname  -\u003e      %identifier {% getType %}\nCOMMA -\u003e       \",\"        {% getType %}\nLP    -\u003e       \"(\"        {% getType %}\nRP    -\u003e       \")\"        {% getType %}\nEND   -\u003e      %end        {% getType %}\nDO    -\u003e      %dolua      {% getType %}\nFUN   -\u003e      %fun        {% getType %}\nSEMICOLON -\u003e  \";\"         {% getType %}\n```\n\nAlternatively, you can set to ignore some tokens at construction time in the call to `makeLexer`:\n\n```js\nlet lexer = makeLexer(tokens, [\"ws\", \"comment\"]);\n```\n\nOr you can also combine both ways:\n\n```js\nlet lexer = makeLexer(tokens, [\"ws\"]);\nlexer.ignore(\"comment\");\n```\n\nFor sake of completeness, here is the contents of the file `tokens.js` we have used in the former code:\n\n```js\nconst { moo } = require(\"moo-ignore\");\n\nmodule.exports = {\n    ws: { match: /\\s+/, lineBreaks: true },\n    comment: /#[^\\n]*/,\n    lp: \"(\",\n    rp: \")\",\n    comma: \",\",\n    semicolon: \";\",\n    identifier: {\n        match: /[a-z_][a-z_0-9]*/,\n        type: moo.keywords({\n            fun: \"fun\",\n            end: \"end\",\n            dolua: \"do\"\n        })\n    }\n}\n```\n\nSee the [tests](https://github.com/ULL-ESIT-PL/moo-ignore/tree/main/test) folder in this distribution for more examples of use. Here is a program that tests the former example:\n\n```js\nconst nearley = require(\"nearley\");\nconst grammar = require(\"./test-grammar.js\");\n\nlet s = `\nfun (id, idtwo, idthree)  \n  do   #hello\n    do end;\n    do end # another comment\n  end \nend`;\n\ntry {\n  const parser = new nearley.Parser(nearley.Grammar.fromCompiled(grammar));\n  parser.feed(s);\n  console.log(parser.results[0]) /* [ 'fun', 'lp', 'identifier', 'comma',\n          'identifier', 'comma', 'identifier', 'rp',\n          'dolua',      'dolua', 'end', 'semicolon',\n          'dolua',      'end', 'end', 'end' */\n} catch (e) {\n    console.log(e);\n}\n```\n\n## The eof option: Emitting a token to signal the End Of File\n\nThe last  argument of `makeLexer` is an object with configuration options:\n\n```js\nlet lexer = makeLexer(Tokens, [ tokens, to, ignore ], { options });\n```\n\n\nCurrently, the only `option` supported in this version is `eof`. \n\nRemember that lexers generated by moo emit `undefined` when the end of the input is reached. This option changes this behavior.\n\nIf the option `{ eof : true }` is specified,  and a token with the name `EOF: \"termination string\"` appears in the tokens specification, `moo-ignore` will concat the `\"termination string\"`  at the end of the input stream. \n\n```js\nconst { makeLexer } = require(\"moo-ignore\");\nconst Tokens = {\n  EOF: \"__EOF__\",\n  WHITES: { match: /\\s+/, lineBreaks: true },\n  /* etc. */\n};\n\nlet lexer = makeLexer(Tokens, [\"WHITES\"], { eof: true });\n```\n\nThe generated lexer will emit this `EOF` token when the end of the input is reached. \n\nInside your grammar you'll have to explicit the use of the `EOF` token. Something like this:\n\n```js\n@{%\nconst { lexer } = require('./lex.js');\n%}\n@lexer lexer\nprogram -\u003e expression %EOF {% id %}\n# ... other rules\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Full-esit-pl%2Fmoo-ignore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Full-esit-pl%2Fmoo-ignore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Full-esit-pl%2Fmoo-ignore/lists"}