{"id":30048337,"url":"https://github.com/segmentio/fql-ts","last_synced_at":"2025-10-29T21:32:12.561Z","repository":{"id":42995092,"uuid":"162624380","full_name":"segmentio/fql-ts","owner":"segmentio","description":"A TypeScript FQL lexer","archived":false,"fork":false,"pushed_at":"2025-06-30T20:15:45.000Z","size":593,"stargazers_count":3,"open_issues_count":9,"forks_count":2,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-08-07T10:17:46.750Z","etag":null,"topics":["lexer","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/segmentio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-12-20T19:43:15.000Z","updated_at":"2025-05-27T16:26:40.000Z","dependencies_parsed_at":"2024-08-07T21:34:28.219Z","dependency_job_id":"9f3ebde0-fef7-4fb6-9890-f60c5119fe47","html_url":"https://github.com/segmentio/fql-ts","commit_stats":null,"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/segmentio/fql-ts","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Ffql-ts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Ffql-ts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Ffql-ts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Ffql-ts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/segmentio","download_url":"https://codeload.github.com/segmentio/fql-ts/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Ffql-ts/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274583806,"owners_count":25311899,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lexer","typescript"],"created_at":"2025-08-07T10:09:52.116Z","updated_at":"2025-10-29T21:32:12.453Z","avatar_url":"https://github.com/segmentio.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003ch2 align=\"center\"\u003eFQL (TypeScript)\u003c/h2\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\nLanguage for filtering json streams\n\u003cbr\u003e\u003cbr\u003e\n\u003c/p\u003e\n\n`fql-ts` is a Typescript lexer for [Segment's](https://segment.com) [Filter Query Language](https://segment.com/docs/api/public-api/fql/) (FQL).\n\nThe goal is to allow us to use FQL on the browser and generate React components on top of FQL. It could\nalso be used to give our FQL in-browser REPLs and integrate into existing tooling.\n\n## Usage\n\nInstall it with:\n\n```\nyarn add @segment/fql-ts\n```\n\n### Lexing\n\nTo convert some code into some strings, you can use `lex`:\n\n```js\nimport { lex } from '@segment/fql-ts'\n\nconst { tokens } = lex('message.event \u003e 30')\n```\n\nIf some thing goes wrong lexing you'll get an `error` that you could surface to the user.\n\n```js\nconst { error } = lex('message.event \u003e 30.30.30')\n\nerror.cursor.line // 0\nerror.cursor.column // 22\nerror.message // 'multiple decimal points in one number'\n```\n\n### Tokens\n\nIn `fql-ts`, a token is an object with a `value` and an enum `type`.\n\nFor example, a string like `\"Order Completed\"` has a token that looks like this:\n\n```js\nimport { types } from '@segment/fql-ts'\n\n{\n    type: types.String,\n    value: \"Order Completed\"\n}\n```\n\nSome tokens don't have a unique value, so theirs is set to the name of the\ntoken. This is helpful for when we want to \"unlex\" or convert tokens back to strings.\n\n```js\n{\n    type: types.Comma,\n    value: \",\"\n}\n```\n\nBut `fql-ts` also comes with helper functions to create tokens. We can create\nthat same String token with:\n\n```js\nimport { t } from '@segment/fql-ts'\n\nt.String('\"Order Completed\"')\n```\n\nLikewise, tokens that don't need values don't need string inputs in their helper\nfunctions:\n\n```js\nt.Comma()\n```\n\nBelow is the full list of token types. There's a corresponding helper for each of\nthem as well.\n\n```ts\n// types\nenum TokenType {\n  Err = 'err',\n  Ident = 'ident',\n  Dot = 'dot',\n  Operator = 'operator',\n  Conditional = 'conditional',\n  String = 'string',\n  Number = 'number',\n  Null = 'null',\n  BrackLeft = 'brackleft',\n  BrackRight = 'brackright',\n  ParenLeft = 'parenleft',\n  ParenRight = 'parenright',\n  Comma = 'comma',\n  EOS = 'eos' // End of statement (the final token)\n}\n\n// example helpers\nt.Err()\nt.Ident('someIdentifier')\nt.Dot()\nt.Operator('=')\nt.Conditional('or')\nt.String('\"Something Something Something\"') // note the extra \"\" quotes here\nt.Number('23') // Still needs to be a string\nt.Null()\nt.BrackLeft()\nt.BrackRight()\nt.ParenLeft()\nt.ParenRight()\nt.Comma()\nt.EOS()\n```\n\n### Unlexing\n\nIf you can lex, can you un-lex? Why of course! `fql-ts` comes with full\nsupport for turning tokens back into code.\n\nThis is pretty helpful for _generating_ FQL code, either through UI pieces\nor programatically.\n\n```js\nimport { lex, unlex } from '@segment/fql-ts'\n\nconst { tokens } = lex('some.code \u003e= 30')\nconst { code } = unlex(tokens)\n\ncode // 'some.code \u003e= 30' (give or take some whitespace)\n```\n\nIf we have errors during the unlexing, we'll return an `error` object, just like the lexer.\n\n```js\nconst { error } = unlex(dangerousErrorTokens)\n```\n\n### AST\n\nWe can create a typed AST from an array of tokens.\n\n```js\nimport { ast, lex } from '@segment/fql-ts'\n\nconst { tokens } = lex(`message = \"foo\"`)\nconst { node } = ast(tokens)\n```\n\nThe AST returns the _root_ node of the tree which will always have the `node.type` of `ROOT`.\n\nThe Tree structure type looks like this:\n\n```ts\n{\n  children: [],\n  type: \"expr\"\n}\n```\n\nLeaves are arrays of tokens, nodes are other nodes. The type is an enum defined with `AbstractSyntaxType`. If you're using typescript, they're defined as:\n\n```ts\nexport enum AbstractSyntaxType {\n  ROOT = 'root',\n  EXPR = 'expr',\n  PATH = 'path',\n  FUNC = 'func',\n  ERR = 'err',\n  CONDITIONAL = 'conditional',\n  OPERATOR = 'operator'\n}\n\ninterface ASTNode {\n  children: Array\u003cToken | ASTNode\u003e\n  type: AbstractSyntaxType\n}\n\ninterface Token {\n  type: TokenType\n  value: string\n}\n```\n\n#### Errors\n\nIf something went wrong in the parsing, the AST will return an error and the last node will be of type `ERR`:\n\n```js\nimport { ast, lex } from '@segment/fql-ts'\n\nconst { tokens } = lex(`message = `)\nconst { node, error } = ast(tokens)\n\nconsole.error(error) // \"ParserError: ...\"\n```\n\n#### All Tokens are wrapped by a Node\n\nFor consistency and parsing reasons, *there are no heterogenous children on nodes.* In other words, a node NEVER has BOTH a token and another node as a children.\n\nThis is impossible in FQL-TS:\n```\n// BAD\nnode\n - token \n - node\n   - token\n```\n\nInstead, tokens are wrapped in their own node:\n```\n// GOOD\nnode \n - node\n   - token\n - node\n   - token\n```\n\nThis is incredibly important to note in `conditional` (like \"and\" and \"or\") and `operator` nodes, as it's often easy to assume that FQL like `message.event = \"foo\"` might look like this:\n\n```\n// BAD\nnode(expr)\n  - node(path)\n    - ...\n  - token('=')\n  - node(string)\n    - ...\n```\n\nInstead, we'd see something like this:\n```\n// GOOD\nnode(expr)\n  - node(path)\n    - ...\n  - node(operator)\n    - token('=')\n  - node(string)\n    - ...\n```\n\n#### There and Back Again\n\nYou can go from an AST to tokens pretty easily with `astToTokens`:\n\n```js\nimport { astToTokens, lex, ast } from '@segment/fql-ts'\n\nconst { tokens } = lex(`message = \"foo\"`)\nconst { node } = ast(tokens)\n\nastToTokens(node) // same thing as tokens\n```\n\n## Contributing\n\nInstall the dependencies!\n\n```sh\nyarn\n```\n\nRun the tests!\n\n```sh\nyarn test --coverage\n```\n\nCreate a build!\n\n```sh\nyarn build\n```\n\n## License\n\nMIT License\n\nCopyright (c) 2021 Segment\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n\n## Contributing\n\nAll third party contributors acknowledge that any contributions they provide will be made under the same open source license that the open source project is provided under.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsegmentio%2Ffql-ts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsegmentio%2Ffql-ts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsegmentio%2Ffql-ts/lists"}