{"id":26989996,"url":"https://github.com/flex-development/fsm-tokenizer","last_synced_at":"2025-04-03T21:32:11.098Z","repository":{"id":245332991,"uuid":"815760853","full_name":"flex-development/fsm-tokenizer","owner":"flex-development","description":"finite state machine tokenizer","archived":false,"fork":false,"pushed_at":"2025-03-26T00:13:11.000Z","size":2519,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-26T00:19:46.534Z","etag":null,"topics":["finite-state-machine","fsm","lexer","parse","state","tokenize","tokenizer"],"latest_commit_sha":null,"homepage":"https://github.com/flex-development/fsm-tokenizer","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flex-development.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/funding.yml","license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["flex-development"]}},"created_at":"2024-06-16T04:21:59.000Z","updated_at":"2025-03-26T00:12:18.000Z","dependencies_parsed_at":"2025-03-10T05:21:05.101Z","dependency_job_id":"4ad6392a-a3f3-48e6-aef2-004eb69cad62","html_url":"https://github.com/flex-development/fsm-tokenizer","commit_stats":null,"previous_names":["flex-development/vfile-lexer","flex-development/vfile-tokenizer","flex-development/fsm-tokenizer"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flex-development%2Ffsm-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flex-development%2Ffsm-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flex-development%2Ffsm-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flex-development%2Ffsm-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flex-development","download_url":"https://codeload.github.com/flex-development/fsm-tokenizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247082768,"owners_count":20880713,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["finite-state-machine","fsm","lexer","parse","state","tokenize","tokenizer"],"created_at":"2025-04-03T21:30:29.567Z","updated_at":"2025-04-03T21:32:11.082Z","avatar_url":"https://github.com/flex-development.png","language":"TypeScript","funding_links":["https://github.com/sponsors/flex-development"],"categories":[],"sub_categories":[],"readme":"# fsm-tokenizer\n\n[![github release](https://img.shields.io/github/v/release/flex-development/fsm-tokenizer.svg?include_prereleases\\\u0026sort=semver)](https://github.com/flex-development/fsm-tokenizer/releases/latest)\n[![npm](https://img.shields.io/npm/v/@flex-development/fsm-tokenizer.svg)](https://npmjs.com/package/@flex-development/fsm-tokenizer)\n[![codecov](https://codecov.io/gh/flex-development/fsm-tokenizer/graph/badge.svg?token=iA1BvaucoZ)](https://codecov.io/gh/flex-development/fsm-tokenizer)\n[![module type: esm](https://img.shields.io/badge/module%20type-esm-brightgreen)](https://github.com/voxpelli/badges-cjs-esm)\n[![license](https://img.shields.io/github/license/flex-development/fsm-tokenizer.svg)](LICENSE.md)\n[![conventional commits](https://img.shields.io/badge/-conventional%20commits-fe5196?logo=conventional-commits\\\u0026logoColor=ffffff)](https://conventionalcommits.org)\n[![typescript](https://img.shields.io/badge/-typescript-3178c6?logo=typescript\\\u0026logoColor=ffffff)](https://typescriptlang.org)\n[![vitest](https://img.shields.io/badge/-vitest-6e9f18?style=flat\\\u0026logo=vitest\\\u0026logoColor=ffffff)](https://vitest.dev)\n[![yarn](https://img.shields.io/badge/-yarn-2c8ebb?style=flat\\\u0026logo=yarn\\\u0026logoColor=ffffff)](https://yarnpkg.com)\n\nfinite state machine tokenizer\n\n## Contents\n\n- [What is this?](#what-is-this)\n- [When should I use this?](#when-should-i-use-this)\n- [Install](#install)\n- [Use](#use)\n- [API](#api)\n- [Types](#types)\n- [Contribute](#contribute)\n\n## What is this?\n\nThis package is a tokenization utility. The underlying tokenizer is implemented as a state machine that produces tokens\nwith positional information, as well as any other fields attached when the token was opened. Tokens are turned into\nevents, with the tokenizer calling any defined resolvers in the process.\n\n## When should I use this?\n\nThis package can be used to tokenize a [file][vfile], `Uint8Array`, string, or a list containing these values,\nlike command-line arguments. Developers familiar with [micromark][] will find that tokenizers and constructs (objects\nused to define how to parse text) in this package work similarly.\n\n## Install\n\nThis package is [ESM only][esm].\n\nIn Node.js with [yarn][]:\n\n```sh\nyarn add @flex-development/fsm-tokenizer\n```\n\n\u003cblockquote\u003e\n  \u003csmall\u003e\n    See \u003ca href='https://yarnpkg.com/protocol/git'\u003eGit - Protocols | Yarn\u003c/a\u003e\n    \u0026nbsp;for details regarding installing from Git.\n  \u003c/small\u003e\n\u003c/blockquote\u003e\n\nIn Deno with [`esm.sh`][esmsh]:\n\n```ts\nimport { tokenize } from 'https://esm.sh/@flex-development/fsm-tokenizer'\n```\n\nIn browsers with [`esm.sh`][esmsh]:\n\n```html\n\u003cscript type=\"module\"\u003e\n  import { tokenize } from 'https://esm.sh/@flex-development/fsm-tokenizer'\n\u003c/script\u003e\n```\n\n## Use\n\n**TODO**: use\n\n## API\n\n**TODO**: api\n\n\u003e Please refer to the [source code](./src/) for documentation.\n\n## Types\n\nThis package is fully typed with [TypeScript][].\n\n## Contribute\n\nSee [`CONTRIBUTING.md`](CONTRIBUTING.md).\n\nThis project has a [code of conduct](CODE_OF_CONDUCT.md). By interacting with this repository, organization, or\ncommunity you agree to abide by its terms.\n\n[esm]: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c\n\n[esmsh]: https://esm.sh\n\n[micromark]: https://github.com/micromark/micromark\n\n[typescript]: https://www.typescriptlang.org\n\n[vfile]: https://github.com/vfile/vfile\n\n[yarn]: https://yarnpkg.com\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflex-development%2Ffsm-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflex-development%2Ffsm-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflex-development%2Ffsm-tokenizer/lists"}