https://github.com/flex-development/vfile-tokenizer

vfile utility to tokenize a file
https://github.com/flex-development/vfile-tokenizer

lexer parse tokenize vfile vfile-util

Last synced: 3 months ago
JSON representation

vfile utility to tokenize a file

• Host: GitHub
• URL: https://github.com/flex-development/vfile-tokenizer
• Owner: flex-development
• License: bsd-3-clause
• Created: 2024-06-16T04:21:59.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2025-03-17T04:56:56.000Z (4 months ago)
• Last Synced: 2025-03-17T05:27:38.589Z (4 months ago)
• Topics: lexer, parse, tokenize, vfile, vfile-util
• Language: TypeScript
• Homepage: https://github.com/flex-development/vfile-tokenizer
• Size: 2.07 MB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • Contributing: CONTRIBUTING.md
  • Funding: .github/funding.yml
  • License: LICENSE.md
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
# vfile-tokenizer

[![github release](https://img.shields.io/github/v/release/flex-development/vfile-tokenizer.svg?include_prereleases\&sort=semver)](https://github.com/flex-development/vfile-tokenizer/releases/latest)

[![npm](https://img.shields.io/npm/v/@flex-development/vfile-tokenizer.svg)](https://npmjs.com/package/@flex-development/vfile-tokenizer)

[![codecov](https://codecov.io/gh/flex-development/vfile-tokenizer/graph/badge.svg?token=iA1BvaucoZ)](https://codecov.io/gh/flex-development/vfile-tokenizer)

[![module type: esm](https://img.shields.io/badge/module%20type-esm-brightgreen)](https://github.com/voxpelli/badges-cjs-esm)

[![license](https://img.shields.io/github/license/flex-development/vfile-tokenizer.svg)](LICENSE.md)

[![conventional commits](https://img.shields.io/badge/-conventional%20commits-fe5196?logo=conventional-commits\&logoColor=ffffff)](https://conventionalcommits.org)

[![typescript](https://img.shields.io/badge/-typescript-3178c6?logo=typescript\&logoColor=ffffff)](https://typescriptlang.org)

[![vitest](https://img.shields.io/badge/-vitest-6e9f18?style=flat\&logo=vitest\&logoColor=ffffff)](https://vitest.dev)

[![yarn](https://img.shields.io/badge/-yarn-2c8ebb?style=flat\&logo=yarn\&logoColor=ffffff)](https://yarnpkg.com)

[vfile][] utility to tokenize a file

## Contents

- [What is this?](#what-is-this)

- [When should I use this?](#when-should-i-use-this)

- [Install](#install)

- [Use](#use)

- [API](#api)

- [Types](#types)

- [Related](#related)

- [Contribute](#contribute)

## What is this?

**TODO**: what is this?

## When should I use this?

**TODO**: when should I use this?

## Install

This package is [ESM only][esm].

In Node.js with [yarn][]:

```sh

yarn add @flex-development/vfile-tokenizer

```



  

    See Git - Protocols | Yarn

     for details regarding installing from Git.

  



In Deno with [`esm.sh`][esmsh]:

```ts

import {} from 'https://esm.sh/@flex-development/vfile-tokenizer'

```

In browsers with [`esm.sh`][esmsh]:

```html

  import {} from 'https://esm.sh/@flex-development/vfile-tokenizer'

```

## Use

**TODO**: use

## API

**TODO**: api

## Types

This package is fully typed with [TypeScript][].

## Related

- [`vfile-location`][vfile-location] — [vfile][] utility to convert between point and offset based locations

- [`vfile-reader`][vfile-reader] — [vfile][] utility to read from a file

## Contribute

See [`CONTRIBUTING.md`](CONTRIBUTING.md).

This project has a [code of conduct](CODE_OF_CONDUCT.md). By interacting with this repository, organization, or

community you agree to abide by its terms.

[esm]: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

[esmsh]: https://esm.sh

[typescript]: https://www.typescriptlang.org

[vfile-location]: https://github.com/flex-development/vfile-location

[vfile-reader]: https://github.com/flex-development/vfile-reader

[vfile]: https://github.com/vfile/vfile

[yarn]: https://yarnpkg.com