{"id":19653007,"url":"https://github.com/csstools/tokenizer","last_synced_at":"2025-04-28T17:31:10.649Z","repository":{"id":45522964,"uuid":"258403088","full_name":"csstools/tokenizer","owner":"csstools","description":"Tokenize CSS according to the CSS Syntax","archived":false,"fork":false,"pushed_at":"2023-12-28T16:29:50.000Z","size":1518,"stargazers_count":66,"open_issues_count":0,"forks_count":5,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-04-14T05:31:43.958Z","etag":null,"topics":["css","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/csstools.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-04-24T04:13:48.000Z","updated_at":"2024-03-26T00:52:59.000Z","dependencies_parsed_at":"2024-01-06T01:05:29.066Z","dependency_job_id":null,"html_url":"https://github.com/csstools/tokenizer","commit_stats":{"total_commits":23,"total_committers":4,"mean_commits":5.75,"dds":"0.13043478260869568","last_synced_commit":"d8a669c3b36343888f939ff4ffdc1d0fb51c1b2c"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csstools%2Ftokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csstools%2Ftokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csstools%2Ftokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csstools%2Ftokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/csstools","download_url":"https://codeload.github.com/csstools/tokenizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224124924,"owners_count":17259746,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["css","tokenizer"],"created_at":"2024-11-11T15:12:54.393Z","updated_at":"2024-11-11T15:12:55.359Z","avatar_url":"https://github.com/csstools.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CSS Tokenizer\n\n[\u003cimg alt=\"npm version\" src=\"https://img.shields.io/npm/v/@csstools/tokenizer.svg\" height=\"20\"\u003e](https://www.npmjs.com/package/@csstools/tokenizer)\n[\u003cimg alt=\"build status\" src=\"https://img.shields.io/travis/csstools/tokenizer/main.svg\" height=\"20\"\u003e](https://travis-ci.org/github/csstools/tokenizer)\n[\u003cimg alt=\"code coverage\" src=\"https://img.shields.io/codecov/c/github/csstools/tokenizer\" height=\"20\"\u003e](https://codecov.io/gh/csstools/tokenizer)\n[\u003cimg alt=\"issue tracker\" src=\"https://img.shields.io/github/issues/csstools/tokenizer.svg\" height=\"20\"\u003e](https://github.com/csstools/tokenizer/issues)\n[\u003cimg alt=\"pull requests\" src=\"https://img.shields.io/github/issues-pr/csstools/tokenizer.svg\" height=\"20\"\u003e](https://github.com/csstools/tokenizer/pulls)\n[\u003cimg alt=\"support chat\" src=\"https://img.shields.io/badge/support-chat-blue.svg\" height=\"20\"\u003e](https://gitter.im/postcss/postcss)\n\nThis tools lets you tokenize CSS according to the [CSS Syntax Specification](https://drafts.csswg.org/css-syntax/).\nTokenizing CSS is separating a string of CSS into its smallest, semantic parts — otherwise known as tokens.\n\nThis tool is intended to be used in other tools on the front and back end. It seeks to maintain:\n\n- 100% compliance with the CSS syntax specification. ✨\n- 100% code coverage. 🦺\n- 100% static typing. 💪\n- 1kB maximum contribution size. 📦\n- Superior quality over Shark P. 🦈\n\n## Usage\n\nAdd the [CSS tokenizer](https://github.com/csstools/tokenizer) to your project:\n\n```sh\nnpm install @csstools/tokenizer\n```\n\nTokenize CSS in JavaScript:\n\n```js\nimport { tokenize } from '@csstools/tokenizer'\n\nfor (const token of tokenize(cssText)) {\n  console.log(token) // logs an individual CSSToken\n}\n```\n\nTokenize CSS in _classical_ NodeJS:\n\n```js\nconst { tokenizer } = require('@csstools/tokenizer')\n\nlet iterator = tokenizer(cssText), iteration\n\nwhile (!(iteration = iterator()).done) {\n  console.log(iteration.value) // logs an individual CSSToken\n}\n```\n\nTokenize CSS in client-side scripts:\n\n```html\n\u003cscript type=\"module\"\u003e\n\nimport { tokenize } from 'https://unpkg.com/@csstools/tokenizer?module'\n\nfor (const token of tokenize(cssText)) {\n  console.log(token) // logs an individual CSSToken\n}\n\n\u003c/script\u003e\n```\n\nTokenize CSS in _classical_ client-side scripts:\n\n```html\n\u003cscript src=\"http://unpkg.com/@csstools/tokenizer\"\u003e\u003c/script\u003e\n\u003cscript\u003e\n\nconst tokens = Array.from(tokenizeCSS(cssText)) // an array of CSSTokens\n\n\u003c/script\u003e\n```\n\n### Serialize tokens\n\n```js\nimport { tokenize } from '@csstools/tokenizer'\n\nlet cssOutput = '';\nfor (const token of tokenize(cssText)) {\n  // mutate some tokens\n\n  cssOutput += token.lead + token.data + token.tail\n}\n\nconsole.log(cssOutput) // logs the CSS string\n```\n\n## How it works\n\nThe CSS tokenizer separates a string of CSS into tokens.\n\n```ts\ninterface CSSToken {\n  /** Position in the string at which the token was retrieved. */\n  tick: number\n\n  /** Number identifying the kind of token. */\n  type:\n    | 1 // Symbol\n    | 2 // Comment\n    | 3 // Space\n    | 4 // Word\n    | 5 // Function\n    | 6 // Atword\n    | 7 // Hash\n    | 8 // String\n    | 9 // Number\n  \n  /** Code, like the character code of a symbol, or the character code of the opening parenthesis of a function. */\n  code: number\n\n  /** Lead, like the opening of a comment, the quotation mark of a string, or the name of a function. */\n  lead: string,\n\n  /** Data, like the numbers before a unit, the word after an at-sign, or the opening parenthesis of a Function. */\n  data: string,\n\n  /** Tail, like the unit after a number, or the closing of a comment. */\n  tail: string,\n}\n```\n\nAs an example, the CSS string `@media` would become a **Atword** token where `@` and `media` are recognized as distinct parts of that token. As another example, the CSS string `5px` would become a **Number** token where `5` and `px` are recognized as distinct parts of that token. As a final example, the string `5px 10px` would become 3 tokens; the **Number** as mentioned before (`5px`), a **Space** token that represents a single space (` `), and then another **Number** token (`10px`).\n\n## Benchmarks\n\nAs of August 23, 2021, these benchmarks were averaged from my local machine:\n\n```\nBenchmark: Tailwind CSS\n  ┌────────────────────────────────────────────────────┬───────┬────────┬────────┐\n  │                      (index)                       │  ms   │ ms/50k │ tokens │\n  ├────────────────────────────────────────────────────┼───────┼────────┼────────┤\n  │ CSSTree 1 x 35.04 ops/sec ±6.55% (64 runs sampled) │ 28.54 │  1.51  │ 946205 │\n  │ CSSTree 2 x 41.76 ops/sec ±7.57% (58 runs sampled) │ 23.95 │  1.27  │ 946205 │\n  │ PostCSS 8 x 14.18 ops/sec ±3.31% (40 runs sampled) │ 70.54 │  3.77  │ 935282 │\n  │ Tokenizer x 17.40 ops/sec ±0.98% (48 runs sampled) │ 57.48 │  3.04  │ 946206 │\n  └────────────────────────────────────────────────────┴───────┴────────┴────────┘\n\nBenchmark: Bootstrap\n  ┌───────────────────────────────────────────────────┬──────┬────────┬────────┐\n  │                      (index)                      │  ms  │ ms/50k │ tokens │\n  ├───────────────────────────────────────────────────┼──────┼────────┼────────┤\n  │ CSSTree 1 x 600 ops/sec ±0.87% (96 runs sampled)  │ 1.67 │  1.41  │ 59236  │\n  │ CSSTree 2 x 695 ops/sec ±0.08% (100 runs sampled) │ 1.44 │  1.21  │ 59236  │\n  │ PostCSS 8 x 432 ops/sec ±0.94% (94 runs sampled)  │ 2.31 │  2.26  │ 51170  │\n  │ Tokenizer x 288 ops/sec ±0.40% (93 runs sampled)  │ 3.48 │  2.93  │ 59237  │\n  └───────────────────────────────────────────────────┴──────┴────────┴────────┘\n```\n\n## Development\n\nYou wanna take a deeper dive? Awesome! Here are a few useful development commands.\n\n### npm run build\n\nThe **build** command creates all the files needed to run this tool in many different JavaScript environments.\n\n```sh\nnpm run build\n```\n\n### npm run benchmark\n\nThe **benchmark** command builds the project and then tests its performance as compared to [PostCSS].\nThese benchmarks are run against [Boostrap] and [Tailwind CSS].\n\n```sh\nnpm run benchmark\n```\n\n### npm run test\n\nThe **test** command tests the coverage and accuracy of the tokenizer.\n\nAs of September 26, 2020, this tokenizer has 100% test coverage:\n\n```sh\nnpm run test\n```\n\n[Boostrap]: https://getbootstrap.com\n[PostCSS]: https://postcss.org\n[Tailwind CSS]: https://tailwindcss.com\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcsstools%2Ftokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcsstools%2Ftokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcsstools%2Ftokenizer/lists"}