{"id":15368233,"url":"https://github.com/keithamus/csslex","last_synced_at":"2025-04-15T12:53:41.029Z","repository":{"id":176433224,"uuid":"654232629","full_name":"keithamus/csslex","owner":"keithamus","description":"A very small and very fast spec compliant css lexer","archived":false,"fork":false,"pushed_at":"2023-07-19T17:15:58.000Z","size":3356,"stargazers_count":47,"open_issues_count":2,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-01T15:13:40.554Z","etag":null,"topics":["css","lexer","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/keithamus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-15T17:05:19.000Z","updated_at":"2024-11-02T21:45:43.000Z","dependencies_parsed_at":null,"dependency_job_id":"49bf096e-7cc8-46c1-b201-1a374140cade","html_url":"https://github.com/keithamus/csslex","commit_stats":{"total_commits":10,"total_committers":1,"mean_commits":10.0,"dds":0.0,"last_synced_commit":"7182786dfcd1a553ac60f82d7a66bb4a48607d29"},"previous_names":["keithamus/csstokenize"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithamus%2Fcsslex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithamus%2Fcsslex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithamus%2Fcsslex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithamus%2Fcsslex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/keithamus","download_url":"https://codeload.github.com/keithamus/csslex/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248555089,"owners_count":21123825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["css","lexer","tokenizer"],"created_at":"2024-10-01T13:28:47.267Z","updated_at":"2025-04-15T12:53:41.008Z","avatar_url":"https://github.com/keithamus.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# csslex\n\nThis aims to be a very small and very fast spec compliant css lexer (or scanner\nor tokenizer depending on your favourite nomenclature).\n\nIt is not _the_ fastest, nor is it _the_ smallest, but it chooses to trade size\nfor speed and speed for correctness. Smaller lexers exist but they sacrifice\nspeed and correctness. Faster lexers exist but they sacrifice code size, and the\nability to easily run in the browser. More clearly written lexers exist, but\nusually at the sacrifice of both speed and size. For details on how fast, how\nsmall, and how correct, see below.\n\n## What is this good for?\n\nThe applications are quite limited. If you know what CSS is, and you know what a\nlexer/scanner/tokenizer is, then you probably know why you would want this. If\nyou don't know those things or how you could use them, then this probably won't\nbe helpful for you.\n\n## How do I import this?\n\nIf you're using node.js then running `npm i csslex` which will install the\ndependency in your `node_modules` folder. Then import it with:\n\n```ts\nimport { lex, types, value } from \"csslex\";\n```\n\nIf you're using Deno, then you can try the following line:\n\n```ts\nimport { lex, types, value } from \"https://deno.land/x/csslex/mod.ts\";\n```\n\nIf you're using a Browser, you can import using unpkg or esm.sh:\n\n```ts\nimport { lex, types, value } from \"https://esm.sh/csslex\";\n```\n\n## How do I use this?\n\nIf you can understand typescript, this will be helpful:\n\n```ts\ntype Token = [type: typeof types[keyof typeof types], start: number, end: number]\nlex(css: string): Generator\u003cToken\u003e\n```\n\nThe main `lex` function takes a css string, and creates an iterable of \"Tokens\".\nEach \"Token\" is a tuple 3 (an array always with 3 elements inside it). The first\nitem in the array is the number representing the type, the second is the start\nposition of that token in the css string, the second is the end of that token in\nthe string.\n\nSo for example:\n\n```js\nimport { lex, types, value } from \"https://esm.sh/csslex\";\nArray.from(lex(\"margin: 1px\"))[ // -\u003e output\n  ([types.IDENT, 0, 6],\n  [types.COLON, 6, 7],\n  [types.WHITESPACE, 7, 8][(types.DIMENSION, 8, 11)])\n];\n```\n\nIf you want to know the raw value of a token, simply take your original string\nand call `.slice(start, end)`. However you can also give the string and a token\ntuple to `value` which will also do extra things like normalise escape\ncharacters and give you structural values:\n\n```js\nimport { lex, types, value } from \"https://esm.sh/csslex\";\nvalue(\"margin: 1px\", [types.IDENT, 0, 6]) == \"margin\";\nvalue(\"margin: 1px\", [types.COLON, 6, 7]) == \":\";\nvalue(\"margin: 1px\", [types.DIMENSION, 8, 11]) ==\n  { type: \"integer\", value: 1, unit: \"px\" };\n```\n\n## Test Coverage\n\nThis uses [`css-tokenizer-tests`][1] which provides a set of difficult inputs\nintended to test the edge cases of the spec.\n\nIt also uses \"snapshot testing\" to avoid regressions, it tokenizes the\n[`postcss-parser-tests`][2] series of css files, as well as [`open-props`][3].\n\n[1]: https://github.com/romainmenke/css-tokenizer-tests\n[2]: https://www.npmjs.com/package/postcss-parser-tests/v/8.3.1\n[3]: https://github.com/argyleink/open-props\n\n## Spec Conformance\n\n[@romainmenke][4] maintains a comparison of\n[CSS tokenizers with scores pertaining to each][5]. `csslex` aims to always\nachieve a perfect score here, so if you visit the [scores page][5] an it does\nnot have a perfect score, please file an issue!\n\n[4]: https://github.com/romainmenke\n[5]: https://romainmenke.github.io/css-tokenizer-tests/\n\n## Size Differentials\n\nThis package aims to be the smallest minified css tokenizer codebase. Here's a\ncomparison of popular alternatives:\n\n|           Name            | Minified | Gzipped |\n| :-----------------------: | :------- | :------ |\n|   `@csstools/tokenizer`   | 4.1kb    | 1.1kb   |\n|      `csslex` (this)      | 4.7kb    | 1.9kb   |\n| `@csstools/css-tokenizer` | 15.5kb   | 3.4kb   |\n|      `css-tokenize`       | 19.1kb   | 5.7kb   |\n|        `parse-css`        | 16kb     | 4.1kb   |\n|        `css-tree`         | 157.9kb  | 45kb    |\n\n## Speed differentials\n\nYou can run `node bench.js` to get some benchmark numbers. Here's some I ran on\nthe machine I developed the library on:\n\n|          Name           | ops/sec                                |\n| :---------------------: | :------------------------------------- |\n|        css-tree         | 3,080 ops/sec ±0.43% (96 runs sampled) |\n|      csslex (this)      | 2,314 ops/sec ±0.45% (93 runs sampled) |\n| @csstools/css-tokenizer | 1,622 ops/sec ±0.76% (96 runs sampled) |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeithamus%2Fcsslex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkeithamus%2Fcsslex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeithamus%2Fcsslex/lists"}