{"id":21028762,"url":"https://github.com/jgmdev/lite-xl-tokenizer","last_synced_at":"2025-07-12T09:33:37.518Z","repository":{"id":150318538,"uuid":"581447029","full_name":"jgmdev/lite-xl-tokenizer","owner":"jgmdev","description":"Native tokenizer implementation by Adam, extracted from https://github.com/adamharrison/lite-xl-simplified/tree/c-tokenizer","archived":false,"fork":false,"pushed_at":"2022-12-25T20:30:28.000Z","size":22,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-13T19:16:13.490Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jgmdev.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-23T07:59:57.000Z","updated_at":"2022-12-23T08:02:08.000Z","dependencies_parsed_at":"2023-04-04T12:48:08.508Z","dependency_job_id":null,"html_url":"https://github.com/jgmdev/lite-xl-tokenizer","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/jgmdev/lite-xl-tokenizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jgmdev%2Flite-xl-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jgmdev%2Flite-xl-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jgmdev%2Flite-xl-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jgmdev%2Flite-xl-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jgmdev","download_url":"https://codeload.github.com/jgmdev/lite-xl-tokenizer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jgmdev%2Flite-xl-tokenizer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264972152,"owners_count":23691375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T11:58:54.777Z","updated_at":"2025-07-12T09:33:37.497Z","avatar_url":"https://github.com/jgmdev.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lite XL C-Tokenizer Plugin\n\nThis repository provides a standalone copy of the work done by **Adam** on the\nnative implementation of the Lua tokenizer found on his simplified branch\nhttps://github.com/adamharrison/lite-xl-simplified/tree/c-tokenizer for\ntesting purposes.\n\n## Building\n\nYou will need to have meson and a working build environment for your operating\nsystem. Then, to build just execute the following commands:\n\n```sh\nmeson setup build\nmeson compile -C build\n```\n\nBy default we configured meson to download Lua and pcre2 and statically link\nto it. To disable this behavior and instead link against system libraries,\nuse `--wrap-mode default`\n\n```sh\nmeson setup --wrap-mode default build\n```\n\nAlso you can link to luajit instead by adding `-Djit=true`\n\n```sh\nmeson setup -Djit=true build\n```\n\n## Installation\n\nTo install just copy the generated library file to your libraries directory:\n\n```sh\ncp build/tokenizer.so ~/.config/lite-xl/libraries/\n```\n\nThen install the Lua plugin to take advantage of the native tokenizer plugin:\n\n```sh\ncp plugins/ctokenizer.lua ~/.config/lite-xl/plugins/\n```\n\n## Usage:\n\nYou can enable and disable the native tokenizer from the settings or by\nmodifying the value of `config.plugins.ctokenizer.enabled` to `true` or `false`.\n\nAlso, you can enable `config.plugins.ctokenizer.log_time` to measure the time\nit takes to tokenize a range of lines.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjgmdev%2Flite-xl-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjgmdev%2Flite-xl-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjgmdev%2Flite-xl-tokenizer/lists"}