{"id":13578315,"url":"https://github.com/starwing/luautf8","last_synced_at":"2025-04-05T07:07:14.377Z","repository":{"id":13588821,"uuid":"16281562","full_name":"starwing/luautf8","owner":"starwing","description":"a utf-8 support module for Lua and LuaJIT.","archived":false,"fork":false,"pushed_at":"2024-01-08T07:11:06.000Z","size":676,"stargazers_count":387,"open_issues_count":9,"forks_count":66,"subscribers_count":23,"default_branch":"master","last_synced_at":"2024-02-16T07:36:56.783Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/starwing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2014-01-27T14:36:49.000Z","updated_at":"2024-01-31T13:49:30.000Z","dependencies_parsed_at":"2024-02-16T07:46:39.265Z","dependency_job_id":null,"html_url":"https://github.com/starwing/luautf8","commit_stats":{"total_commits":58,"total_committers":9,"mean_commits":6.444444444444445,"dds":0.3275862068965517,"last_synced_commit":"697b4a0bc95aa8c1bd82f1c875f1c1b2775a03f3"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/starwing%2Fluautf8","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/starwing%2Fluautf8/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/starwing%2Fluautf8/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/starwing%2Fluautf8/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/starwing","download_url":"https://codeload.github.com/starwing/luautf8/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247299832,"owners_count":20916190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:01:29.387Z","updated_at":"2025-04-05T07:07:14.349Z","avatar_url":"https://github.com/starwing.png","language":"C","funding_links":[],"categories":["C"],"sub_categories":[],"readme":"UTF-8 module for Lua 5.x\n========================\n[![Build Status](https://travis-ci.org/starwing/luautf8.svg?branch=master)](https://travis-ci.org/starwing/luautf8)\n[![Coverage Status](https://coveralls.io/repos/github/starwing/luautf8/badge.svg?branch=master)](https://coveralls.io/github/starwing/luautf8?branch=master)\n\nThis module adds UTF-8 support to Lua.\n\nIt uses data extracted from\n[Unicode Character Database](http://www.unicode.org/reports/tr44/),\nand tested on Lua 5.2.3, Lua 5.3.0 and LuaJIT.\n\nparseucd.lua is a pure Lua script which generates unidata.h, to support\nconverting characters and checking characters' category.\n\nIt is compatible with Lua's own string module and passes all\nstring and pattern matching tests in the Lua test suite[2].\n\nIt also adds some useful routines against UTF-8 features, such as:\n- a convenient interface to escape Unicode sequences in strings.\n- string insert/remove, since UTF-8 substring extraction may be expensive.\n- calculate Unicode width, useful when implementing e.g. console emulator.\n- a useful interface to translate Unicode offsets and byte offsets.\n- checking UTF-8 strings for validity and removing invalid byte sequences.\n- converting Unicode strings to normal form.\n\nNote that to avoid conflict with Lua5.3's built-in library 'utf8',\nthis library produces a file like lua-utf8.dll or lua-utf8.so. so use\nit like this:\n\n```lua\nlocal utf8 = require 'lua-utf8'\n```\n\nin your code :-(\n\n[2]: http://www.lua.org/tests/5.2/\n\n\nLuaRocks Installation\n---------------------\n`luarocks install luautf8`\n\nIt's now fully-compatible with Lua 5.3's utf8 library, so replacing this\nfile (and headers) with lutf8lib.c from the Lua 5.3 sources is also okay.\n\nUsage\n-----\n\nMany routines are the same as Lua's string module:\n- `utf8.byte`\n- `utf8.char`\n- `utf8.find`\n- `utf8.gmatch`\n- `utf8.gsub`\n- `utf8.len`\n- `utf8.lower`\n- `utf8.match`\n- `utf8.reverse`\n- `utf8.sub`\n- `utf8.upper`\n\nThe documentation of these functions can be found in the Lua manual[3].\n\n[3]: http://www.lua.org/manual/5.2/manual.html#6.4\n\n\nSome routines in string module needn't support Unicode:\n- `string.dump`\n- `string.format`\n- `string.rep`\n\nThey are NOT in utf8 module.\n\nSome routines are for compatibility with Lua 5.3's basic UTF-8 support library:\n- `utf8.offset`\n- `utf8.codepoint`\n- `utf8.codes`\n\nSee Lua5.3's manual for usage.\n\nSome routines are new, with some Unicode-spec functions:\n\n### utf8.escape(str) -\u003e utf8 string\nescape a str to UTF-8 format string. It supports several escape formats:\n\n * `%ddd` - which ddd is a decimal number at any length:\n   change Unicode code point to UTF-8 format.\n * `%{ddd}` - same as `%nnn` but has bracket around.\n * `%uddd` - same as `%ddd`, u stands Unicode\n * `%u{ddd}` - same as `%{ddd}`\n * `%xhhh` - hexadigit version of `%ddd`\n * `%x{hhh}` same as `%xhhh`.\n * `%?` - '?' stands for any other character: escape this character.\n\n#### Examples:\n```lua\nlocal u = utf8.escape\nprint(u\"%123%u123%{123}%u{123}%xABC%x{ABC}\")\nprint(u\"%%123%?%d%%u\")\n```\n\n\n### utf8.charpos(s[[, charpos], index]) -\u003e charpos, code point\nconvert UTF-8 position to byte offset.\nif only `index` is given, return byte offset of this UTF-8 char index.\nif both `charpos` and `index` is given, a new `charpos` will be\ncalculated, by adding/subtracting UTF-8 char `index` to current `charpos`.\nin all cases, it returns a new char position, and code point (a\nnumber) at this position.\n\n\n### utf8.next(s[, charpos[, index]]) -\u003e charpos, code point\niterate though the UTF-8 string s.\nIf only s is given, it can used as a iterator:\n```lua\nfor pos, code in utf8.next, \"utf8-string\" do\n   -- ...\nend\n```\nif only s and `charpos` are given, return the byte offset of the next codepoint\nin the string.\nif `charpos` and `index` are given, a new `charpos` will be calculated, by\nadding/subtracting UTF-8 char offset to current charpos.\nin all cases, it returns a new char position (in bytes), and code point\n(a number) at this position.\n\n\n### utf8.insert(s[, idx], substring) -\u003e new_string\ninsert a substring into s. If `idx` is given, insert the substring before\nthe char at this index; otherwise, substring will be concatenated onto s.\n`idx` can be negative.\n\n\n### utf8.remove(s[, start[, stop]]) -\u003e new_string\ndelete a substring in s. If neither `start` nor `stop` is given, delete the\nlast UTF-8 char in s, otherwise delete chars from `start` to the end of s. if\n`stop` is given, delete chars from `start` to `stop` (including `start` and `stop`).\n`start` and `stop` can be negative.\n\n\n### utf8.width(s[, ambi_is_double[, default_width]]) -\u003e width\ncalculate the width of UTF-8 string s. if `ambi_is_double` is given,\ncharacters with ambiguous width will be treated as having width 2.\nOtherwise, they will be treated as having width 1.\nthe width of fullwidth/doublewidth characters is 2, and the width of other\ncharacters is 1.\nif `default_width` is given, it will be used as the width of unprintable\ncharacters. (If you will replace unprintable characters with a placeholder,\npass its width as `default_width`.)\nif s is a code point, return the width of this code point.\n\n\n### utf8.widthindex(s, location[, ambi_is_double[, default_width]]) -\u003e idx, offset, width\nreturn the character index at given location in string s, where location is\nin width units. this is the inverse operation of utf8.width().\nif the requested location does not fall at a character boundary, `offset` will be\ngreater than 1; specifically, if the location is at the second column (middle)\nof a wide char, `offset` will be 2. the width of the character at idx is returned also.\n\n\n### utf8.title(s) -\u003e new_string\n### utf8.fold(s) -\u003e new_string\nconverts UTF-8 string s to title-case, or folded case (used for\ncase-insensitive comparison).\nif s is a number, it's treated as a code point and a converted code\npoint (number) is returned.\nutf8.lower/utf8.upper has the same extension.\n\n\n### utf8.ncasecmp(a, b) -\u003e [-1,0,1]\ncompare a and b without case, -1 means a \u003c b, 0 means a == b and 1 means a \u003e b.\n\n\n### utf8.isvalid(s) -\u003e boolean\ncheck whether s is a valid UTF-8 string or not.\n\n\n### utf8.clean(s[, replacement_string]) -\u003e cleaned_string, was_valid\nreplace any invalid UTF-8 byte sequences in s with the replacement string.\nif no replacement string is provided, the default is \"�\" (REPLACEMENT CHARACTER U+FFFD).\nnote that *any* number of consecutive invalid bytes will be replaced by a single copy of the replacement string.\nthe 2nd return value is true if the original string was already valid (meaning no replacements were made).\n\n\n### utf8.invalidoffset(s[, init]) -\u003e offset\nreturn the byte offset within s of the first invalid UTF-8 byte sequence.\n(1 is the first byte of the string.)\nif s is a valid UTF-8 string, return nil.\nthe optional numeric argument init specifies where to start the search; its default value is 1 and can be negative.\n\n\n### utf8.isnfc(s) -\u003e boolean\ncheck whether s is in Normal Form C or not.\n\"Normal Form C\" means that whenever possible, combining marks are combined with a preceding codepoint. For example, instead of U+0041 (LATIN CAPITAL LETTER A) U+00B4 (ACUTE ACCENT), an NFC string will use U+00C1 (LATIN CAPITAL LETTER A WITH ACUTE). Also, some deprecated codepoints are converted to the recommended replacements.\nsince the same sequence of characters can be represented in more than one way in Unicode, it is better to ensure strings are in Normal Form before comparing them.\nan error may be raised if s is not a valid UTF-8 string.\n\n\n### utf8.normalize_nfc(s) -\u003e normal_string, was_nfc\nconvert s to Normal Form C.\nthe 2nd return value is true if the original string was already in NFC (meaning no modifications were made).\nan error will be raised if s is not a valid UTF-8 string.\n\n\n### utf8.grapheme_indices(s[, start[, stop]]) -\u003e iterator\nreturn an iterator which yields the starting and ending byte index of each successive grapheme cluster in s. This range of bytes is inclusive of the endpoints, so the yielded values can be passed to `string.sub` to extract the grapheme cluster.\nif you provide `start` and `stop` byte indices, then the iterator will only cover the requested byte range. `start` and `stop` should fall on character boundaries, since an error will be raised if the requested byte range is not a valid UTF-8 string.\n```lua\nlocal i = 1\nfor from,to in utf8.grapheme_indices(s) do\n  print(\"grapheme cluster \"..i..\" is from byte \"..from..\" to byte \"..to)\n  i = i + 1\nend\n```\n\n\nImprovement needed\n------------------\n\n- add Lua 5.3 spec test-suite.\n- more test case.\n- grapheme-compose support, and affect in utf8.reverse and utf8.width\n\n\nLicense\n-------\nIt uses the same license as Lua: http://www.lua.org/license.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstarwing%2Fluautf8","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstarwing%2Fluautf8","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstarwing%2Fluautf8/lists"}