{"id":48298613,"url":"https://github.com/defname/str8","last_synced_at":"2026-04-04T23:40:04.781Z","repository":{"id":322101681,"uuid":"1088215745","full_name":"defname/str8","owner":"defname","description":"str8 - A memory optimized, UTF-8 friendly string library","archived":false,"fork":false,"pushed_at":"2025-11-14T22:13:31.000Z","size":148,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-15T00:18:10.254Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/defname.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-02T14:44:23.000Z","updated_at":"2025-11-08T13:17:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/defname/str8","commit_stats":null,"previous_names":["defname/str8"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/defname/str8","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defname%2Fstr8","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defname%2Fstr8/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defname%2Fstr8/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defname%2Fstr8/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/defname","download_url":"https://codeload.github.com/defname/str8/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defname%2Fstr8/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31419538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T20:09:54.854Z","status":"ssl_error","status_checked_at":"2026-04-04T20:09:44.350Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-04T23:40:04.545Z","updated_at":"2026-04-04T23:40:04.718Z","avatar_url":"https://github.com/defname.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# str8 - UTF-8 friendly string library\n\nWork in progress...\n\n## Goals\n\n- Minimal overhead\n- Fast access to single characters\n- Full compatibility with C strings\n- Decreased memory allocations\n- Maybe providing a string arena for multiple small edits\n- Const correctness (so no lazy-caching or anything like this)\n\n## Strategy\n\n- Preserve additional memory for adding strings without reallocation\n- Header before the actual string that stores:\n    - Size in bytes\n    - Capacity available in bytes\n    - Length in characters\n    - A `checkpoints` list holding the number of characters up to specified positions\n- Use different header versions for minimal memory use\n- Use a variable size per entry of the checkpoints list\n\n## Header Layout\n\n### Type 0\n\nFor strings up to 31 bytes.\n\n```\n│ 5 Bits                                    │ 3 Bits      │\n┌───────────────────────────────────────────┬─────────────┐\n│ size                                      │ Type 0      │\n└───────────────────────────────────────────┴─────────────┘\n```\n\n### Other Types\n\nFor strings larger than 31 bytes, with increasing capacity:\n\n```\n                                      Type 1    Type 2    Type 4    Type 8\n              ╭ ┌───────────────┐ \n              │ │ checkpoints   │\nonly used     │ ┆               ┆\nif not        ⎨ ┆               ┆\npure ASCII    │ │               │\n              │ ├───────────────┤\n              │ │ length        │     1 Byte    2 Byte    4 Byte    8 Byte\n              ╰ ├───────────────┤\n                │ capacity      │     1 Byte    2 Byte    4 Byte    8 Byte\n                ├───────────────┤\n                │ size          │     1 Byte    2 Byte    4 Byte    8 Byte\n                ├───────────────┤\n                │ type          │     1 Byte    1 Byte    1 Byte    1 Byte\n                └───────────────┘\n```\n\nThe `type` byte contains multiple pieces of information, interpreted contextually:\n- The lowest 3 bits (`byte \u0026 0x07`) always store the string type (`TYPE0`, `TYPE1`, etc.).\n- **For `TYPE0` strings:** Bits 3-7 store the string's size.\n- **For `TYPE1` and higher strings:** The highest bit (`type \u0026 0x80`) is a flag. If not set, the string is pure ASCII, and the `length` field and `checkpoints` list are omitted to save space.\n\n## Checkpoints List: A Packed, Variable-Size Structure\n\nTo maximize memory efficiency, the `checkpoints` list is not a simple array of a fixed type. Instead, it's a highly optimized, packed data structure where the size of each entry is variable.\n\n**The Principle:** The size of an entry is determined by its *position* (index) in the list, not by the overall string's type. For each entry, the smallest possible integer type (`uint16_t`, `uint32_t`, `uint64_t`) is used that can store the maximum possible character count at that checkpoint's byte position.\n\n**How it Works and Transition Points:**\n\nA checkpoint at index `i` corresponds to a byte position of `(i + 1) * CHECKPOINTS_GRANULARITY` (where `CHECKPOINTS_GRANULARITY` is 512 bytes). The character count at this position can never be greater than the byte position itself. This leads to fixed points where the entry size changes:\n\n*   **`uint16_t` Zone:** The maximum value for a `uint16_t` is 65,535. The last checkpoint whose character count can safely be stored in a `uint16_t` is at index 126 (corresponding to byte position `127 * 512 = 65,024`). Therefore, the **first 127 entries** (indices 0-126) of the list are always stored as `uint16_t`.\n\n*   **`uint32_t` Zone:** The 128th entry (index 127) corresponds to byte position `128 * 512 = 65,536`. This character count requires at least a `uint32_t`. All subsequent entries are stored as `uint32_t` until their corresponding byte position exceeds the `uint32_t` limit.\n\n*   **`uint64_t` Zone:** For extremely large strings, entries will eventually be stored as `uint64_t`.\n\n**Layout Example:**\n\n```\n\u003c-- 127 entries of 2 Bytes --\u003e | \u003c--  N entries of 4 Bytes  --\u003e | \u003c-- M entries of 8 Bytes --\u003e\n┌────┬────┬.............┬──────┬────────┬..............┬────────┬──────────┬...\n│ E0 │ E1 │.............│ E126 │  E127  │..............│  EN-1  │    EN    │... \n└────┴────┴.............┴──────┴────────┴..............┴────────┴──────────┴...\n```\n\n**Advantages of this Design:**\n\n1.  **Maximum Memory Efficiency:** It uses the absolute minimum required memory for the `checkpoints` list.\n2.  **Efficient Reallocation:** When appending to a string, the existing part of the `checkpoints` list can be copied with a single `memcpy`, as its layout is static and does not change when the string's main `type` is promoted.\n\n## Further Tweaks\n\nUse SIMD instructions to analyze the string and build the checkpoints list.\n\n## Interface\n\n```C\ntypedef char* str8;\n\nstr8 str8new(const char *s);\nvoid str8free(str8 s);\n\nsize_t str8len(const str8 s);\nsize_t str8size(const str8 s);\nsize_t str8cap(const str8 s);\n\nstr8 str8append(str8 s1, const char *s2);\n....\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefname%2Fstr8","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdefname%2Fstr8","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefname%2Fstr8/lists"}