https://github.com/yeslogic/unicode-general-category
Fast lookup of the Unicode General Category property for `char` in Rust.
https://github.com/yeslogic/unicode-general-category
general-category rust ucd unicode
Last synced: 11 months ago
JSON representation
Fast lookup of the Unicode General Category property for `char` in Rust.
- Host: GitHub
- URL: https://github.com/yeslogic/unicode-general-category
- Owner: yeslogic
- License: apache-2.0
- Created: 2019-12-05T05:16:37.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-10-30T01:49:24.000Z (over 1 year ago)
- Last Synced: 2024-10-30T03:59:02.382Z (over 1 year ago)
- Topics: general-category, rust, ucd, unicode
- Language: Rust
- Size: 73.2 KB
- Stars: 16
- Watchers: 14
- Forks: 6
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
unicode-general-category
========================
Fast lookup of the Unicode General Category property for `char` in Rust using
Unicode 16.0 data. This crate is no-std compatible.
Usage
-----
```rust
use unicode_general_category::{get_general_category, GeneralCategory};
fn main() {
assert_eq!(get_general_category('A'), GeneralCategory::UppercaseLetter);
}
```
Performance & Implementation Notes
----------------------------------
[ucd-generate] is used to generate `tables.rs`. A build script (`build.rs`)
compiles this into a two level look up table. The look up time is constant as
it is just indexing into two arrays.
The two level approach maps a code point to a block, then to a position within
a block. The allows the second level of block to be deduplicated, saving space.
The code is parameterised over the block size, which must be a power of 2. The
value in the build script is optimal for the data set.
This approach trades off some space for faster lookups. The tables take up
about 45KiB. Benchmarks showed this approach to be ~5–10× faster than the
typical binary search approach.
It's possible there are further optimisations that could be made to eliminate
some runs of repeated values in the first level array.
[ucd-generate]: https://github.com/yeslogic/ucd-generate