{"id":13439262,"url":"https://github.com/hsivonen/encoding_rs","last_synced_at":"2025-05-14T01:02:26.237Z","repository":{"id":41333614,"uuid":"48985043","full_name":"hsivonen/encoding_rs","owner":"hsivonen","description":"A Gecko-oriented implementation of the Encoding Standard in Rust","archived":false,"fork":false,"pushed_at":"2024-11-13T16:38:58.000Z","size":4819,"stargazers_count":405,"open_issues_count":21,"forks_count":56,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-05-03T01:57:17.211Z","etag":null,"topics":["charset","encoding","rust","unicode","web"],"latest_commit_sha":null,"homepage":"https://docs.rs/encoding_rs/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hsivonen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-01-04T08:47:55.000Z","updated_at":"2025-04-19T04:58:25.000Z","dependencies_parsed_at":"2024-04-09T20:34:48.236Z","dependency_job_id":"955f003a-dd97-41cd-a1f9-e861e1858809","html_url":"https://github.com/hsivonen/encoding_rs","commit_stats":{"total_commits":918,"total_committers":22,"mean_commits":41.72727272727273,"dds":0.05119825708060999,"last_synced_commit":"ca7487a568e06df610c4696b43fe76c935b8e957"},"previous_names":[],"tags_count":65,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsivonen%2Fencoding_rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsivonen%2Fencoding_rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsivonen%2Fencoding_rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsivonen%2Fencoding_rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hsivonen","download_url":"https://codeload.github.com/hsivonen/encoding_rs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254043221,"owners_count":22004912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["charset","encoding","rust","unicode","web"],"created_at":"2024-07-31T03:01:12.457Z","updated_at":"2025-05-14T01:02:26.183Z","avatar_url":"https://github.com/hsivonen.png","language":"Rust","readme":"# encoding_rs\n\n[![Build Status](https://github.com/hsivonen/encoding_rs/actions/workflows/ci.yml/badge.svg)](https://github.com/hsivonen/encoding_rs/actions/workflows/ci.yml)\n[![crates.io](https://img.shields.io/crates/v/encoding_rs.svg)](https://crates.io/crates/encoding_rs)\n[![docs.rs](https://docs.rs/encoding_rs/badge.svg)](https://docs.rs/encoding_rs/)\n\nencoding_rs an implementation of the (non-JavaScript parts of) the\n[Encoding Standard](https://encoding.spec.whatwg.org/) written in Rust.\n\nThe Encoding Standard defines the Web-compatible set of character encodings,\nwhich means this crate can be used to decode Web content. encoding_rs is\nused in Gecko starting with Firefox 56. Due to the notable overlap between\nthe legacy encodings on the Web and the legacy encodings used on Windows,\nthis crate may be of use for non-Web-related situations as well; see below\nfor links to adjacent crates.\n\nAdditionally, the `mem` module provides various operations for dealing with\nin-RAM text (as opposed to data that's coming from or going to an IO boundary).\nThe `mem` module is a module instead of a separate crate due to internal\nimplementation detail efficiencies.\n\n## Functionality\n\nDue to the Gecko use case, encoding_rs supports decoding to and encoding from\nUTF-16 in addition to supporting the usual Rust use case of decoding to and\nencoding from UTF-8. Additionally, the API has been designed to be FFI-friendly\nto accommodate the C++ side of Gecko.\n\nSpecifically, encoding_rs does the following:\n\n* Decodes a stream of bytes in an Encoding Standard-defined character encoding\n  into valid aligned native-endian in-RAM UTF-16 (units of `u16` / `char16_t`).\n* Encodes a stream of potentially-invalid aligned native-endian in-RAM UTF-16\n  (units of `u16` / `char16_t`) into a sequence of bytes in an Encoding\n  Standard-defined character encoding as if the lone surrogates had been\n  replaced with the REPLACEMENT CHARACTER before performing the encode.\n  (Gecko's UTF-16 is potentially invalid.)\n* Decodes a stream of bytes in an Encoding Standard-defined character\n  encoding into valid UTF-8.\n* Encodes a stream of valid UTF-8 into a sequence of bytes in an Encoding\n  Standard-defined character encoding. (Rust's UTF-8 is guaranteed-valid.)\n* Does the above in streaming (input and output split across multiple\n  buffers) and non-streaming (whole input in a single buffer and whole\n  output in a single buffer) variants.\n* Avoids copying (borrows) when possible in the non-streaming cases when\n  decoding to or encoding from UTF-8.\n* Resolves textual labels that identify character encodings in\n  protocol text into type-safe objects representing the those encodings\n  conceptually.\n* Maps the type-safe encoding objects onto strings suitable for\n  returning from `document.characterSet`.\n* Validates UTF-8 (in common instruction set scenarios a bit faster for Web\n  workloads than the standard library; hopefully will get upstreamed some\n  day) and ASCII.\n\nAdditionally, `encoding_rs::mem` does the following:\n\n* Checks if a byte buffer contains only ASCII.\n* Checks if a potentially-invalid UTF-16 buffer contains only Basic Latin (ASCII).\n* Checks if a valid UTF-8, potentially-invalid UTF-8 or potentially-invalid UTF-16\n  buffer contains only Latin1 code points (below U+0100).\n* Checks if a valid UTF-8, potentially-invalid UTF-8 or potentially-invalid UTF-16\n  buffer or a code point or a UTF-16 code unit can trigger right-to-left behavior\n  (suitable for checking if the Unicode Bidirectional Algorithm can be optimized\n  out).\n* Combined versions of the above two checks.\n* Converts valid UTF-8, potentially-invalid UTF-8 and Latin1 to UTF-16.\n* Converts potentially-invalid UTF-16 and Latin1 to UTF-8.\n* Converts UTF-8 and UTF-16 to Latin1 (if in range).\n* Finds the first invalid code unit in a buffer of potentially-invalid UTF-16.\n* Makes a mutable buffer of potential-invalid UTF-16 contain valid UTF-16.\n* Copies ASCII from one buffer to another up to the first non-ASCII byte.\n* Converts ASCII to UTF-16 up to the first non-ASCII byte.\n* Converts UTF-16 to ASCII up to the first non-Basic Latin code unit.\n\n## Integration with `std::io`\n\nNotably, the above feature list doesn't include the capability to wrap\na `std::io::Read`, decode it into UTF-8 and presenting the result via\n`std::io::Read`. The [`encoding_rs_io`](https://crates.io/crates/encoding_rs_io)\ncrate provides that capability.\n\n## `no_std` Environment\n\nThe crate works in a `no_std` environment. By default, the `alloc` feature,\nwhich assumes that an allocator is present is enabled. For a no-allocator\nenvironment, the default features (i.e. `alloc`) can be turned off. This\nmakes the part of the API that returns `Vec`/`String`/`Cow` unavailable.\n\n## Decoding Email\n\nFor decoding character encodings that occur in email, use the\n[`charset`](https://crates.io/crates/charset) crate instead of using this\none directly. (It wraps this crate and adds UTF-7 decoding.)\n\n## Windows Code Page Identifier Mappings\n\nFor mappings to and from Windows code page identifiers, use the\n[`codepage`](https://crates.io/crates/codepage) crate.\n\n## DOS Encodings\n\nThis crate does not support single-byte DOS encodings that aren't required by\nthe Web Platform, but the [`oem_cp`](https://crates.io/crates/oem_cp) crate does.\n\n## Preparing Text for the Encoders\n\nNormalizing text into Unicode Normalization Form C prior to encoding text into\na legacy encoding minimizes unmappable characters. Text can be normalized to\nUnicode Normalization Form C using the\n[`icu_normalizer`](https://crates.io/crates/icu_normalizer) crate.\n\nThe exception is windows-1258, which after normalizing to Unicode Normalization\nForm C requires tone marks to be decomposed in order to minimize unmappable\ncharacters. Vietnamese tone marks can be decomposed using the\n[`detone`](https://crates.io/crates/detone) crate.\n\n## Licensing\n\nTL;DR: `(Apache-2.0 OR MIT) AND BSD-3-Clause` for the code and data combination.\n\nPlease see the file named\n[COPYRIGHT](https://github.com/hsivonen/encoding_rs/blob/master/COPYRIGHT).\n\nThe non-test code that isn't generated from the WHATWG data in this crate is\nunder Apache-2.0 OR MIT. Test code is under CC0.\n\nThis crate contains code/data generated from WHATWG-supplied data. The WHATWG\nupstream changed its license for portions of specs incorporated into source code\nfrom CC0 to BSD-3-Clause between the initial release of this crate and the present\nversion of this crate. The in-source licensing legends have been updated for the\nparts of the generated code that have changed since the upstream license change.\n\n## Documentation\n\nGenerated [API documentation](https://docs.rs/encoding_rs/) is available\nonline.\n\nThere is a [long-form write-up](https://hsivonen.fi/encoding_rs/) about the\ndesign and internals of the crate.\n\n## C and C++ bindings\n\nAn FFI layer for encoding_rs is available as a\n[separate crate](https://github.com/hsivonen/encoding_c). The crate comes\nwith a [demo C++ wrapper](https://github.com/hsivonen/encoding_c/blob/master/include/encoding_rs_cpp.h)\nusing the C++ standard library and [GSL](https://github.com/Microsoft/GSL/) types.\n\nThe bindings for the `mem` module are in the\n[encoding_c_mem crate](https://github.com/hsivonen/encoding_c_mem).\n\nFor the Gecko context, there's a\n[C++ wrapper using the MFBT/XPCOM types](https://searchfox.org/mozilla-central/source/intl/Encoding.h#100).\n\nThere's a [write-up](https://hsivonen.fi/modern-cpp-in-rust/) about the C++\nwrappers.\n\n## Sample programs\n\n* [Rust](https://github.com/hsivonen/recode_rs)\n* [C](https://github.com/hsivonen/recode_c)\n* [C++](https://github.com/hsivonen/recode_cpp)\n\n## Optional features\n\nThere are currently these optional cargo features:\n\n### `simd-accel`\n\nEnables SIMD acceleration using the nightly-dependent `portable_simd` standard\nlibrary feature.\n\nThis is an opt-in feature, because enabling this feature _opts out_ of Rust's\nguarantees of future compilers compiling old code (aka. \"stability story\").\n\nCurrently, this has not been tested to be an improvement except for these\ntargets and enabling the `simd-accel` feature is expected to break the build\non other targets:\n\n* x86_64\n* i686\n* aarch64\n* thumbv7neon\n\nIf you use nightly Rust, you use targets whose first component is one of the\nabove, and you are prepared _to have to revise your configuration when updating\nRust_, you should enable this feature. Otherwise, please _do not_ enable this\nfeature.\n\nUsed by Firefox.\n\n### `serde`\n\nEnables support for serializing and deserializing `\u0026'static Encoding`-typed\nstruct fields using [Serde][1].\n\n[1]: https://serde.rs/\n\nNot used by Firefox.\n\n### `fast-legacy-encode`\n\nA catch-all option for enabling the fastest legacy encode options. _Does not\naffect decode speed or UTF-8 encode speed._\n\nAt present, this option is equivalent to enabling the following options:\n * `fast-hangul-encode`\n * `fast-hanja-encode`\n * `fast-kanji-encode`\n * `fast-gb-hanzi-encode`\n * `fast-big5-hanzi-encode`\n\nAdds 176 KB to the binary size.\n\nNot used by Firefox.\n\n### `fast-hangul-encode`\n\nChanges encoding precomposed Hangul syllables into EUC-KR from binary\nsearch over the decode-optimized tables to lookup by index making Korean\nplain-text encode about 4 times as fast as without this option.\n\nAdds 20 KB to the binary size.\n\nDoes _not_ affect decode speed.\n\nNot used by Firefox.\n\n### `fast-hanja-encode`\n\nChanges encoding of Hanja into EUC-KR from linear search over the\ndecode-optimized table to lookup by index. Since Hanja is practically absent\nin modern Korean text, this option doesn't affect perfomance in the common\ncase and mainly makes sense if you want to make your application resilient\nagaist denial of service by someone intentionally feeding it a lot of Hanja\nto encode into EUC-KR.\n\nAdds 40 KB to the binary size.\n\nDoes _not_ affect decode speed.\n\nNot used by Firefox.\n\n### `fast-kanji-encode`\n\nChanges encoding of Kanji into Shift_JIS, EUC-JP and ISO-2022-JP from linear\nsearch over the decode-optimized tables to lookup by index making Japanese\nplain-text encode to legacy encodings 30 to 50 times as fast as without this\noption (about 2 times as fast as with `less-slow-kanji-encode`).\n\nTakes precedence over `less-slow-kanji-encode`.\n\nAdds 36 KB to the binary size (24 KB compared to `less-slow-kanji-encode`).\n\nDoes _not_ affect decode speed.\n\nNot used by Firefox.\n\n### `less-slow-kanji-encode`\n\nMakes JIS X 0208 Level 1 Kanji (the most common Kanji in Shift_JIS, EUC-JP and\nISO-2022-JP) encode less slow (binary search instead of linear search) making\nJapanese plain-text encode to legacy encodings 14 to 23 times as fast as\nwithout this option.\n\nAdds 12 KB to the binary size.\n\nDoes _not_ affect decode speed.\n\nNot used by Firefox.\n\n### `fast-gb-hanzi-encode`\n\nChanges encoding of Hanzi in the CJK Unified Ideographs block into GBK and\ngb18030 from linear search over a part the decode-optimized tables followed\nby a binary search over another part of the decode-optimized tables to lookup\nby index making Simplified Chinese plain-text encode to the legacy encodings\n100 to 110 times as fast as without this option (about 2.5 times as fast as\nwith `less-slow-gb-hanzi-encode`).\n\nTakes precedence over `less-slow-gb-hanzi-encode`.\n\nAdds 36 KB to the binary size (24 KB compared to `less-slow-gb-hanzi-encode`).\n\nDoes _not_ affect decode speed.\n\nNot used by Firefox.\n\n### `less-slow-gb-hanzi-encode`\n\nMakes GB2312 Level 1 Hanzi (the most common Hanzi in gb18030 and GBK) encode\nless slow (binary search instead of linear search) making Simplified Chinese\nplain-text encode to the legacy encodings about 40 times as fast as without\nthis option.\n\nAdds 12 KB to the binary size.\n\nDoes _not_ affect decode speed.\n\nNot used by Firefox.\n\n### `fast-big5-hanzi-encode`\n\nChanges encoding of Hanzi in the CJK Unified Ideographs block into Big5 from\nlinear search over a part the decode-optimized tables to lookup by index\nmaking Traditional Chinese plain-text encode to Big5 105 to 125 times as fast\nas without this option (about 3 times as fast as with\n`less-slow-big5-hanzi-encode`).\n\nTakes precedence over `less-slow-big5-hanzi-encode`.\n\nAdds 40 KB to the binary size (20 KB compared to `less-slow-big5-hanzi-encode`).\n\nDoes _not_ affect decode speed.\n\nNot used by Firefox.\n\n### `less-slow-big5-hanzi-encode`\n\nMakes Big5 Level 1 Hanzi (the most common Hanzi in Big5) encode less slow\n(binary search instead of linear search) making Traditional Chinese\nplain-text encode to Big5 about 36 times as fast as without this option.\n\nAdds 20 KB to the binary size.\n\nDoes _not_ affect decode speed.\n\nNot used by Firefox.\n\n## Performance goals\n\nFor decoding to UTF-16, the goal is to perform at least as well as Gecko's old\nuconv. For decoding to UTF-8, the goal is to perform at least as well as\nrust-encoding. These goals have been achieved.\n\nEncoding to UTF-8 should be fast. (UTF-8 to UTF-8 encode should be equivalent\nto `memcpy` and UTF-16 to UTF-8 should be fast.)\n\nSpeed is a non-goal when encoding to legacy encodings. By default, encoding to\nlegacy encodings should not be optimized for speed at the expense of code size\nas long as form submission and URL parsing in Gecko don't become noticeably\ntoo slow in real-world use.\n\nIn the interest of binary size, by default, encoding_rs does not have\nencode-specific data tables beyond 32 bits of encode-specific data for each\nsingle-byte encoding. Therefore, encoders search the decode-optimized data\ntables. This is a linear search in most cases. As a result, by default, encode\nto legacy encodings varies from slow to extremely slow relative to other\nlibraries. Still, with realistic work loads, this seemed fast enough not to be\nuser-visibly slow on Raspberry Pi 3 (which stood in for a phone for testing)\nin the Web-exposed encoder use cases.\n\nSee the cargo features above for optionally making CJK legacy encode fast.\n\nA framework for measuring performance is [available separately][2].\n\n[2]: https://github.com/hsivonen/encoding_bench/\n\n## Rust Version Compatibility\n\nIt is a goal to support the latest stable Rust, the latest nightly Rust and\nthe version of Rust that's used for Firefox Nightly.\n\nAt this time, there is no firm commitment to support a version older than\nwhat's required by Firefox, and there is no commitment to treat MSRV changes\nas semver-breaking, because this crate depends on `cfg-if`, which doesn't\nappear to treat MSRV changes as semver-breaking, so it would be useless for\nthis crate to treat MSRV changes as semver-breaking.\n\nAs of 2024-11-01, MSRV appears to be Rust 1.40.0 for using the crate and\n1.42.0 for doc tests to pass without errors about the global allocator.\nWith the `simd-accel` feature, the MSRV is even higher.\n\n## Compatibility with rust-encoding\n\nA compatibility layer that implements the rust-encoding API on top of\nencoding_rs is\n[provided as a separate crate](https://github.com/hsivonen/encoding_rs_compat)\n(cannot be uploaded to crates.io). The compatibility layer was originally\nwritten with the assuption that Firefox would need it, but it is not currently\nused in Firefox.\n\n## Regenerating Generated Code\n\nTo regenerate the generated code:\n\n * Have Python 2 installed.\n * Clone [`https://github.com/hsivonen/encoding_c`](https://github.com/hsivonen/encoding_c)\n   next to the `encoding_rs` directory.\n * Clone [`https://github.com/hsivonen/codepage`](https://github.com/hsivonen/codepage)\n   next to the `encoding_rs` directory.\n * Clone [`https://github.com/whatwg/encoding`](https://github.com/whatwg/encoding)\n   next to the `encoding_rs` directory.\n * Checkout revision `1d519bf8e5555cef64cf3a712485f41cd1a6a990` of the `encoding` repo.\n   (Note: `f381389` was the revision of `encoding` used from before the `encoding` repo\n   license change.)\n * With the `encoding_rs` directory as the working directory, run\n   `python generate-encoding-data.py`.\n\n## Roadmap\n\n- [x] Design the low-level API.\n- [x] Provide Rust-only convenience features.\n- [x] Provide an stl/gsl-flavored C++ API.\n- [x] Implement all decoders and encoders.\n- [x] Add unit tests for all decoders and encoders.\n- [x] Finish BOM sniffing variants in Rust-only convenience features.\n- [x] Document the API.\n- [x] Publish the crate on crates.io.\n- [x] Create a solution for measuring performance.\n- [x] Accelerate ASCII conversions using SSE2 on x86.\n- [x] Accelerate ASCII conversions using ALU register-sized operations on\n      non-x86 architectures (process an `usize` instead of `u8` at a time).\n- [x] Split FFI into a separate crate so that the FFI doesn't interfere with\n      LTO in pure-Rust usage.\n- [x] Compress CJK indices by making use of sequential code points as well\n      as Unicode-ordered parts of indices.\n- [x] Make lookups by label or name use binary search that searches from the\n      end of the label/name to the start.\n- [x] Make labels with non-ASCII bytes fail fast.\n- [ ] ~Parallelize UTF-8 validation using [Rayon](https://github.com/nikomatsakis/rayon).~\n      (This turned out to be a pessimization in the ASCII case due to memory bandwidth reasons.)\n- [x] Provide an XPCOM/MFBT-flavored C++ API.\n- [x] Investigate accelerating single-byte encode with a single fast-tracked\n      range per encoding.\n- [x] Replace uconv with encoding_rs in Gecko.\n- [x] Implement the rust-encoding API in terms of encoding_rs.\n- [x] Add SIMD acceleration for Aarch64.\n- [x] Investigate the use of NEON on 32-bit ARM.\n- [ ] ~Investigate Björn Höhrmann's lookup table acceleration for UTF-8 as\n      adapted to Rust in rust-encoding.~\n- [x] Add actually fast CJK encode options.\n- [ ] ~Investigate [Bob Steagall's lookup table acceleration for UTF-8](https://github.com/BobSteagall/CppNow2018/blob/master/FastConversionFromUTF-8/Fast%20Conversion%20From%20UTF-8%20with%20C%2B%2B%2C%20DFAs%2C%20and%20SSE%20Intrinsics%20-%20Bob%20Steagall%20-%20C%2B%2BNow%202018.pdf).~\n- [x] Provide a build mode that works without `alloc` (with lesser API surface).\n- [x] Migrate to `std::simd` ~once it is stable and declare 1.0.~\n- [ ] Migrate `unsafe` slice access by larger types than `u8`/`u16` to `align_to`.\n\n## Release Notes\n\n### 0.8.35\n\n* Implement changes for GB18030-2022. (Intentionally not treated as a semver break in practice even if this could be argued to be a breaking change in theory.)\n\n### 0.8.34\n\n* Use the `portable_simd` nightly feature of the standard library instead of the `packed_simd` crate. Only affects the `simd-accel` optional nightly feature.\n* Internal documentation improvements and minor code improvements around `unsafe`.\n* Added `rust-version` to `Cargo.toml`.\n\n### 0.8.33\n\n* Use `packed_simd` instead of `packed_simd_2` again now that updates are back under the `packed_simd` name. Only affects the `simd-accel` optional nightly feature.\n\n### 0.8.32\n\n* Removed `build.rs`. (This removal should resolve false positives reported by some antivirus products. This may break some build configurations that have opted out of Rust's guarantees against future build breakage.)\n* Internal change to what API is used for reinterpreting the lane configuration of SIMD vectors.\n* Documentation improvements.\n\n### 0.8.31\n\n* Use SPDX with parentheses now that crates.io supports parentheses.\n\n### 0.8.30\n\n* Update the licensing information to take into account the WHATWG data license change.\n\n### 0.8.29\n\n* Make the parts that use an allocator optional.\n\n### 0.8.28\n\n* Fix error in Serde support introduced as part of `no_std` support.\n\n### 0.8.27\n\n* Make the crate works in a `no_std` environment (with `alloc`).\n\n### 0.8.26\n\n* Fix oversights in edition 2018 migration that broke the `simd-accel` feature.\n\n### 0.8.25\n\n* Do pointer alignment checks in a way where intermediate steps aren't defined to be Undefined Behavior.\n* Update the `packed_simd` dependency to `packed_simd_2`.\n* Update the `cfg-if` dependency to 1.0.\n* Address warnings that have been introduced by newer Rust versions along the way.\n* Update to edition 2018, since even prior to 1.0 `cfg-if` updated to edition 2018 without a semver break.\n\n### 0.8.24\n\n* Avoid computing an intermediate (not dereferenced) pointer value in a manner designated as Undefined Behavior when computing pointer alignment.\n\n### 0.8.23\n\n* Remove year from copyright notices. (No features or bug fixes.)\n\n### 0.8.22\n\n* Formatting fix and new unit test. (No features or bug fixes.)\n\n### 0.8.21\n\n* Fixed a panic with invalid UTF-16[BE|LE] input at the end of the stream.\n\n### 0.8.20\n\n* Make `Decoder::latin1_byte_compatible_up_to` return `None` in more\n  cases to make the method actually useful. While this could be argued\n  to be a breaking change due to the bug fix changing semantics, it does\n  not break callers that had to handle the `None` case in a reasonable\n  way anyway.\n\n### 0.8.19\n\n* Removed a bunch of bound checks in `convert_str_to_utf16`.\n* Added `mem::convert_utf8_to_utf16_without_replacement`.\n\n### 0.8.18\n\n* Added `mem::utf8_latin1_up_to` and `mem::str_latin1_up_to`.\n* Added `Decoder::latin1_byte_compatible_up_to`.\n\n### 0.8.17\n\n* Update `bincode` (dev dependency) version requirement to 1.0.\n\n### 0.8.16\n\n* Switch from the `simd` crate to `packed_simd`.\n\n### 0.8.15\n\n* Adjust documentation for `simd-accel` (README-only release).\n\n### 0.8.14\n\n* Made UTF-16 to UTF-8 encode conversion fill the output buffer as\n  closely as possible.\n\n### 0.8.13\n\n* Made the UTF-8 to UTF-16 decoder compare the number of code units written\n  with the length of the right slice (the output slice) to fix a panic\n  introduced in 0.8.11.\n\n### 0.8.12\n\n* Removed the `clippy::` prefix from clippy lint names.\n\n### 0.8.11\n\n* Changed minimum Rust requirement to 1.29.0 (for the ability to refer\n  to the interior of a `static` when defining another `static`).\n* Explicitly aligned the lookup tables for single-byte encodings and\n  UTF-8 to cache lines in the hope of freeing up one cache line for\n  other data. (Perhaps the tables were already aligned and this is\n  placebo.)\n* Added 32 bits of encode-oriented data for each single-byte encoding.\n  The change was performance-neutral for non-Latin1-ish Latin legacy\n  encodings, improved Latin1-ish and Arabic legacy encode speed\n  somewhat (new speed is 2.4x the old speed for German, 2.3x for\n  Arabic, 1.7x for Portuguese and 1.4x for French) and improved\n  non-Latin1, non-Arabic legacy single-byte encode a lot (7.2x for\n  Thai, 6x for Greek, 5x for Russian, 4x for Hebrew).\n* Added compile-time options for fast CJK legacy encode options (at\n  the cost of binary size (up to 176 KB) and run-time memory usage).\n  These options still retain the overall code structure instead of\n  rewriting the CJK encoders totally, so the speed isn't as good as\n  what could be achieved by using even more memory / making the\n  binary even langer.\n* Made UTF-8 decode and validation faster.\n* Added method `is_single_byte()` on `Encoding`.\n* Added `mem::decode_latin1()` and `mem::encode_latin1_lossy()`.\n\n### 0.8.10\n\n* Disabled a unit test that tests a panic condition when the assertion\n  being tested is disabled.\n\n### 0.8.9\n\n* Made `--features simd-accel` work with stable-channel compiler to\n  simplify the Firefox build system.\n\n### 0.8.8\n\n* Made the `is_foo_bidi()` not treat U+FEFF (ZERO WIDTH NO-BREAK SPACE\n  aka. BYTE ORDER MARK) as right-to-left.\n* Made the `is_foo_bidi()` functions report `true` if the input contains\n  Hebrew presentations forms (which are right-to-left but not in a\n  right-to-left-roadmapped block).\n\n### 0.8.7\n\n* Fixed a panic in the UTF-16LE/UTF-16BE decoder when decoding to UTF-8.\n\n### 0.8.6\n\n* Temporarily removed the debug assertion added in version 0.8.5 from\n  `convert_utf16_to_latin1_lossy`.\n\n### 0.8.5\n\n* If debug assertions are enabled but fuzzing isn't enabled, lossy conversions\n  to Latin1 in the `mem` module assert that the input is in the range\n  U+0000...U+00FF (inclusive).\n* In the `mem` module provide conversions from Latin1 and UTF-16 to UTF-8\n  that can deal with insufficient output space. The idea is to use them\n  first with an allocation rounded up to jemalloc bucket size and do the\n  worst-case allocation only if the jemalloc rounding up was insufficient\n  as the first guess.\n\n### 0.8.4\n\n* Fix SSE2-specific, `simd-accel`-specific memory corruption introduced in\n  version 0.8.1 in conversions between UTF-16 and Latin1 in the `mem` module.\n\n### 0.8.3\n\n* Removed an `#[inline(never)]` annotation that was not meant for release.\n\n### 0.8.2\n\n* Made non-ASCII UTF-16 to UTF-8 encode faster by manually omitting bound\n  checks and manually adding branch prediction annotations.\n\n### 0.8.1\n\n* Tweaked loop unrolling and memory alignment for SSE2 conversions between\n  UTF-16 and Latin1 in the `mem` module to increase the performance when\n  converting long buffers.\n\n### 0.8.0\n\n* Changed the minimum supported version of Rust to 1.21.0 (semver breaking\n  change).\n* Flipped around the defaults vs. optional features for controlling the size\n  vs. speed trade-off for Kanji and Hanzi legacy encode (semver breaking\n  change).\n* Added NEON support on ARMv7.\n* SIMD-accelerated x-user-defined to UTF-16 decode.\n* Made UTF-16LE and UTF-16BE decode a lot faster (including SIMD\n  acceleration).\n\n### 0.7.2\n\n* Add the `mem` module.\n* Refactor SIMD code which can affect performance outside the `mem`\n  module.\n\n### 0.7.1\n\n* When encoding from invalid UTF-16, correctly handle U+DC00 followed by\n  another low surrogate.\n\n### 0.7.0\n\n* [Make `replacement` a label of the replacement\n  encoding.](https://github.com/whatwg/encoding/issues/70) (Spec change.)\n* Remove `Encoding::for_name()`. (`Encoding::for_label(foo).unwrap()` is\n  now close enough after the above label change.)\n* Remove the `parallel-utf8` cargo feature.\n* Add optional Serde support for `\u0026'static Encoding`.\n* Performance tweaks for ASCII handling.\n* Performance tweaks for UTF-8 validation.\n* SIMD support on aarch64.\n\n### 0.6.11\n\n* Make `Encoder::has_pending_state()` public.\n* Update the `simd` crate dependency to 0.2.0.\n\n### 0.6.10\n\n* Reserve enough space for NCRs when encoding to ISO-2022-JP.\n* Correct max length calculations for multibyte decoders.\n* Correct max length calculations before BOM sniffing has been\n  performed.\n* Correctly calculate max length when encoding from UTF-16 to GBK.\n\n### 0.6.9\n\n* [Don't prepend anything when gb18030 range decode\n  fails](https://github.com/whatwg/encoding/issues/110). (Spec change.)\n\n### 0.6.8\n\n* Correcly handle the case where the first buffer contains potentially\n  partial BOM and the next buffer is the last buffer.\n* Decode byte `7F` correctly in ISO-2022-JP.\n* Make UTF-16 to UTF-8 encode write closer to the end of the buffer.\n* Implement `Hash` for `Encoding`.\n\n### 0.6.7\n\n* [Map half-width katakana to full-width katana in ISO-2022-JP\n  encoder](https://github.com/whatwg/encoding/issues/105). (Spec change.)\n* Give `InputEmpty` correct precedence over `OutputFull` when encoding\n  with replacement and the output buffer passed in is too short or the\n  remaining space in the output buffer is too small after a replacement.\n\n### 0.6.6\n\n* Correct max length calculation when a partial BOM prefix is part of\n  the decoder's state.\n\n### 0.6.5\n\n* Correct max length calculation in various encoders.\n* Correct max length calculation in the UTF-16 decoder.\n* Derive `PartialEq` and `Eq` for the `CoderResult`, `DecoderResult`\n  and `EncoderResult` types.\n\n### 0.6.4\n\n* Avoid panic when encoding with replacement and the destination buffer is\n  too short to hold one numeric character reference.\n\n### 0.6.3\n\n* Add support for 32-bit big-endian hosts. (For real this time.)\n\n### 0.6.2\n\n* Fix a panic from subslicing with bad indices in\n  `Encoder::encode_from_utf16`. (Due to an oversight, it lacked the fix that\n  `Encoder::encode_from_utf8` already had.)\n* Micro-optimize error status accumulation in non-streaming case.\n\n### 0.6.1\n\n* Avoid panic near integer overflow in a case that's unlikely to actually\n  happen.\n* Address Clippy lints.\n\n### 0.6.0\n\n* Make the methods for computing worst-case buffer size requirements check\n  for integer overflow.\n* Upgrade rayon to 0.7.0.\n\n### 0.5.1\n\n* Reorder methods for better documentation readability.\n* Add support for big-endian hosts. (Only 64-bit case actually tested.)\n* Optimize the ALU (non-SIMD) case for 32-bit ARM instead of x86_64.\n\n### 0.5.0\n\n* Avoid allocating an excessively long buffers in non-streaming decode.\n* Fix the behavior of ISO-2022-JP and replacement decoders near the end of the\n  output buffer.\n* Annotate the result structs with `#[must_use]`.\n\n### 0.4.0\n\n* Split FFI into a separate crate.\n* Performance tweaks.\n* CJK binary size and encoding performance changes.\n* Parallelize UTF-8 validation in the case of long buffers (with optional\n  feature `parallel-utf8`).\n* Borrow even with ISO-2022-JP when possible.\n\n### 0.3.2\n\n* Fix moving pointers to alignment in ALU-based ASCII acceleration.\n* Fix errors in documentation and improve documentation.\n\n### 0.3.1\n\n* Fix UTF-8 to UTF-16 decode for byte sequences beginning with 0xEE.\n* Make UTF-8 to UTF-8 decode SSE2-accelerated when feature `simd-accel` is used.\n* When decoding and encoding ASCII-only input from or to an ASCII-compatible\n  encoding using the non-streaming API, return a borrow of the input.\n* Make encode from UTF-16 to UTF-8 faster.\n\n### 0.3\n\n* Change the references to the instances of `Encoding` from `const` to `static`\n  to make the referents unique across crates that use the refernces.\n* Introduce non-reference-typed `FOO_INIT` instances of `Encoding` to allow\n  foreign crates to initialize `static` arrays with references to `Encoding`\n  instances even under Rust's constraints that prohibit the initialization of\n  `\u0026'static Encoding`-typed array items with `\u0026'static Encoding`-typed\n  `statics`.\n* Document that the above two points will be reverted if Rust changes `const`\n  to work so that cross-crate usage keeps the referents unique.\n* Return `Cow`s from Rust-only non-streaming methods for encode and decode.\n* `Encoding::for_bom()` returns the length of the BOM.\n* ASCII-accelerated conversions for encodings other than UTF-16LE, UTF-16BE,\n  ISO-2022-JP and x-user-defined.\n* Add SSE2 acceleration behind the `simd-accel` feature flag. (Requires\n  nightly Rust.)\n* Fix panic with long bogus labels.\n* Map [0xCA to U+05BA in windows-1255](https://github.com/whatwg/encoding/issues/73).\n  (Spec change.)\n* Correct the [end of the Shift_JIS EUDC range](https://github.com/whatwg/encoding/issues/53).\n  (Spec change.)\n\n### 0.2.4\n\n* Polish FFI documentation.\n\n### 0.2.3\n\n* Fix UTF-16 to UTF-8 encode.\n\n### 0.2.2\n\n* Add `Encoder.encode_from_utf8_to_vec_without_replacement()`.\n\n### 0.2.1\n\n* Add `Encoding.is_ascii_compatible()`.\n\n* Add `Encoding::for_bom()`.\n\n* Make `==` for `Encoding` use name comparison instead of pointer comparison,\n  because uses of the encoding constants in different crates result in\n  different addresses and the constant cannot be turned into statics without\n  breaking other things.\n\n### 0.2.0\n\nThe initial release.\n","funding_links":[],"categories":["Libraries","库 Libraries","库","Rust"],"sub_categories":["Encoding","编码 Encoding","编码(Encoding)","加密 Encoding"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhsivonen%2Fencoding_rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhsivonen%2Fencoding_rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhsivonen%2Fencoding_rs/lists"}