{"id":16572666,"url":"https://github.com/lemire/validateutf8-experiments","last_synced_at":"2025-03-16T20:31:28.365Z","repository":{"id":66054982,"uuid":"265680018","full_name":"lemire/validateutf8-experiments","owner":"lemire","description":"Reproducible experimeents on UTF-8 validation using SIMD instructions","archived":false,"fork":false,"pushed_at":"2024-08-21T13:52:41.000Z","size":488,"stargazers_count":40,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-02-27T13:22:11.913Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lemire.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-20T20:32:56.000Z","updated_at":"2024-08-22T13:24:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"b86bb808-bf79-4711-afe8-77e095aba69c","html_url":"https://github.com/lemire/validateutf8-experiments","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fvalidateutf8-experiments","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fvalidateutf8-experiments/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fvalidateutf8-experiments/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fvalidateutf8-experiments/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lemire","download_url":"https://codeload.github.com/lemire/validateutf8-experiments/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243830912,"owners_count":20354848,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T21:28:13.733Z","updated_at":"2025-03-16T20:31:26.839Z","avatar_url":"https://github.com/lemire.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# validateutf8-experiments\n\nThis project contains benchmarks regarding fast UTF-8 validation. It is for research purposes only: not for production use. If you are not doing research, this repository is not for you!\n\nThe algorithm of this repository has been included in production-ready libraries:\n\n- [simdutf](https://github.com/simdutf/simdutf) C++ library is part of important systems such as Bun, Node.js, WebKit/Safari, etc.\n- [SimdUnicode](https://github.com/simdutf/SimdUnicode) is a C# port of the validation algorithm, adapted for the .NET runtime.\n\n## Software requirements\n\nAn x64 system with Linux, docker, bash and git.\n\nBecause we use performance counters, you need to have privileged access to the docker containers.\n\n## Hardware requirements\n\n\nThough the code is generic, the benchmarks assume that you have a processor with AVX2 support (there is no runtime check). Thus please ensure that you have a recent Intel or AMD processor with AVX2 support. We recommend AMD Zen 2 or better, or Intel Skylake and better.\n\nUnder Linux, you might be able to know about your CPU with the following command:\n\n```\nlscpu | grep Model\n```\n\n- We recommend that you have bare-metal access to the hardware. Cloud nodes are often noisier.\n- We recommend against testing on a laptop.\n- It is best if you can set the governor (or the equivalent) to `Performance`.\n- Some people prefer to disable hyper-threading and to lock the process on a given processor: if you are using a quiet machine, we believe that it is unnecessary with this benchmark.\n\n\n## Reproducible experiments\n\nTo ensure that experiments are reproducible, we rely on a docker image. We recommend that you install docker under Linux. \n\n\n\n\n\nStarting in a bash shell do:\n\n```\ngit clone https://github.com/lemire/validateutf8-experiments.git\ncd validateutf8-experiments\n./run bash\nmake\n./unit\n./benchmark\n./benchstream 1000\n```\n\n\n## Want a production-ready function?\n\nThe validateutf8-experiments repository is for research purposes only.\n\nIf you want access to a fast validation function for production use, you can rely on the simdjson library. [It is as simple as the following](https://github.com/simdjson/simdjson/blob/master/doc/basics.md#utf-8-validation-alone):\n\n```C++\n  const char * some_string = \"[ 1, 2, 3, 4] \";\n  size_t length = strlen(some_string);\n  bool is_ok = simdjson::validate_utf8(some_string, length);\n```\n\nSee https://github.com/simdjson/\n\n\nThe simdjson library supports a wide-range of platforms and offers runtime dispatching as well as the most up-to-date algorithms. It is not necessary that your data is made of JSON though this was the original motivation.\n\n## Example\n\n```\n$ git clone https://github.com/lemire/validateutf8-experiments.git\n$ cd validateutf8-experiments\n$ make\n$ ./unit\ntests ok.\n$ ./benchmark\n$ ./benchmark\n\nRunning UTF8 validation benchmark.\nThe speed is normalized by the number of input bytes.\nfile: examples/hongkong.html (1807KB)\nOverhead ==\u003e nanoseconds : 20.000, instructions 0.000\n                                  memcpy                                  48.281 GB/s  ( 1.94 %)\n                                  fushia                                   1.323 GB/s  (11.26 %)\n                            fushia_ascii                                   3.660 GB/s  ( 0.88 %)\n                           fushia_ascii2                                   4.334 GB/s  ( 1.30 %)\n                           fushia_ascii4                                   4.210 GB/s  ( 0.94 %)\n                                 utf8lib                                   0.377 GB/s  ( 0.19 %)\n                                     dfa                                   0.664 GB/s  ( 0.96 %)\n                                    dfa2                                   1.130 GB/s  ( 0.53 %)\n                                    dfa3                                   1.968 GB/s  ( 7.13 %)\n                                    dfa4                                   0.678 GB/s  ( 0.42 %)\n                              zwegneravx                                  14.039 GB/s  ( 1.05 %)\n                              lookup2avx                                  17.426 GB/s  ( 2.40 %)\n                              lookup3avx                                  17.672 GB/s  ( 2.86 %)\n                              lookup4avx                                  18.185 GB/s  ( 2.48 %)\n                                basicavx                                  13.374 GB/s  ( 2.13 %)\n                                rangeavx                                  14.758 GB/s  ( 3.07 %)\nfile: examples/twitter.json (631KB)\nOverhead ==\u003e nanoseconds : 20.000, instructions 0.000\n                                  memcpy                                  41.992 GB/s  ( 2.83 %)\n                                  fushia                                   1.537 GB/s  (14.57 %)\n                            fushia_ascii                                   3.437 GB/s  ( 1.10 %)\n                           fushia_ascii2                                   4.414 GB/s  ( 1.09 %)\n                           fushia_ascii4                                   4.077 GB/s  ( 1.33 %)\n                                 utf8lib                                   0.394 GB/s  ( 0.92 %)\n                                     dfa                                   0.667 GB/s  ( 2.14 %)\n                                    dfa2                                   1.131 GB/s  ( 0.48 %)\n                                    dfa3                                   1.970 GB/s  ( 1.67 %)\n                                    dfa4                                   0.679 GB/s  ( 0.65 %)\n                              zwegneravx                                  24.299 GB/s  ( 1.24 %)\n                              lookup2avx                                  27.537 GB/s  ( 4.99 %)\n                              lookup3avx                                  29.649 GB/s  ( 3.38 %)\n                              lookup4avx                                  30.436 GB/s  ( 7.83 %)\n                                basicavx                                  18.816 GB/s  ( 1.29 %)\n                                rangeavx                                  22.463 GB/s  ( 6.10 %)\n\nRunning UTF8 validation benchmark.\nThe speed is normalized by the number of input bytes.\nInput size: (UTF8) 16384\n- ASCII characters\nOverhead ==\u003e nanoseconds : 20.000, instructions 0.000\n                                  memcpy                                  74.477 GB/s  (10.12 %)\n                                  fushia                                   1.697 GB/s  ( 0.91 %)\n                            fushia_ascii                                   9.038 GB/s  ( 1.15 %)\n                           fushia_ascii2                                  17.986 GB/s  ( 2.22 %)\n                           fushia_ascii4                                  35.620 GB/s  ( 1.98 %)\n                                 utf8lib                                   0.424 GB/s  ( 0.54 %)\n                                     dfa                                   0.653 GB/s  ( 0.30 %)\n                                    dfa2                                   1.132 GB/s  ( 0.39 %)\n                                    dfa3                                   1.961 GB/s  ( 1.85 %)\n                                    dfa4                                   0.680 GB/s  ( 1.95 %)\n                              zwegneravx                                  52.855 GB/s  ( 6.15 %)\n                              lookup2avx                                  71.239 GB/s  ( 6.48 %)\n                              lookup3avx                                  71.239 GB/s  ( 6.74 %)\n                              lookup4avx                                  71.239 GB/s  ( 6.28 %)\n                                basicavx                                  71.239 GB/s  ( 6.30 %)\n                                rangeavx                                  68.271 GB/s  ( 2.70 %)\n- 1 or 2 UTF8 bytes\nOverhead ==\u003e nanoseconds : 20.000, instructions 0.000\n                                  memcpy                                  74.482 GB/s  ( 9.76 %)\n                                  fushia                                   0.416 GB/s  ( 3.77 %)\n                            fushia_ascii                                   0.362 GB/s  ( 2.14 %)\n                           fushia_ascii2                                   0.390 GB/s  ( 3.67 %)\n                           fushia_ascii4                                   0.364 GB/s  ( 2.91 %)\n                                 utf8lib                                   0.230 GB/s  ( 1.01 %)\n                                     dfa                                   0.653 GB/s  ( 0.31 %)\n                                    dfa2                                   1.132 GB/s  ( 0.99 %)\n                                    dfa3                                   1.975 GB/s  ( 0.95 %)\n                                    dfa4                                   0.680 GB/s  ( 0.24 %)\n                              zwegneravx                                   9.401 GB/s  ( 1.47 %)\n                              lookup2avx                                  11.857 GB/s  ( 1.46 %)\n                              lookup3avx                                  12.395 GB/s  ( 1.83 %)\n                              lookup4avx                                  13.747 GB/s  ( 1.66 %)\n                                basicavx                                   6.441 GB/s  ( 1.31 %)\n                                rangeavx                                   7.751 GB/s  ( 1.49 %)\n- 1, 2, 3 UTF8 bytes\nOverhead ==\u003e nanoseconds : 20.000, instructions 0.000\n                                  memcpy                                  74.145 GB/s  ( 8.25 %)\n                                  fushia                                   0.392 GB/s  ( 1.53 %)\n                            fushia_ascii                                   0.372 GB/s  ( 2.83 %)\n                           fushia_ascii2                                   0.363 GB/s  ( 3.29 %)\n                           fushia_ascii4                                   0.364 GB/s  ( 3.22 %)\n                                 utf8lib                                   0.258 GB/s  ( 0.76 %)\n                                     dfa                                   0.657 GB/s  ( 0.94 %)\n                                    dfa2                                   1.131 GB/s  ( 1.19 %)\n                                    dfa3                                   1.973 GB/s  ( 1.29 %)\n                                    dfa4                                   0.679 GB/s  ( 0.30 %)\n                              zwegneravx                                   9.401 GB/s  ( 1.28 %)\n                              lookup2avx                                  11.857 GB/s  ( 1.85 %)\n                              lookup3avx                                  12.489 GB/s  ( 2.57 %)\n                              lookup4avx                                  13.632 GB/s  ( 1.65 %)\n                                basicavx                                   6.441 GB/s  ( 1.20 %)\n                                rangeavx                                   7.792 GB/s  ( 1.99 %)\n- 1, 2, 3, 4 UTF8 bytes\nOverhead ==\u003e nanoseconds : 20.000, instructions 0.000\n                                  memcpy                                  74.486 GB/s  (10.24 %)\n                                  fushia                                   0.580 GB/s  ( 3.17 %)\n                            fushia_ascii                                   0.608 GB/s  ( 6.12 %)\n                           fushia_ascii2                                   0.537 GB/s  ( 5.54 %)\n                           fushia_ascii4                                   0.490 GB/s  ( 3.89 %)\n                                 utf8lib                                   0.286 GB/s  ( 1.89 %)\n                                     dfa                                   0.653 GB/s  ( 0.23 %)\n                                    dfa2                                   1.131 GB/s  ( 0.41 %)\n                                    dfa3                                   1.971 GB/s  ( 1.19 %)\n                                    dfa4                                   0.680 GB/s  ( 0.32 %)\n                              zwegneravx                                   9.402 GB/s  ( 1.05 %)\n                              lookup2avx                                  11.857 GB/s  ( 0.92 %)\n                              lookup3avx                                  12.396 GB/s  ( 0.98 %)\n                              lookup4avx                                  13.736 GB/s  ( 1.94 %)\n                                basicavx                                   6.441 GB/s  ( 1.11 %)\n                                rangeavx                                   7.792 GB/s  ( 1.73 %)\n```\n\n\n## Reference\n\n- John Keiser, Daniel Lemire, [Validating UTF-8 In Less Than One Instruction Per Byte](https://arxiv.org/abs/2010.03090), Software: Practice and Experience 51 (5), 2021\n\n\n## Credit\n\nA lot of the hard work is due to Keiser. Some of the code is based on code by Muła. The first SIMD UTF-8 validator was based on work by Willets. Some of our improvments were motivated by work by Zwegner who produced some of the finest SIMD-based UTF-8 validators.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemire%2Fvalidateutf8-experiments","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flemire%2Fvalidateutf8-experiments","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemire%2Fvalidateutf8-experiments/lists"}