{"id":17769853,"url":"https://github.com/wojciechmula/base64simd","last_synced_at":"2025-04-05T05:05:13.192Z","repository":{"id":66099957,"uuid":"67354721","full_name":"WojciechMula/base64simd","owner":"WojciechMula","description":"Base64 coding and decoding with SIMD instructions (SSE/AVX2/AVX512F/AVX512BW/AVX512VBMI/ARM Neon)","archived":false,"fork":false,"pushed_at":"2025-02-21T23:00:12.000Z","size":425,"stargazers_count":163,"open_issues_count":4,"forks_count":14,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-03-29T04:06:09.807Z","etag":null,"topics":["avx2","avx512","base64","neon","simd","sse"],"latest_commit_sha":null,"homepage":"http://0x80.pl/articles/index.html#base64-algorithm-new","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WojciechMula.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-04T16:02:47.000Z","updated_at":"2025-03-06T04:03:02.000Z","dependencies_parsed_at":"2024-03-28T20:04:43.023Z","dependency_job_id":"c297fc90-7939-4dee-a59c-fa4b6f9789bb","html_url":"https://github.com/WojciechMula/base64simd","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WojciechMula%2Fbase64simd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WojciechMula%2Fbase64simd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WojciechMula%2Fbase64simd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WojciechMula%2Fbase64simd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WojciechMula","download_url":"https://codeload.github.com/WojciechMula/base64simd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247289426,"owners_count":20914464,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avx2","avx512","base64","neon","simd","sse"],"created_at":"2024-10-26T21:19:23.627Z","updated_at":"2025-04-05T05:05:13.177Z","avatar_url":"https://github.com/WojciechMula.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"================================================================================\n                        base64 using SIMD instructions\n================================================================================\n\nOverview\n--------------------------------------------------\n\nRepository contains code for encoding and decoding base64 using SIMD instructions.\nDepending on CPU's architecture, vectorized encoding is faster than scalar\nversions by factor from **2 to 4**; decoding is faster **2 .. 2.7** times.\n\nThere are several versions of procedures utilizing following instructions sets:\n\n* SSE,\n* AVX2,\n* AVX512F,\n* AVX512BW,\n* AVX512VBMI,\n* AVX512VL,\n* BMI2, and\n* ARM Neon.\n\nVectorization approaches were described in a series of articles:\n\n* `Base64 encoding with SIMD instructions`__,\n* `Base64 decoding with SIMD instructions`__,\n* `Base64 encoding \u0026 decoding using AVX512BW instructions`__ (includes AVX512VBMI and AVX512VL),\n* `AVX512F base64 coding and decoding`__.\n\n__ http://0x80.pl/notesen/2016-01-12-sse-base64-encoding.html\n__ http://0x80.pl/notesen/2016-01-17-sse-base64-decoding.html\n__ http://0x80.pl/notesen/2016-04-03-avx512-base64.html\n__ http://0x80.pl/articles/avx512-foundation-base64.html\n\n`Daniel Lemire`__ and I wrote also paper `Faster Base64 Encoding\nand Decoding Using AVX2 Instructions`__ which was published\nby `ACM Transactiona on the Web`__.\n\n__ http://lemire.me\n__ https://arxiv.org/abs/1704.00605\n__ https://tweb.acm.org/\n\nPerformance results from various machines are located\nin subdirectories ``results``.\n\n\nProject organization\n--------------------------------------------------\n\nThere are separate subdirectories for both algorithms, however both have\nthe same structure. Each project contains four programs:\n\n* ``verify`` --- does simple validation of particular parts of algorithms,\n* ``check`` --- validates whole procedures,\n* ``speed`` --- compares speed of different variants of procedures,\n* ``benchmark`` --- similarly to ``speed`` but works on small buffers and\n  calculates CPU cycle rate (available only for Intel architectures).\n\nBuilding\n--------------------------------------------------\n\nChange to either directory ``encode`` or ``decode`` and then use following\n``make`` commands.\n\n.. list-table::\n    :header-rows: 1\n\n    * - command\n      - tools\n      - instruction sets\n\n    * - ``make``\n      - ``verify``, ``check``, ``speed``, ``benchmark``\n      - scalar, SSE, BMI2\n\n    * - ``make avx2``\n      - ``verify_avx2``, ``check_avx2``, ``speed_avx2``, ``benchmark_avx2``\n      - scalar, SSE, BMI2, AVX2\n\n    * - ``make avx512``\n      - ``verify_avx512``, ``check_avx512``, ``speed_avx512``, ``benchmark_avx512``\n      - scalar, SSE, BMI2, AVX2, AVX512F\n\n    * - ``make avx512bw``\n      - ``verify_avx512bw``, ``check_avx512bw``, ``speed_avx512bw``, ``benchmark_avx512bw``\n      - scalar, SSE, BMI2, AVX2, AVX512F, AVX512BW\n\n    * - ``make avx512vbmi``\n      - ``verify_avx512vbmi``, ``check_avx512vbmi``, ``benchmark_avx512vbmi``\n      - scalar, SSE, BMI2, AVX2, AVX512F, AVX512BW, AVX512VBMI \n    \n    * - ``make xop``\n      - ``verify_xop``, ``check_xop``, ``speed_xop``, ``benchmark_xop``\n      - scalar, SSE and AMD XOP\n\n    * - ``make arm``\n      - ``verify_arm``, ``check_arm``, ``speed_arm``\n      - scalar, ARM Neon\n\nType ``make run`` (for SSE) or ``make run_ARCH`` to run all programs for given\ninstruction sets; ``ARCH`` can be \"sse\", \"avx2\", \"avx512\", \"avx512bw\",\n\"avx512vbmi\", \"avx512vl\".\n\nBMI2 presence is determined based on ``/proc/cpuinfo`` or a counterpart.\nWhen an AVX2 or AVX512 targets are used then BMI2 is enabled by default.\n\n\nAVX512\n--------------------------------------------------\n\nTo compile AVX512 versions of the programs at least GCC 5.3 is required.\nGCC 4.9.2 doesn't have AVX512 support.\n\nPlease download `Intel Software Development Emulator`__ in order to run AVX512\nvariants via ``make run_avx512``, ``run_avx512bw`` or ``run_avx512vbmi``.\nThe emulator path should be added to the ``PATH``.\n\n__ https://software.intel.com/en-us/articles/intel-software-development-emulator\n\n\nKnown problems\n--------------------------------------------------\n\nBoth encoding and decoding don't match the base64 specification,\nthere is no processing of data tail, i.e. encoder never produces\n'=' chars at the end, and decoder doesn't handle them at all.\n\nAll these shortcoming are not present in a brilliant library\nby Alfred Klomp: https://github.com/aklomp/base64.\n\n\nSee also\n--------------------------------------------------\n\n* Daniel's benchmarks and comparison with state of the art solutions\n  https://github.com/lemire/fastbase64\n\n\nWho uses our algorithms?\n--------------------------------------------------\n\n* C/C++ library by **Alfred Klomp** https://github.com/aklomp/base64\n* .NET library by **Günther Foidl** https://github.com/gfoidl/Base64\n* there was attempt to include an assembly implementation into Go:\n  https://github.com/golang/go/issues/20206\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwojciechmula%2Fbase64simd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwojciechmula%2Fbase64simd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwojciechmula%2Fbase64simd/lists"}