{"id":13434983,"url":"https://github.com/lemire/fastbase64","last_synced_at":"2025-10-25T23:04:01.564Z","repository":{"id":43061085,"uuid":"75234804","full_name":"lemire/fastbase64","owner":"lemire","description":"SIMD-accelerated base64 codecs","archived":false,"fork":false,"pushed_at":"2024-04-04T15:32:41.000Z","size":4194,"stargazers_count":432,"open_issues_count":0,"forks_count":34,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-03-29T03:11:16.956Z","etag":null,"topics":["avx2","simd"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lemire.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-30T23:00:39.000Z","updated_at":"2025-03-22T10:58:20.000Z","dependencies_parsed_at":"2024-01-14T09:17:58.252Z","dependency_job_id":"c823faa8-629d-4095-a69e-6b7ba3041a51","html_url":"https://github.com/lemire/fastbase64","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Ffastbase64","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Ffastbase64/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Ffastbase64/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Ffastbase64/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lemire","download_url":"https://codeload.github.com/lemire/fastbase64/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247284954,"owners_count":20913704,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avx2","simd"],"created_at":"2024-07-31T03:00:29.379Z","updated_at":"2025-10-25T23:04:01.463Z","avatar_url":"https://github.com/lemire.png","language":"C","funding_links":[],"categories":["C"],"sub_categories":[],"readme":"# fastbase64\n\n**This is an unmaintained  software package that was used for research purposes. Do not use in production. [Use the simdutf library instead](https://github.com/simdutf/simdutf).** The simdutf library has superior support for base64.\n\n## Story\n\nWe are investigating the possibility of SIMD-accelerated base64 codecs. \n\n* Wojciech Muła, Daniel Lemire, Faster Base64 Encoding and Decoding Using AVX2 Instructions, ACM Transactions on the Web (to appear) https://arxiv.org/abs/1704.00605\n\nOur AVX2 decoder does not use floating-point numbers.\n\n## Sample usage\n\n\nWe extend's Nick Galbreath's base64 library (this high-performance library is used in Chromium).\n\nWe assume that you have an AVX2-capable machine (recent Intel processor from 2013 and up). In practice,\nwhich function is called should be determined based on the underlying hardware.\n\n### Encoding\n\nOriginal encoding...\n```\n char* src = ...;\n int srclen = ...; //the length of number of bytes in src\n char* dest = (char*) malloc(chromium_base64_encode_len(srclen)); // decode_len effectively multiplies by 4/3\n int len = chromium_base64_encode(dest, src, sourcelen); // returns how many bytes were decoded.\n```\nEncoding with AVX2...\n```\n char* src = ...;\n int srclen = ...; //the length of number of bytes in src\n char* dest = (char*) malloc(chromium_base64_encode_len(srclen));\n int len = fast_avx2_base64_encode(dest, src, sourcelen); // returns how many bytes were decoded.\n```\n\n### Decoding\n\nOriginal decoding...\n```\nchar* src = ...;\nint srclen = ...; // or if you don't know use strlen(src)\nchar* dest = (char*) malloc(chromium_base64_decode_len(srclen)); // effectively multiplies by 3/4\nint len = chromium_base64_decode(dest, src, sourcelen);\nif (len == MODP_B64_ERROR) { error }\n```\n\n\n\nDecoding with AVX2...\n\n```\nchar* src = ...;\nint srclen = ...; // or if you don't know use strlen(src)\nchar* dest = (char*) malloc(chromium_base64_decode_len(srclen)); // effectively multiplies by 3/4\nint len = fast_avx2_base64_decode(dest, src, sourcelen);\nif (len == MODP_B64_ERROR) { error }\n```\n\n\n\n\n\n## Results\n\nWe compare SIMD decoding with competitive alternatives.  One particularly fast decoder is used by Google Chrome (and available in Chromium). Chromium uses code produced by Nick Galbreath  called \"high performance base64 encoder / decoder\". We also use the decoder found in the Linux kernel as well as the one found in the QuickTime code (which was derived from code from the Apache HTTP server).\n\nLet us look at real data (images and text):\n\n```\n$ ./basic_benchmark\nrdtsc_overhead set to 30\nTesting first with random data.\nSee files encodingperf.txt decodingperf.txt ...\nTesting with real data.\nlena [jpg]\ndecoding a base64 input of  141020 bytes, original size = 105764\nmemcpy(buffer, data, datalength)                                :  0.09 cycles per operation (best)     0.09 cycles per operation (avg)\nlinux_base64_decode(buffer, data, data + datalength)            :  19.73 cycles per operation (best)     19.79 cycles per operation (avg)\nquicktime_base64_decode(buffer, data)                           :  3.10 cycles per operation (best)     3.17 cycles per operation (avg)\nchromium_base64_decode(buffer, data, datalength)                :  1.84 cycles per operation (best)     1.84 cycles per operation (avg)\nscalar_base64_decode(data,datalength,buffer,\u0026outputlength)      :  2.07 cycles per operation (best)     2.09 cycles per operation (avg)\nklomp_avx2_base64_decode(data,datalength,buffer,\u0026outputlength)    :  0.43 cycles per operation (best)     0.44 cycles per operation (avg)\nfast_avx2_base64_decode(buffer, data, datalength)               :  0.21 cycles per operation (best)     0.21 cycles per operation (avg)\n\npeppers [jpg]\ndecoding a base64 input of  12640 bytes, original size = 9478\nmemcpy(buffer, data, datalength)                                :  0.03 cycles per operation (best)     0.04 cycles per operation (avg)\nlinux_base64_decode(buffer, data, data + datalength)            :  15.05 cycles per operation (best)     15.94 cycles per operation (avg)\nquicktime_base64_decode(buffer, data)                           :  3.15 cycles per operation (best)     3.17 cycles per operation (avg)\nchromium_base64_decode(buffer, data, datalength)                :  1.82 cycles per operation (best)     1.82 cycles per operation (avg)\nscalar_base64_decode(data,datalength,buffer,\u0026outputlength)      :  2.08 cycles per operation (best)     2.09 cycles per operation (avg)\nklomp_avx2_base64_decode(data,datalength,buffer,\u0026outputlength)    :  0.44 cycles per operation (best)     0.44 cycles per operation (avg)\nfast_avx2_base64_decode(buffer, data, datalength)               :  0.21 cycles per operation (best)     0.21 cycles per operation (avg)\n\nmandril [jpg]\ndecoding a base64 input of  329632 bytes, original size = 247222\nmemcpy(buffer, data, datalength)                                :  0.11 cycles per operation (best)     0.12 cycles per operation (avg)\nlinux_base64_decode(buffer, data, data + datalength)            :  20.02 cycles per operation (best)     20.08 cycles per operation (avg)\nquicktime_base64_decode(buffer, data)                           :  3.10 cycles per operation (best)     3.17 cycles per operation (avg)\nchromium_base64_decode(buffer, data, datalength)                :  1.84 cycles per operation (best)     1.84 cycles per operation (avg)\nscalar_base64_decode(data,datalength,buffer,\u0026outputlength)      :  2.07 cycles per operation (best)     2.08 cycles per operation (avg)\nklomp_avx2_base64_decode(data,datalength,buffer,\u0026outputlength)    :  0.43 cycles per operation (best)     0.44 cycles per operation (avg)\nfast_avx2_base64_decode(buffer, data, datalength)               :  0.22 cycles per operation (best)     0.22 cycles per operation (avg)\n\nmoby_dick [text]\ndecoding a base64 input of  1484 bytes, original size = 1111\nmemcpy(buffer, data, datalength)                                :  0.04 cycles per operation (best)     0.05 cycles per operation (avg)\nlinux_base64_decode(buffer, data, data + datalength)            :  3.61 cycles per operation (best)     4.33 cycles per operation (avg)\nquicktime_base64_decode(buffer, data)                           :  3.20 cycles per operation (best)     3.21 cycles per operation (avg)\nchromium_base64_decode(buffer, data, datalength)                :  1.83 cycles per operation (best)     1.83 cycles per operation (avg)\nscalar_base64_decode(data,datalength,buffer,\u0026outputlength)      :  2.11 cycles per operation (best)     2.13 cycles per operation (avg)\nklomp_avx2_base64_decode(data,datalength,buffer,\u0026outputlength)    :  0.53 cycles per operation (best)     0.54 cycles per operation (avg)\nfast_avx2_base64_decode(buffer, data, datalength)               :  0.28 cycles per operation (best)     0.29 cycles per operation (avg)\n\ngoogle logo [png]\ndecoding a base64 input of  3144 bytes, original size = 2357\nmemcpy(buffer, data, datalength)                                :  0.05 cycles per operation (best)     0.05 cycles per operation (avg)\nlinux_base64_decode(buffer, data, data + datalength)            :  4.36 cycles per operation (best)     6.37 cycles per operation (avg)\nquicktime_base64_decode(buffer, data)                           :  3.16 cycles per operation (best)     3.18 cycles per operation (avg)\nchromium_base64_decode(buffer, data, datalength)                :  1.81 cycles per operation (best)     1.82 cycles per operation (avg)\nscalar_base64_decode(data,datalength,buffer,\u0026outputlength)      :  2.09 cycles per operation (best)     2.10 cycles per operation (avg)\nklomp_avx2_base64_decode(data,datalength,buffer,\u0026outputlength)    :  0.47 cycles per operation (best)     0.47 cycles per operation (avg)\nfast_avx2_base64_decode(buffer, data, datalength)               :  0.23 cycles per operation (best)     0.24 cycles per operation (avg)\n\nbing.com social icons [png]\ndecoding a base64 input of  1808 bytes, original size = 1355\nmemcpy(buffer, data, datalength)                                :  0.04 cycles per operation (best)     0.04 cycles per operation (avg)\nlinux_base64_decode(buffer, data, data + datalength)            :  3.97 cycles per operation (best)     5.25 cycles per operation (avg)\nquicktime_base64_decode(buffer, data)                           :  3.13 cycles per operation (best)     3.20 cycles per operation (avg)\nchromium_base64_decode(buffer, data, datalength)                :  1.82 cycles per operation (best)     1.83 cycles per operation (avg)\nscalar_base64_decode(data,datalength,buffer,\u0026outputlength)      :  2.11 cycles per operation (best)     2.27 cycles per operation (avg)\nklomp_avx2_base64_decode(data,datalength,buffer,\u0026outputlength)    :  0.46 cycles per operation (best)     0.47 cycles per operation (avg)\nfast_avx2_base64_decode(buffer, data, datalength)               :  0.24 cycles per operation (best)     0.24 cycles per operation (avg)\n```\n\nNext plot shows results using random data of varying size:\n\n\u003cimg src=\"https://github.com/lemire/fastbase64/blob/master/results/skylake_decoding_cyclesperinputbyte.png\" width=\"50%\" /\u003e\n\nWe see that for base64 inputs of 100 bytes or more the AVX2 decoder is much faster, being more than three times faster.\n\n\n## How does SIMD base64 decoding works?\n\nLet us focus on decoding, the most performance-sensitive task.\n\n### Character decoding (lookup)\n\nBase64 writes 6-bit bytes in text form, not as byte values in [0,64). It is useful to take the text input and convert it to values in [0,64) if we want to decode base64 text. (This is not a necessary step however: some high performance base64 decoders do not include such a separate step, decoding base64 in one pass instead.) Muła calls this a lookup, possibly because it is commonly solved using a lookup table.\n\nMuła showed (https://github.com/WojciechMula/base64simd) that you could quickly take a 32-byte vector of base64 encoded text and convert it to an array of integers in [0,64) using shifts, bitwise logical operations and shuffles. It is fast.\n\n### Bit packing\n\nGiven the byte values in [0,64), i.e., 6-bit values, we must then pack them to finish the decoding. Base64 works by packing 4 bytes into 3 bytes as follows. The normal 4-byte to 3-byte base64 decoding routine goes as follows...\n\n```\noutput[0] =  ( input[0] \u003c\u003c 2 ) | ( input[1] \u003e\u003e 4)\noutput[1] =  ( input[1] \u003c\u003c 4 ) | ( input[2] \u003e\u003e 2)\noutput[2] =  ( input[3] \u003c\u003c 6 ) |  input[3]\n```\n\nSee https://en.wikipedia.org/wiki/Base64#Sample_Implementation_in_Java for a reference implementation.\n\n(Base64 decoders such as the one in the Chromium code base avoid shifts entirely by looking up bytes as \"pre-shifted\" 32-bit values.)\n\n\nMuła addresses the issue of \"gathering data\" from the result of the lookup:\nhttp://0x80.pl/notesen/2016-01-17-sse-base64-decoding.html#gathering-data\n\n\nIn a naive form, Muła suggests we use code as this :\n\n```\nconst __m128i bits_a = _mm_and_si128(values, _mm256_set1_epi32(0x0000003f));\nconst __m128i bits_b = _mm_srli_epi32(_mm_and_si128(values, _mm256_set1_epi32(0x00003f00)), 2);\nconst __m128i bits_c = _mm_srli_epi32(_mm_and_si128(values, _mm256_set1_epi32(0x003f0000)), 4);\nconst __m128i bits_d = _mm_srli_epi32(_mm_and_si128(values, _mm256_set1_epi32(0x3f000000)), 6);\n\nresult = _mm_or_si128(bits_a, _mm_or_si128(bits_b, _mm_or_si128(bits_c, bits_d)));\n```\n\nThis almost correct, but base64 works in big endian mode so proper byte shuffling is needed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemire%2Ffastbase64","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flemire%2Ffastbase64","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemire%2Ffastbase64/lists"}