{"id":13465058,"url":"https://github.com/ronomon/crypto-async","last_synced_at":"2025-03-25T13:32:52.522Z","repository":{"id":46207151,"uuid":"70810182","full_name":"ronomon/crypto-async","owner":"ronomon","description":"Fast, reliable cipher, hash and hmac methods executed in Node's threadpool for multi-core throughput.","archived":true,"fork":false,"pushed_at":"2021-11-07T01:21:34.000Z","size":67,"stargazers_count":173,"open_issues_count":6,"forks_count":15,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-25T03:51:42.697Z","etag":null,"topics":["cipher","crypto","hash","hmac"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ronomon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-13T13:37:47.000Z","updated_at":"2024-11-28T16:32:57.000Z","dependencies_parsed_at":"2022-09-24T17:22:12.313Z","dependency_job_id":null,"html_url":"https://github.com/ronomon/crypto-async","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ronomon%2Fcrypto-async","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ronomon%2Fcrypto-async/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ronomon%2Fcrypto-async/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ronomon%2Fcrypto-async/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ronomon","download_url":"https://codeload.github.com/ronomon/crypto-async/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245471236,"owners_count":20620904,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cipher","crypto","hash","hmac"],"created_at":"2024-07-31T14:00:56.594Z","updated_at":"2025-03-25T13:32:52.162Z","avatar_url":"https://github.com/ronomon.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# @ronomon/crypto-async\nFast, reliable cipher, hash and hmac methods executed in Node's threadpool for\nmulti-core throughput.\n\n## Motivation\n#### Some longstanding issues with Node's `crypto` module\n* Did you know that Node's cipher, hash and hmac streams are not truly\nasynchronous? They execute in C, but only in the main thread and so the `crypto`\nmodule **blocks your event loop**. Encrypting 64 MB of data might block your\nevent loop for +/- 70ms. Hashing 64 MB of data might block your event loop for\n+/- 190ms. This will spike any concurrent user-visible request latencies.\n* Worse, the `crypto` module **does not take advantage of multiple CPU cores**.\nYour server may have four CPU cores but `crypto` will use only one of these four\nCPU cores for all encrypting and hashing operations. The `cluster` module with\nits IPC overhead is not an efficient solution to multi-core crypto.\n* The `crypto` module was sadly **not designed to use statically allocated\nbuffers**, allocating a new output buffer when encrypting or hashing data,\neven if you already have an output buffer available. If you want to hash only a\nportion of a buffer you must first create a slice. Creating thousands of\nJavascript objects in this way **strains the GC**, leads to longer GC pauses and\nfurther blocks your event loop.\n* The `crypto` module forces **multiple unnecessary roundtrips between JS and\nC** even if you are only encrypting or hashing a single buffer. When your buffer\nis small (less than a few hundred bytes), this calling overhead alone, of a few\nhundred nanoseconds per call, can double your latencies and halve your\nthroughput.\n* In summary, the `crypto` module is **not suitable for high-throughput network\nprotocols or storage systems**, which need to checksum and encrypt/decrypt huge\namounts of data concurrently. Such a user-space network protocol or storage\nsystem using the `crypto` module might saturate a single CPU core with crypto\noperations well before saturating a fast local network or SSD disk.\n\n#### Some new ideas for `@ronomon/crypto-async`\n* **Truly asynchronous.** All operations can execute asynchronously in Node's\nthreadpool. This keeps your event loop free from blocking.\n* **Scales across multiple CPU cores.** While `@ronomon/crypto-async` is a\nfraction slower per call than `crypto` because of the overhead of pushing tasks\ninto the threadpool, for buffers larger than 1024 bytes it shines and provides\nnearly N-cores more throughput. Don't let your CPU cores go to waste.\n* **Zero-copy.** All keys, ivs, source and target arguments can be passed\ndirectly using offsets into existing buffers, without requiring any slices and\nwithout allocating any temporary output buffers. This enables predictable memory\nusage for programs with tight memory budgets.\n* **Fast.** Supports the common use case of encrypting or hashing a single\nbuffer, to avoid multiple round-trips between JS and C. This halves latencies\nand doubles throughput for small buffers.\n* **Synchronous where it makes sense.** While you should use asynchronous\nmethods for large buffers to improve throughput, you can also use synchronous\nmethods for small buffers to achieve optimal latency.\n\n## Performance\n```\n\n                CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz\n              Cores: 8\n            Threads: 4\n\n========================================================================\n\n        aes-256-ctr: 16384 x 256 Bytes\n               node: Latency: 0.008ms Throughput: 29.09 MB/s\n      sync @ronomon: Latency: 0.003ms Throughput: 76.70 MB/s\n     async @ronomon: Latency: 0.047ms Throughput: 21.04 MB/s\n\n        aes-256-ctr: 16384 x 1024 Bytes\n               node: Latency: 0.007ms Throughput: 132.43 MB/s\n      sync @ronomon: Latency: 0.003ms Throughput: 340.46 MB/s\n     async @ronomon: Latency: 0.045ms Throughput: 88.86 MB/s\n\n        aes-256-ctr: 16384 x 4096 Bytes\n               node: Latency: 0.009ms Throughput: 439.00 MB/s\n      sync @ronomon: Latency: 0.004ms Throughput: 1010.11 MB/s\n     async @ronomon: Latency: 0.043ms Throughput: 376.36 MB/s\n\n        aes-256-ctr: 1024 x 65536 Bytes\n               node: Latency: 0.046ms Throughput: 1402.22 MB/s\n      sync @ronomon: Latency: 0.030ms Throughput: 2154.93 MB/s\n     async @ronomon: Latency: 0.088ms Throughput: 2938.77 MB/s\n\n        aes-256-ctr: 64 x 1048576 Bytes\n               node: Latency: 0.717ms Throughput: 1460.21 MB/s\n      sync @ronomon: Latency: 0.452ms Throughput: 2314.90 MB/s\n     async @ronomon: Latency: 1.372ms Throughput: 3013.60 MB/s\n\n========================================================================\n\n        aes-256-gcm: 16384 x 256 Bytes\n               node: Latency: 0.009ms Throughput: 27.99 MB/s\n      sync @ronomon: Latency: 0.003ms Throughput: 82.62 MB/s\n     async @ronomon: Latency: 0.042ms Throughput: 24.11 MB/s\n\n        aes-256-gcm: 16384 x 1024 Bytes\n               node: Latency: 0.009ms Throughput: 105.41 MB/s\n      sync @ronomon: Latency: 0.004ms Throughput: 253.50 MB/s\n     async @ronomon: Latency: 0.042ms Throughput: 94.61 MB/s\n\n        aes-256-gcm: 16384 x 4096 Bytes\n               node: Latency: 0.013ms Throughput: 314.20 MB/s\n      sync @ronomon: Latency: 0.006ms Throughput: 621.70 MB/s\n     async @ronomon: Latency: 0.043ms Throughput: 375.18 MB/s\n\n        aes-256-gcm: 1024 x 65536 Bytes\n               node: Latency: 0.091ms Throughput: 719.20 MB/s\n      sync @ronomon: Latency: 0.061ms Throughput: 1065.52 MB/s\n     async @ronomon: Latency: 0.113ms Throughput: 2285.47 MB/s\n\n        aes-256-gcm: 64 x 1048576 Bytes\n               node: Latency: 1.063ms Throughput: 986.12 MB/s\n      sync @ronomon: Latency: 0.944ms Throughput: 1109.59 MB/s\n     async @ronomon: Latency: 1.516ms Throughput: 2715.93 MB/s\n\n========================================================================\n\n             sha256: 16384 x 256 Bytes\n               node: Latency: 0.007ms Throughput: 36.79 MB/s\n      sync @ronomon: Latency: 0.002ms Throughput: 101.47 MB/s\n     async @ronomon: Latency: 0.042ms Throughput: 24.05 MB/s\n\n             sha256: 16384 x 1024 Bytes\n               node: Latency: 0.008ms Throughput: 124.19 MB/s\n      sync @ronomon: Latency: 0.004ms Throughput: 224.30 MB/s\n     async @ronomon: Latency: 0.043ms Throughput: 92.59 MB/s\n\n             sha256: 16384 x 4096 Bytes\n               node: Latency: 0.016ms Throughput: 240.94 MB/s\n      sync @ronomon: Latency: 0.013ms Throughput: 319.26 MB/s\n     async @ronomon: Latency: 0.040ms Throughput: 398.04 MB/s\n\n             sha256: 2048 x 65536 Bytes\n               node: Latency: 0.201ms Throughput: 325.30 MB/s\n      sync @ronomon: Latency: 0.188ms Throughput: 349.06 MB/s\n     async @ronomon: Latency: 0.273ms Throughput: 955.41 MB/s\n\n             sha256: 128 x 1048576 Bytes\n               node: Latency: 3.013ms Throughput: 347.94 MB/s\n      sync @ronomon: Latency: 3.003ms Throughput: 349.09 MB/s\n     async @ronomon: Latency: 3.310ms Throughput: 1257.44 MB/s\n\n========================================================================\n\n        hmac-sha256: 16384 x 256 Bytes\n               node: Latency: 0.009ms Throughput: 27.94 MB/s\n      sync @ronomon: Latency: 0.003ms Throughput: 69.70 MB/s\n     async @ronomon: Latency: 0.038ms Throughput: 26.30 MB/s\n\n        hmac-sha256: 16384 x 1024 Bytes\n               node: Latency: 0.010ms Throughput: 97.52 MB/s\n      sync @ronomon: Latency: 0.006ms Throughput: 176.88 MB/s\n     async @ronomon: Latency: 0.036ms Throughput: 111.07 MB/s\n\n        hmac-sha256: 16384 x 4096 Bytes\n               node: Latency: 0.019ms Throughput: 212.33 MB/s\n      sync @ronomon: Latency: 0.014ms Throughput: 285.50 MB/s\n     async @ronomon: Latency: 0.039ms Throughput: 411.16 MB/s\n\n        hmac-sha256: 2048 x 65536 Bytes\n               node: Latency: 0.198ms Throughput: 330.22 MB/s\n      sync @ronomon: Latency: 0.191ms Throughput: 342.88 MB/s\n     async @ronomon: Latency: 0.256ms Throughput: 1019.00 MB/s\n\n        hmac-sha256: 128 x 1048576 Bytes\n               node: Latency: 3.025ms Throughput: 346.55 MB/s\n      sync @ronomon: Latency: 2.926ms Throughput: 358.31 MB/s\n     async @ronomon: Latency: 3.214ms Throughput: 1298.56 MB/s\n\n```\n\n## Installation\nThis will install `@ronomon/crypto-async` and compile the native binding\nautomatically:\n```\nnpm install @ronomon/crypto-async\n```\n\n## Usage\n\n#### Adjust threadpool size and control concurrency\nNode runs filesystem and DNS operations in the threadpool. The threadpool\nconsists of 4 threads by default, which is far from optimal. This means that at\nmost 4 operations can be running at any point in time. If any operation is slow\nto complete, it will cause head-of-line blocking, otherwise known as the Convoy\neffect.\n\nThe size of the threadpool should therefore be increased at startup time (at the\ntop of your script, before requiring any modules) by setting the\n`UV_THREADPOOL_SIZE` environment variable. The absolute maximum is 128 threads,\nwhich requires only ~1 MB memory in total according to the\n[libuv docs](http://docs.libuv.org/en/v1.x/threadpool.html).\n\nAgain, conventional wisdom would set the number of threads to the number of CPU\ncores, but most operations running in the threadpool are not run hot, they are\nnot CPU-intensive and block mostly on IO. Issuing more IO operations than there\nare CPU cores will increase throughput and will decrease latency per operation\nby decreasing queueing time. On the other hand, `@ronomon/crypto-async` is\nCPU-intensive. Issuing more `@ronomon/crypto-async` operations than there\nare CPU cores will not increase throughput and will increase latency per\noperation by increasing queueing time.\n\nYou should therefore:\n\n1. Set the threadpool size to `IO` + `N`, where `IO` is the number of filesystem\nand DNS operations you expect to be running concurrently, and where `N` is the\nnumber of CPU cores available. This will reduce head-of-line blocking.\n\n2. Allow or design for at most `N` `@ronomon/crypto-async` operations to be\nrunning concurrently, where `N` is the number of CPU cores available. This will\nkeep latency within reasonable bounds.\n\n```javascript\n// At the top of your script, before requiring any modules:\nprocess.env['UV_THREADPOOL_SIZE'] = 128;\n```\n\n#### Synchronous method alternatives\nAll methods have a synchronous method alternative: just leave out the callback\nwhen calling the method. These are convenient for small buffers and outperform the\n`crypto` module equivalents.\n\n#### Cipher whitelist\n`@ronomon/crypto-async` disables slow, complicated ciphers such as CCM and\n[dangerous ciphers](https://blog.cloudflare.com/padding-oracles-and-the-decline-of-cbc-mode-ciphersuites) such as CBC and ECB.\nA limited whitelist of stream ciphers and AEAD ciphers are supported. This is a\ngood thing in the interest of a safe implementation.\n\n##### Supported stream ciphers\nThese are dangerous if you do not [encrypt-then-mac](http://www.daemonology.net/blog/2009-06-24-encrypt-then-mac.html):\n\n* **chacha20** (keySize=32, ivSize=16)\n* **aes-256-ctr** (keySize=32, ivSize=16)\n* **aes-192-ctr** (keySize=24, ivSize=16)\n* **aes-128-ctr** (keySize=16, ivSize=16)\n\n##### Supported AEAD ciphers\nThese are recommended over stream ciphers for safety, ease-of-use and\nefficiency:\n\n* **chacha20-poly1305** (keySize=32, ivSize=12, tagSize=16)\n* **aes-256-gcm** (keySize=32, ivSize=12, tagSize=16)\n* **aes-128-gcm** (keySize=16, ivSize=12, tagSize=16)\n\n#### Cipher\n```javascript\nvar cryptoAsync = require('@ronomon/crypto-async');\nvar algorithm = 'aes-256-ctr';\nvar encrypt = 1; // Encrypt\nvar key = Buffer.alloc(32);\nvar iv = Buffer.alloc(16);\nvar plaintext = Buffer.alloc(128);\ncryptoAsync.cipher(algorithm, encrypt, key, iv, plaintext,\n  function(error, ciphertext) {\n    if (error) throw error;\n    console.log('ciphertext:', ciphertext.toString('hex'));\n    var encrypt = 0; // Decrypt\n    cryptoAsync.cipher(algorithm, encrypt, key, iv, ciphertext,\n      function(error, plaintext) {\n        if (error) throw error;\n        console.log('plaintext:', plaintext.toString('hex'));\n      }\n    );\n  }\n);\n```\n\n#### Cipher (AEAD)\n```javascript\nvar cryptoAsync = require('@ronomon/crypto-async');\nvar algorithm = 'chacha20-poly1305';\nvar encrypt = 1; // Encrypt\nvar key = Buffer.alloc(32);\nvar iv = Buffer.alloc(12);\nvar plaintext = Buffer.alloc(128);\nvar aad = Buffer.alloc(256);\nvar tag = Buffer.alloc(16);\ncryptoAsync.cipher(algorithm, encrypt, key, iv, plaintext, aad, tag,\n  function(error, ciphertext) {\n    if (error) throw error;\n    console.log('ciphertext:', ciphertext.toString('hex'));\n    console.log('tag:', tag.toString('hex'));\n    var encrypt = 0; // Decrypt\n    cryptoAsync.cipher(algorithm, encrypt, key, iv, ciphertext, aad, tag,\n      function(error, plaintext) {\n        if (error) {\n          if (error.message === cryptoAsync.E_CORRUPT) {\n            throw new Error('key/iv/source/aad/tag failed authentication');\n          } else {\n            throw error;\n          }\n        }\n        console.log('plaintext:', plaintext.toString('hex'));\n      }\n    );\n  }\n);\n```\n\n#### Hash\n```javascript\nvar cryptoAsync = require('@ronomon/crypto-async');\nvar algorithm = 'sha256';\nvar source = Buffer.alloc(1024 * 1024);\ncryptoAsync.hash(algorithm, source,\n  function(error, hash) {\n    if (error) throw error;\n    console.log('hash:', hash.toString('hex'));\n  }\n);\n```\n\n#### HMAC\n```javascript\nvar cryptoAsync = require('@ronomon/crypto-async');\nvar algorithm = 'sha256';\nvar key = Buffer.alloc(1024);\nvar source = Buffer.alloc(1024 * 1024);\ncryptoAsync.hmac(algorithm, key, source,\n  function(error, hmac) {\n    if (error) throw error;\n    console.log('hmac:', hmac.toString('hex'));\n  }\n);\n```\n\n### Zero-Copy Methods\n\nThese methods require more arguments but support zero-copy crypto\noperations for reduced memory overhead and GC pressure.\n\n#### Cipher (Zero-Copy)\n```javascript\nvar cryptoAsync = require('@ronomon/crypto-async');\nvar algorithm = 'aes-256-ctr';\nvar encrypt = 1; // Encrypt\nvar key = Buffer.alloc(1024);\nvar keyOffset = 4;\nvar keySize = 32;\nvar iv = Buffer.alloc(32);\nvar ivOffset = 2;\nvar ivSize = 16;\nvar source = Buffer.alloc(1024 * 1024);\nvar sourceOffset = 512;\nvar sourceSize = 32;\nvar target = Buffer.alloc(sourceSize + cryptoAsync.CIPHER_BLOCK_MAX);\nvar targetOffset = 0;\ncryptoAsync.cipher(\n  algorithm,\n  encrypt,\n  key,\n  keyOffset,\n  keySize,\n  iv,\n  ivOffset,\n  ivSize,\n  source,\n  sourceOffset,\n  sourceSize,\n  target,\n  targetOffset,\n  function(error, targetSize) {\n    if (error) throw error;\n    var slice = target.slice(targetOffset, targetOffset + targetSize);\n    console.log('ciphertext:', slice.toString('hex'));\n  }\n);\n```\n\n#### Cipher (Zero-Copy, AEAD)\n```javascript\nvar cryptoAsync = require('@ronomon/crypto-async');\nvar algorithm = 'chacha20-poly1305';\nvar encrypt = 1; // Encrypt\nvar key = Buffer.alloc(1024);\nvar keyOffset = 4;\nvar keySize = 32;\nvar iv = Buffer.alloc(32);\nvar ivOffset = 2;\nvar ivSize = 12;\nvar source = Buffer.alloc(1024 * 1024);\nvar sourceOffset = 512;\nvar sourceSize = 32;\nvar target = Buffer.alloc(sourceSize + cryptoAsync.CIPHER_BLOCK_MAX);\nvar targetOffset = 0;\nvar aad = Buffer.alloc(1024);\nvar aadOffset = 0;\nvar aadSize = 10;\nvar tag = Buffer.alloc(16);\nvar tagOffset = 0;\nvar tagSize = 16;\ncryptoAsync.cipher(\n  algorithm,\n  encrypt,\n  key,\n  keyOffset,\n  keySize,\n  iv,\n  ivOffset,\n  ivSize,\n  source,\n  sourceOffset,\n  sourceSize,\n  target,\n  targetOffset,\n  aad,\n  aadOffset,\n  aadSize,\n  tag,\n  tagOffset,\n  tagSize,\n  function(error, targetSize) {\n    if (error) {\n      if (error.message === cryptoAsync.E_CORRUPT) {\n        throw new Error('key/iv/source/aad/tag failed authentication');\n      } else {\n        throw error;\n      }\n    }\n    var slice = target.slice(targetOffset, targetOffset + targetSize);\n    console.log('ciphertext:', slice.toString('hex'));\n    console.log('tag:', tag.toString('hex', tagOffset, tagOffset + tagSize));\n  }\n);\n```\n\n#### Hash (Zero-Copy)\n```javascript\nvar cryptoAsync = require('@ronomon/crypto-async');\nvar algorithm = 'sha256';\nvar source = Buffer.alloc(1024 * 1024);\nvar sourceOffset = 512;\nvar sourceSize = 65536;\nvar target = Buffer.alloc(1024 * 1024);\nvar targetOffset = 32768;\ncryptoAsync.hash(\n  algorithm,\n  source,\n  sourceOffset,\n  sourceSize,\n  target,\n  targetOffset,\n  function(error, targetSize) {\n    if (error) throw error;\n    var slice = target.slice(targetOffset, targetOffset + targetSize);\n    console.log('hash:', slice.toString('hex'));\n  }\n);\n```\n\n#### HMAC (Zero-Copy)\n```javascript\nvar cryptoAsync = require('@ronomon/crypto-async');\nvar algorithm = 'sha256';\nvar key = Buffer.alloc(1024);\nvar keyOffset = 4;\nvar keySize = 8;\nvar source = Buffer.alloc(1024 * 1024);\nvar sourceOffset = 512;\nvar sourceSize = 65536;\nvar target = Buffer.alloc(1024 * 1024);\nvar targetOffset = 32768;\ncryptoAsync.hmac(\n  algorithm,\n  key,\n  keyOffset,\n  keySize,\n  source,\n  sourceOffset,\n  sourceSize,\n  target,\n  targetOffset,\n  function(error, targetSize) {\n    if (error) throw error;\n    var slice = target.slice(targetOffset, targetOffset + targetSize);\n    console.log('hmac:', slice.toString('hex'));\n  }\n);\n```\n\n## Tests\n`@ronomon/crypto-async` ships with comprehensive fuzz tests, which have\nuncovered multiple bugs in OpenSSL:\n\n* [CVE-2019-1543: chacha20-poly1305 fails to detect IV tampering, where IV \u003e 12 and IV \u003c= CHACHA_CTR_SIZE](https://www.openssl.org/news/secadv/20190306.txt)\n\n* [EVP_CTRL_AEAD_SET_TAG fails for OCB](https://github.com/openssl/openssl/issues/8331)\n\n* [AEAD: EVP_CIPHER_CTX_iv_length is oblivious to EVP_CTRL_AEAD_SET_IVLEN](https://github.com/openssl/openssl/issues/8330)\n\n* [EVP_CipherUpdate() setting AAD for AES-256-OCB returns incorrect `outlen`](https://github.com/openssl/openssl/issues/8310)\n\nTo run the tests:\n```\nnode test.js\n```\n\n## Benchmark\nTo benchmark `@ronomon/crypto-async` vs Node's `crypto`:\n```\nnode benchmark.js\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fronomon%2Fcrypto-async","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fronomon%2Fcrypto-async","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fronomon%2Fcrypto-async/lists"}