{"id":32578318,"url":"https://github.com/animetosho/rapidyenc","last_synced_at":"2025-10-29T14:56:16.092Z","repository":{"id":97179507,"uuid":"590448679","full_name":"animetosho/rapidyenc","owner":"animetosho","description":"SIMD accelerated yEnc en/decode C library","archived":false,"fork":false,"pushed_at":"2024-08-06T04:32:39.000Z","size":467,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-08-06T07:01:23.271Z","etag":null,"topics":["crc32","usenet","yenc","yencode"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/animetosho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-18T12:51:56.000Z","updated_at":"2024-08-06T04:32:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"f6010a31-df06-4ec3-b629-6ae48ebabbc1","html_url":"https://github.com/animetosho/rapidyenc","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/animetosho/rapidyenc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/animetosho%2Frapidyenc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/animetosho%2Frapidyenc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/animetosho%2Frapidyenc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/animetosho%2Frapidyenc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/animetosho","download_url":"https://codeload.github.com/animetosho/rapidyenc/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/animetosho%2Frapidyenc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281642036,"owners_count":26536358,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-29T02:00:06.901Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crc32","usenet","yenc","yencode"],"created_at":"2025-10-29T14:56:10.787Z","updated_at":"2025-10-29T14:56:16.086Z","avatar_url":"https://github.com/animetosho.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"This C compatible library provides functions for implementing [yEnc](http://www.yenc.org/yenc-draft.1.3.txt) where speed is important.\n\nNote that it only handles the underlying encoding/decoding routines - yEnc headers aren’t handled.\n\nFeatures\n---------\n\n-   implementation uses x86/ARM/RISC-V SIMD capabilities, with support for ARMv7 NEON, ARMv8 ASIMD or the following x86 SIMD extensions: SSE2, SSSE3, AVX, AVX2, AVX512-BW (128/256-bit), AVX512-VBMI2 (or AVX10.1/256)\n-   CPU detection and dynamic dispatch (i.e. select best implementation for currently running CPU)\n-   incremental processing, including detection of yEnc/NNTP end sequences in decoder\n-   raw yEnc encoding with the ability to specify line length. A single thread can achieve \\\u003e450MB/s on a Raspberry Pi 3, or \\\u003e5GB/s on a Core-i series CPU.\n-   yEnc decoding, with and without NNTP layer dot unstuffing. A single thread can achieve \\\u003e300MB/s on a Raspberry Pi 3, or \\\u003e4.5GB/s on a Core-i series CPU.\n-   CRC32 implementation via [crcutil](https://code.google.com/p/crcutil/) or [PCLMULQDQ instruction](http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf), ARMv8’s CRC instructions, or RISC-V’s Zb(k)c extension (\\\u003e1GB/s on a low power Atom/ARM CPU, \\\u003e15GB/s on a modern Intel CPU)\n-   ability to combine two CRC32 hashes into one (useful for amalgamating *pcrc32s* into a *crc32* for yEnc), as well as quickly compute the CRC32 of a sequence of null bytes\n\nBuilding\n==========\n\nA build file/project can be created using CMake.\n\n```\nmkdir build\ncd build\ncmake ..\ncmake --build . --config Release\n```\n\nAfter compilation, a shared and static library should be generated, as well as a benchmark and sample CLI application.\n\n## Build Options\n\nThe following options can be passed into CMake:\n\n* **BUILD_NATIVE**: Optimise for and target only the build host’s CPU; this build may not be re-distributable\n* **DISABLE_AVX256**: Disable the use of 256-bit AVX instructions on x86 processors\n* **DISABLE_CRCUTIL**: Disable crcutil usage. crcutil is only ever enabled for x86 builds\n* **DISABLE_ENCODE**: Remove yEnc encode functionality from build. `rapidyenc_encode`* functions, except `rapidyenc_encode_max_length`, will be unavailable\n* **DISABLE_DECODE**: Remove yEnc decode functionality from build. `rapidyenc_decode`* functions will be unavailable\n* **DISABLE_CRC**: Remove CRC32 functionality from build. `rapidyenc_crc`* functions will be unavailable. Implies *DISABLE_CRCUTIL*\n\nAPI\n===\n\nBefore any encoding/decoding/CRC functions can be used, the respective `_init` function must be called. These functions set up the necessary state for computation. Note that `_init` functions aren’t thread-safe, but all others are.\n\nFunctions documented in the [header file](rapidyenc.h).\n\n[cli.c](tool/cli.c) is a simple command-line application which encodes/decodes stdin to stdout. It demonstrates how to do incremental encoding/decoding/CRC32 using this library.\n\n# Other Language Bindings\n\n* [node-yencode](https://github.com/animetosho/node-yencode): for NodeJS/Bun\n* [go-yencode](https://github.com/mnightingale/go-yencode): for Golang\n* [sabctools](https://github.com/sabnzbd/sabctools): for Python (Sabnzbd specific binding)\n\nAlgorithm\n=========\n\nA brief description of how the SIMD yEnc encoding algorithm works [can be found here](https://github.com/animetosho/node-yencode/issues/4#issuecomment-330025192).\nI may eventually write up something more detailed, regarding optimizations and such used.\n\nLicense\n=======\n\nThis module is Public Domain or [CC0](https://creativecommons.org/publicdomain/zero/1.0/legalcode) (or equivalent) if PD isn’t recognised.\n\n[crcutil](https://code.google.com/p/crcutil/), used for CRC32 calculation, is licensed under the [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0)\n\n[zlib-ng](https://github.com/Dead2/zlib-ng), from where the CRC32 calculation using folding approach was stolen, is under a [zlib license](https://github.com/Dead2/zlib-ng/blob/develop/LICENSE.md)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanimetosho%2Frapidyenc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanimetosho%2Frapidyenc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanimetosho%2Frapidyenc/lists"}