{"id":43255728,"url":"https://github.com/ljishen/bitar","last_synced_at":"2026-02-01T13:32:46.237Z","repository":{"id":41546483,"uuid":"490108576","full_name":"ljishen/bitar","owner":"ljishen","description":"Simplify accessing hardware compression/decompression accelerators","archived":false,"fork":false,"pushed_at":"2023-02-25T23:35:55.000Z","size":554,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-01-26T11:52:52.398Z","etag":null,"topics":["apache-arrow","compression","cpp","dpdk","hardware-acceleration"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ljishen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-05-09T02:24:52.000Z","updated_at":"2024-08-07T14:36:42.000Z","dependencies_parsed_at":"2025-04-13T11:01:09.390Z","dependency_job_id":null,"html_url":"https://github.com/ljishen/bitar","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/ljishen/bitar","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ljishen%2Fbitar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ljishen%2Fbitar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ljishen%2Fbitar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ljishen%2Fbitar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ljishen","download_url":"https://codeload.github.com/ljishen/bitar/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ljishen%2Fbitar/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28979124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T12:13:08.691Z","status":"ssl_error","status_checked_at":"2026-02-01T12:13:08.356Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-arrow","compression","cpp","dpdk","hardware-acceleration"],"created_at":"2026-02-01T13:32:46.096Z","updated_at":"2026-02-01T13:32:46.201Z","avatar_url":"https://github.com/ljishen.png","language":"C++","readme":"# bitar\n\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/3f22cce4889a4261997ae300e8f4e595)](https://app.codacy.com/gh/ljishen/bitar?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=ljishen/bitar\u0026utm_campaign=Badge_Grade_Settings)\n[![GitHub Super-Linter](https://github.com/ljishen/bitar/workflows/Lint%20Code%20Base/badge.svg)](https://github.com/marketplace/actions/super-linter)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![DOI](https://zenodo.org/badge/490108576.svg)](https://zenodo.org/badge/latestdoi/490108576)\n\n\u003c!-- markdownlint-disable-next-line no-inline-html --\u003e\n\u003cimg src=\"assets/logo.png\" width=\"25%\"\u003e\nBitar is a C++ library to simplify accessing hardware compression/decompression accelerators.\n\n## Features\n\n- Zero-copy of data input and output\n- Synchronous and asynchronous operations\n- Multi-core and multi-device support\n- No sudo permission requirement\n\nBitar can run either on the host machine or on the NVIDIA BlueField DPU target\nin either DPU mode or separated host mode.\n\n## Prerequisites\n\n- Linux (with kernel \u003e= 4.4) or FreeBSD\n- For Linux, glibc \u003e= 2.7 (reported by `ldd --version`)\n- GCC \u003e= 9 (C++17 compliant compiler)\n- [DPDK](https://github.com/DPDK/dpdk) \u003e= v21.11 (can be installed via vcpkg)\n- [Apache Arrow](https://github.com/apache/arrow) \u003e= 11.0.0 (build automatically\n  if not found)\n\n## Supported Hardware\n\n- [NVIDIA BLUEFIELD-2 DPU](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/documents/datasheet-nvidia-bluefield-2-dpu.pdf)\n\n## Integration\n\nBitar can be easily installed and integrated via\n[vcpkg](https://github.com/microsoft/vcpkg)\n\n```bash\nvcpkg install bitar\n```\n\n## Development\n\n- The DPDK library will be built from source by vcpkg if `dpdk_ROOT` is not\n  specified.\n\n- The Arrow parquet library is required if `BITAR_BUILD_APPS` is `ON`.\n  Otherwise, just having the Arrow library is sufficient.\n\n- Loading the Arrow parquet library will create a CMake target for the arrow and\n  the parquet library, respectively.\n\n- Use `Arrow_ROOT` to specify the installation prefix of the Arrow library if it\n  is not installed at the default location. By default, the system-installed\n  Arrow library will be at `/usr`.\n\n- If the Arrow library is not found, or it is found but the parquet library is\n  not found when needed, the Arrow library will be built from source.\n\n```bash\n$ # Reserve hugepages\n$ sudo sh -c 'echo 1024 \u003e /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages'\n$ # On NUMA machines, we may need\n$ # sudo sh -c 'echo 1024 \u003e /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages'\n\n$ # Install prerequisites for development\n$ scripts/install-deps.sh\n\n$ CC=clang CXX=clang++ cmake -S . -B ./build-$(uname -m) -G Ninja \\\n[-Ddpdk_ROOT:PATH=\u003cdpdk-install-prefix\u003e] \\\n[-DArrow_ROOT:PATH=\u003carrow-install-prefix\u003e] \\\n-DBITAR_BUILD_APPS:BOOL=ON -DBITAR_BUILD_TESTS:BOOL=ON \\\n-DENABLE_DEVELOPER_MODE:BOOL=ON -DCMAKE_BUILD_TYPE:BOOL=Debug\n\n$ cmake --build ./build-$(uname -m)\n$ cmake --install ./build-$(uname -m) --prefix \u003cinstall-prefix\u003e\n\n$ ./build-$(uname -m)/apps/demo_app --in-memory --lcores 5@(0-7),6-7 \\\n-a \u003cdevice-pci-id\u003e,class=compress -- --file \u003cfile\u003e \\\n[--bytes \u003csize-to-read-from-file\u003e] [--mode \u003cfile-read-mode\u003e] [--help]\n```\n\n### Advanced CMake Configuration Options\n\n- `BITAR_FETCHCONTENT_OVERWRITE_CONFIGURATION`: set this option to `OFF` to have\n  separate debug and release builds without overwriting each others\n  configurations (default: `ON`)\n\n- `VCPKG_ROOT`: the prefix to an installed vcpkg instance (install automatically\n  if not specified)\n\n- `BITAR_BUILD_ARROW`: set this option to `ON` to force building the Arrow\n  dependency from source (default: `OFF`)\n\n- `BITAR_ARROW_GIT_REPOSITORY`: the git repository to fetch the Arrow source\n  (default: the official repository)\n\n- `BITAR_ARROW_GIT_TAG`: use the source at the git branch, tag or commit hash\n  from the Arrow repository for building when needed (default\n  `apache-arrow-11.0.0`)\n\n- `BITAR_INSTALL_ARROW`: install the Arrow library as part of the cmake\n  installation process if Arrow is built by this project (default: `OFF`)\n\n- Any\n  [Arrow supported CMake options](https://github.com/apache/arrow/blob/apache-arrow-11.0.0/cpp/cmake_modules/DefineOptions.cmake),\n  e.g., `ARROW_WITH_LZ4`, `ARROW_WITH_ZSTD`, and `ARROW_WITH_SNAPPY`.\n\n## Known Issues\n\n- (July 23, 2022) DPDK mistakenly assumes the support of `aes`, `pmull`, `sha1`,\n  and `sha2` CPU flags on crypto-disabled BlueField-2 DPUs (e.g.,\n  `MBF2H516A-CENO_Ax`, the one on the CloudLab r7525 machine) with **LLVM\n  Clang**, resulting in the following error when executing a program compiled\n  with bitar\n  ([relevant code](https://github.com/DPDK/dpdk/blob/v22.07/config/arm/meson.build#L652-L655)):\n\n  ```bash\n  ERROR: This system does not support \"AES\".\n  Please check that RTE_MACHINE is set correctly.\n  ```\n\n  There is no such problem when DPDK is compiled with GCC. Note that bitar can\n  still be compiled with Clang and linked with DPDK that is compiled with GCC.\n\n- ```bash\n  [Out of memory: Reserving memzone of XXXXXX bytes failed. [Error 12: Cannot allocate memory]]\n  ```\n\n  (Aug 18, 2022) If you see this error, check whether DPDK is running in the\n  Virtual Addresses mode (VA). Normally, you should see\n  `EAL: Selected IOVA mode 'VA'` during the program initialization. But if you\n  see `EAL: Selected IOVA mode 'PA'`, it's probably that the IOMMU is disable\n  either in the BIOS or in the kernel. If IOMMU is enabled,\n  `/sys/kernel/iommu_groups` path will contain kernel IOMMU groups and thus is\n  not be empty. To enable IOMMU in the kernel, you can add\n  `intel_iommu=on iommu=pt` in GRUB command-line on x86_64 systems, or add\n  `iommu.passthrough=1` on aarch64 systems.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fljishen%2Fbitar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fljishen%2Fbitar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fljishen%2Fbitar/lists"}