{"id":16661402,"url":"https://github.com/bodigrim/bitvec","last_synced_at":"2025-04-09T21:17:21.782Z","repository":{"id":37951265,"uuid":"168411954","full_name":"Bodigrim/bitvec","owner":"Bodigrim","description":"Bit vectors: 8x less memory, up to 3500x faster than Vector Bool","archived":false,"fork":false,"pushed_at":"2025-03-29T21:29:29.000Z","size":412,"stargazers_count":72,"open_issues_count":3,"forks_count":7,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-09T21:17:17.597Z","etag":null,"topics":["bit-array","bit-vectors","bitmap","bitmask","libgmp","vectors"],"latest_commit_sha":null,"homepage":"https://hackage.haskell.org/package/bitvec","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bodigrim.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-30T20:41:14.000Z","updated_at":"2025-03-29T21:29:32.000Z","dependencies_parsed_at":"2024-04-28T17:25:28.576Z","dependency_job_id":"6a3dd57a-7087-49c8-9ec0-0ac8edbda2b9","html_url":"https://github.com/Bodigrim/bitvec","commit_stats":{"total_commits":329,"total_committers":9,"mean_commits":36.55555555555556,"dds":0.1185410334346505,"last_synced_commit":"b3c8df3ffe647124af7bf2b5f21de59a2e884c3c"},"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodigrim%2Fbitvec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodigrim%2Fbitvec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodigrim%2Fbitvec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodigrim%2Fbitvec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bodigrim","download_url":"https://codeload.github.com/Bodigrim/bitvec/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248111973,"owners_count":21049578,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bit-array","bit-vectors","bitmap","bitmask","libgmp","vectors"],"created_at":"2024-10-12T10:34:51.337Z","updated_at":"2025-04-09T21:17:21.761Z","avatar_url":"https://github.com/Bodigrim.png","language":"Haskell","readme":"# bitvec [![Hackage](https://img.shields.io/hackage/v/bitvec.svg)](https://hackage.haskell.org/package/bitvec) [![Stackage LTS](https://www.stackage.org/package/bitvec/badge/lts)](https://www.stackage.org/lts/package/bitvec) [![Stackage Nightly](https://www.stackage.org/package/bitvec/badge/nightly)](https://www.stackage.org/nightly/package/bitvec)\n\nA newtype over `Bool` with a better `Vector` instance: 8x less memory, up to 3500x faster.\n\nThe [`vector`](https://hackage.haskell.org/package/vector)\npackage represents unboxed arrays of `Bool`s\nspending 1 byte (8 bits) per boolean.\nThis library provides a newtype wrapper `Bit` and a custom instance\nof an unboxed `Vector`, which packs bits densely,\nachieving an __8x smaller memory footprint.__\nThe performance stays mostly the same;\nthe most significant degradation happens for random writes\n(up to 10% slower).\nOn the other hand, for certain bulk bit operations\n`Vector Bit` is up to 3500x faster than `Vector Bool`.\n\n## Thread safety\n\n* `Data.Bit` is faster, but writes and flips are not thread-safe.\n  This is because naive updates are not atomic:\n  they read the whole word from memory,\n  then modify a bit, then write the whole word back.\n  Concurrently modifying non-intersecting slices of the same underlying array\n  may also lead to unexpected results, since they can share a word in memory.\n* `Data.Bit.ThreadSafe` is slower (usually 10-20%),\n  but writes and flips are thread-safe.\n  Additionally, concurrently modifying non-intersecting slices of the same underlying array\n  works as expected. However, operations that affect multiple elements are not\n  guaranteed to be atomic.\n\n## Quick start\n\nConsider the following (very naive) implementation of\n[the sieve of Eratosthenes](https://en.wikipedia.org/wiki/Sieve_of_Eratosthenes). It returns a vector with `True`\nat prime indices and `False` at composite indices.\n\n```haskell\nimport Control.Monad\nimport Control.Monad.ST\nimport qualified Data.Vector.Unboxed as U\nimport qualified Data.Vector.Unboxed.Mutable as MU\n\neratosthenes :: U.Vector Bool\neratosthenes = runST $ do\n  let len = 100\n  sieve \u003c- MU.replicate len True\n  MU.write sieve 0 False\n  MU.write sieve 1 False\n  forM_ [2 .. floor (sqrt (fromIntegral len))] $ \\p -\u003e do\n    isPrime \u003c- MU.read sieve p\n    when isPrime $\n      forM_ [2 * p, 3 * p .. len - 1] $ \\i -\u003e\n        MU.write sieve i False\n  U.unsafeFreeze sieve\n```\n\nWe can switch from `Bool` to `Bit` just by adding newtype constructors:\n\n```haskell\nimport Data.Bit\n\nimport Control.Monad\nimport Control.Monad.ST\nimport qualified Data.Vector.Unboxed as U\nimport qualified Data.Vector.Unboxed.Mutable as MU\n\neratosthenes :: U.Vector Bit\neratosthenes = runST $ do\n  let len = 100\n  sieve \u003c- MU.replicate len (Bit True)\n  MU.write sieve 0 (Bit False)\n  MU.write sieve 1 (Bit False)\n  forM_ [2 .. floor (sqrt (fromIntegral len))] $ \\p -\u003e do\n    Bit isPrime \u003c- MU.read sieve p\n    when isPrime $\n      forM_ [2 * p, 3 * p .. len - 1] $ \\i -\u003e\n        MU.write sieve i (Bit False)\n  U.unsafeFreeze sieve\n```\n\nThe `Bit`-based implementation requires 8x less memory to store\nthe vector. For large sizes it allows to crunch more data in RAM\nwithout swapping. For smaller arrays it helps to fit into\nCPU caches.\n\n```haskell\n\u003e listBits eratosthenes\n[2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67,71,73,79,83,89,97]\n```\n\nThere are several high-level helpers, digesting bits in bulk,\nwhich makes them up to 64x faster than the respective counterparts\nfor `Vector Bool`. One can query the population count (popcount)\nof a vector (giving us [the prime-counting function](https://en.wikipedia.org/wiki/Prime-counting_function)):\n\n```haskell\n\u003e countBits eratosthenes\n25\n```\n\nAnd vice versa, query an address of the _n_-th set bit\n(which corresponds to the _n_-th prime number here):\n\n```haskell\n\u003e nthBitIndex (Bit True) 10 eratosthenes\nJust 29\n```\n\nOne may notice that the order of the inner traversal by `i`\ndoes not matter and get tempted to run it in several parallel threads.\nIn this case it is vital to switch from `Data.Bit` to `Data.Bit.ThreadSafe`,\nbecause the former is not thread-safe with regards to writes.\nThere is a moderate performance penalty (usually 10-20%)\nfor using the thread-safe interface.\n\n## Sets\n\nBit vectors can be used as a blazingly fast representation of sets,\nas long as their elements are `Enum`eratable and sufficiently dense,\nleaving `IntSet` far behind.\n\nFor example, consider three possible representations of a set of `Word16`:\n\n* As an `IntSet` with a readily available `union` function.\n* As a 64k-long unboxed `Vector Bool`, implementing union as `zipWith (||)`.\n* As a 64k-long unboxed `Vector Bit`, implementing union as `zipBits (.|.)`.\n\nWhen the `simd` flag is enabled,\naccording to our benchmarks (see `bench` folder),\nthe union of `Vector Bit` evaluates magnitudes faster\nthan the union of not-too-sparse `IntSet`s\nand stunningly outperforms `Vector Bool`.\nHere are benchmarks on MacBook M2:\n\n```\nunion\n  16384\n    Vector Bit:\n      61.2 ns ± 3.2 ns\n    Vector Bool:\n      96.1 μs ± 4.5 μs, 1570.84x\n    IntSet:\n      2.15 μs ± 211 ns, 35.06x\n  32768\n    Vector Bit:\n      143  ns ± 7.4 ns\n    Vector Bool:\n      225  μs ±  16 μs, 1578.60x\n    IntSet:\n      4.34 μs ± 429 ns, 30.39x\n  65536\n    Vector Bit:\n      249  ns ±  18 ns\n    Vector Bool:\n      483  μs ±  28 μs, 1936.42x\n    IntSet:\n      8.77 μs ± 835 ns, 35.18x\n  131072\n    Vector Bit:\n      322  ns ±  30 ns\n    Vector Bool:\n      988  μs ±  53 μs, 3071.83x\n    IntSet:\n      17.6 μs ± 1.6 μs, 54.79x\n  262144\n    Vector Bit:\n      563  ns ±  27 ns\n    Vector Bool:\n      2.00 ms ± 112 μs, 3555.36x\n    IntSet:\n      36.8 μs ± 3.3 μs, 65.40x\n```\n\n## Binary polynomials\n\nBinary polynomials are polynomials with coefficients modulo 2.\nTheir applications include coding theory and cryptography.\nWhile one can successfully implement them with the [`poly`](https://hackage.haskell.org/package/poly) package,\noperating on `UPoly Bit`,\nthis package provides even faster arithmetic routines\nexposed via the `F2Poly` data type and its instances.\n\n```haskell\n\u003e :set -XBinaryLiterals\n\u003e -- (1 + x) * (1 + x + x^2) = 1 + x^3 (mod 2)\n\u003e 0b11 * 0b111 :: F2Poly\nF2Poly {unF2Poly = [1,0,0,1]}\n```\n\nUse `fromInteger` / `toInteger` to convert binary polynomials\nfrom `Integer` to `F2Poly` and back.\n\n## Package flags\n\n* Flag `simd`, enabled by default.\n\n  Use a C SIMD implementation for the ultimate performance of `zipBits`, `invertBits` and `countBits`.\n\n## Similar packages\n\n* [`bv`](https://hackage.haskell.org/package/bv) and\n  [`bv-little`](https://hackage.haskell.org/package/bv-little)\n  do not offer mutable vectors.\n\n* [`array`](https://hackage.haskell.org/package/array)\n  is memory-efficient for `Bool`, but lacks\n  a handy `Vector` interface and is not thread-safe.\n\n## Additional resources\n\n* __Bit vectors without compromises__, Haskell Love, 31.07.2020:\n  [slides](https://github.com/Bodigrim/my-talks/raw/master/haskelllove2020/slides.pdf), [video](https://youtu.be/HhpH8DKFBls).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbodigrim%2Fbitvec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbodigrim%2Fbitvec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbodigrim%2Fbitvec/lists"}