https://github.com/oz4462/aletheia

Entropy-aware containers for Roaring bitmaps: near-Shannon-H0 space in the dense-irregular regime where Roaring spends a flat 8 KiB, never larger than Roaring, with a faster select. Pure Rust, zero deps, no unsafe.
https://github.com/oz4462/aletheia
bitmap bitmap-index bitset compression data-structures elias-fano entropy-coding information-retrieval roaring roaring-bitmaps rust rust-crate succinct succinct-data-structures zero-dependencies
Last synced: about 3 hours ago
JSON representation
Host: GitHub
URL: https://github.com/oz4462/aletheia
Owner: Oz4462
License: mit
Created: 2026-06-08T18:25:57.000Z (about 1 month ago)
Default Branch: master
Last Pushed: 2026-06-08T20:18:02.000Z (about 1 month ago)
Last Synced: 2026-06-08T20:24:41.988Z (about 1 month ago)
Topics: bitmap, bitmap-index, bitset, compression, data-structures, elias-fano, entropy-coding, information-retrieval, roaring, roaring-bitmaps, rust, rust-crate, succinct, succinct-data-structures, zero-dependencies
Language: Rust
Size: 32.2 KB
Stars: 1
Watchers: 0
Forks: 0
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project

README

          


# 🔓 ALETHEIA

### Near-entropy containers for Roaring bitmaps

*aletheia* (ἀλήθεια) — *"unconcealment"*: the 8 KiB a flat bitset hides, revealed.

[![CI](https://github.com/Oz4462/aletheia/actions/workflows/ci.yml/badge.svg)](https://github.com/Oz4462/aletheia/actions/workflows/ci.yml)

[![License: PolyForm--Noncommercial](https://img.shields.io/badge/License-PolyForm--Noncommercial-blue.svg)](LICENSE)

[![unsafe: forbidden](https://img.shields.io/badge/unsafe-forbidden-success.svg)](src/lib.rs)

[![runtime deps: 0](https://img.shields.io/badge/runtime%20deps-0-success.svg)](Cargo.toml)

[![Rust: 1.94+](https://img.shields.io/badge/rust-1.94%2B-orange.svg)](https://www.rust-lang.org)




> ### Up to **4× smaller** than Roaring — **never larger** — with a **14× faster `select`**.

≤ ¼

_{the size
on its worst-case chunks}

0×

_{ever larger
mathematically guaranteed}

14×

_{faster select
vs a flat bitset}

0

_{runtime deps
pure std · no unsafe}



---

## ✨ At a glance

- **The problem.** Roaring stores any 2¹⁶ chunk with cardinality > 4096 as a **flat, uncompressed 8 KiB bitset** — no matter how compressible the chunk is. In the *dense-but-irregular* band (≈10 % or ≈90 % full) that's **2–4× bigger than the Shannon entropy floor**, and Roaring has no smaller option.

- **The fix.** ALETHEIA adds two **Elias-Fano** encodings — of the set positions *and* of the complement — as extra candidates, and keeps the smallest per chunk.

- **The guarantee.** Its candidate set is a **strict superset** of Roaring's, so the result is **never larger than Roaring** — and *strictly smaller* across the whole dense-irregular band.

- **The proof.** Every claim is backed by reproducible, CPU-only, zero-dependency tests — including an independent cross-check against the real [`roaring`](https://crates.io/crates/roaring) crate's actual serialized bytes.

```rust

use aletheia::Container;

// a dense-but-irregular chunk Roaring would store as a flat 8 KiB bitset

let values: Vec = (0u16..65_535).step_by(7).collect();

let c = Container::from_sorted(&values);

assert!(c.size_bytes() < 8192);   // ← Roaring spends a full 8192 here

```

---

An **entropy-aware container** for Roaring-style bitmaps. It closes a real,

literature-grounded gap: Roaring stores any 2¹⁶ chunk with cardinality > 4096

as a **flat, uncompressed 8 KiB bitset**, no matter the chunk's entropy. In the

dense-but-irregular regime (too many runs for a run container, too dense for an

array) that bitset can be 2–4× larger than the Shannon-H₀ floor — and Roaring

has no smaller option there.

ALETHEIA adds two Elias-Fano candidate encodings (of the set positions, and of

the **complement** for the symmetric high-density tail) to the container

selector and keeps the smallest representation per chunk. Because the candidate

set is a strict superset of Roaring's (array, run, bitset **+** EF-set,

EF-clear), the result is **never larger than the container Roaring would

choose**, and strictly smaller across the dense-irregular band.

## 👥 Who is this for?

Anyone who stores **large sets of integer IDs** with Roaring and is **memory- or

storage-bound** — and whose access pattern leans on `select`/`rank`/`iterate`

rather than point membership:

| Domain | Typical use |

|---|---|

| 🔎 **Search engines** (Lucene, Elasticsearch, Solr) | inverted-index posting lists — "which documents contain term *X*" |

| 📊 **Analytical / OLAP databases** (Druid, ClickHouse, Pinot) | bitmap indexes & fast filters over billions of rows |

| ⚡ **Big-data engines** (Spark) | set operations over huge ID sets |

| 🚩 **Feature flags / permissions** | "which users are in segment *Y*" |

Best fit: **medium-cardinality columns** with many chunks in the dense-irregular

band, where shrinking each chunk means more fits in cache/RAM and less I/O.

**Not the goal:** membership-heavy hot loops — see the honest speed tradeoff below.

## 🚀 Quickstart

```toml

[dependencies]

aletheia = { git = "https://github.com/Oz4462/aletheia" }

```

```rust

use aletheia::Container;

// a dense-but-irregular chunk Roaring would store as a flat 8 KiB bitset

let values: Vec = (0u16..65_535).step_by(7).collect();

let c = Container::from_sorted(&values);

assert!(c.contains(7));

assert_eq!(c.iter().len(), c.cardinality() as usize);

println!("{} bytes (Roaring would use 8192)", c.size_bytes());

```

> **Tip:** `from_sorted` trusts that its input is sorted & distinct (the fast

> path). If you're not sure, use `Container::from_values`, which sorts and

> de-duplicates for you.

### Full `u32` bitmap, set operations & serialization

```rust

use aletheia::Bitmap;

let a = Bitmap::from_values(&[1, 2, 3, 1_000_000, 5_000_000]);

let b = Bitmap::from_values(&[3, 4, 1_000_000]);

assert_eq!((&a & &b).to_vec(), vec![3, 1_000_000]);    // ∩ (operators or .and())

assert_eq!((&a | &b).cardinality(), 6);                // ∪

assert_eq!((&a ^ &b).to_vec(), vec![1, 2, 4, 5_000_000]);

assert_eq!((&a - &b).to_vec(), vec![1, 2, 5_000_000]);

// rank / select / mutation over the whole u32 universe

assert_eq!(a.rank(1_000_000), 3);

assert_eq!(a.select(0), Some(1));

let mut m = a.clone();

m.insert(7);

m.remove(2);

// ALETHEIA's own compact format (validated: malformed bytes are refused

// with a DecodeError naming the violated check — never a panic)

let bytes = a.serialize();

assert_eq!(Bitmap::deserialize(&bytes).unwrap(), a);

// the official Roaring interop format — byte-identical to what

// roaring-rs itself writes, readable by every Roaring implementation

let portable = a.serialize_portable();

assert_eq!(Bitmap::deserialize_portable(&portable).unwrap(), a);

```

`Bitmap` is the Roaring-shaped wrapper: one entropy-aware `Container` per 2¹⁶

chunk, so the never-larger-than-Roaring guarantee holds chunk by chunk.

Both deserializers validate everything (magic/version, per-container

structure, cardinality cross-checks, ordering) and refuse malformed input

with a typed `DecodeError` — never a panic, never attacker-driven allocation.

## 📐 Verified result (reproducible, CPU-only, zero dependencies)

**Space** — `cargo run --release --example space_report`:

| density | card  | roaring | aletheia | kind    | ratio | H₀ floor |

|--------:|------:|--------:|---------:|---------|------:|---------:|

| 0.065   | 4260  | 8192    | **3259** | EfSet   | 2.51× | 2843     |

| 0.111   | 7260  | 8192    | **4783** | EfSet   | 1.71× | 4115     |

| 0.199   | 13073 | 8192    | 7187     | EfSet   | 1.14× | 5906     |

| 0.299   | 19575 | 8192    | 8192     | Bitset  | 1.00× | 7207     |

| 0.496   | 32525 | 8192    | 8192     | Bitset  | 1.00× | 8192     |

| 0.851   | 55750 | 8192    | **5931** | EfClear | 1.38× | 4982     |

| 0.935   | 61305 | 8192    | **3244** | EfClear | 2.53× | 2829     |

| 0.964   | 63147 | 8192    | **2062** | EfClear | 3.97× | 1850     |

| 0.985   | 64583 | 3766¹   | **982**  | EfClear | 3.84× | 898      |

¹ at this density Roaring picks a *run* container (3766 B), not a bitset —

ALETHEIA still beats it 3.84×. The container tracks the H₀ floor to within

~10–16% across the band, and degrades to exactly Roaring's bitset (1.00×, never

worse) in the middle.

**Speed** — `cargo run --release --example speed_report` (card 6486, d≈0.10,

min-of-7):

> 💡 **The tradeoff in one line:** ALETHEIA is a *different Pareto point*, not a

> free lunch — **smaller and far faster at `select`**, at the cost of slower

> point membership.

| op       | aletheia | bitset  | factor          |

|----------|---------:|--------:|-----------------|

| contains | ~73 ns   | ~1.6 ns | **~45× slower** |

| select   | ~60 ns   | ~893 ns | **~15× faster** |

| iter     | ~4.6 ns  | ~2.2 ns | ~2× slower      |

(Numbers swing between runs on a clock-scaling CPU; min-of-7 methodology,

fresh run 2026-07-04 — full tables with machine info in `docs/CLAIMS.md`.)

`contains` started at 176 ns and a single-`select0` fast path brought it to

`select`-parity — both are now dominated by one directory lookup. Set

operations cost ~0.4–1.1 ms per 2¹⁶ chunk (materialize + full re-selection —

the documented single-code-path trade-off).

This is the honest tradeoff, not a hidden one: ALETHEIA is a **different Pareto

point**, not a free lunch. It is smaller *and* dramatically faster at `select`

(Roaring's bitset `select` is an O(U) popcount scan), at the cost of ~40× slower

membership (a directory lookup vs a flat bit test). It fits memory-constrained,

select-/rank-/iterate-heavy, membership-light workloads — e.g. bitmap-index

posting lists on medium-cardinality columns.

## ✅ Correctness

`cargo test` runs a deterministic splitmix64 fuzz oracle that verifies every

`contains`/`rank`/`select`/`iter` against a brute-force `Vec` truth table

across the **full 65536-value universe** for ~110 configurations (all

densities, uniform + clustered + edge cases), and asserts the never-larger-than-

Roaring guarantee on every one. `space_win.rs` additionally asserts the ≥1.5×

win at the band edges, so CI fails if any change erases it. A failure prints the

exact seed to replay.

As a second, independent witness, `roaring_crosscheck.rs` checks every

cardinality, full-universe membership, iteration, and set-operation result

against the real `roaring` crate (roaring-rs — a separate codebase), and

confirms never-larger against its *actual serialized output* — including a

whole-bitmap footprint check (the measured edge-band ratios, 1.2×–6.5× on

the current run, are printed by the test; the CI-gated bound is

never-larger plus the ≥1.5× band-edge floor in `space_win.rs`). (`roaring` is a dev-only dependency; the

library itself stays pure std with zero runtime dependencies.)

Beyond that: `setops_oracle.rs` proves never-larger **after every set

operation** across an 8×8 density grid; `malformed.rs` fuzzes both

deserializers with random buffers, single-byte mutations of every container

kind, and every truncation point (refusal or provable self-consistency,

never a panic); `portable.rs` holds the interop format **byte-identical**

to roaring-rs's own output across eight shapes and round-trips both

directions. The full claim→evidence map is `docs/CLAIMS.md`.

```

cargo test                                       # correctness + win assertions

cargo test --test roaring_crosscheck -- --nocapture  # vs real roaring-rs

cargo clippy --all-targets -- -D warnings        # lint-clean

cargo run --release --example space_report       # the space table above

cargo run --release --example speed_report       # the honest cost table above

```

## 🔬 How it works

A value `v` splits into `high = v >> l` and `l` low bits. Low bits are packed;

high bits are a count-unary bitvector (`m + max_high + 1` bits) with a coarse

512-bit-block prefix-popcount directory for near-constant-time `select1`,

`select0` and `rank1`. With `l = floor(log2(U/m))` the structure sits within a

couple of bits per element of the H₀ floor. The complement variant captures the

high-density tail symmetrically.

## 🗺️ Scope

This started as an MVP proving the invention with exact, unfakeable numbers; it

now also ships the bitmap machinery built on top of it.

**Done in v0.2.0:**

* **Set operations** — `and` / `or` / `xor` / `andnot` (+ `& | ^ -` operator

  traits) on both `Container` and `Bitmap`, via materialize-to-bitset with the

  result re-run through the full selector (so the never-larger guarantee holds

  on results too — proven per-op by `tests/setops_oracle.rs`).

* **Multi-chunk `Bitmap`** — the Roaring-shaped `u32` wrapper: one container per

  2¹⁶ chunk, with `contains` / `rank` / `select` / `min` / `max`, mutation

  (`insert` / `remove` with automatic encoding re-selection), lazy iteration,

  and the std traits (`FromIterator`, `Extend`, `IntoIterator`, `PartialEq`).

* **Native serialization** — a compact, self-describing, versioned byte format

  (magic `ALTH`) for both `Container` and `Bitmap`. Deserialization validates

  every structural invariant and cross-checks the stored cardinality against

  the payload; malformed input is refused with a typed `DecodeError`, never a

  panic, and untrusted lengths never drive allocation (fuzz-proven).

* **Roaring-portable serialization** — the official RoaringFormatSpec,

  **byte-identical** to what roaring-rs writes (golden-tested) and readable by

  every Roaring implementation; the validating reader re-selects each chunk

  back into ALETHEIA's smallest encoding.

* **`SparseVec`** — compressed `u32` indices paired with values, for sparse

  workloads (see below). Optional `serde` / `serde_bincode` features (default

  build stays zero-dependency).

* **Cross-checked** — `Bitmap` and every set operation are validated against the

  real `roaring` crate over the full `u32` universe, not just our own oracle.

**Done since v0.1.0:** a single-`select0` `contains` fast path (176 ns → ~73 ns);

a hardened `EliasFano::select` (bounds-checked rather than spinning on an

out-of-range index).

**Measured & rejected:**

* A BMI2 `PDEP` broadword `select_in_u64` — 2.8× *slower* than the naive

  clear-lowest-bit loop here (per-call feature dispatch plus the

  `#[target_feature]` inlining barrier outweigh `PDEP`; within-word select is

  only ~15 ns of a ~67 ns `select` — the directory block search is the floor).

* An **RRR candidate** for the mid-band — the math doesn't pay off. At d≈0.3,

  15-bit RRR blocks cost ~9.4 KB (the 4-bit-per-block class header dominates),

  *worse* than the flat 8 KB bitset; even 63-bit blocks reach only ~8.0 KB, a

  ~2 % win that a sampled rank/select directory erases. Where the bitset already

  ties the H₀ floor (d≈0.5) RRR can't help at all. Not worth the complexity for a

  break-even encoding — revisit only with a fundamentally cheaper class coding.

  Re-measured rigorously against a pre-registered rule (63-bit blocks,

  combinadic offsets, a directory every 32 blocks; full uniform density

  sweep, 3 seeds/step): win peaked at 12.7% (d≈0.76), but the widest

  contiguous ≥10%-win band was only 0.04 wide against the pre-registered

  0.05 threshold — rejected on the numbers; full sweep in `docs/DECISIONS.md`.

**Out of scope** (deliberately, with reasons in `docs/DECISIONS.md` and

`docs/AUDIT.md` §6): the Roaring 64-bit extension format, SIMD (`unsafe` is

forbidden), specialized non-materializing set-op kernels, in-library

parallelism.

## 🧠 Sparse indices for LLM / agent workloads

`SparseVec` pairs a compressed `Bitmap` index set with a parallel value

array — positions compress toward the entropy floor, values stay contiguous:

```rust

use aletheia::SparseVec;

// pruned weights: (index, value) pairs in any order

let sv = SparseVec::from_pairs(vec![(7u32, 0.5f32), (1 << 20, 0.25), (42, 0.9)]);

assert_eq!(sv.len(), 3);

assert_eq!(sv.get(1), Some((42, &0.9)));            // i-th pair, index order

let total: f32 = sv.iter().map(|(_, v)| v).sum();   // lazy iteration

```

Typical fits: pruned LLM weight matrices (>90 % sparse), sparse KV-cache

positions, MoE routing masks, large agent memory ID sets — build once, then

`select`/`rank`/iterate on the compressed form.

## 📦 Optional features

| Feature | Adds | Default |

|---|---|---|

| *(none)* | the whole library — pure std, zero dependencies | ✅ |

| `serde` | `Serialize`/`Deserialize` for `Bitmap` + `SparseVec` (value-list representation; hostile input is validated, never trusted) | off |

| `serde_bincode` | `SparseVec::to_bytes` / `from_bytes` (both return `Result` — no silent failure) | off |

## 🔍 Audit trail

Written to be walked file-by-file by an external reviewer:

| Document | Contents |

|---|---|

| [`docs/AUDIT.md`](docs/AUDIT.md) | the ~90-minute verification runbook, incl. a falsify-each-claim attack table |

| [`docs/CLAIMS.md`](docs/CLAIMS.md) | every claim → reproduction command → guarding test → last measured value |

| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | file-by-file walkthrough, exact byte layouts, data flow, cost model |

| [`docs/DECISIONS.md`](docs/DECISIONS.md) | every non-obvious choice with alternatives and evidence — incl. the full RRR rejection experiment |

## 📄 License

[PolyForm Noncommercial 1.0.0](LICENSE), © Ozan Kuesmez. In short: the

source is open; use, modify, and share freely for **any noncommercial

purpose** (personal projects, research, education, evaluation — charities

and public institutions included). **Commercial use requires a separate

license — just ask** (GitHub issue or profile contact). Copyright and

ownership remain with the author; required notices must be preserved.

Dual licensing: one codebase, free for the community, licensed for

business.

_{Built in Rust 🦀 · zero runtime dependencies · #![forbid(unsafe_code)]}
ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/oz4462/aletheia

Awesome Lists containing this project

README

≤ ¼

0×

14×

0
