https://github.com/tsaarni/raft-inspector
Tool for inspecting OpenBao / HashiCorp Vault integrated storage (raft)
https://github.com/tsaarni/raft-inspector
ai-generated-code hashicorp-vault openbao raft
Last synced: 20 days ago
JSON representation
Tool for inspecting OpenBao / HashiCorp Vault integrated storage (raft)
- Host: GitHub
- URL: https://github.com/tsaarni/raft-inspector
- Owner: tsaarni
- License: apache-2.0
- Created: 2026-05-15T13:28:36.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2026-06-26T05:50:37.000Z (21 days ago)
- Last Synced: 2026-06-26T07:19:42.755Z (21 days ago)
- Topics: ai-generated-code, hashicorp-vault, openbao, raft
- Language: Go
- Homepage:
- Size: 145 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# raft-inspector
Developer tool for offline inspection of OpenBao or Hashicorp Vault raft storage. Reads `raft.db` and `vault.db` directly without requiring a running server. Works while the server is running (copies files to bypass locks).
See [raft-inspector.md](raft-inspector.md) for a full walkthrough with example output against a 3-node cluster.
## Install
```bash
go install github.com/tsaarni/raft-inspector@latest
```
or run directly
```bash
go run github.com/tsaarni/raft-inspector@latest status --data-dir /opt/openbao/data
```
## Usage
```
raft-inspector status Health overview from raft/raft.db and vault.db
raft-inspector log List or inspect raft log entries with decoded operations
raft-inspector fsm Inspect the FSM state (vault.db data bucket)
raft-inspector snapshot Inspect an external snapshot archive
```
### Flags
**Data source** (mutually exclusive, for status/log/fsm):
--data-dir PATH Data directory (expects raft/raft.db and vault.db inside)
--raft-db PATH Path to raft.db directly
--vault-db PATH Path to vault.db directly
**Decryption** (Shamir seal threshold=1, for log/fsm/snapshot):
--unseal-key-file FILE JSON file from `bao operator init -format=json`
--unseal-key KEY Unseal key as hex or base64 string
**log:**
[range] Entry selector: N (single index), N..M (index range),
~N (last N entries), or date ranges:
5 Single entry by index
100..110 Index range (inclusive)
~10 Last 10 entries
2024-01-01..2024-01-31 Date range (inclusive)
2024-06-01.. From date to end of log
..2024-01-31 From start of log to date
Dates: YYYY-MM-DD or RFC 3339 (2024-01-15T10:30:00Z)
--stats Show operation distribution and hot keys instead of entries
**fsm / snapshot:**
Snapshot file path (positional, required for snapshot)
--prefix PREFIX List keys matching prefix (shows size per key)
--limit N Max keys to display (0=unlimited)
**Global:**
--max-value-length N Max bytes of decrypted value to display (default 256, 0=unlimited)
### Examples
```bash
raft-inspector status --data-dir /opt/openbao/data
raft-inspector log --data-dir /opt/openbao/data ~10
raft-inspector log --data-dir /opt/openbao/data --stats
raft-inspector fsm --data-dir /opt/openbao/data --prefix core/
# Direct file paths
raft-inspector status --raft-db ./raft.db --vault-db ./vault.db
# Decryption
raft-inspector log --data-dir /data --unseal-key-file init.json ~5
raft-inspector fsm --vault-db ./vault.db --prefix core/ \
--unseal-key ""
```
## How the database files work
Integrated storage uses two database files per node:
**`raft/raft.db`** holds the replicated write-ahead log. Every write operation (secret put, lease creation, policy change) is first appended here as a log entry, then replicated to other nodes. OpenBao automatically takes periodic snapshots of the current state and truncates old log entries, keeping only a trailing window (default ~10000 entries) for follower catch-up. The file grows with write activity but freed space from truncated entries is reused internally. The on-disk file size stays roughly stable after initial growth.
**`vault.db`** holds the current state: all secrets, configuration, leases, and internal metadata. Log entries from `raft.db` are applied here as key/value writes and deletes. This is the "result" of replaying the log. Its size reflects the actual volume of stored data. When secrets or leases are deleted, the freed space is reused for future writes but the file does not shrink.
**Manual snapshot restore** (`bao/vault operator raft snapshot restore`) replaces `vault.db` with a fresh file built from the snapshot contents. The `raft.db` is not modified — existing log entries remain and are reused for follower catch-up. The new `vault.db` contains only live data with no accumulated free pages, which is why snapshot restore is the primary way to reclaim disk space for `vault.db`.
**Index and term** are the two values that identify any log entry. The **index** is a cluster-wide sequence number that increases by one for each write. It never resets and all nodes agree on what each index contains. The **term** is an epoch counter that increments each time a new leader is elected. It is used internally for leader election and consistency checks between nodes but does not affect log replay. Both are stored in `raft/raft.db` as part of each log entry. The `vault.db` config bucket records the last applied index so that after a restart, the node knows where to resume applying log entries.
**Buckets** are BoltDB's equivalent of tables — each database file contains a few named buckets that organize data by purpose.
`raft/raft.db` has two buckets:
- **`logs`** — the raft log entries, keyed by index. Each entry is a protobuf-encoded message containing one or more operations (put, delete, etc.).
- **`conf`** — election state: current term, last vote candidate, and last vote term. Persisted so the node can safely resume after a restart without violating election rules.
`vault.db` has two buckets:
- **`data`** — all application state (secrets, leases, policies, mounts, auth config, and internal cryptographic material like the keyring and unseal keys under `core/`).
- **`config`** — FSM metadata: last applied log index/term, cluster membership, and this node's desired role (voter/nonvoter).
## Field descriptions
### status
| Field | Source | Description |
|-------|--------|-------------|
| Current Term | raft/raft.db | Raft election epoch; increments on each leader election. Rapidly increasing = network instability. |
| First Log Index | raft/raft.db | Oldest log entry retained. Advances as snapshots compact old entries away. |
| Last Log Index | raft/raft.db | Most recent log entry written. Continuously increasing on an active cluster. |
| Entry Count | raft/raft.db | Retained log entries (last − first + 1). Typically near `trailing_logs` config (default 10000). |
| Last Vote Cand | raft/raft.db | Node this server last voted for in a leader election. |
| Last Vote Term | raft/raft.db | Term in which the last vote was cast. Should be close to Current Term. |
| Applied Index | vault.db | Last log entry applied to the FSM. Should equal or be very close to Last Log Index. |
| Applied Term | vault.db | Term of the last applied log entry. |
| Config Index | vault.db | Log index at which current cluster membership was committed. Changes on add/remove. |
| Servers | vault.db | Cluster members: voter = participates in quorum, nonvoter = read replica only. |
| Desired Suffrage | vault.db | Role this node wants (voter/nonvoter). Mismatch with actual = promotion/demotion pending. |
| Unapplied Entries | computed | Last Log Index − Applied Index. Should be 0 on a healthy idle node; large gap = FSM falling behind. |
| Trailing Entries | computed | Applied entries kept in log for follower catch-up without full snapshot transfer. |
| Snapshot Index | computed | First Log Index − 1. Highest index compacted into a snapshot; never advancing = snapshots broken. |
| File Size | os.Stat | Total on-disk size of the database file. |
| DB Logical Size | bolt.Tx.Size | Portion of the file actively used by the database. Difference from File Size is preallocation. |
| Page Size | bolt.DB.Info | Internal allocation unit (typically 4096 bytes). |
| Free Pages | bolt.DB.Stats | Unused space from past deletions, reused for future writes but not returned to OS. High % = snapshot restore would shrink the file. |
| Pending Pages | bolt.DB.Stats | Pages being freed; will become reusable after the next write. Normally 0 on an idle node. |
| Freelist In-Use | bolt.DB.Stats | Overhead for tracking free pages internally. |
| Space Efficiency | computed | Percentage of on-disk file size occupied by live data (excludes free pages and preallocation). For `vault.db`, the live data size is the estimated file size after snapshot restore. For `raft.db`, snapshot restore has no effect. |
| Bucket (per-bucket) | bolt.Bucket.Stats | Key count, tree depth, branch/leaf page utilization %. Branch pages hold internal routing keys; leaf pages hold actual data. Utilization = in-use / allocated bytes. 50–100% is normal. Lower values occur after heavy deletions (e.g., lease revocations). Space is reused on new writes; snapshot restore reclaims disk. |
| Integrity Check | bolt.Tx.Check | Consistency check across all database pages. OK = healthy. FAILED = data corruption, investigate immediately. |
### log
| Field | Source | Description |
|-------|--------|-------------|
| Index | raft/raft.db | Monotonically increasing sequence number identifying this entry in the raft log. |
| Term | raft/raft.db | Election term when the leader created this entry. |
| Type | raft/raft.db | `LogCommand` = data op, `LogConfiguration` = membership change, `LogBarrier` = consistency fence, `LogNoop` = leader establishment after election. |
| AppendedAt | raft/raft.db | Wall-clock time when the leader appended this entry. Shows `(+offset)` relative to first displayed entry. |
| Operations | raft/raft.db | Decoded ops: put (write key), delete (remove key), beginTx/commitTx (transaction boundaries), verifyRead/verifyList (optimistic concurrency checks), restoreCallback (post-snapshot-restore signal). |
### log --stats
| Field | Source | Description |
|-------|--------|-------------|
| Time Range | raft/raft.db | Wall-clock span from oldest to newest retained log entry. |
| Entry Count | raft/raft.db | Total retained log entries. |
| Total/Avg/Max Size | raft/raft.db | Byte sizes of log entry payloads (encrypted). Large max = bulk writes or big secrets. |
| Op Distribution | raft/raft.db | Count per operation type. Helps identify workload pattern (write-heavy, transactional, etc.). |
| Hot Keys | raft/raft.db | Top storage paths by write frequency. Normal hot keys: `core/lock`, `sys/expire/`, `core/leader`. Unexpected ones may indicate misbehaving plugins or lease storms. |
### fsm
| Field | Source | Description |
|-------|--------|-------------|
| Keys | vault.db | Plaintext storage paths in the data bucket. Values are AES-GCM encrypted with the keyring. |
| Top-level segments | vault.db | First path segment groups: `core/` = internal state, `sys/` = system backend (policies, leases, mounts), `logical/` = secrets engines, `auth/` = auth methods. Key count per segment shows relative data volume. |
| Largest keys | vault.db | Top 10 entries by encrypted value size. |
### snapshot
| Field | Source | Description |
|-------|--------|-------------|
| Index | meta.json | Raft log index captured in this snapshot. All state up to this index is included. |
| Term | meta.json | Raft term at snapshot time. |
| Servers | meta.json | Cluster membership at snapshot time. May be stale if snapshot is old. |
| Checksums | SHA256SUMS | SHA-256 integrity verification. ✓ = intact, ✗ = corruption during transfer or storage. |
| Total Keys | state.bin | Number of key/value entries in the full FSM state dump. Useful for comparing snapshots over time. |
| Top-level segments | state.bin | First path segment groups and their key counts (same as fsm command). |
| Largest keys | state.bin | Top 10 entries by encrypted value size. |