https://github.com/harperfast/rocksdb-js
RocksDB binding for Node.js
https://github.com/harperfast/rocksdb-js
Last synced: about 2 months ago
JSON representation
RocksDB binding for Node.js
- Host: GitHub
- URL: https://github.com/harperfast/rocksdb-js
- Owner: HarperFast
- License: apache-2.0
- Created: 2024-09-14T14:05:11.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2026-04-23T23:25:33.000Z (2 months ago)
- Last Synced: 2026-04-24T01:31:17.204Z (2 months ago)
- Language: C++
- Homepage:
- Size: 31.8 MB
- Stars: 5
- Watchers: 4
- Forks: 0
- Open Issues: 20
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
- Agents: AGENTS.md
Awesome Lists containing this project
README
# rocksdb-js
A Node.js binding for the RocksDB library.
## Features
- Supports optimistic and pessimistic transactions
- Hybrid sync/async data retrieval
- Range queries return an iterable with array-like methods and lazy evaluation
- Transaction log system for recording transaction related data
- Custom stores provide ability to override default database interactions
- Efficient binary key and value encoding
- Access to internal RocksDB statistics
- Designed for Node.js and Bun on Linux, macOS, and Windows
## Example
```typescript
const db = RocksDatabase.open('/path/to/db');
for (const key of ['a', 'b', 'c', 'd', 'e']) {
await db.put(key, `value ${key}`);
}
console.log(await db.get('b')); // `value b`
for (const { key, value } of db.getRange({ start: 'b', end: 'd' })) {
console.log(`${key} = ${value}`);
}
await db.transaction(async (txn: Transaction) => {
await txn.put('f', 'value f');
await txn.remove('c');
});
```
## Usage
### `new RocksDatabase(path, options?)`
Creates a new database instance.
- `path: string` The path to write the database files to. This path does not need to exist, but the
parent directories do.
- `options: object` [optional]
- `disableWAL: boolean` Whether to disable the RocksDB write ahead log. Defaults to `false`.
- `enableStats: boolean` When `true` and the database is open, RocksDB will captures stats that
are retrieved by calling `db.getStats()`. Enabling statistics imposes 5-10% in overhead.
Defaults to `false`.
- `name: string` The column family name. Defaults to `"default"`.
- `noBlockCache: boolean` When `true`, disables the block cache. Block caching is enabled by
default and the cache is shared across all database instances.
- `parallelismThreads: number` The number of background threads to use for flush and compaction.
Defaults to `1`.
- `pessimistic: boolean` When `true`, throws conflict errors when they occur instead of waiting
until commit. Defaults to `false`.
- `readOnly: boolean` When `true`, the database is opened in read-only mode. Read operations are
permitted. Write operations will throw an error with code `ERR_DATABASE_READONLY`. Transactions
are a no-op in read-only mode.
- `statsLevel: StatsLevel` Controls which type of statistics to skip and reduce statistic
overhead. Defaults to `StatsLevel.ExceptDetailedTimers`.
- `store: Store` A custom store that handles all interaction between the `RocksDatabase` or
`Transaction` instances and the native database interface. See [Custom Store](#custom-store) for
more information.
- `transactionLogMaxAgeThreshold: number` The threshold for the transaction log file's last
modified time to be older than the retention period before it is rotated to the next sequence
number. Value must be between `0.0` and `1.0`. A threshold of `0.0` means ignore age check.
Defaults to `0.75`.
- `transactionLogMaxSize: number` The maximum size of a transaction log file. If a log file is
empty, the first log entry will always be added regardless if it's larger than the max size. If
a log file is not empty and the entry is larger than the space available, the log file is
rotated to the next sequence number. Defaults to 16 MB.
- `transactionLogRetention: string | number` The number of minutes to retain transaction logs
before purging. Defaults to `'3d'` (3 days).
- `transactionLogsPath: string` The path to store transaction logs. Defaults to
`"${db.path}/transaction_logs"`.
### `db.close()`
Closes a database. This function can be called multiple times and will only close an opened
database. A database instance can be reopened once its closed.
```typescript
const db = RocksDatabase.open('foo');
db.close();
```
### `db.config(options)`
Sets global database settings.
- `options: object`
- `blockCacheSize: number` The amount of memory in bytes to use to cache uncompressed blocks.
Defaults to 32MB. Set to `0` (zero) disables block cache for future opened databases. Existing
block cache for any opened databases is resized immediately. Negative values throw an error.
```typescript
RocksDatabase.config({
blockCacheSize: 100 * 1024 * 1024, // 100MB
});
```
### `db.isOpen(): boolean`
Returns `true` if the database is open, otherwise false.
```typescript
console.log(db.isOpen()); // true or false
```
### `db.name: string`
Returns the database column family's name.
```typescript
const db = new RocksDatabase('path/to/db');
console.log(db.name); // 'default'
const db2 = new RocksDatabase('path/to/db', { name: 'users' });
console.log(db.name); // 'users'
```
### `db.open(): RocksDatabase`
Opens the database at the given path. This must be called before performing any data operations.
```typescript
import { RocksDatabase } from '@harperfast/rocksdb-js';
const db = new RocksDatabase('path/to/db');
db.open();
```
There's also a static `open()` method for convenience that performs the same thing:
```typescript
const db = RocksDatabase.open('path/to/db');
```
### `db.status: 'opened' | 'closed'`
Returns a string `'opened'` or `'closed'` indicating if the database is opened or closed.
```typescript
console.log(db.status);
```
## Data Operations
### `db.clear(options?): Promise`
Asychronously removes all data in the current database.
- `options: object`
- `batchSize?: number` The number of records to remove at once. Defaults to `10000`.
Returns the number of entries that were removed.
Note: This does not remove data from other column families within the same database path.
```typescript
for (let i = 0; i < 10; i++) {
db.putSync(`key${i}`, `value${i}`);
}
const entriesRemoved = await db.clear();
console.log(entriesRemoved); // 10
```
### `db.clearSync(options?): number`
Synchronous version of `db.clear()`.
- `options: object`
- `batchSize?: number` The number of records to remove at once. Defaults to `10000`.
```typescript
for (let i = 0; i < 10; i++) {
db.putSync(`key${i}`, `value${i}`);
}
const entriesRemoved = db.clearSync();
console.log(entriesRemoved); // 10
```
### `db.destroy(): void`
Completely removes a database based on the `db` instance's path including all data, column families,
and files on disk.
```typescript
db.destroy();
console.log(fs.existsSync(db.path)); // false
```
### `db.drop(): Promise`
Removes all entries in the database. If the database was opened with a `name`, the database will be
deleted on close.
```typescript
const db = RocksDatabase.open('path/to/db', { name: 'users' });
await db.drop();
db.close();
```
### `db.dropSync(): void`
Synchronous version of `db.drop()`.
```typescript
const db = RocksDatabase.open('path/to/db');
db.dropSync();
db.close();
```
### `db.get(key: Key, options?: GetOptions): MaybePromise`
Retreives the value for a given key. If the key does not exist, it will resolve `undefined`.
```typescript
const result = await db.get('foo');
assert.equal(result, 'foo');
```
If the value is in the memtable or block cache, `get()` will immediately return the value
synchronously instead of returning a promise.
```typescript
const result = db.get('foo');
const value = result instanceof Promise ? await result : result;
assert.equal(result, 'foo');
```
Note that all errors are returned as rejected promises.
### `db.getSync(key: Key, options?: GetOptions): any`
Synchronous version of `get()`.
### `db.getKeys(options?: IteratorOptions): ExtendedIterable`
Retrieves all keys within a range.
```typescript
for (const key of db.getKeys()) {
console.log(key);
}
```
### `db.getKeysCount(options?: RangeOptions): number`
Retrieves the number of keys within a range.
```typescript
const total = db.getKeysCount();
const range = db.getKeysCount({ start: 'a', end: 'z' });
```
### `db.getMonotonicTimestamp(): number`
Returns the current timestamp as a monotonically increasing timestamp in milliseconds represented as
a decimal number.
```typescript
const ts = db.getMonotonicTimestamp();
console.log(ts); // 1764307857213.739
```
### `db.getOldestSnapshotTimestamp(): number`
Returns a number representing a unix timestamp of the oldest unreleased snapshot.
Snapshots are only created during transactions. When the database is opened in optimistic mode (the
default), the snapshot will be created on the first read. When the database is opened in pessimistic
mode, the snapshot will be created on the first read or write.
```typescript
console.log(db.getOldestSnapshotTimestamp()); // returns `0`, no snapshots
const promise = db.transaction(async (txn) => {
// perform a write to create a snapshot
await txn.get('foo');
await setTimeout(100);
});
console.log(db.getOldestSnapshotTimestamp()); // returns `1752102248558`
await promise;
// transaction completes, snapshot released
console.log(db.getOldestSnapshotTimestamp()); // returns `0`, no snapshots
```
### `db.getDBProperty(propertyName: string): string`
Gets a RocksDB database property as a string.
- `propertyName: string` The name of the property to retrieve (e.g., ) `'rocksdb.levelstats'`.
```typescript
const db = RocksDatabase.open('/path/to/database');
const levelStats = db.getDBProperty('rocksdb.levelstats');
const stats = db.getDBProperty('rocksdb.stats');
```
### `db.getDBIntProperty(propertyName: string): number`
Gets a RocksDB database property as an integer.
- `propertyName: string` The name of the property to retrieve (e.g., ) `'rocksdb.num-blob-files'`.
```typescript
const db = RocksDatabase.open('/path/to/database');
const blobFiles = db.getDBIntProperty('rocksdb.num-blob-files');
const numKeys = db.getDBIntProperty('rocksdb.estimate-num-keys');
```
### `db.getRange(options?: IteratorOptions): ExtendedIterable`
Retrieves a range of keys and their values. Supports both synchronous and asynchronous iteration.
```typescript
// sync
for (const { key, value } of db.getRange()) {
console.log({ key, value });
}
// async
for await (const { key, value } of db.getRange()) {
console.log({ key, value });
}
// key range
for (const { key, value } of db.getRange({ start: 'a', end: 'z' })) {
console.log({ key, value });
}
```
### `db.getUserSharedBuffer(key: Key, defaultBuffer: ArrayBuffer, options?)`
Creates a new buffer with the contents of `defaultBuffer` that can be accessed across threads. This
is useful for storing data such as flags, counters, or any ArrayBuffer-based data.
- `options?: object`
- `callback?: () => void` A optional callback is called when `notify()` on the returned buffer is
called.
Returns a new `ArrayBuffer` with two additional methods:
- `notify()` - Invokes the `options.callback`, if specified.
- `cancel()` - Removes the callback; future `notify()` calls do nothing
Note: If a shared buffer already exists for the given `key`, the returned `ArrayBuffer` will
reference this existing shared buffer. Once all `ArrayBuffer` instances have gone out of scope and
garbage collected, the underlying memory and notify callback will be freed.
```typescript
const buffer = new Uint8Array(db.getUserSharedBuffer('isDone', new ArrayBuffer(1)));
done[0] = 0;
if (done[0] !== 1) {
done[1] = 1;
}
```
```typescript
const incrementer = new BigInt64Array(
db.getUserSharedBuffer('next-id', new BigInt64Array(1).buffer)
);
incrementer[0] = 1n;
function getNextId() {
return Atomics.add(incrementer, 0, 1n);
}
```
### `db.put(key: Key, value: any, options?: PutOptions): Promise`
Stores a value for a given key.
```typescript
await db.put('foo', 'bar');
```
### `db.putSync(key: Key, value: any, options?: PutOptions): void`
Synchronous version of `put()`.
### `db.remove(key: Key): Promise`
Removes the value for a given key.
```typescript
await db.remove('foo');
```
### `db.removeSync(key: Key): void`
Synchronous version of `remove()`.
## Transactions
### `db.transaction(callback: TransactionCallback, options?: DBTransactionOptions): Promise`
Executes all database operations within the specified callback within a single transaction. If the
callback completes without error, the database operations are automatically committed. However, if
an error is thrown during the callback, all database operations will be rolled back.
```typescript
import type { Transaction } from '@harperfast/rocksdb-js';
await db.transaction(async (txn: Transaction) => {
await txn.put('foo', 'baz');
});
```
Additionally, you may pass the transaction into any database data method:
```typescript
await db.transaction(async (transaction: Transaction) => {
await db.put('foo', 'baz', { transaction });
});
```
Note that `db.transaction()` returns whatever value the transaction callback returns:
```typescript
const isBar = await db.transaction(async (txn: Transaction) => {
const foo = await txn.get('foo');
return foo === 'bar';
});
console.log(isBar ? 'Foo is bar' : 'Foo is not bar');
```
### `db.transactionSync(callback: TransactionCallback, options?: TransactionOptions): T`
Executes a transaction callback and commits synchronously. Once the transaction callback returns,
the commit is executed synchronously and blocks the current thread until finished.
Inside a synchronous transaction, use `getSync()`, `putSync()`, and `removeSync()`.
```typescript
import type { Transaction } from '@harperfast/rocksdb-js';
db.transactionSync((txn: Transaction) => {
txn.putSync('foo', 'baz');
});
```
### Optimistic and Pessimistic Modes
`rocksdb-js` supports two different transaction modes: optimistic and pessimistic. The default mode
is optimistic.
- Optimistic: Conflicts detected at commit time.
- Pessimistic: Conflicts throw immediately on detection.
When a database is opened in optimistic mode, transactions are not locked and can be retried if
they fail with a conflict. When a database is opened in pessimistic mode, transactions are aborted
and cannot be retried if they fail with a conflict.
Optimistic mode is the default mode and is recommended for most use cases. Pessimistic mode is
recommended for use cases where you need to know immediately if a conflict occurs.
If a database is opened in one mode, it cannot be opened in a different mode. An error will be
thrown when trying to open it in a different mode without closing the database first.
### `TransactionCallback`
`(txn: Transaction, attempt: number) => T | PromiseLike`
A sync or async function to encapsulate all of the transaction operations. Once the function is
executed, the transaction is automatically committed. If the function returns a value, it will be
returned from the transaction call.
The `txn` parameter is a `Transaction`. See the [Transaction](#class-transaction) section for more
details.
The `attempt` parameter is the number of times the transaction has been retried.
### `TransactionOptions`
- `disableSnapshot?: boolean` Whether to disable snapshots. Defaults to `false`.
- `maxRetries?: number` The maximum number of times to retry the transaction. Defaults to `3`.
- `retryOnBusy?: boolean` Whether to retry the transaction if the commit fails with `IsBusy`.
Defaults to `true` when the transaction is bound to a transaction log, otherwise `false`.
### Transaction Retry Logic
The retry mechanism will only be active when the `retryOnBusy` option is `true` or when
`retryOnBusy` is `undefined` and the transaction is bound to a transaction log. The attempts starts
at `1` and ends at `maxRetries`.
When using a transaction log and the commit fails with `ERR_BUSY`, the transaction log will be in
a bad state and the transaction will need to be retried. If the max retries is reached or the
transaction is not retried, a `ERR_TRANSACTION_ABANDONED` error will be thrown.
Users should use the `attempt` transaction callback parameter to ensure duplicate transaction log
entries are not added.
### Class: `Transaction`
The transaction callback is passed in a `Transaction` instance which contains all of the same data
operations methods as the `RocksDatabase` instance plus:
- `txn.abort()` Rolls back and closes the transaction. This method is automatically called after the
transaction callback returns, so you shouldn't need to call it, but it's ok to do so. Once called,
no further transaction operations are permitted. Calling this method multiple times has no effect.
- `txn.commit(): Promise` Asynchronously commits the transaction and closes the transaction.
- `txn.commitSync()` Synchronously commits and closes the transaction.
- `txn.getTimestamp(): number` Retrieves the transaction start timestamp in seconds as a decimal. It
defaults to the time at which the transaction was created.
- `txn.id: number` The read-only transaction ID. Transaction IDs are unique to the RocksDB database
path, regardless the database name/column family.
- `txn.setTimestamp(ts?: number): void` Overrides the transaction start timestamp. If called without
a timestamp, it will set the timestamp to the current time. The value must be in seconds with
higher precision in the decimal.
#### `txn.abort(): void`
Rolls back and closes the transaction. This method is automatically called after the transaction
callback returns, so you shouldn't need to call it, but it's ok to do so. Once called, no further
transaction operations are permitted.
#### `txn.commit(): Promise`
Commits and closes the transaction. This is a non-blocking operation and runs on a background
thread. Once called, no further transaction operations are permitted.
#### `txn.commitSync(): void`
Synchronously commits and closes the transaction. This is a blocking operation on the main thread.
Once called, no further transaction operations are permitted.
#### `txn.getTimestamp(): number`
Retrieves the transaction start timestamp in seconds as a decimal. It defaults to the time at which
the transaction was created.
#### `txn.id`
Type: `number`
The transaction ID represented as a 32-bit unsigned integer. Transaction IDs are unique to the
RocksDB database path, regardless the database name/column family.
#### `txn.setTimestamp(ts: number?): void`
Overrides the transaction start timestamp. If called without a timestamp, it will set the timestamp
to the current time. The value must be in seconds with higher precision in the decimal.
```typescript
await db.transaction(async (txn) => {
txn.setTimestamp(Date.now() / 1000);
});
```
## Events
### Event: `'aftercommit'`
The `'aftercommit'` event is emitted after a transaction has been committed and the transaction has
completed including waiting for the async worker thread to finish.
- `result: object`
- `next: null`
- `last: null`
- `txnId: number` The id of the transaction that was just committed.
### Event: `'beforecommit'`
The `'beforecommit'` event is emitted before a transaction is about to be committed.
### Event: `'begin-transaction'`
The `'begin-transaction'` event is emitted right before the transaction function is executed.
### Event: `'committed'`
The `'committed'` event is emitted after the transaction has been written. When this event is
emitted, the transaction is still cleaning up. If you need to know when the transaction is fully
complete, use the `'aftercommit'` event.
## Event API
`rocksdb-js` provides a EventEmitter-like API that lets you asynchronously notify events to one or
more synchronous listener callbacks. Events are scoped by database path.
Unlike `EventEmitter`, events are emitted asynchronously, but in the same order that the listeners
were added.
```typescript
const callback = (name) => console.log(`Hi from ${name}`);
db.addListener('foo', callback);
db.notify('foo');
db.notify('foo', 'bar');
db.removeListener('foo', callback);
```
### `addListener(event: string, callback: () => void): void`
Adds a listener callback for the specific key.
```typescript
db.addListener('foo', () => {
// this callback will be executed asynchronously
});
db.addListener(1234, (...args) => {
console.log(args);
});
```
### `listeners(event: string): number`
Gets the number of listeners for the given key.
```typescript
db.listeners('foo'); // 0
db.addListener('foo', () => {});
db.listeners('foo'); // 1
```
### `on(event: string, callback: () => void): void`
Alias for `addListener()`.
### `once(event: string, callback: () => void): void`
Adds a one-time listener, then automatically removes it.
```typescript
db.once('foo', () => {
console.log('This will only ever be called once');
});
```
### `removeListener(event: string, callback: () => void): boolean`
Removes an event listener. You must specify the exact same callback that was used in
`addListener()`.
```typescript
const callback = () => {};
db.addListener('foo', callback);
db.removeListener('foo', callback); // return `true`
db.removeListener('foo', callback); // return `false`, callback not found
```
### `off(event: string, callback: () => void): boolean`
Alias for `removeListener()`.
### `notify(event: string, ...args?): boolean`
Call all listeners for the given key. Returns `true` if any callbacks were found, otherwise `false`.
Unlike `EventEmitter`, events are emitted asynchronously, but in the same order that the listeners
were added.
You can optionally emit one or more arguments. Note that the arguments must be serializable. In
other words, `undefined`, `null`, strings, booleans, numbers, arrays, and objects are supported.
```typescript
db.notify('foo');
db.notify(1234);
db.notify({ key: 'bar' }, { value: 'baz' });
```
## Statistics
Retrieve RocksDB statistics at runtime. You must set `enableStats: true` when calling `db.open()`.
Statistics are captured at the database level and include all column families.
RocksDB has two types of statistics: tickers and histograms. Tickers are 64-bit unsigned integers
that measure counters. Histograms are objects containing various measurements of statistic
distribution across all operations.
```typescript
import { RocksDatabase, stats } from '@harperfast/rocksdb-js';
const db = RocksDatabase.open('/path/to/db', {
enableStats: true,
statsLevel: stats.StatsLevel.ExceptDetailedTimers, // default
});
console.log(db.getStats());
```
### `db.getStat(statName: string): RocksDBStat`
Retrieves a single statistic value. Return value is either a `number` or `StatsHistogramData`
object.
```typescript
console.log(db.getStat('rocksdb.block.cache.miss'));
```
### `db.getStats(all?: boolean): RocksDBStats`
Returns an object containing a curated list of column family-level properties, internal tickers
stats, and internal histogram stats. Return value is an object with the stat name as the key and
a `RocksDBStat` as the value.
By default, it only returns the most meaningful internal stats. When `all = true`, it returns the
same column family-level properties, but includes all internal tickers and histogram stats.
Column family and ticker stat values are 64-bit unsigned integers and histogram values are
`StatsHistogramData` objects.
```typescript
// get essential stats
console.log(db.getStats());
// get all stats
console.log(db.getStats(true));
```
### `stats`
An object containing stat specific constants including ticker and histogram names.
#### `stats.histograms`
An array of internal histogram stat names.
```typescript
import { stats } from '@harperfast/rocksdb-js';
console.log(stats.histograms);
```
#### `stats.tickers`
An array of internal ticker stat names.
```typescript
import { stats } from '@harperfast/rocksdb-js';
console.log(stats.tickers);
```
#### `stats.StatsLevel`
The `stats.StatsLevel` contains constants used to set which types of skip and reduce statistic overhead.
- `StatsLevel.DisableAll` Disable all metrics.
- `StatsLevel.ExceptTickers` Disable all tickers.
- `StatsLevel.ExceptHistogramOrTimers` Disable timer stats and skip histogram stats.
- `StatsLevel.ExceptTimers` Skip timer stats
- `StatsLevel.ExceptDetailedTimers` Skip time waiting for mutex locks and compression.
- `StatsLevel.ExceptTimeForMutex` Skip time waiting for mutex locks.
- `StatsLevel.All` Collects all stats.
### `type RocksDBStat = number | StatsHistogramData`
A `RocksDBStat` is either a `number` or `StatsHistogramData` object.
### `type RocksDBStats = Record`
A `RocksDBStats` is an object with the stat name as the key and a `RocksDBStat` as the value.
### `type StatsHistogramData`
An object is a record with the following properties:
- `average: number` A double containing the average value.
- `count: number` An unsigned 64-bit integer containing the number of values.
- `max: number` A double containing the maximum value.
- `median: number` A double containing the median value.
- `min: number` A double containing the minimum value.
- `percentile95: number` A double containing the 95th percentile value.
- `percentile99: number` A double containing the 99th percentile value.
- `standardDeviation: number` A double containing the standard deviation.
- `sum: number` An unsigned 64-bit integer containing the sum of all values.
## Exclusive Locking
`rocksdb-js` includes a handful of functions for executing thread-safe mutually exclusive functions.
### `db.hasLock(key: Key): boolean`
Returns `true` if the database has a lock for the given key, otherwise `false`.
```typescript
db.hasLock('foo'); // false
db.tryLock('foo'); // true
db.hasLock('foo'); // true
```
### `db.tryLock(key: Key, onUnlocked?: () => void): boolean`
Attempts to acquire a lock for a given key. If the lock is available, the function returns `true`
and the optional `onUnlocked` callback is never called. If the lock is not available, the function
returns `false` and the `onUnlocked` callback is queued until the lock is released.
When a database is closed, all locks associated to it will be unlocked.
```typescript
db.tryLock('foo', () => {
console.log('never fired');
}); // true, callback ignored
db.tryLock('foo', () => {
console.log('hello world');
}); // false, already locked, callback queued
db.unlock('foo'); // fires second lock callback
```
The `onUnlocked` callback function can be used to signal to retry acquiring the lock:
```typescript
function doSomethingExclusively() {
// if lock is unavailable, queue up callback to recursively retry
if (db.tryLock('foo', () => doSomethingExclusively())) {
// lock acquired, do something exclusive
db.unlock('foo');
}
}
```
### `db.unlock(key): boolean`
Releases the lock on the given key and calls any queued `onUnlocked` callback handlers. Returns
`true` if the lock was released or `false` if the lock did not exist.
```typescript
db.tryLock('foo');
db.unlock('foo'); // true
db.unlock('foo'); // false, already unlocked
```
### `db.withLock(key: Key, callback: () => void | Promise): Promise`
Runs a function with guaranteed exclusive access across all threads.
```typescript
await db.withLock('key', async () => {
// do something exclusive
console.log(db.hasLock('key')); // true
});
```
If there are more than one simultaneous lock requests, it will block them until the lock is
available.
```typescript
await Promise.all([
db.withLock('key', () => {
console.log('first lock blocking for 100ms');
return new Promise((resolve) => setTimeout(resolve, 100));
}),
db.withLock('key', () => {
console.log('second lock blocking for 100ms');
return new Promise((resolve) => setTimeout(resolve, 100));
}),
db.withLock('key', () => {
console.log('third lock acquired');
}),
]);
```
Note: If the `callback` throws an error, Node.js suppress the error. Node.js 18.3.0 introduced a
`--force-node-api-uncaught-exceptions-policy` flag which will cause errors to emit the
`'uncaughtException'` event. Future Node.js releases will enable this flag by default.
### `db.flush(): Promise`
Flushes all in-memory data to disk asynchronously.
```typescript
await db.flush();
```
### `db.flushSync(): void`
Flushes all in-memory data to disk synchronously. Note that this can be an expensive operation, so
it is recommended to use `flush()` if you want to keep the event loop free.
```typescript
db.flushSync();
```
## Transaction Log
A user controlled API for logging transactions. This API is designed to be generic so that you can
log gets, puts, and deletes, but also arbitrary entries.
Transaction logs are isolated by the database path allowing different column families in the same
database to share the transaction log store, but not other databases.
### `db.listLogs(): string[]`
Returns an array of log store names.
```typescript
const names = db.listLogs();
```
### `db.purgeLogs(options?): string[]`
Deletes transaction log files older than the `transactionLogRetention` (defaults to 3 days).
- `options: object`
- `before?: number` Remove all transaction log files older than the specified timestamp.
- `destroy?: boolean` When `true`, deletes transaction log stores including all log sequence files
on disk.
- `name?: string` The name of a store to limit the purging to.
Returns an array with the full path of each log file deleted.
```typescript
const removed = db.purgeLogs();
console.log(`Removed ${removed.length} log files`);
```
### `db.useLog(name): TransactionLog`
Gets or creates a `TransactionLog` instance. Internally, the `TransactionLog` interfaces with a
shared transaction log store that is used by all threads. Multiple worker threads can use the same
log at the same time.
- `name: string | number` The name of the log. Numeric log names are converted to a string.
```typescript
const log1 = db.useLog('foo');
const log2 = db.useLog('foo'); // gets existing instance (e.g. log1 === log2)
const log3 = db.useLog(123);
```
`Transaction` instances also provide a `useLog()` method that binds the returned transaction log to
the transaction so you don't need to pass in the transaction id every time you add an entry.
```typescript
await db.transaction(async (txn) => {
const log = txn.useLog('foo');
log.addEntry(Buffer.from('hello'));
});
```
### Class: `TransactionLog`
A `TransactionLog` lets you add arbitrary data bound to a transaction that is automatically written
to disk right before the transaction is committed. You may add multiple enties per transaction. The
underlying architecture is thread safe.
- `log.addEntry()`
- `log.path`
- `log.query()`
#### `log.addEntry(data, transactionId): void`
Adds an entry to the transaction log.
- `data: Buffer | UInt8Array` The entry data to store. There is no inherent limit beyond what
Node.js can handle.
- `transactionId: Number` A related transaction used to batch entries on commit.
```typescript
const log = db.useLog('foo');
await db.transaction(async (txn) => {
log.addEntry(Buffer.from('hello'), txn.id);
});
```
If using `txn.useLog()` (instead of `db.useLog()`), you can omit the transaction id from
`addEntry()` calls.
```typescript
await db.transaction(async (txn) => {
const log = txn.useLog('foo');
log.addEntry(Buffer.from('hello'));
});
```
Note that the `TransactionLog` class also has internal methods `_getMemoryMapOfFile`,
`_findPosition`, and `_getLastCommittedPosition` that should not be used directly and may change in
any version.
#### `log.path: string`
Returns the path to the transaction log store files.
```typescript
const log = db.useLog('foo');
console.log(log.path);
```
#### `log.query(options?): IterableIterator`
Returns an iterable/iterator that streams all log entries for the given filter.
- `options: object`
- `start?: number` The transaction start timestamp.
- `end?: string` The transction end timestamp.
- `exclusiveStart?: boolean` When `true`, this will only match transactions with timestamps after
the start timestamp.
- `exactStart?: boolean` When `true`, this will only match and iterate starting from a transaction
with the given start timestamp. Once the specified transaction is found, all subsequent
transactions will be returned (regardless of whether their timestamp comes before the `start`
time). This can be combined with `exactStart`, finding the specified transaction, and returning
all transactions that follow. By default, all transactions equal to or greater than the start
timestamp will be included.
- `readUncommitted?: boolean` When `true`, this will include uncommitted transaction entries.
Normally transaction entries that haven't finished committed are not included. This is
particularly useful for replaying transaction logs on startup where many entries may have been
written to the log but are no longer considered committed if they were not flushed to disk.
- `startFromLastFlushed?: boolean` When `true`, this will only match transactions that have been
flushed from RocksDB's memtables to disk (and are within any provided `start` and `end` filters,
if included). This is useful for replaying transaction logs on startup where many entries may
have been written to the log but are no longer considered committed if they were not flushed to
disk.
The iterator produces an object with the log entry timestamp and data.
- `object`
- `data: Buffer` The entry data.
- `timestamp: number` The entry timestamp used to collate entries by transaction.
- `endTxn: boolean` This is `true` when the entry is the last entry in a transaction.
```typescript
const log = db.useLog('foo');
const iter = log.query({});
for (const entry of iter) {
console.log(entry);
}
const lastHour = Date.now() - 60 * 60 * 1000;
const rangeIter = log.query({ start: lastHour, end: Date.now() });
for (const entry of rangeIter) {
console.log(entry.timestamp, entry.data);
}
```
#### `log.getLogFileSize(sequenceNumber?: number): number`
Returns the size of the given transaction log sequence file in bytes. Omit the sequence number to
get the total size of all the transaction log sequence files for this log.
### Transaction Log Parser
#### `parseTransactionLog(file)`
In general, you should use `log.query()` to query the transaction log, however, if you need to load
an entire transaction log into memory and want detailed information about entries, you can use the
`parseTransactionLog()` utility function.
```typescript
const everything = parseTransactionLog('/path/to/file.txnlog');
console.log(everything);
```
Returns an object containing all of the information in the log file.
- `size: number` The size of the file.
- `version: number` The log file format version.
- `entries: LogEntry[]` An array of transaction log entries.
- `data: Buffer` The entry data.
- `flags: number` Transaction related flags.
- `length: number` The size of the entry data.
- `timestamp: number` The entry timestamp.
### `currentThreadId(): number`
Returns the current thread ID.
```typescript
import { currentThreadId } from '@harperfast/rocksdb-js';
console.log(currentThreadId());
```
### `registryStatus(): RegistryStatus`
Returns an array containing that status of all active RocksDB instances.
- `path: string` The database path.
- `refCount: number` The number of JavaScript database instances plus the registry's reference.
- `columnFamiles: object` A map of column family names and their their info.
- `userSharedBuffers: number` The count of active user shared buffers.
- `transactions: number` The count of active transactions.
- `closables: number` The count of active database, transactions, and iterators.
- `locks: number` The count of active locks.
- `listenerCallbacks: number` The count of in-flight callbacks.
```typescript
import { registryStatus } from '@harperfast/rocksdb-js';
console.log(registryStatus());
```
### `shutdown(): void`
The `shutdown()` will flush all in-memory data to disk and wait for any outstanding compactions to
finish, for all open databases. It is highly recommended to call this in a `process` `exit` event
listener (on the main thread), to ensure that all data is flushed to disk before the process exits:
```typescript
import { shutdown } from '@harperfast/rocksdb-js';
process.on('exit', shutdown);
```
### `versions: { 'rocksdb': string; 'rocksdb-js': string }`
Returns the `rocksdb-js` and RocksDB version.
```typescript
import { versions } from '@harperfast/rocksdb-js';
console.log(versions); // { "rocksdb": "10.10.1", "rocksdb-js": "0.1.2" }
```
## Custom Store
The store is a class that sits between the `RocksDatabase` or `Transaction` instance and the native
RocksDB interface. It owns the native RocksDB instance along with various settings including
encoding and the db name. It handles all interactions with the native RocksDB instance.
The default `Store` contains the following methods which can be overridden:
- `constructor(path, options?)`
- `close()`
- `decodeKey(key)`
- `decodeValue(value)`
- `encodeKey(key)`
- `encodeValue(value)`
- `get(context, key, resolve, reject, txnId?)`
- `getCount(context, options?, txnId?)`
- `getKeys(context, options?)`
- `getKeysCount(context, options?)`
- `getRange(context, options?)`
- `getSync(context, key, options?)`
- `getUserSharedBuffer(key, defaultBuffer?)`
- `hasLock(key)`
- `isOpen()`
- `listLogs()`
- `open()`
- `putSync(context, key, value, options?)`
- `removeSync(context, key, options?)`
- `tryLock(key, onUnlocked?)`
- `unlock(key)`
- `useLog(context, name)`
- `withLock(key, callback?)`
To use it, extend the default `Store` and pass in an instance of your store into the `RocksDatabase`
constructor.
```typescript
import { RocksDatabase, Store } from '@harperfast/rocksdb-js';
class MyStore extends Store {
get(context, key, resolve, reject, txnId) {
console.log('Getting:' key);
return super.get(context, key, resolve, reject, txnId);
}
putSync(context, key, value, options) {
console.log('Putting:', key);
return super.putSync(context, key, value, options);
}
}
const myStore = new MyStore('path/to/db');
const db = RocksDatabase.open(myStore);
await db.put('foo', 'bar');
console.log(await db.get('foo'));
```
> [!IMPORTANT]
> If your custom store overrides `putSync()` without calling `super.putSync()` and it performs its
> own `this.encodeKey(key)`, then you MUST encode the VALUE before you encode the KEY.
>
> Keys are encoded into a shared buffer. If the database is opened with the `sharedStructuresKey`
> option, encoding the value will load and save the structures which encodes the
> `sharedStructuresKey` overwriting the encoded key in the shared key buffer, so it's ultra
> important that you encode the value first!
## Interfaces
### `RocksDBOptions`
- `options: object`
- `adaptiveReadahead: boolean` When `true`, RocksDB will do some enhancements for prefetching the
data. Defaults to `true`. Note that RocksDB defaults this to `false`.
- `asyncIO: boolean` When `true`, RocksDB will prefetch some data async and apply it if reads are
sequential and its internal automatic prefetching. Defaults to `true`. Note that RocksDB
defaults this to `false`.
- `autoReadaheadSize: boolean` When `true`, RocksDB will auto-tune the readahead size during scans
internally based on the block cache data when block caching is enabled, an end key (e.g. upper
bound) is set, and prefix is the same as the start key. Defaults to `true`.
- `backgroundPurgeOnIteratorCleanup: boolean` When `true`, after the iterator is closed, a
background job is scheduled to flush the job queue and delete obsolete files. Defaults to
`true`. Note that RocksDB defaults this to `false`.
- `fillCache: boolean` When `true`, the iterator will fill the block cache. Filling the block
cache is not desirable for bulk scans and could impact eviction order. Defaults to `false`. Note
that RocksDB defaults this to `true`.
- `readaheadSize: number` The RocksDB readahead size. RocksDB does auto-readahead for iterators
when there is more than two reads for a table file. The readahead starts at 8KB and doubles on
every additional read up to 256KB. This option can help if most of the range scans are large and
if a larger readahead than that enabled by auto-readahead is needed. Using a large readahead
size (> 2MB) can typically improve the performance of forward iteration on spinning disks.
Defaults to `0`.
- `tailing: boolean` When `true`, creates a "tailing iterator" which is a special iterator that
has a view of the complete database including newly added data and is optimized for sequential
reads. This will return records that were inserted into the database after the creation of the
iterator. Defaults to `false`.
### `RangeOptions`
Extends `RocksDBOptions`.
- `options: object`
- `end: Key | Uint8Array` The range end key, otherwise known as the "upper bound". Defaults to the
last key in the database.
- `exclusiveStart: boolean` When `true`, the iterator will exclude the first key if it matches the
start key. Defaults to `false`.
- `inclusiveEnd: boolean` When `true`, the iterator will include the last key if it matches the
end key. Defaults to `false`.
- `start: Key | Uint8Array` The range start key, otherwise known as the "lower bound". Defaults to
the first key in the database.
### `IteratorOptions`
Extends `RangeOptions`.
- `options: object`
- `reverse: boolean` When `true`, the iterator will iterate in reverse order. Defaults to `false`.
## Development
This package requires Node.js 18 or higher, pnpm, and a C++ compiler.
> [!TIP]
> Enable pnpm log streaming to see full build output:
>
> ```
> pnpm config set stream true
> ```
### Building
There are two things being built: the native binding and the TypeScript code. Each of those can be
built to be debug friendly.
| Description | Command |
| -------------------------------------------- | ---------------------------------------- |
| Production build (minified + native binding) | `pnpm build` |
| TypeScript only (minified) | `pnpm build:bundle` |
| TypeScript only (unminified) | `pnpm build:debug` |
| Native binding only (prod) | `pnpm rebuild` |
| Native binding only (with debug logging) | `pnpm rebuild:debug` |
| Debug build everything | `pnpm build:debug && pnpm rebuild:debug` |
When building the native binding, it will download the appropriate prebuilt RocksDB library for your
platform and architecture from the
[rocksdb-prebuilds](https://github.com/HarperFast/rocksdb-prebuilds) GitHub repository. It defaults
to the pinned version in the `package.json` file. You can override this by setting the
`ROCKSDB_VERSION` environment variable. For example:
```bash
ROCKSDB_VERSION=10.9.1 pnpm build
```
You may also specify `latest` to use the latest prebuilt version.
```bash
ROCKSDB_VERSION=latest pnpm build
```
Optionally, you may also create a `.env` file in the root of the project to specify various
settings. For example:
```bash
echo "ROCKSDB_VERSION=10.9.1" >> .env
```
### Linux C runtime versions
When you compile `rocksdb-js`, you can specify the `ROCKSDB_LIBC` environment variable to choose
either `glibc` (default) or `musl`.
```bash
ROCKSDB_LIBC=musl pnpm rebuild
```
### Windows C runtime versions
By default on Windows, `rocksdb-js` is compiled with the `/MT` flag. This will statically link the
C runtime making the binary self-contained and portable.
### Building RocksDB from Source
To build RocksDB from source, simply set the `ROCKSDB_PATH` environment variable to the path of the
local `rocksdb` repo:
```bash
git clone https://github.com/facebook/rocksdb.git /path/to/rocksdb
echo "ROCKSDB_PATH=/path/to/rocksdb" >> .env
pnpm rebuild
```
### Debugging
It is often helpful to do a debug build and see the internal debug logging of the native binding.
You can do a debug build by running:
```bash
pnpm rebuild:debug
```
Each debug log message is prefixed with the thread id. Most debug log messages include the instance
address making it easier to trace through the log output.
#### Debugging on macOS
In the event Node.js crashes, re-run Node.js in `lldb`:
```bash
lldb node
# Then in lldb:
# (lldb) run your-program.js
# When the crash occurs, print the stack trace:
# (lldb) bt
```
### Testing
To run the tests, run:
```bash
pnpm coverage
```
To run the tests without code coverage, run:
```bash
pnpm test
```
To run a specific test suite, for example `"ranges"`, run:
```bash
pnpm test ranges
# or
pnpm test test/ranges
```
To run a specific unit test, for example all tests that mention `"column family"`, run:
```bash
pnpm test -t "column family"
```
Vitest's terminal renderer will often overwrite the debug log output, so it's highly recommended to
specify the `CI=1` environment variable to prevent Vitest from erasing log output:
```bash
CI=1 pnpm test
```
By default, the test runner deletes all test databases after the tests finish. To keep the temp
databases for closer inspection, set the `KEEP_FILES=1` environment variable:
```bash
CI=1 KEEP_FILES=1 pnpm test
```