Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/firstbatchxyz/hollowdb-vector
A decentralized vector database for building vector search applications
https://github.com/firstbatchxyz/hollowdb-vector
ai ann arweave blockchain decentralized embedding embeddings nearest-neighbor-search vector-database
Last synced: 8 days ago
JSON representation
A decentralized vector database for building vector search applications
- Host: GitHub
- URL: https://github.com/firstbatchxyz/hollowdb-vector
- Owner: firstbatchxyz
- License: apache-2.0
- Created: 2023-04-06T16:07:45.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2024-02-16T09:05:16.000Z (9 months ago)
- Last Synced: 2024-05-21T06:05:06.854Z (6 months ago)
- Topics: ai, ann, arweave, blockchain, decentralized, embedding, embeddings, nearest-neighbor-search, vector-database
- Language: TypeScript
- Homepage: https://firstbatch.xyz
- Size: 872 KB
- Stars: 88
- Watchers: 5
- Forks: 6
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-distributed-system-projects - DANNY - a decentralized vector database for building vector search applications
README
HollowDB Vector
Implementation of Hierarchical Navigable Small Worlds (HNSW) index over HollowDB.
## Installation
Install the package:
```sh
yarn add hollowdb-vector
pnpm add hollowdb-vector
npm install hollowdb-vector
```
## Usage
You can create the VectorDB as follows:
```ts
import HollowDBVector from "hollowdb-vector";
import { WarpFactory, defaultCacheOptions } from "warp-contracts";
import { SetSDK } from "hollowdb";
import { Redis } from "ioredis";
import { RedisCache } from "warp-contracts-redis";
// connect to Redis
const redis = new Redis();
// create Warp instance with Redis cache
const warp = WarpFactory.forMainnet().useKVStorageFactory(
(contractTxId: string) =>
new RedisCache({ ...defaultCacheOptions, dbLocation: `${contractTxId}` }, { client: redis }),
);
// create HollowDB SDK
const wallet = JSON.parse(readFileSync("./path/to/wallet.json", "utf-8"));
const contractTxId = "your-contract-tx-id";
const hollowdb = new SetSDK(wallet, contractTxId, warp);
// create HollowDB Vector
const vectordb = new HollowDBVector(hollowdb);
```
### Inserting a Vector
With this, you can insert a new point:
```ts
const point = [-0.28571999073028564 /* and many more... */, 0.13964000344276428];
// any object
const metadata = {
name: "My favorite vector!",
};
// insert a point
await vectordb.insert(point, metadata);
```
Metadata is optional, and you can leave it out during `insert`. If you would like to set it a later time, you can always do:
```ts
vectordb.db.set_metadata(index, metadata);
```
> [!NOTE]
>
> The complexity of inserting a point may increase with more points in the DB.
### Fetching a Vector
You can get a vector by its index, which returns its point value and metadata:
```ts
const { point, metadata } = await vectordb.get_vector(index);
```
### Querying a Vector
You can make a query and return top K relevant results:
```ts
// a query point
const query = [-0.28571999073028564 /* and many more... */, 0.13964000344276428];
// number of top results to return
const K = 10;
// make a KNN search
const results = await vectordb.knn_search(query, K);
// each result contains the vector id, its distance to query, and metadata
const { id, distance, metadata } = results[0];
```
### Deploying your own Contract
HollowDB Vector exports a static function that allows you two deploy a new contract that you own. Assuming that you have a `wallet` and a `warp` instance as described [above](#usage), you can create a new contract with:
```ts
const { contractTxId } = await HollowDBVector.deploy(wallet, warp);
console.log("Deployed at:", contractTxId);
```
## Setup
For local setup of this repo, first clone it.
```sh
git clone https://github.com/firstbatchxyz/hollowdb-vector
```
Then, install packages:
```sh
pnpm install
```
Peer dependencies should be installed automatically.
### Protobuffers
We include the pre-compiled protobuf within the repo, but if you were to change the protobuf later, you can generate the compiled code as follows:
```sh
# HNSW protobufs
pnpm proto:code:hnsw # generate js code
pnpm proto:type:hnsw # generate types
# Request protobufs
pnpm proto:code:req # generate js code
pnpm proto:type:req # generate types
```
## Testing
Tests are ran over a few cases for a fixed set of $N, K$ parameters that are prepared in [Python](./test/python/main.py), and are compared in Typescript. Run the tests via:
```sh
pnpm test
```
> [!WARNING]
>
> You need a live Redis server for the HollowDB test to work. Furthermore, the HollowDB tests may take some time.
## Styling
Check the formatting with:
```sh
pnpm format
```
Lint everything with:
```sh
pnpm lint
```
You can also check types with:
```sh
pnpm check
```
## Legacy
HollowDB Vector replaces DANNY, for the legacy code please [refer to this branch](https://github.com/firstbatchxyz/danny/tree/legacy).
## License
HollowDB Vector is licensed under [Apache 2.0](./LICENSE).