https://github.com/blackglory/extra-deduplicator
🌿 A data deduplication library
https://github.com/blackglory/extra-deduplicator
deno jsr-package library typescript
Last synced: about 1 month ago
JSON representation
🌿 A data deduplication library
- Host: GitHub
- URL: https://github.com/blackglory/extra-deduplicator
- Owner: BlackGlory
- License: mit
- Created: 2024-01-20T02:21:20.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2026-02-11T06:02:52.000Z (4 months ago)
- Last Synced: 2026-02-11T13:38:58.820Z (4 months ago)
- Topics: deno, jsr-package, library, typescript
- Language: TypeScript
- Homepage: https://jsr.io/@blackglory/deduplicator
- Size: 193 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# extra-deduplicator
A data deduplication library for Deno,
which is designed for extracting new data.
## Install
```sh
deno add jsr:@blackglory/deduplicator
```
## Usage
```ts
#!/usr/bin/env -S deno run --allow-all
import { Deduplicator } from 'jsr:@blackglory/deduplicator'
const deduplicator = await Deduplicator.create({ filename: 'dedup.db' })
const records = ['record-1', 'record-2']
const newRecords = await deduplicator.addAndDidff(records)
```
## API
```ts
interface IOptions {
filename?: string
hash?: (value: T) => JSONValue
shrink?: {
target?: number
threshold?: number
}
}
class Deduplicator {
static create(options?: IOptions): Promise>
add(values: T[]): Promise
diff(values: T[]): Promise
diffLast(value: T): Promise
addAndDiff(values: T[]): Promise
addAndDiffLast(value: T): Promise
close(): void
}
```