Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/F483/dejavu
Quickly detect already witnessed data.
https://github.com/F483/dejavu
command-line command-line-tool deduplication duplicate-values duplicates go golang history memory probabilistic
Last synced: about 2 months ago
JSON representation
Quickly detect already witnessed data.
- Host: GitHub
- URL: https://github.com/F483/dejavu
- Owner: F483
- License: mit
- Created: 2017-05-29T18:36:42.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2024-07-11T14:59:46.000Z (2 months ago)
- Last Synced: 2024-07-12T13:07:21.885Z (2 months ago)
- Topics: command-line, command-line-tool, deduplication, duplicate-values, duplicates, go, golang, history, memory, probabilistic
- Language: Go
- Homepage: https://f483.github.io
- Size: 305 KB
- Stars: 157
- Watchers: 7
- Forks: 7
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/f483/dejavu/actions/workflows/go.yml)
[](https://github.com/f483/dejavu/issues)
[](https://goreportcard.com/report/github.com/f483/dejavu)
[](https://raw.githubusercontent.com/f483/dejavu/master/LICENSE)
[](https://godoc.org/github.com/f483/dejavu)
# Déjà vu
Quickly detect already witnessed data, ideal for deduplication.
Limited memory of witnessed data, oldest are forgotten. Library is thread safe.
Offers deterministic and probabilistic (over an order of magnitude less memory
consuming) implementation. The probabilistic implementation uses bloom filters,
meaning false positives are possible but not false negatives.
# Installation
## Download binary release
Compiled binaries for many platforms are available and can be downloaded for
the [latest release](https://github.com/F483/dejavu/releases/latest).
Extract the binary for your platform and add it to your system path.
## Compile from source
Requires golang [environment/workspace](https://golang.org/doc/code.html).
```
# compile and install library
go get github.com/f483/dejavu
# compile and install binary
go install github.com/f483/dejavu/dejavu
```
# Command line usage
```
$ dejavu -h
Usage: dejavu [OPTION]... [FILE]...
Concatenate FILE(s) and filter or output duplicate lines.
With no FILE, or when FILE is -, read standard input.
Options:
-D use deterministic mode instead of probabilistic
WARNING requires order of magnitude more memory
-d output only duplicates instead of filtering
-f float
chance of false positive, between 0.0 and 1.0
only for probabilistic mode (default 1e-06)
-l uint
limit after which entries are forgotton (default 1000000)
-o string
output file, defaults to stdout
-v output version information and exit
Examples:
dejavu
default probabilistic deduplication from stdin to std out with
1mil entry limit and 1/1mil chance of false positive (~8M mem usage)
dejavu -o s f - g
deduplicat f, then stdin, then g, to output s
dejavu -l 10000000 -fp 0.000000001
probabilistic deduplication with 10mil entry limit
and 1/1bil chance of false positive (~70M mem usage)
dejavu -d -D -l 65536
output duplicates and avoid false positives with deterministic mode
lower entry limit to avoid excessive memory usage
Implementation:
Efficient probabilistic and deterministic duplicate detection with O(1)
detection time and O(n) memory usage in relation to entry limit. Default
probabilistic implementation uses bloom filters, meaning false
positives are possible but not false negatives.
Author: Fabian Barkhau
Project: https://github.com/f483/dejavu
License: MIT https://raw.githubusercontent.com/f483/dejavu/master/LICENSE
```
# Library usage (golang)
## Probabilistic example
```
package main
import (
"fmt"
"github.com/f483/dejavu"
)
func main() {
// probably remembers last 65536 with 0.000001 chance of false positive
p := dejavu.NewProbabilistic(65536, 0.000001)
fmt.Println(p.Witness([]byte("bar"))) // entry added
fmt.Println(p.Witness([]byte("bar"))) // probably remembers entry
}
```
## Deterministic example
```
package main
import (
"fmt"
"github.com/f483/dejavu"
)
func main() {
// always remembers last 1024 entries
d := dejavu.NewDeterministic(1024)
fmt.Println(d.Witness([]byte("foo"))) // entry added
fmt.Println(d.Witness([]byte("foo"))) // remembers entry
}
```
# Performance
## Linear memory usage: O(n)
### Probabilistic
0.000001 chance of false positive.
### Deterministic
## Constant witness time: O(1)
### Probabilistic
0.000001 chance of false positive.
### Deterministic
