Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with deduplicate

A curated list of projects in awesome lists tagged with deduplicate .

https://github.com/knjcode/imgdupes

Identifying and removing near-duplicate images using perceptual hashing.

dedupe deduplicate image perceptual-hashes perceptual-hashing

Last synced: 18 Jan 2026

https://github.com/tc39/proposal-array-unique

ECMAScript proposal for Deduplicating method of Array

array deduplicate ecmascript javascript polyfill proposal unique

Last synced: 19 Aug 2025

https://github.com/sysulq/dataloader-go

Go implementation of Facebook's DataLoader with 200+ lines of code.

batch cache dataloader deduplicate facebook go golang lru opentelemetry

Last synced: 21 Aug 2025

https://github.com/routineLife1/MultiPassDedup

Efficient Deduplicate for Anime Video Frame Interpolation

animation anime deduplicate gmfss rife slomo smooth vfi video-frame-interpolation

Last synced: 22 Jul 2025

https://github.com/wsh032/sd-webui-fast-dataset-maker

A funny extension that integrates image-browsing , downloader , deduplicate , cluster , can quickly collect, classify and process your images. | 一个有趣的扩展，整合了 图库，下载，去重，聚类 ，可以快速搜集、分类、处理你的图片。

cluster dataset-maker deduplicate gelbooru image-browsing image-dataset image-downloader stable-duffusion webui

Last synced: 07 Apr 2025

https://github.com/rsalmei/refine

Refine your file collections using Rust!

batch deduplicate duplicates files rename rust scan

Last synced: 17 Jul 2025

https://github.com/routinelife1/multipassdedup

Efficient Deduplicate for Anime Video Frame Interpolation

animation anime deduplicate gmfss rife slomo smooth vfi video-frame-interpolation

Last synced: 27 Sep 2025

https://github.com/svandriel/cachify-promise

Smart caching for promises. Like memoization, but better.

async cache deduplicate memoization promise

Last synced: 23 Apr 2025

https://github.com/thijse/dsphotosorter

A command line tool for sorting photo's from the Synology DSPhoto auto-upload tool

csharp deduplicate ds-photo duplicate-images photo-organizer synology

Last synced: 20 Mar 2025

https://github.com/yaroslaff/hashget

Deduplication/backup tool with extremely high 'compression' rate

archive backup compression deduplicate deduplication restic

Last synced: 13 Apr 2025

https://github.com/infogulch/uniq

Package uniq provides primitives for getting the first unique elements of (aka deduplicate) your existing sorted sort.Interface.

deduplicate distinct go golang uniq

Last synced: 25 Jun 2025

https://github.com/gblach/reflicate

Deduplicate data by creating reflinks between identical files.

btrfs deduplicate deduplication ocfs2 reflinks rust xfs

Last synced: 26 Mar 2025

https://github.com/mterron/swuniq

A command-line tool for deduplicating entries in a file or stream with constant memory usage

cli dedupe deduping deduplicate deduplication filter sliding-window uniq

Last synced: 22 Feb 2026

https://github.com/ojarva/maildir-deduplicate

Deduplicates maildir contents using hard links.

compress deduplicate email maildir

Last synced: 15 Jun 2025

https://github.com/stdlib-js/iter-unique-by

Create an iterator which returns unique values according to a predicate function.

dedupe deduplicate distinct iterable iterate iterator javascript node node-js nodejs predicate stdlib uniq unique util utilities utility utils

Last synced: 17 Apr 2026

https://github.com/stdlib-js/iter-unique-by-hash

Create an iterator which returns unique values according to a hash function.

dedupe deduplicate distinct hash iterable iterate iterator javascript node node-js nodejs stdlib uniq unique util utilities utility utils

Last synced: 26 Feb 2026

https://github.com/stdlib-js/array-base-to-deduped

Copy elements to a new generic array after removing consecutive duplicated values.

array compress copy data dedupe deduplicate deduplication duplicate generic javascript node node-js nodejs stdlib structure types uniq unique

Last synced: 14 Jun 2025

https://github.com/gkjohnson/webpack-script-guard

Webpack loader for guarding against duplicate scripts in separate bundles

deduplicate deduplication html import imports javascript loader polymer webcomponents webpack

Last synced: 16 Apr 2026

https://github.com/stdlib-js/iter-dedupe-by

Create an iterator which removes consecutive values that resolve to the same value according to a provided function.

compress dedupe deduplicate deduplication duplicate iterable iterate iteration iterator javascript node node-js nodejs stdlib uniq unique util utilities utility utils

Last synced: 16 Aug 2025

https://github.com/hekmon/deduper

Analyse 2 paths to found identical files and hard link them to save space

dedup deduplicate deduplication deduplicator duplicate-files filesystem hardlink hardlinking hardlinks linux saving scan scanner space tool utility

Last synced: 28 Feb 2025

https://github.com/localghost/dedup

Find and remove duplicated files.

deduplicate files

Last synced: 24 Jul 2025

https://github.com/franchoy/coldkeep

coldkeep is an experimental local-first content-addressed file storage engine with verifiable integrity written in Go. Files are split into content-addressed chunks, packed into container files on disk, and tracked through PostgreSQL metadata to attempt deduplication

backup cold-storage content-defined-chunking deduplicate go research-project storage-engines

Last synced: 09 Apr 2026

https://github.com/oleg-putseiko/function-performer

Performer providing API for debounce, throttle and deduplication functions

debounce deduplicate function performer schedule throttle

Last synced: 22 Jan 2026

https://github.com/grjan7/deduplicate-array

Returns duplicates-removed array -- has options for case-sensitivity and strict-typing.

array deduplicate javascript json nodejs npm npm-package objects unique

Last synced: 11 Apr 2026