Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with duplicates

A curated list of projects in awesome lists tagged with duplicates .

https://github.com/qarmin/czkawka

Multi functional app to find duplicates, empty folders, similar images etc.

cleaner duplicates gtk-rs multiplatform rust similar-images similar-music similar-videos

Last synced: 14 Feb 2026

https://github.com/sahib/rmlint

Extremely fast tool to remove duplicates and other lint from your filesystem

c deduplication duplicates fdupes filesystem lint python

Last synced: 14 May 2025

https://github.com/scinos/yarn-deduplicate

Deduplication tool for yarn.lock files

dedupe duplicated-packages duplicates lock-file yarn yarn-lock

Last synced: 11 May 2025

https://github.com/sean-public/python-hashes

Interesting (non-cryptographic) hashes implemented in pure Python.

bloom-filter duplicates geohashes hash hash-functions hashes-implemented nilsimsa python python3 simhash

Last synced: 17 Jan 2026

https://github.com/kristiankoskimaki/vidupe

Vidupe is a program that can find duplicate and similar video files. V1.211 released on 2019-09-18, Windows exe here:

duplicate-detection duplicate-videos duplicates videos

Last synced: 10 Apr 2025

https://github.com/kouhin/redux-dataloader

Loads async data for Redux apps focusing on preventing duplicated requests and dealing with async dependencies.

action async duplicates react redux thunk

Last synced: 21 Aug 2025

https://github.com/canop/backdown

A deduplicator

duplicates rust

Last synced: 06 Apr 2025

https://github.com/Canop/backdown

A deduplicator

duplicates rust

Last synced: 19 Jul 2025

https://github.com/scrubbbbs/cbird

Command-line program for Content-Based Image Retrieval of images and videos. Includes tools for general search and de-duplication.

command-line-interface computer-vision content-based-image-retrieval duplicate-detection duplicate-files duplicates ffmpeg opencv qt6 similarity-search

Last synced: 06 Apr 2025

https://github.com/matteodelabre/mongoose-beautiful-unique-validation

Plugin for Mongoose that turns duplicate errors into regular Mongoose validation errors

duplicates errors mongoose plugin schema validation

Last synced: 06 Apr 2025

https://github.com/jvirkki/dupd

CLI utility to find duplicate files

c deduplication duplicate-files duplicatefilefinder duplicates fdupes

Last synced: 21 Mar 2025

https://github.com/stephanecouturier/katalog

Katalog is an application to manage catalogs of disks and files to search and get statistics.

backup cataloging catalogue differences duplicates external-content external-drives file file-manager file-statistics filemanager filesystem indexer katalog search storage

Last synced: 07 Feb 2026

https://github.com/StephaneCouturier/Katalog

Katalog is an application to manage catalogs of disks and files to search and get statistics.

backup cataloging catalogue differences duplicates external-content external-drives file file-manager file-statistics filemanager filesystem indexer katalog search storage

Last synced: 29 Apr 2025

https://github.com/PJDude/dude

Duplicates Detector is a cross-platform GUI utility for finding duplicate files, allowing you to delete or link them to save space. Duplicate files are displayed and processed on two synchronized panels for efficient and convenient operation.

cli deduplication duplicate duplicate-detection duplicate-files duplicates duplicates-removal easy easy-to-use easyui gui gui-application python python3 sha1 threads tkinter utility utility-application

Last synced: 06 Mar 2025

https://github.com/src-d/gemini

Advanced similarity and duplicate source code at scale.

duplicate-detection duplicates hash source-code-analysis spark

Last synced: 05 May 2025

https://github.com/deric/es-dedupe

Tool for removing duplicate documents from Elasticsearch

duplicates duplicity elasticsearch

Last synced: 23 Apr 2025

https://github.com/src-d/apollo

Advanced similarity and duplicate source code proof of concept for our research efforts.

duplicate-detection duplicates python similarity similarity-search source-code

Last synced: 05 May 2025

https://github.com/lyonsyonii/akin

Rust crate for writing repetitive code easier and faster.

code duplicate duplicates repeat repetition rust rust-crate similar similars simpler write

Last synced: 08 Aug 2025

https://github.com/bartozzz/potential-duplicates-bot

A configurable GitHub App which checks for potential issue duplicates using Damerau–Levenshtein distance algorithm.

bot duplicates github issues probot probot-app probot-plugin

Last synced: 23 Oct 2025

https://github.com/navid2zp/dups

A CLI tool to find/remove duplicate files supporting multi-core and different algorithms (MD5, SHA256, and XXHash).

duplicates go golang md5 sha256 xxhash

Last synced: 15 May 2025

https://github.com/gacarrillor/appendfeaturestolayer

QGIS Processing plugin to add an algorithm for upserting features from a source vector layer to an existing target vector layer.

append copy copy-paste duplicates etl export load load-data qgis qgis-processing qgis-processing-provider qgis3 qgis3-plugin update upsert

Last synced: 22 Mar 2025

https://github.com/Navid2zp/dups

A CLI tool to find/remove duplicate files supporting multi-core and different algorithms (MD5, SHA256, and XXHash).

duplicates go golang md5 sha256 xxhash

Last synced: 27 Feb 2025

https://github.com/arikw/outlook-duplicated-items-remover

A VBA script that finds and moves duplicated items in selected outlook folders

duplicate-detection duplicates outlook vba vba-script vba-snippets

Last synced: 12 Feb 2026

https://github.com/rasheedsulayman/duplicatecontactsremover

📒 A simple app to optimize your address book and remove duplicate contacts.

androidx architecture-components contacts-manager dependency-injection duplicates kotlin material-design mvvm

Last synced: 14 Oct 2025

https://github.com/lmammino/indexed-string-variation

Experimental JavaScript module to generate all possible variations of strings over an alphabet using an n-ary virtual tree

algorithm alphabet characters duplicates javascript javascript-library library node nodejs string strings tree variations virtual

Last synced: 06 May 2025

https://github.com/kevinpollet/pocket-deduper

Remove duplicates from your Pocket list.

cli dedupe duplicates go golang pocket tool

Last synced: 11 Apr 2025

https://github.com/raspi/duplikaatti

Remove duplicate files.

duplicate-files duplicates files go golang

Last synced: 19 Oct 2025

https://github.com/rsalmei/refine

Refine your file collections using Rust!

batch deduplicate duplicates files rename rust scan

Last synced: 17 Jul 2025

https://github.com/rmdm/nodups

No dups, no doubts

duplicates nodups uniq unique

Last synced: 12 Apr 2025

https://github.com/deadsoul/dugu

Find, remove and avoid duplicates with dugu: The Duplicates Guru

deduplication dugu duplicate-detection duplicate-files duplicatefilefinder duplicates duplicates-guru python

Last synced: 05 Apr 2025

https://github.com/nicolasbizzozzero/dupe_eraser

A command-line tool which automate the deletion of duplicate files based on their hash or perceptual-hash.

cli duplicate-detection duplicate-files duplicates file-management

Last synced: 12 Apr 2025

https://github.com/mesqueeb/compare-anything

Compares objects and tells you which props are duplicate, and props are only present once.

compare compare-arrays compare-objects count-if countif duplicates find-duplicates javascript remove-duplicates typescript

Last synced: 10 Aug 2025

https://github.com/rix4uni/unew

A tool combined of 2 commands features in 1 sort and tee for adding new lines to files, skipping duplicates

bug-bounty bugbounty bugbountytips duplicates hacking infosec osint osint-resources osint-tool penetration-testing pentest-tool pentesting recon reconnaissance security security-tools threat-intelligence

Last synced: 15 Apr 2025

https://github.com/redonkulus/dump-deps

Dump NPM package dependencies to display packages with multiple versions.

deps dump-deps duplicates npm

Last synced: 28 Oct 2025

https://github.com/innovatrics/dedubcheck

dedubcheck - De-Duplicate Dependency Checker for Node.js monorepos

deduplication duplicates duplicity javascript nodejs nodejs-modules

Last synced: 13 Apr 2025

https://github.com/ant-js/compare-similarity

👁 Compare the similarity of two strings

compare duplicates similarity utils

Last synced: 15 May 2025

https://github.com/keyweeusr/bear

:bear: The decluttering deduplicator

cli-app clutterremoval duplicate-detection duplicates python

Last synced: 25 Jul 2025

https://github.com/ajmalshahabudeen/Bitwarden-Duplicate-remover

When Importing multiple CSV files Bitwarden creates Duplicate Entries. So this Python script will remove duplicate entries and keep ONE.

bitwarden bitwarden-password-vault duplicate-detection duplicates duplicates-removal python

Last synced: 27 Mar 2025

https://github.com/codecliff/fdupesanalyzer

A script to analyze output of fdupes linux utility to find level of overlap between directories. Written in R

bash bash-script directory duplicates fdupes fdupes-linux-utility files r rstudio

Last synced: 20 Apr 2026

https://github.com/r-darwish/dupfiles-cpp

Find duplicate files

boost cpp cpp11 duplicates

Last synced: 25 Mar 2025

https://github.com/ajmalshahabudeen/bitwarden-duplicate-remover

When Importing multiple CSV files Bitwarden creates Duplicate Entries. So this Python script will remove duplicate entries and keep ONE.

bitwarden bitwarden-password-vault duplicate-detection duplicates duplicates-removal python

Last synced: 04 Mar 2026

https://github.com/softonus-io/prettier-plugin-duplicate-remover

A Prettier plugin that removes duplicate class names in class and className attributes, ensuring cleaner, more efficient code in frontend projects like React, Vue.js, and Angular.

duplicate duplicate-detection duplicate-removal duplicates duplicates-removal prettier prettier-config prettier-eslint prettier-plugin prettier-plugin-tailwindcss prettierconfig prettierrc

Last synced: 25 Jun 2026

https://github.com/mckael/goduf

A simple (but fast) duplicate file finder written in Go [Mirror repository]

cli duplicate-files duplicates go golang

Last synced: 03 Apr 2025

https://github.com/jordicorbilla/duplicatechecker

☑️ .Net service that allows you to check duplicate rows on a sql table using Levenshtein distance

duplicates levenshtein-distance

Last synced: 13 Apr 2025

https://github.com/davidefiocco/dockerized-elasticsearch-duplicate-finder

Attempt to use MinHash to find duplicates in an Elasticsearch index

duplicates elasticsearch minhash python

Last synced: 19 May 2026

https://github.com/pouyakary/dup

a tiny and fast command line utility to find the duplicate files within a directory

cli cmd duplicate-detection duplicate-files duplicates filesystem gnu-utilities utility

Last synced: 14 May 2025

https://github.com/nbari/backup

Command line tool for creating encrypted backups avoiding duplicates

backup backup-utility duplicates encryption-decryption restore

Last synced: 06 Oct 2025

https://github.com/jackra1n/czkawka-web

A lightning-fast, minimalist web frontend for the czkawka duplicate file finder

cleaner duplicates multiplatform rust similar-images similar-music similar-videos svelte sveltekit

Last synced: 14 Jun 2026

https://github.com/plandes/clj-filedup

Find Duplicate Files by Content

clojure duplicates filesystem

Last synced: 16 Jun 2026

https://github.com/theohbrothers/get-duplicatephotos

A script to locate duplicate image files between two sets of folders (e.g. Camera Roll folders vs other folders).

criteria date-taken duplicate-files duplicate-photos duplicates export file-metadata powershell pwsh script search

Last synced: 20 Mar 2025

https://github.com/derhuerst/key-map

Keep track of old keys when removing duplicates.

duplicates object rename

Last synced: 05 Mar 2026

https://github.com/cfurrow/fzf-rdfind

Use rdfind + fzf to find and remove duplicate files

duplicates fzf fzf-scripts rdfind

Last synced: 19 Feb 2026

https://github.com/miroslav-reiter/microsoft_outlook_vba

📧 Makrá a skripty pre Microsoft Outlook VBA - Automatizácia úloh, opravy a úpravy Microsoft Outlook

duplicates emails macro microsoft office outlook vba

Last synced: 19 Mar 2026

https://github.com/dr8co/doppel

A simple, modern duplicate file detector.

deduplication duplicate-detection duplicate-files duplicatefilefinder duplicates

Last synced: 18 Jan 2026

https://github.com/exitare/duplicateimagefinder

A tool to find duplicate images for given paths

duplicate-detection duplicates filemanagement images python python3

Last synced: 13 Aug 2025

https://github.com/busterc/similars

:dancers: Find similar objects and partial duplicates in collections

arrays collections duplicate-detection duplicates similar-objects similarity-search

Last synced: 16 Jun 2025

https://github.com/cepr0/duplicate-parent-entities

Spring Data JPA - duplicated parent entities in 'join fetch' repository query methods

duplicates spring-boot spring-data-jpa

Last synced: 19 May 2026

https://github.com/davdiv/hashfolder

Simple command line tool that can create/update an sqlite database that contains the hash (by default SHA256) of all files inside a specified root folder.

checksum duplicate-detection duplicate-files duplicates sha256

Last synced: 16 May 2025

https://github.com/americanhanko/columnmapper

A simple VLOOKUP-like function that can handle duplicates.

csv duplicate-detection duplicates vlookup

Last synced: 05 Sep 2025

https://github.com/intera/redmine_subject_autocomplete

makes the new issue subject field show an autocomplete that lists existing issues to prevent duplicate tickets

didyoumean duplicates redmine redmine-plugin

Last synced: 06 Feb 2026

https://github.com/clmnin/dupper

Find duplicate files in a directory, in Rust

duplicates filesystem lint rust

Last synced: 16 Mar 2025

https://github.com/ndsvw/clone-the-tab

A Firefox Add-on that clones the active tab with just 1 click on the icon or with "Ctrl+Alt+D" / "Cmd+Alt+D".

clone duplicates firefox firefox-addon tab

Last synced: 05 Jul 2025

https://github.com/hansalemaos/screwduplicates

provides a simple and efficient way to remove duplicates from an iterable (even with unhashable elements, optional order preservation)

duplicates hashable list remove tuple unhashable

Last synced: 14 Jan 2026

https://github.com/hansalemaos/duplicateindexer

Find duplicates in multiple lists and return their indices and values.

duplicates fast find hash index list list-of-lists locate nested python set

Last synced: 09 Apr 2026

https://github.com/writetome51/get-indexes-of-item-duplicates

Function returns indexes of duplicates of one item in array

array duplicates element index indexes javascript typescript

Last synced: 13 May 2026

https://github.com/theohbrothers/get-duplicatecontact

A script to locate duplicate or non-duplicate contacts between two `.csv` contact lists.

compare-object contact-lists contacts csv duplicates hashtable json keys nonduplicates object objects powershell pwsh script xml

Last synced: 21 May 2026

https://github.com/prajjwol09/data-cleaning-project

This project is dedicated to cleaning, standardizing a dataset, dealing with null values from a CSV file named "layoffs" using MySQL, with MySQL Workbench as the workspace environment. The goal is to prepare the data for analysis.

cleaning-data columns data-analysis database duplicates mysql rows standard

Last synced: 20 Apr 2026

https://github.com/flowm/mdupes

Find and delete duplicate media files based on filename

duplicate-detection duplicates jellyfin media plex

Last synced: 15 Jun 2026

https://github.com/dav-m85/hashsnap

Remove files that have duplicates elsewhere.

duplicates hash snapshot

Last synced: 11 Oct 2025

https://github.com/jmsv/listset

remove duplicates from lists

duplicates list listset set

Last synced: 14 Oct 2025

https://github.com/clement-berard/go-imap-backup

A collection of Go tools for managing IMAP emails, featuring backup capabilities and duplicate detection/cleanup.

backup duplicates golang imap

Last synced: 14 Sep 2025

https://github.com/hansalemaos/stridesduplicatefinder

Calculate overlapping values between two arrays and return the results as a DataFrame

duplicates fast numexpr numpy strides

Last synced: 07 Mar 2026

https://github.com/hansalemaos/drop_duplicates_nested_list

Drops duplicates from nested list

drop duplicates list python

Last synced: 01 Jun 2026

https://github.com/bkb3/duplicate-bib-fix

Small python script to check and replace duplicated bib entries in your .tex files

bib biber bibliography bibtex duplicates entries latex tex

Last synced: 11 Mar 2025

https://github.com/writetome51/array-remove-duplicates

Function that removes any duplicate items in the array

array array-manipulations duplicates javascript remove remove-duplicates

Last synced: 18 May 2026

https://github.com/pklatka/photo-organizer

Portable and lightweight application for photo segregation.

duplicates lightweight organizer photo-organizer photos portable segregation

Last synced: 29 Mar 2025

https://github.com/paralevel/bookmark-checker

Shortcut to find or delete bookmarks for the current Safari web page on iOS and macOS

bookmarks duplicates ios macos safari shortcut shortcuts shortcuts-app shortcuts-automation

Last synced: 11 Jun 2026

https://github.com/kandekore/advanced-duplicate-post-manager-main

A powerful tool for WordPress administrators to detect and manage duplicate content across posts, pages, categories, media, and custom post types. Assign 301 redirects, clean up media, and manage .htaccess rules from a user-friendly interface.

301 301-redirect 301-redirects admon csv duplicates htaccess media redirect seo slug wordpress

Last synced: 16 May 2026

https://github.com/fiddyschmitt/udp_dedupe

Deduplicate UDP datagrams

deduplication duplicates udp windivert

Last synced: 04 Apr 2026