{"id":13525931,"url":"https://github.com/mk-fg/image-deduplication-tool","last_synced_at":"2025-04-23T23:31:33.675Z","repository":{"id":2445261,"uuid":"3415765","full_name":"mk-fg/image-deduplication-tool","owner":"mk-fg","description":"Tool to detect (and get rid of) similar images using perceptual hashing (pHash lib)","archived":false,"fork":false,"pushed_at":"2016-11-06T21:27:19.000Z","size":14,"stargazers_count":82,"open_issues_count":0,"forks_count":8,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-02T22:23:10.355Z","etag":null,"topics":["ctypes","image-comparison","image-deduplication","perceptual-hashing","phash","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"wtfpl","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mk-fg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-02-11T15:03:41.000Z","updated_at":"2025-02-12T10:09:16.000Z","dependencies_parsed_at":"2022-09-09T23:10:28.708Z","dependency_job_id":null,"html_url":"https://github.com/mk-fg/image-deduplication-tool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fimage-deduplication-tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fimage-deduplication-tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fimage-deduplication-tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fimage-deduplication-tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mk-fg","download_url":"https://codeload.github.com/mk-fg/image-deduplication-tool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250532041,"owners_count":21446105,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ctypes","image-comparison","image-deduplication","perceptual-hashing","phash","python"],"created_at":"2024-08-01T06:01:23.632Z","updated_at":"2025-04-23T23:31:33.404Z","avatar_url":"https://github.com/mk-fg.png","language":"Python","funding_links":[],"categories":["Image Deduplication"],"sub_categories":[],"readme":"image-deduplication-tool: simple tool to detect (and get rid of) similar images using perceptual hashing\n--------------------\n\nThere's gonna be a lot of duplicates in almost any arbitrary collection of\nimages, and it can actually be surprising how many.\n\npHash lib, which is the core of the tool, easily detects cropped and retouched\nimages, or same thing in different resolutions and formats.\n\nTool goes over the specified paths, calculating the hashes of all the images\nthere, pickling them into a db file between runs (to save lots of time on\nre-calculating all of them).\nThen it just compares the hashes, showing closest results first.\n\npHash lib seem to be able to utilize multiple cpu cores for the hashing when\nbuilt with with openmp flag, but it didn't seem to work for me, so put much\nsimplier solution in place to scale such task - just forking worker pid for each\nhardware thread.\n\nOptinally, tool can start handy [feh](http://derf.homelinux.org/projects/feh/)\nviewer, where human can make a decision to remove one image version or the other\n(with pre-configured \"rm\" action) for each duplicate pair, skip to the next pair\nor stop the comparison.\n\n\nWarning\n--------------------\n\nAs illustrated in\n[#1](https://github.com/mk-fg/image-deduplication-tool/issues/1) and\n[CImg#49](https://sourceforge.net/p/cimg/bugs/49/), libpHash/CImg will fall back\nto using potentially unsafe (exploitable with crafted pathnames) \"sh -c\"\ncommands for non-image file formats and might not get filename-escaping\ncorrectly there (especially with CImg versions up to 1.5.3).\n\nSimple safeguard for that particular issue would be only to run the tool on\nimage paths (where CImg doesn't run \"sh\"), not paths that contain mixed-type\nfiles, or at least make sure there's no funky stuff in the filenames, script\ndoesn't enforce any kind of policy there.\n\nNote also that thing libpHash/CImg runs (usually) is ImageMagick's \"convert\",\nwhich can have all sort of issues with malicious file contents (see e.g.\n[ImageTragick](imagetragick.com) bug there), so maybe it's not a good idea to\nrun the tool on a bunch of unsanitized images, ever.\n\nOne other precaution is that with the --feh option, script will run \"feh\"\nprogram, and --feh-args parameter may contain options (e.g. --info) that will be\nexecuted in the shell by feh, so either don't use --feh for weird and/or\npossibly-malicious (e.g. really weird) filenames or at least remove --info\noption from the --feh-args commandline.\n\n\nRequirements\n--------------------\n\n* [Python 2.7](http://python.org) (with ctypes support)\n* [libpHash](http://phash.org) (used via ctypes)\n* (optional) [feh image viewer](http://derf.homelinux.org/projects/feh/)\n\n\nUsage\n--------------------\n\nJust run as e.g. `./image_matcher.py --feh ~/media/images`.\n\n\t% ./image_matcher.py -h\n\n\tusage: image_matcher.py [-h] [--hash-db PATH] [-d [PATH]] [-p THREADS]\n\t                        [-n COUNT] [--feh] [--feh-args CMDLINE] [--debug]\n\t                        paths [paths ...]\n\n\tpositional arguments:\n\t  paths                 Paths to match images in (can be files or dirs).\n\n\toptional arguments:\n\t  -h, --help            show this help message and exit\n\t  --hash-db PATH        Path to db to store hashes in (default:\n\t                        ./image_matcher.db).\n\t  -d [PATH], --reported-db [PATH]\n\t                        Record already-displayed pairs in a specified file and\n\t                        dont show these again. Can be specified without\n\t                        parameter to use \"reported.db\" file in the current dir.\n\t  -p THREADS, --parallel THREADS\n\t                        How many hashing ops can be done in parallel (default:\n\t                        try cpu_count() or 1).\n\t  -n COUNT, --top-n COUNT\n\t                        Limit output to N most similar results (default: print\n\t                        all).\n\t  --feh                 Run feh for each image match with removal actions\n\t                        defined (see --feh-args).\n\t  --feh-args CMDLINE    Feh commandline parameters (space-separated, unless\n\t                        quoted with \") before two image paths (default: -GNFY\n\t                        --info \"echo '%f %wx%h (diff: {diff}, {diff_n} /\n\t                        {diff_count})'\" --action8 \"rm %f\" --action1 \"kill -INT\n\t                        {pid}\", only used with --feh, python-format keywords\n\t                        available: path1, path2, n, pid, diff, diff_n,\n\t                        diff_count)\n\t  --debug               Verbose operation mode.\n\nfeh can be customized to do any action or show any kind of info alongside images\nwith --feh-args parameter. It's also possible to make it show images\nside-by-side in montage mode or in separate windows in multiwindow mode, see\n\"man feh\" for details.\n\nDefault feh command line:\n\n\tfeh -GNFY --info \"echo '%f %wx%h (diff: {diff}, {diff_n} / {diff_count})'\" --action8 \"rm %f\" --action1 \"kill -INT {pid}\" {path1} {path2}\n\nmakes it show fullscreen image, some basic info (along with difference between\nimage hashes and how much images there are with the same level of difference)\nabout it and action reference, pressing \"8\" there will remove currently\ndisplayed version, \"1\" will stop the comparison and quitting feh (\"q\") will go\nto the next pair.\n\nWithout --feh (non-interactive / non-gui mode), script outputs pairs of images\nand the integer [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance)\nvalue for their perceptual hash values (basically the degree of difference\nbetween the two).\n\nOutput is sorted by this \"distance\", so most similar images (with the lowest\nnumber) should come first (see --top-n parameter).\n\nOptional --reported-db (or \"-d\") parameter allows efficient skipping of\nalready-reported \"similar\" image pairs by recording these in a dbm file.\nIntended usage for this option is to skip repeating same hash-similar pairs on\nrepeated runs, reporting similarity for new images instead.\n\n\nOperation\n--------------------\n\nScript does these steps, in order:\n\n* Try to load pre-calculated image hash values from --hash-db file.\n\n* Calculate missing perceptual hash values (ph_dct_imagehash) for each image\n  found, possibly in multiple subprocesses.\n\n* Dump (pickle) produced hash values (back) to a --hash-db file.\n\n* Calculate the difference between hashes of each image pair for all two-image\n  combinations, sorting the results.\n\n* Print (or run \"feh\" on) each found image-pair, in most-similar-first order,\n  optionally skipping pairs matching those in --reported-db file.\n\nIt's fairly simple, with all the magic and awesomeness in calculation of that\n\"perceptual hash\" values, which is contained in [libpHash](http://phash.org).\n\n\nKnown Issues\n--------------------\n\npHash seem to be prone to hanging indefinitely on some non-image files without\nconsuming much resources. Use `./image_matcher.py --debug -p 1` to see on which\nexact file it hangs on in such cases.\nMight add some check for file magic to see if it's image before running pHash\nover it in the future.\n\npHash also gives zero as a hash value for some images. No idea why it does that\natm, but these \"0\" values obviously can't be meaningfully compared to anything,\nso tool skips them, issuing a log message (seen only with --debug).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmk-fg%2Fimage-deduplication-tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmk-fg%2Fimage-deduplication-tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmk-fg%2Fimage-deduplication-tool/lists"}