{"id":20789597,"url":"https://github.com/victorqribeiro/dtf","last_synced_at":"2025-07-26T21:34:54.086Z","repository":{"id":128646939,"uuid":"179887312","full_name":"victorqribeiro/dtf","owner":"victorqribeiro","description":"DTF - Duplicate Thumbnail Files - A method to identify duplicate files.","archived":false,"fork":false,"pushed_at":"2019-04-07T03:42:54.000Z","size":66,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-19T19:04:14.101Z","etag":null,"topics":["duplicate-detection","duplicate-files","systematic-mapping","systematic-reviews"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/victorqribeiro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-06T21:13:29.000Z","updated_at":"2025-03-18T08:05:31.000Z","dependencies_parsed_at":"2023-06-08T01:00:44.676Z","dependency_job_id":null,"html_url":"https://github.com/victorqribeiro/dtf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/victorqribeiro/dtf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorqribeiro%2Fdtf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorqribeiro%2Fdtf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorqribeiro%2Fdtf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorqribeiro%2Fdtf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/victorqribeiro","download_url":"https://codeload.github.com/victorqribeiro/dtf/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorqribeiro%2Fdtf/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264785800,"owners_count":23663839,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duplicate-detection","duplicate-files","systematic-mapping","systematic-reviews"],"created_at":"2024-11-17T15:25:06.145Z","updated_at":"2025-07-11T10:34:40.595Z","avatar_url":"https://github.com/victorqribeiro.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DTF\n\nDuplicate Thumbnail File - A mathod to indetify duplicate articles when doing a [Systematic review](https://en.wikipedia.org/wiki/Systematic_review).\n\n## The problem\n\nA systematic review could spawn countless articles, when running your search string on different data bases.\nMany times articles were exported as PDF files and submited to diferent jornals or publications. The problem is, depending on how the PDF was exported, it will produce a different file, making it hard to check for duplicates by looking at their hash.\n\n## The experiment\n\nFive articles were written and exported to PDF, each one with a small difference.\n\n* **01-jpg-100.pdf** - was exported as a JPG compression with 100% quality.\n\n* **02-lossless.pdf** - was exported as a LOSSLESS compression.\n\n* **03-lossless-2newlines.pdf** - was exported as a LOSSLESS compression with two new lines after the end of the document.\n\n* **04-jpg-80-2newlines.pdf** - was exported as a JPG compression with 80% quality and two new lines after the end of the document.\n\n* **05-jpg-80.pdf** - was exported as a JPG compression with 80% quality.\n\nAll documents were created using [Libre Office](https://www.libreoffice.org/).\n\nLet's look at their hash:\n\n| Hash | File |\n| -- | -- |\n|1db6c720061004b740b87320f0d1d2a68bd9d312 | 01-jpg-100.pdf |\n|c9ecd0d02f04637ab81f8f383a74929d8a9f1a40 | 02-lossless.pdf |\n|dd8f35b79ded6b61f53eaeb0a9a2a7f1bf34eae6 | 03-lossless-2newlines.pdf |\n|9b660130bba4c2c7b63be832aab08a48a2b2e6f5 | 04-jpg-80-2newlines.pdf |\n|ffc4a3fcff380929f7c38379f5804e7bbd87ea44 | 05-jpg-80.pdf |\n\nAccording to the hash of each file, they're different from each other, when in fact, is the \"same\" document.\n\n## The solution\n\nAnd what if instead of looking at the hash of each file, we look at their thumbnail? It might work a little better.\n\nSo a python script was written to generate the thumbnail of each file and measure how different those thumbnails ware.\n\nHere's the result:\n\n| File 01 | File 02 | Difference |\n| -- | -- | -- |\n| 03-lossless-2newlines.png | 02-lossless.png | 0.0 |\n| 03-lossless-2newlines.png | 05-jpg-80.png | 0.0 |\n| 03-lossless-2newlines.png | 01-jpg-100.png | 0.0 |\n| 03-lossless-2newlines.png | 04-jpg-80-2newlines.png | 0.0 |\n| 02-lossless.png | 05-jpg-80.png | 0.0 |\n| 02-lossless.png | 01-jpg-100.png | 0.0 |\n| 02-lossless.png | 04-jpg-80-2newlines.png | 0.0 |\n| 05-jpg-80.png | 01-jpg-100.png | 0.0 |\n| 05-jpg-80.png | 04-jpg-80-2newlines.png | 0.0 |\n| 01-jpg-100.png | 04-jpg-80-2newlines.png | 0.0 |\n\nSo, as we can see, there is 0.0 difference between them, so they must have the same content.\n\n## Conclusion\n\nThis experiment offers a diffent approach to a common problem when doing systemic review, and in this particular case, it worked better. This method could be used alongside with other methods to indentify and exclude duplicate files.\n\nThe *dtf.py* script could be used as **base** to a **more robust** script that compares any files that could be visually compared.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictorqribeiro%2Fdtf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvictorqribeiro%2Fdtf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictorqribeiro%2Fdtf/lists"}