{"id":19879484,"url":"https://github.com/pvlbzn/itoh","last_synced_at":"2025-10-07T03:38:41.190Z","repository":{"id":260192368,"uuid":"104550421","full_name":"pvlbzn/itoh","owner":"pvlbzn","description":"Research: image store","archived":false,"fork":false,"pushed_at":"2017-09-23T10:04:44.000Z","size":490,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-15T22:58:16.486Z","etag":null,"topics":["blake2","hashing","sha256","storage"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pvlbzn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-23T07:40:44.000Z","updated_at":"2020-02-19T15:04:12.000Z","dependencies_parsed_at":"2024-10-30T03:52:56.807Z","dependency_job_id":null,"html_url":"https://github.com/pvlbzn/itoh","commit_stats":null,"previous_names":["pvlbzn/itoh"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pvlbzn/itoh","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvlbzn%2Fitoh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvlbzn%2Fitoh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvlbzn%2Fitoh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvlbzn%2Fitoh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pvlbzn","download_url":"https://codeload.github.com/pvlbzn/itoh/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvlbzn%2Fitoh/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278717080,"owners_count":26033535,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blake2","hashing","sha256","storage"],"created_at":"2024-11-12T17:08:40.077Z","updated_at":"2025-10-07T03:38:41.164Z","avatar_url":"https://github.com/pvlbzn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Image to Hash\n\nSmall research project: attempt to solve a question \"how to store images\".\n\n\n## Problem\n\nYou want to store image, or effectively, any other files on a server. Things\nwhich you most likely will concern are:\n\n* Security\n* Consistent naming\n* Name collisions\n* Reduce # of duplicates\n\n\n## Solution\n\nDue to the nature of widespread compression algorithms for images, each image\nmay have many instances. Trivially, a number of instances depends on image's origin, its purpose.\nFor examples images on Instagram has usually only one instance due to its nature:\nsharing personal photos. However, images on Pinterest have a high repetition rate\nbecause Pinterest is about sharing, and people share a lot.\n\nTherefore we can **store images using a hash string representation as a name.**\n\n\n## Proof of Concept\n\nLet's take one instance of an image and store it in different places.\n\n1. [Image on Google Drive](https://drive.google.com/open?id=0B1rsXetG2YsjNjJfWjJuMGR2d1E)\n2. [Image on Dropbox](https://www.dropbox.com/s/kko780u7qypseo0/original_image.jpg?dl=0)\n3. [Image on GitHub](https://github.com/pvlbzn/itoh/raw/master/img/original_image.jpg)\n4. [Image on Imgur](https://i.imgur.com/AhfqaQW.jpg)\n\nAfter, let's save them back on a disk in a separate directory adding prefixes\nabout their origin.\n\n```\nsha256\n\nImgur:      975aa7f42346a406dcfa0dfef408d45f389795a4d16b12c4a9eda861da29d474\nDropbox:    c380db894d40144b9cc91f9821ae4833085df10201313be1f32b5c3f41038c3d\nGDrive:     cc26a121692fd12b356f783ed7f863da0f6dc478cb074570dde87fda929bd04a\nGitHub:     cc26a121692fd12b356f783ed7f863da0f6dc478cb074570dde87fda929bd04a\nOriginal:   cc26a121692fd12b356f783ed7f863da0f6dc478cb074570dde87fda929bd04a\n```\n\nOriginal, GitHub, Google Drive versions are identical, while Imgur uses compression\nas well as some kind of similar naming techniques, as can be inferred from the image's\nname.\n\n\n#### Hashing Algorithm\n\nIt is better to prefer [Blake2](https://blake2.net/) algorithm, due to its speed\nand interface advantages. In the code, [`./itoh.py`](https://github.com/pvlbzn/itoh/blob/master/itoh.py)\nblake2s aliased to blake.\n\n\n## Conclusion\n\nHashing method for sure can solve the naming problem with not much of an overhead.\nDepending on the nature of your application it also may save considerable space\non duplicates, however, if service used mostly for unique content, such as Instagram,\nhashing won't save any space.\n\nHowever, hashing always has a probability of a collision. Therefore it is not\na good idea to have only hashing check. In case of collision, some additional\ncomparison algorithm may be used together with a sanity check assertions.\n\nFor example, if two images have the same hash *and* the same size, then they are\nthe same.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpvlbzn%2Fitoh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpvlbzn%2Fitoh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpvlbzn%2Fitoh/lists"}