{"id":13737636,"url":"https://github.com/cceyda/image-checker","last_synced_at":"2025-04-13T07:33:22.459Z","repository":{"id":48175553,"uuid":"309986706","full_name":"cceyda/image-checker","owner":"cceyda","description":"Fast Image Integrity Checker: Scan for corrupted images using Nvidia DALI","archived":false,"fork":false,"pushed_at":"2021-06-20T10:01:44.000Z","size":72,"stargazers_count":20,"open_issues_count":2,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-05T14:12:13.456Z","etag":null,"topics":["image-processing","nvidia-dali"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cceyda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-04T11:46:37.000Z","updated_at":"2024-12-30T05:05:27.000Z","dependencies_parsed_at":"2022-08-28T14:03:07.814Z","dependency_job_id":null,"html_url":"https://github.com/cceyda/image-checker","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cceyda%2Fimage-checker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cceyda%2Fimage-checker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cceyda%2Fimage-checker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cceyda%2Fimage-checker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cceyda","download_url":"https://codeload.github.com/cceyda/image-checker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240045016,"owners_count":19739186,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image-processing","nvidia-dali"],"created_at":"2024-08-03T03:01:55.741Z","updated_at":"2025-02-23T02:30:34.206Z","avatar_url":"https://github.com/cceyda.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Fast Image Integrity Checker\nCheck for corrupted JPEG,PNG (and more) images in bulk using GPU jpeg decoding powered by NVIDIA DALI!\n\nSuper fast compared to alternatives because it uses GPU decoding and checks in batches.\n\n![](dali_checker.jpg)\n\nWhat is DALI? [blogpost](https://cceyda.github.io/blog/dali/cv/image_processing/2020/11/10/nvidia_dali.html)\n\n# Requirements\nDepending on your cuda version install:\n[NVIDIA DALI](https://docs.nvidia.com/deeplearning/dali/user-guide/docs/installation.html#id1)\nIf it fails try `pip3 install –upgrade pip`\n\n# Usage\n\n`pip3 install image-checker`\n\nor\n\n`git clone https://github.com/cceyda/image-checker.git`\n\n## CLI\nThere is a CLI that takes a folder `--path` to scan for images with the given extensions `--ext`. Outputs one line to `error.log` containing image path per broken file. Format of log file can be modified by `--log_conf` error handler using python logging [example](/master/dali_image_checker/logging_config.json). If you don't want to use gpu provide `--use_cpu` flag. Use `--recursive` to \ntraverse sub-folders.\n\n`image-check --path /mnt/data/dali_test/corrupt --recursive`\n\n```bash\nusage: image-checker [-h] [-p PATH] [-b BATCH_SIZE] [-g DEVICE_ID]\n                        [-ext EXTENSIONS] [-l LOG_CONG] [-r] [-d] [-c]\n\nCheck a folder of images for broken/misidentified images\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -p PATH, --path PATH  Path for folder to be checked\n  -b BATCH_SIZE, --batch_size BATCH_SIZE\n                        Number of files checked per iteration (Recommend \u003c100)\n  -g DEVICE_ID, --device_id DEVICE_ID\n                        Gpu ID\n  -ext EXTENSIONS, --extensions EXTENSIONS\n                        (comma delimited) list of extentions to test for (only types supported by DALI)\n  -l LOG_CONG, --log_conf LOG_CONF\n                        Config file path\n  -r, --recursive\n  -d, --debug\n  -c, --use_cpu\n```\n\n## Code\n\n```python\nfrom image_checker.checker import checker_batch,checker_single\nfrom image_checker.iterators import folder_iterator\n\nargs = {\n     \"path\": \"/mnt/data/images/main/\",\n     \"batch_size\": 50,\n     \"prefetch\": 1,\n     \"debug\": False,\n     \"extensions\": [\"jpeg\", \"jpg\", \"png\"],\n     \"recursive\":False,\n     \"log_cong\":\"logging_config.json\",\n     \"device\":\"mixed\",\n     \"device_id\":0\n }\n\nds = folder_iterator(args[\"path\"], args[\"extensions\"], args[\"recursive\"])\nbad_files=checker_batch(ds, args)\n\n```\n\n# FAQ\n\n- What kind of corrupted images will this catch?\n    - Images that can't be decoded by DALI.\n    - GIFs pretending to be JPEGs (with a jpg,jpeg extension)\n    - (Won't catch) files that can't be opened (TODO)\n    - (Won't catch) empty image files\n    \n- Supported image types?\n\nSame as [DALI supported formats](https://docs.nvidia.com/deeplearning/dali/user-guide/docs/supported_ops.html?highlight=supported%20image#nvidia.dali.ops.ImageDecoder): JPG, BMP, PNG, TIFF, PNM, PPM, PGM, PBM, JPEG 2000. Please note that GPU acceleration for JPEG 2000 decoding is only available for CUDA 11.\n    \n- What is batch_size \u0026 prefetch?\n\nDALI works with a batching+prefetching system. So batch_size * prefetch number of images are read at a time. If there is a corrupted file in the batch that batch is rechecked 1-by-1. So keep batch_size reasonable (0\u003c100). Currently, only **prefetch=1** works but it is a TODO item.\n\n\n- Package versioning follows dali for major.minor (since it heavily depends on it), patch is this packages version changes.\n\n# Alternatives\n[check-media-integrity](https://github.com/ftarlao/check-media-integrity): supports more types but uses PIL thus slow.\n\n[Bad Peggy](https://github.com/llaith-oss/BadPeggy): Checks JPEG images, maybe detects more types of errors than this one.\n\n# TODOs\nAdd tests\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcceyda%2Fimage-checker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcceyda%2Fimage-checker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcceyda%2Fimage-checker/lists"}