{"id":13411938,"url":"https://github.com/deliton/idt","last_synced_at":"2025-03-14T17:31:20.578Z","repository":{"id":51228290,"uuid":"288855026","full_name":"deliton/idt","owner":"deliton","description":"Image Dataset Tool (idt) is a cli tool designed to make the otherwise repetitive and slow task of creating image datasets into a fast and intuitive process.","archived":false,"fork":false,"pushed_at":"2021-01-09T22:50:25.000Z","size":108,"stargazers_count":228,"open_issues_count":2,"forks_count":28,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-08T13:12:00.778Z","etag":null,"topics":["bing","bing-api","deep-learning","download","download-dataset","download-images","duckgo","flickr-api","idt","image-datasets","image-download","mount","scraping","scraping-web","search-engine"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deliton.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-19T22:59:21.000Z","updated_at":"2025-01-30T14:17:10.000Z","dependencies_parsed_at":"2022-09-05T01:00:42.010Z","dependency_job_id":null,"html_url":"https://github.com/deliton/idt","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deliton%2Fidt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deliton%2Fidt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deliton%2Fidt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deliton%2Fidt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deliton","download_url":"https://codeload.github.com/deliton/idt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243618703,"owners_count":20320280,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bing","bing-api","deep-learning","download","download-dataset","download-images","duckgo","flickr-api","idt","image-datasets","image-download","mount","scraping","scraping-web","search-engine"],"created_at":"2024-07-30T20:01:18.916Z","updated_at":"2025-03-14T17:31:17.223Z","avatar_url":"https://github.com/deliton.png","language":"Python","readme":"# IDT - Image Dataset Tool\n\n## Version 0.0.6 beta\n\n![idt-logo](https://user-images.githubusercontent.com/47995046/96403078-d675f080-11ad-11eb-8435-c8ce69a6c871.png)\n\n\n## Description\n\nThe image dataset tool (IDT) is a CLI app developed to make it easier and faster to create image datasets to be used for deep learning. The tool achieves this by scraping images from several search engines such as duckgo, bing and deviantart. IDT also optimizes the image dataset, although this feature is optional, the user can downscale and compress the images for optimal file size and dimensions. A sample dataset created using **idt** that contains  a total amount of 23.688 image files weights only 559,2 megabytes.\n\n## NEW UPDATE!\nI am proud to announce our newest version! 🎉🎉\n\n**What changed**\n* Added auto duplicate images remover\n* Added longer side resize method. With this option, the image is resized to its longer side.\n* Added shorter side resize method. With this option, the image is resized to its shorter side.\n* Added Smart Crop. This method tries to crop and resize exactly the main subject of the image. The algorithm is based on SmartCrop.js and SmartCrop.py.\n* Removed verbose mode. This was used in earlier stages of development but now don't add value to the experience.\n* The official documentation is almost ready. A link will be available soon\n\n## Installing\n\nYou can install it via pip or cloning this repository.\n\n```console\nuser@admin:~$ pip3 install idt\n\n```\n\n**OR**\n\n\n```console\nuser@admin:~$ git clone https://github.com/deliton/idt.git \u0026\u0026 cd idt\nuser@admin:~/idt$ sudo python3 setup.py install\n\n```\n\n\n## Getting Started\n\n![idt-gif](https://user-images.githubusercontent.com/47995046/96406740-6d46ab00-11b6-11eb-980b-a40968ed38b4.gif)\n\nThe quickest way to get started with IDT is running the simple \"run\" command. Just write in your favorite console something like:\n\n```console\nuser@admin:~$ idt run -i apples \n```\n\nThis will quickly download 50 images of apples. By default it uses the duckgo search engine to do so. \nThe run command accepts the following options:\n\n| Option | Description |\n| ----------- | ----------- |\n| **-i** or **--input** | the keyword to find the desired images. | \n| **-s** or **--size** | the amount of images to be downloaded. |\n| **-e** or **--engine** | the desired search engine (options: duckgo, bing, bing_api and flickr_api) |\n| **--resize-method** | choose a resize method of images. (options: longer_side, shorter_side and smartcrop) |\n| **-is** or **--image-size** | option to set the desired image size ratio. default=512 |\n| **-ak** or **--api-key** | If you are using a search engine that requires an API key, this option is required |\n\n\n## Usage\n\nIDT requires a config file that tells it how your dataset should be organized. You can create it using the following command:\n\n```console\nuser@admin:~$ idt init\n```\n\nThis command will trigger the config file creator and will ask for the desired dataset parameters. In this example let's create a dataset containing images of your favorite cars. The first parameters this command will ask is what name should your dataset have? In this example, let's name our dataset \"My favorite cars\"\n\n```console\nInsert a name  for your dataset: : My favorite cars\n```\n\nThen the tool will ask how many samples per search are required to mount your dataset. In order to build a good dataset for deep learning, many images are required and since we're using a search engine to scrape images, many searches with different keywords are required to mount a good sized dataset. This value will correspond to how many images should be downloaded at every search. In this example we need a dataset with 250 images in each class, and we'll use 5 keywords to mount each class. So if we type the number 50 here, IDT will download 50 images of every keyword provided. If we provide 5 keywords we should get the required 250 images.\n\n```console\nHow many samples per search will be necessary?  : 50\n```\n\nThe tool will now ask for and image size ratio. Since using large images to train neural networks is not a viable thing, we can optionally choose one of the following image size ratios and scale down our images to that size. In this example, we'll go for 512x512, although 256x256 would be an even better option for this task.\n\n```console\nChoose images resolution:\n\n[1] 512 pixels / 512 pixels (recommended)\n[2] 1024 pixels / 1024 pixels\n[3] 256 pixels / 256 pixels\n[4] 128 pixels / 128 pixels\n[5] Keep original image size\n\nps: note that the aspect ratio of the image will not be changed, \nso possibly the images received will have slightly different size\n\nWhat is the desired image size ratio: 1\n```\n\nAnd then choose \"longer_side\" for resize method.\n\n```console\n[1] Resize image based on longer side\n[2] Resize image based on shorter side\n[3] Smartcrop\n\nps: note that the aspect ratio of the image will not be changed,\nso possibly the images received will have slightly different size\n\nDesired Image resize method: : longer_side\n\n```\n\nNow you must choose how many classes/folders your dataset should have. In this example, this part can be very personal, but my favorite cars are: Chevrolet Impala, Range Rover Evoque, Tesla Model X and (why not) AvtoVAZ Lada. So in this case we have 4 classes, one for each favorite.\n\n```console\nHow many image classes are required? : 4\n```\n\nAfterwards, you'll be asked to choose between one of the search engines available. In this example, we'll use DuckGO to search images for us.\n\n```console\nChoose a search engine:\n\n[1] Duck GO (recommended)\n[2] Bing\n[3] Bing API \n[4] Flickr API\n\nSelect option:: 1\n```\n\nNow we have to do some repetitive form filling. We must name each class and all the keywords that will be used to find the images. Note that this part can be later changed by your own code, to generate more classes and keywords.\n\n```console\nClass 1 name: : Chevrolet Impala\n```\n\nAfter typing the first class name, we'll be asked to provide all the keywords to find the dataset. Remember that we told the program to download 50 images of each keyword so we must provide 5 keywords in this case to get all 250 images. Each keyword MUST be separated by commas(,)\n\n```console\nIn order to achieve better results, choose several keywords that will\nbe provided to the search engine to find your class in different settings.\n\nExample: \n\nClass Name: Pineapple\nkeywords: pineapple, pineapple fruit, ananas, abacaxi, pineapple drawing\n\nType in all keywords used to find your desired class, separated by commas: Chevrolet Impala 1967 car photos,\nchevrolet impala on the road, chevrolet impala vintage car, chevrolet impala convertible 1961, chevrolet impala 1964 lowrider\n\n```\n\nThen repeat the process of filling class name and its keywords until you fill all the 4 classes required.\n\n```console\nDataset YAML file has been created successfully. Now run idt build to mount your dataset!\n```\n\nYour dataset configuration file has been created. Now just rust the following command and see the magic happen:\n\n```console\nuser@admin:~$ idt build\n```\n\nAnd wait while the dataset is being mounted:\n\n```console\nCreating Chevrolet Impala class\nDownloading Chevrolet Impala 1967 car photos  [#########################-----------]   72%  00:00:12\n\n```\n\nAt the end, all your images will be available in a folder with the dataset name. Also, a csv file with the dataset stats are also included in the dataset's root folder.\n\n![idt-results](https://user-images.githubusercontent.com/47995046/93012667-808fa680-f578-11ea-82fc-7ebcb8ce3c41.png)\n\n\n## Split image dataset for Deep Learning\n\nSince deep learning often requires you to split your dataset into a subset of training/validation folders, this project can also do this for you! Just run:\n\n```console\nuser@admin:~$ idt split\n```\n\nNow you must choose a train/valid proportion. In this example I've chosen that 70% of the images will be reserved for training, while the rest will be reserved for validation: \n\n```console\nChoose the desired proportion of images of each class to be distributed in train/valid folders.\nWhat percentage of images should be distributed towards training? \n(0-100): 70\n\n70 percent of the images will be moved to a train folder, while 30 percent of the remaining images\nwill be stored in a validation folder.\nIs that ok? [Y/n]: y\n```\n\nAnd that's it! The dataset-split should now be found with the corresponding train/valid subdirectories.\n\n## Issues\n\nThis project is being developed in my spare time and it still needs a lot of effort to be free of bugs. Pull requests and contributors are really appreciated, feel free to contribute in any way you can!\n\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeliton%2Fidt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeliton%2Fidt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeliton%2Fidt/lists"}