{"id":13578570,"url":"https://github.com/tomahim/py-image-dataset-generator","last_synced_at":"2026-01-28T10:39:51.758Z","repository":{"id":74933879,"uuid":"120759229","full_name":"tomahim/py-image-dataset-generator","owner":"tomahim","description":"Get a large image dataset with minimal effort by grabbing image through the web and generate new ones by image augmentation.","archived":false,"fork":false,"pushed_at":"2020-06-29T17:30:56.000Z","size":64,"stargazers_count":218,"open_issues_count":11,"forks_count":41,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-05T16:46:26.365Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tomahim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-02-08T12:34:11.000Z","updated_at":"2024-10-25T06:59:36.000Z","dependencies_parsed_at":"2023-07-11T14:47:52.549Z","dependency_job_id":null,"html_url":"https://github.com/tomahim/py-image-dataset-generator","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomahim%2Fpy-image-dataset-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomahim%2Fpy-image-dataset-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomahim%2Fpy-image-dataset-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomahim%2Fpy-image-dataset-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tomahim","download_url":"https://codeload.github.com/tomahim/py-image-dataset-generator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247393081,"owners_count":20931804,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:01:31.910Z","updated_at":"2026-01-28T10:39:51.752Z","avatar_url":"https://github.com/tomahim.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Image dataset generator for Deep learning projects\n\n[![Join the chat at https://gitter.im/py-image-dataset-generator/Lobby](https://badges.gitter.im/py-image-dataset-generator/Lobby.svg)](https://gitter.im/py-image-dataset-generator/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n### Get a large image dataset with minimal effort\n\nThis tool **automatically collect images** from Google or Bing and optionally resize them. \n\n```\npython download.py \"funny cats\" -limit=100 -dest=folder_name -resize=250x250\n```\n\nThen you can **randomly generate new images** with image augmentation from an existing folder. It will add noise, rotate, transform, flip, blur on random images.\n\n```\npython augmentation.py -folder=my_folder/funny_cats -limit=10000\n```\n\nTADA ! In few seconds you will get 10 000 different images of funny cats to train your favorite deep learning algorithm !\n\n### Table of content\n\n* [Pre-requirements](#pre-requirements)\n* [Installation](#installation)\n* [Run unit tests](#run-unit-tests)\n* [Usage](#usage)\n    * [Download images](#download-images-from-the-web)\n    * [Image augmentation](#image-augmentation)\n    * [Create a custom image augmentation pipeline](#create-a-custom-image-augmentation-pipeline)\n* [Common issues](#common-issues)\n* [Acknowledgments](#acknowledgments)\n\n### Pre-requirements\n\nThis project is tested with Python 3.6.4 and more.\n\n*Linux*\n\n- chromium-browser package (`sudo apt-get install chromium-browser`)\n\n*Windows*\n\n- Chrome should be installed\n- [Microsoft Visual C++ Build Tools](https://www.scivision.co/python-windows-visual-c++-14-required/) (scikit image dependency, [see for more info](https://www.scivision.co/python-windows-visual-c++-14-required/))\n\n### Installation\n\nGit clone the project\n\nGet the python dependencies\n\n```\npip install -r requirements.txt\n```\n\n### Run unit tests\n\n```\npython -m unittest discover\n```\n\n### Usage\n\n#### Download images from the web\n\n```\npython download.py \"red car\" -limit=150 -dest=folder_name -resize=250x250\n```\n    \nAfter running this command, you will have 150 images of *red cars* (resized 250px by 250px) in the /folder_name/red_car folder. \n\nYou can find all possible parameters in the table below (also available with the `--help` parameter) :\n\nParameters  | Description\n---    | --- \nKeyword *(required)* | The first parameter should be a keyword describing the images to search for. \u003cbr\u003e\u003cbr\u003e `python download.py \"red car\"`\nDestination folder \u003cbr\u003e*-dest or -d* | Specify the destination folder to save files (default: images/) \u003cbr\u003e\u003cbr\u003e `python download.py \"red car\" -dest=your_folder`\nLimit number \u003cbr\u003e*-limit or -l* | Specify the number of files to download (default: 50). See the note below for the maximum limit. \u003cbr\u003e\u003cbr\u003e `python download.py \"red car\" -limit=200`\nThumbnail only \u003cbr\u003e*-thumbnail or -thumb* | Download the thumbnail instead of the full original image \u003cbr\u003e\u003cbr\u003e   `python download.py \"red car\" -thumbnail`\nResize image \u003cbr\u003e*-resize* | Resize downloaded images on the fly, to get a dataset formatted with the same size (default: no resizing). The parameter should be a couple of number representing the width and height (32x32 will ouput 32px x 32px image files) \u003cbr\u003e\u003cbr\u003e  `python download.py \"red car\" -resize=32x32\"`\nGrab source \u003cbr\u003e*-source, -src or -allsources* |  Choose the website to grab images : Google and/or Bing (default: Google). *-allsources* parameter can be use to. It will equally mix image files from all available sources \u003cbr\u003e\u003cbr\u003e `python download.py \"red car\" -source=Google` (single source) \u003cbr\u003e `python download.py \"red car\" -source=Google -source=Bing` (multi source)\u003cbr\u003e `python download.py \"red car\" -allsources` (all sources)\n\nNote : There are known limitations for the total number of images you can download in one use of the `download.py` script. Bing and Google won't let you download more than 800 images each, so the maximum for one download is around 1600 images if you use the `-allsources` parameter.\n\n#### Image augmentation\n\n```\npython augmentation.py -folder=your_folder -limit=10000\n```\n\n10 000 augmented images will output by default to the \"output\" folder inside your image folder.\n\nBy default, this command will randomly apply these image transformations :\n\n- Blur image (with a probability of 10%)\n- Add Random noise (with a probability of 50%)\n- Horizontal flip (with a probability of 30%)\n- Left or Right rotation between 0 or 25 degree (with a probability of 50%)\n\n- *... to be completed*\n\nYou can customize these default values by editing the `augmentation_config.py` file or by making [your own image augmentation pipeline](#create-a-custom-image-augmentation-pipeline)\n\nYou can find all possible parameters in the table below (also available with the `--help` parameter) :\n\nParameters  | Description\n---    | --- \nKeyword *(required)* | Folder input path containing images that will be augmented.`\nDestination folder \u003cbr\u003e*-dest or -d* | Specify the destination folder to save augmented files (default: /your_folder/output) \u003cbr\u003e\u003cbr\u003e `python augmentation.py -folder=your_folder -limit=50 -dest=other_folder`\nLimit number \u003cbr\u003e*-limit or -l* | Number of image to generate by augmentation (default: 50)\n\n#### Create a custom image augmentation pipeline\n\n```python\nfrom augmentation.augmentation import DatasetGenerator\n\npipeline = DatasetGenerator(\n    folder_path=\"images/red_car/\",\n    num_files=5000,\n    save_to_disk=True,\n    folder_destination=\"images/red_car/results\"\n)\npipeline.rotate(probability=0.5, max_left_degree=25, max_right_degree=25)\npipeline.random_noise(probability=0.5)\npipeline.blur(probability=0.5)\npipeline.vertical_flip(probability=0.1)\npipeline.horizontal_flip(probability=0.2)\npipeline.resize(probability=1, width=20, height=20)\npipeline.execute()\n```\n\nThat's it !\n\n### Common issues\n\n**WebDriverException: Message: unknown error: cannot find Chrome binary**\n\nMake sure chromedriver is well installed on your PATH (run the `which chromedriver` command on Linux and then `echo $PATH`). Also Chrome should be installed on your machine (or the `chromium-package` for Linux).\n\nYou can install the chromedriver with this command ([more information here](https://pypi.python.org/pypi/chromedriver_installer)):\n`pip install chromedriver_installer --install-option=\"--chromedriver-version=2.35\"`\n\n**error: Microsoft Visual C++ 14.0 is required. Get it with \"Microsoft Visual C++ Build Tools\": [http://landinghub.visualstudio.com/visual-cpp-build-tools](https://www.scivision.co/python-windows-visual-c++-14-required/)**\n\nAs this repo use scikit-image for image processing, on Windows you need Microsoft Visual C++ Build Tools which is provided with Visual Studio (think to check the C++ options on installation). You can install it with the link below.\n\n### Acknowledgments\n\n- This repo is *largely inspired* by the work of Marcus Bloice on his [Augmentor](https://arxiv.org/abs/1708.04680) project. Many thanks for the great work and the useful documentation.\n\n- I also pick some ideas from [this great series of articles](https://www.pyimagesearch.com/2017/12/11/image-classification-with-keras-and-deep-learning/) for the *automatic* part to grab images.\n\nThe goal of this repo is mainly to provide the smaller python library as possible to generate an image dataset, without a big framework like Keras, Tflearn etc, which can be hard to configure and install for new people working on Data Science / AI.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomahim%2Fpy-image-dataset-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftomahim%2Fpy-image-dataset-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomahim%2Fpy-image-dataset-generator/lists"}