https://github.com/anupsv/advance-image-downloader-golang
Concurrent image downloader in Go with configurable options for batch processing, random wait times, and file handling.
https://github.com/anupsv/advance-image-downloader-golang
downloader go golang image-downloader image-downloader-go image-downloading image-processing
Last synced: 9 months ago
JSON representation
Concurrent image downloader in Go with configurable options for batch processing, random wait times, and file handling.
- Host: GitHub
- URL: https://github.com/anupsv/advance-image-downloader-golang
- Owner: anupsv
- License: mit
- Created: 2023-06-03T14:34:15.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-06-05T13:32:47.000Z (over 2 years ago)
- Last Synced: 2025-02-09T10:12:33.922Z (10 months ago)
- Topics: downloader, go, golang, image-downloader, image-downloader-go, image-downloading, image-processing
- Language: Go
- Homepage:
- Size: 77.1 KB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# advance-image-downloader-golang
A Go program to download images from a list of URLs in parallel, with configurable batch size and tsnfom wait timed between batches.
## Prerequisites
- Go (version 1.16 or higher)
## Getting Started
1. Clone the repository:
```shell
git clone https://github.com/anupsv/advance-image-downloader-golang.git
```
2. Navigate to the project directory:
```shell
cd advance-image-downloader-golang
```
3. Create a text file called image_urls.txt and add the URLs of the images you want to download, each on a separate line.
4. Create a configuration file called config.yaml and set the following options:
```makefile
image_url_file: image_urls.txt
download_directory: ./downloads/
batch_size: 2
min_wait_time: 0.8
max_wait_time: 3.0
max_image_size_mb: MAX
replace_downloaded_file_size: true
skip_if_file_exists: false
```
Adjust the values according to your requirements. Ensure that the image_url_file option points to the correct file name and path. The max_image_size_mb option can be set to "MAX" to skip the size check and download all images regardless of their size.
5. Run it using `go run main.go`. The program will download the images from the URLs specified in the image_urls.txt file. The specified batch size and wait time between batches will be respected. Images exceeding the specified size (if max_image_size_mb is not set to "MAX") will be skipped.
6. The downloaded images will be saved in the downloads directory.
## Configuration Options
- image_url_file: The path to the file containing the list of image URLs to download.
- download_directory: The directory where the downloaded images will be saved.
- batch_size: The number of images to download concurrently in each batch.
- min_wait_time: The minimum wait time between batches (in seconds).
- max_wait_time: The maximum wait time between batches (in seconds).
- max_image_size_mb: The maximum allowed size (in megabytes) for an image. Set to "MAX" to skip the size check and download all images regardless of their size.
- replace_downloaded_file_size: Set it to true to replace already downloaded files if their size differs from the newly downloaded ones. Set it to false to keep the existing files without replacement.
- skip_if_file_exists: Set it to true to skip downloading if the file already exists. Set it to false to allow downloading even if the file exists.