https://github.com/nashory/loader-torch
An Multi-threaded Data Loader Module for Torch.
https://github.com/nashory/loader-torch
data dataloader loader module multi-thread preprocessing toolbox torch
Last synced: 9 months ago
JSON representation
An Multi-threaded Data Loader Module for Torch.
- Host: GitHub
- URL: https://github.com/nashory/loader-torch
- Owner: nashory
- License: bsd-3-clause
- Created: 2017-09-15T05:25:47.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2017-09-26T03:53:45.000Z (over 8 years ago)
- Last Synced: 2025-04-02T20:38:31.320Z (10 months ago)
- Topics: data, dataloader, loader, module, multi-thread, preprocessing, toolbox, torch
- Language: Lua
- Homepage:
- Size: 89.8 KB
- Stars: 5
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# loader Package Reference Manual
Multi-threaded data loading module with image preprocessing for torch.
## Core-functionality
+ Multi-threaded : blazingly fast data loading
+ Preprocessing : resizing / cropping / enhancing-contrast / rotating / normalizing / adjusting pixel range
+ Deatailed options : you can minutely control the output image condition.
## Benchmark: Single-thread vs. Multi-threads
__Test: total elapsed time of loading 10x3x64x64 batch for 100 times.__
|\# of threads|progress|result|
|---|---|---|
|1(single)|
| 17.09 sec.|
|2(dual) | 
| 10.03 sec.|
|__8(octa)__ | 
| __8.51 sec.__|
## output batch image examples
__1. random cropping__
__2. random rotation__
__3. random horizontal flip__
__4. padding__
__5. random brightness__
__6. add whitenoise__
|std=0|std=0.2|std=0.4|std=0.6|std=0.8|std=1.0|
|---|---|---|---|---|---|
|
|
|
|
|
|
|
## How to use?
__1. Install the package:__
~~~
luarocks install loader
~~~
__2. set loader options: __
You can pass data-loading options as format of lua table like below.
~~~
local opt = {}
opt['nthreads'] = 8
opt['batchSize'] = 40
opt['loadSize'] = 96
opt['sampleSize'] = 96
opt['trainPath'] = '/home/nashory/data/test'
~~~
__3. create dataloader and get batch: __
~~~
require 'loader'
local myloader = loader.new(opt) -- declaration
for i = 1, 100 do
local batch = myloader:getBatch('train') -- get batch (batchSize x 3 x sampleSize x sampleSize)
--local sample = myloader:getSample -- get single sample image (3 x sampleSize x sampleSize)
end
~~~
## default options
if you do not specify the option, these values is applied by default.
default value "N/A" is for options you "must" specify.
|option|default|help|
|---|---|---|
|nthreads|8|number or workers, (1 means single thread)|
|trainPath|'./data'|'path to train data folder.'|
|batchSize|N/A|image batchsize|
|loadSize|80|image is resized so the smallest length of w/h is equal to loadSize.|
|sampleSize|64|crop size. image is cropped so the w = h = sampleSize|
|split|1.0|split ratio (train/test)|
|crop|'random'|crop option for training. when testing, we force to use 'center' crop. (center \| random)|
|padding|false|true: add padding to make square image before resizing|
|keep_ratio|true|true: will keep the image ratio|
## code examples
~~~
will be updated soon.
~~~
## Acknowledgements
This code is referenced to the data loader of @soumith's [dcgan code](https://github.com/soumith/dcgan.torch)
## Author
MinchulShin, [@nashory](https://github.com/nashory)
__Will keep updating the functionalities.__
__Any insane bug reports or questions are welcome. (min.stellastra[at]gmail.com) :-)__