{"id":20982307,"url":"https://github.com/nashory/loader-torch","last_synced_at":"2025-05-14T16:31:31.381Z","repository":{"id":89673770,"uuid":"103617298","full_name":"nashory/loader-torch","owner":"nashory","description":"An Multi-threaded Data Loader Module for Torch.","archived":false,"fork":false,"pushed_at":"2017-09-26T03:53:45.000Z","size":92,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-02T20:38:31.320Z","etag":null,"topics":["data","dataloader","loader","module","multi-thread","preprocessing","toolbox","torch"],"latest_commit_sha":null,"homepage":"","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nashory.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-15T05:25:47.000Z","updated_at":"2018-05-01T05:13:02.000Z","dependencies_parsed_at":"2024-06-28T03:31:11.150Z","dependency_job_id":null,"html_url":"https://github.com/nashory/loader-torch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nashory%2Floader-torch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nashory%2Floader-torch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nashory%2Floader-torch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nashory%2Floader-torch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nashory","download_url":"https://codeload.github.com/nashory/loader-torch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254182927,"owners_count":22028383,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","dataloader","loader","module","multi-thread","preprocessing","toolbox","torch"],"created_at":"2024-11-19T05:45:12.467Z","updated_at":"2025-05-14T16:31:31.375Z","avatar_url":"https://github.com/nashory.png","language":"Lua","funding_links":[],"categories":[],"sub_categories":[],"readme":"# loader Package Reference Manual\nMulti-threaded data loading module with image preprocessing for torch.  \n\n\n## Core-functionality\n+ Multi-threaded : blazingly fast data loading\n+ Preprocessing : resizing / cropping / enhancing-contrast / rotating / normalizing / adjusting pixel range\n+ Deatailed options : you can minutely control the output image condition.\n\n\n## Benchmark: Single-thread vs. Multi-threads\n\n__Test: total elapsed time of loading 10x3x64x64 batch for 100 times.__\n\n|\\# of threads|progress|result|\n|---|---|---|\n|1(single)|\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/thread_1.gif\" width=\"400\"\u003e| 17.09 sec.|\n|2(dual) | \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/thread_2.gif\" width=\"400\"\u003e| 10.03 sec.|\n|__8(octa)__ | \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/thread_8.gif\" width=\"400\"\u003e| __8.51 sec.__|\n\n## output batch image examples\n\n__1. random cropping__\n\n\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/crop/1.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/crop/2.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/crop/3.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/crop/4.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/crop/5.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/crop/6.png\" width=\"128\"\u003e\n\n__2. random rotation__\n\n\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/rotate/1.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/rotate/2.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/rotate/3.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/rotate/4.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/rotate/5.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/rotate/6.png\" width=\"128\"\u003e\n\n__3. random horizontal flip__\n\n\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/hflip/1.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/hflip/2.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/hflip/3.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/hflip/4.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/hflip/5.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/hflip/6.png\" width=\"128\"\u003e\n\n__4. padding__\n\n\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/padding/11.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/padding/12.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/padding/13.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/padding/14.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/padding/15.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/padding/16.png\" width=\"128\"\u003e\n\n__5. random brightness__\n\n\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/brightness/1.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/brightness/2.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/brightness/3.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/brightness/4.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/brightness/5.png\" width=\"128\"\u003e \u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/brightness/6.png\" width=\"128\"\u003e\n\n__6. add whitenoise__\n\n|std=0|std=0.2|std=0.4|std=0.6|std=0.8|std=1.0|\n|---|---|---|---|---|---|\n|\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/noise/0.000000.png\" width=\"128\"\u003e|\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/noise/0.200000.png\" width=\"128\"\u003e|\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/noise/0.400000.png\" width=\"128\"\u003e|\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/noise/0.600000.png\" width=\"128\"\u003e|\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/noise/0.800000.png\" width=\"128\"\u003e|\u003cimg src=\"https://github.com/nashory/gif/blob/master/_loader/noise/1.000000.png\" width=\"128\"\u003e|\n\n \n\n## How to use?\n__1. Install the package:__\n~~~\nluarocks install loader\n~~~\n\n__2. set loader options:  __\nYou can pass data-loading options as format of lua table like below.\n~~~\nlocal opt = {}\nopt['nthreads'] = 8\nopt['batchSize'] = 40\nopt['loadSize'] = 96\nopt['sampleSize'] = 96\nopt['trainPath'] = '/home/nashory/data/test'\n~~~\n\n\n__3. create dataloader and get batch:  __\n~~~\nrequire 'loader'\nlocal myloader = loader.new(opt)                 -- declaration\nfor i = 1, 100 do\n    local batch = myloader:getBatch('train')     -- get batch (batchSize x 3 x sampleSize x sampleSize)\n\t--local sample = myloader:getSample\t\t -- get single sample image (3 x sampleSize x sampleSize)\nend\n~~~\n\n## default options\nif you do not specify the option, these values is applied by default.\ndefault value \"N/A\" is for options you \"must\" specify. \n\n|option|default|help|\n|---|---|---|\n|nthreads|8|number or workers, (1 means single thread)|\n|trainPath|'./data'|'path to train data folder.'|\n|batchSize|N/A|image batchsize|\n|loadSize|80|image is resized so the smallest length of w/h is equal to loadSize.|\n|sampleSize|64|crop size. image is cropped so the w = h = sampleSize|\n|split|1.0|split ratio (train/test)|\n|crop|'random'|crop option for training. when testing, we force to use 'center' crop. (center \\| random)|\n|padding|false|true: add padding to make square image before resizing|\n|keep_ratio|true|true: will keep the image ratio|\n\n\n## code examples\n\n~~~\nwill be updated soon.\n~~~\n\n## Acknowledgements\nThis code is referenced to the data loader of @soumith's [dcgan code](https://github.com/soumith/dcgan.torch)\n\n\n## Author\nMinchulShin, [@nashory](https://github.com/nashory)  \n__Will keep updating the functionalities.__  \n__Any insane bug reports or questions are welcome. (min.stellastra[at]gmail.com)  :-)__\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnashory%2Floader-torch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnashory%2Floader-torch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnashory%2Floader-torch/lists"}