https://github.com/baggepinnen/diskdataproviders.jl

Disk based, buffered data structures for machine learning
https://github.com/baggepinnen/diskdataproviders.jl

buffered-reader data-provider data-structures machine-learning

Last synced: 24 days ago
JSON representation

Disk based, buffered data structures for machine learning

• Host: GitHub
• URL: https://github.com/baggepinnen/diskdataproviders.jl
• Owner: baggepinnen
• Created: 2019-08-13T09:52:19.000Z (about 5 years ago)
• Default Branch: master
• Last Pushed: 2021-01-04T23:18:17.000Z (almost 4 years ago)
• Last Synced: 2024-10-10T23:17:36.498Z (26 days ago)
• Topics: buffered-reader, data-provider, data-structures, machine-learning
• Language: Julia
• Homepage: https://baggepinnen.github.io/DiskDataProviders.jl/latest
• Size: 102 KB
• Stars: 12
• Watchers: 3
• Forks: 2
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
[![Build Status](https://travis-ci.org/baggepinnen/DiskDataProviders.jl.svg?branch=master)](https://travis-ci.org/baggepinnen/DiskDataProviders.jl)

[![codecov](https://codecov.io/gh/baggepinnen/DiskDataProviders.jl/branch/master/graph/badge.svg)](https://codecov.io/gh/baggepinnen/DiskDataProviders.jl)

[![](https://img.shields.io/badge/docs-latest-blue.svg)](https://baggepinnen.github.io/DiskDataProviders.jl/latest)

# DiskDataProviders.jl

This package implements datastructures that are iterable and backed by a buffer that is fed by data from disk. The reading is done on a separate thread, so make sure Julia is started with at least two threads.

Intended usage: To buffer data reading from disk when training convolutional neural networks (1d or 2d) using [Flux.jl](https://github.com/FluxML/Flux.jl/). This allows the CPU to work with the disk and data while the GPU is working on the training. This package might be useful for other things as well.

For usage example, see [the documentation](https://baggepinnen.github.io/DiskDataProviders.jl/latest)  

For lower-level buffered iterators, see [LengthChannels.jl](https://github.com/baggepinnen/LengthChannels.jl)