https://github.com/spencermountain/sunday-driver
be cool with large files
https://github.com/spencermountain/sunday-driver
large-files stream-processing
Last synced: 11 months ago
JSON representation
be cool with large files
- Host: GitHub
- URL: https://github.com/spencermountain/sunday-driver
- Owner: spencermountain
- Created: 2018-05-07T15:48:21.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2022-04-09T21:41:57.000Z (about 4 years ago)
- Last Synced: 2025-05-19T13:58:06.798Z (about 1 year ago)
- Topics: large-files, stream-processing
- Language: JavaScript
- Homepage:
- Size: 1.91 MB
- Stars: 6
- Watchers: 2
- Forks: 1
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: changelog.md
Awesome Lists containing this project
README
sunday-driver
process a large file, at a steady cruise
🕶️
- slow is smooth, smooth is fast -
**sunday-driver** works through a large file *at a responsible* pace - it pauses to let you consider the data, at given points, and waits to resume working once that's all done.
this allows processing a large file, by sizable chunks, without any race-conditions or memory leaking.
(heavily) inspired by [line-by-line](https://github.com/Osterjour/line-by-line), by [Markus Ostertag](https://github.com/Osterjour)🙏
npm i sunday-driver
```js
const sundayDriver = require('sunday-driver')
let options= {
file: './my/large/file.tsv',
splitter: '\n',
start: '80%', //as percentages, or in bytes
end: '100%',
//do your thing, for each segment
each: (chunk, resume) => {
console.log(chunk)//do your thing..
resume()
}
//log progress-based events
atPercent: {
50: (status) => {
console.log('50%!')
},
75: () => {
console.log('75%!')
},
},
//log time-based events
atInterval: {
'1min': (status) => {
console.log('1 minute')
},
'2mins': () => {
console.log('2 minutes')
},
}
}
sundayDriver(options).then((status)=>{
console.log('done!')
})
```
any events/intervals will provide you with all the details of the current reader's status:
```js
/*{
chunksDone:: 10, // how many times we've called the 'each' function
bytesDone:: 20480, // how many bytes we've processed so far
filesize:: 61440, // size of the whole file
position: 34.42, // where, in percentage, we are in the file. (if we didn't start at the top!)
progress: 68.84 // how far, in percentage, we are to being complete
}*/
```
it was built to support unleashing multiple workers on the same file, and letting them run safely and responsibly, without blowing any fuses.
MIT