https://github.com/cemayan/async-chunk-reader
https://github.com/cemayan/async-chunk-reader
async-streams chunking streaming
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/cemayan/async-chunk-reader
- Owner: cemayan
- Created: 2019-02-13T08:04:35.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2019-03-28T13:56:32.000Z (about 7 years ago)
- Last Synced: 2025-09-30T17:39:49.030Z (8 months ago)
- Topics: async-streams, chunking, streaming
- Language: TypeScript
- Homepage: https://www.npmjs.com/package/async-chunk-reader
- Size: 653 KB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Async Chunk Reader
This library allows you to read large amounts of data in chunks.
[Click to see how it works](https://runkit.com/cemayan/async-chunk-reader)
## Install
```console
npm install --save async-chunk-reader
```
## Changelog
> Added reading from zip file (v1.0.6)
> Added skipRows (v.1.0.8)
> to be added in the future (v.1.0.9)
## API
---
**init(parameters : InitParameters)**
> input : InitParameters
- **chunk_size** : String
- **input_file** : String | Stream
- **encoding** : String
- **selectedFileName** String
**get()**
> output : Async Iterator
## Import
### with require :
```javascript
const reader = require('async-chunk-reader')
```
### with import :
```javascript
import * as reader from "async-chunk-reader"
```
## Usage
---
### with path :
```javascript
async function main(){
const data = await reader
.init({
chunkSize: 100000,
inputFile: 'input/mobile_network_201805.csv.gz'
})
.get()
for await(let chunk of data){
console.log(chunk.map(d=>d.value))
}
}
main();
```
### with stream :
```javascript
async function main(){
const data = await reader
.init({
inputFile: process.stdin
})
.get()
for await(let chunk of data){
console.log(chunk.map(d=>d.value))
}
}
main();
```
### with string :
```javascript
async function main(){
const data = await reader
.init({
inputFile: "Some string"
})
.get()
for await(let chunk of data){
console.log(chunk.map(d=>d.value))
}
}
main();
```
### with zipfile :
```javascript
async function main(){
const data = await reader
.init({
chunkSize: 100000,
inputFile: 'example/Archive.zip',
selectedFileName:'avocado.csv' #file in zip
})
.get()
for await(let chunk of data){
console.log(chunk.map(d=>d.value))
}
}
main();
```
### specialChunkSize :
```javascript
async function main(){
const data = await reader
.init({
chunkSize: 100000,
inputFile: 'example/Archive.zip',
specialChunkSize : {0:10000, 1:40000}
})
.get()
for await(let chunk of data){
console.log(chunk.map(d=>d.value))
}
}
main();
```
### to be added in the future (v.1.0.9)
### skipRows :
```javascript
async function main(){
const data = await reader
.init({
chunkSize: 100000,
inputFile: 'example/Archive.zip',
skipRows : 40000
})
.get()
for await(let chunk of data){
console.log(chunk.map(d=>d.value))
}
}
main();
```