https://github.com/idris-maps/datastream
Building blocks to read and transform data streams
https://github.com/idris-maps/datastream
csv-stream deno ndjson-stream
Last synced: 4 months ago
JSON representation
Building blocks to read and transform data streams
- Host: GitHub
- URL: https://github.com/idris-maps/datastream
- Owner: idris-maps
- Created: 2021-11-27T14:57:23.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2022-03-10T22:19:44.000Z (over 3 years ago)
- Last Synced: 2025-02-24T05:18:35.627Z (5 months ago)
- Topics: csv-stream, deno, ndjson-stream
- Language: TypeScript
- Homepage: https://deno.land/x/datastream
- Size: 9.77 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# datastream
Building blocks to read and transform data streams
## Sources
### fromFile
Opens a file and creates a stream of lines
```ts
type fromFile = (
path: string,
funcs?: PipeFunction[],
) => Promise<{ iterator: AsyncIterableIterator; rid: number }>;
```
### fromStdin
Reads a stream from stdin
```ts
type fromStdin = (
funcs: PipeFunction[] = [],
) => { iterator: AsyncIterableIterator, rid: number }
```
### fromNdjsonFile
Opens a ndjson (new line delimited JSON) file and creates a stream of JSON
objects
```ts
type fromNdjsonFile = (
path: string,
funcs?: PipeFunction[],
) => Promise<{ iterator: AsyncIterableIterator; rid: number }>;
```
### fromNdjsonStdin
Reads and parses a stream of ndjson from stdin
```ts
type fromNdjsonStdin = (
funcs?: PipeFunction[],
) => { iterator: AsyncIterableIterator };
```
### fromDsvFile
Opens a dsv (delimiter separated values) file and creates a stream of JSON
objects
The first line is expected to be the column labels
```ts
type fromDsvFile = (
path: string,
config?: {
delimiter?: string; // default "," (csv)
numeric?: string[]; // list of numeric columns
bool?: string[]; // list of boolean columns
},
funcs?: PipeFunction[],
) => Promise<{ iterator: AsyncIterableIterator; rid: number }>;
```
### fromDsvStdin
Reads a stream of dsv from stdin and parses it as JSON objects
```ts
type fromDsvStdin = (
config?: {
delimiter?: string; // default "," (csv)
numeric?: string[]; // list of numeric columns
bool?: string[]; // list of boolean columns
},
funcs?: PipeFunction[],
) => { iterator: AsyncIterableIterator };
```
## Transforms
Functions to modify the stream of data
All transforms return a `PipeFunction`
```ts
type PipeFunction = (
d: AsyncIterableIterator,
) => AsyncIterableIterator;
```
### map
```ts
type map = (func: (d: A, i: number) => B) => PipeFunction;
```
### filter
```ts
type filter = (func: (d: T, i: number) => boolean) => PipeFunction;
```
### offset
```ts
type offset = (n: number) => PipeFunction;
```
### limit
```ts
type limit = (n: number) => PipeFunction;
```
## Output
### toArray
```ts
type toArray = ({ iterator: AsyncIterableIterator, rid?: number }) => Promise;
```
### find
```ts
type find = (
func: (d: T) => boolean,
) => ({ iterator: AsyncIterableIterator, rid?: number }) =>
Promise;
```
### reduce
```ts
type reduce = (
func: (r: B, d: A, i: number) => B,
start: B,
) => ({
iterator: AsyncIterableIterator,
rid?: number,
}) => Promise;
```
### toNdjsonStdout
```ts
type toNdjsonStdout = ({
iterator: AsyncIterableIterator,
rid?: number,
}) => Promise;
```
### toDsvStdout
```ts
type toDsvStdout = (
{ iterator: AsyncIterableIterator, rid?: number },
delimiter?: string,
) => Promise;
```
### toNdjsonFile
```ts
type toNdjsonFile = (
{ iterator: AsyncIterableIterator, rid?: number },
path: string,
) => Promise;
```
### toDsvFile
```ts
type toDsvFile = (
{ iterator: AsyncIterableIterator, rid?: number },
path: string,
delimiter?: string,
) => Promise;
```