{"id":21153764,"url":"https://github.com/idris-maps/datastream","last_synced_at":"2026-05-17T13:37:30.932Z","repository":{"id":57675580,"uuid":"432489670","full_name":"idris-maps/datastream","owner":"idris-maps","description":"Building blocks to read and transform data streams","archived":false,"fork":false,"pushed_at":"2022-03-10T22:19:44.000Z","size":10,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-11-20T22:07:12.775Z","etag":null,"topics":["csv-stream","deno","ndjson-stream"],"latest_commit_sha":null,"homepage":"https://deno.land/x/datastream","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/idris-maps.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-27T14:57:23.000Z","updated_at":"2022-01-09T21:20:35.000Z","dependencies_parsed_at":"2022-09-15T05:40:57.836Z","dependency_job_id":null,"html_url":"https://github.com/idris-maps/datastream","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/idris-maps/datastream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idris-maps%2Fdatastream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idris-maps%2Fdatastream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idris-maps%2Fdatastream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idris-maps%2Fdatastream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/idris-maps","download_url":"https://codeload.github.com/idris-maps/datastream/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idris-maps%2Fdatastream/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33140834,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T09:28:26.183Z","status":"ssl_error","status_checked_at":"2026-05-17T09:27:52.702Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv-stream","deno","ndjson-stream"],"created_at":"2024-11-20T10:59:20.174Z","updated_at":"2026-05-17T13:37:30.915Z","avatar_url":"https://github.com/idris-maps.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# datastream\n\nBuilding blocks to read and transform data streams\n\n## Sources\n\n### fromFile\n\nOpens a file and creates a stream of lines\n\n```ts\ntype fromFile = (\n  path: string,\n  funcs?: PipeFunction[],\n) =\u003e Promise\u003c{ iterator: AsyncIterableIterator\u003cany\u003e; rid: number }\u003e;\n```\n\n### fromStdin\n\nReads a stream from stdin\n\n```ts\ntype fromStdin = (\n  funcs: PipeFunction[] = [],\n) =\u003e { iterator: AsyncIterableIterator\u003cstring\u003e, rid: number }\n```\n\n### fromNdjsonFile\n\nOpens a ndjson (new line delimited JSON) file and creates a stream of JSON\nobjects\n\n```ts\ntype fromNdjsonFile = (\n  path: string,\n  funcs?: PipeFunction[],\n) =\u003e Promise\u003c{ iterator: AsyncIterableIterator\u003cany\u003e; rid: number }\u003e;\n```\n\n### fromNdjsonStdin\n\nReads and parses a stream of ndjson from stdin\n\n```ts\ntype fromNdjsonStdin = (\n  funcs?: PipeFunction[],\n) =\u003e { iterator: AsyncIterableIterator\u003cany\u003e };\n```\n\n### fromDsvFile\n\nOpens a dsv (delimiter separated values) file and creates a stream of JSON\nobjects\n\nThe first line is expected to be the column labels\n\n```ts\ntype fromDsvFile = (\n  path: string,\n  config?: {\n    delimiter?: string; // default \",\" (csv)\n    numeric?: string[]; // list of numeric columns\n    bool?: string[]; // list of boolean columns\n  },\n  funcs?: PipeFunction[],\n) =\u003e Promise\u003c{ iterator: AsyncIterableIterator\u003cany\u003e; rid: number }\u003e;\n```\n\n### fromDsvStdin\n\nReads a stream of dsv from stdin and parses it as JSON objects\n\n```ts\ntype fromDsvStdin = (\n  config?: {\n    delimiter?: string; // default \",\" (csv)\n    numeric?: string[]; // list of numeric columns\n    bool?: string[]; // list of boolean columns\n  },\n  funcs?: PipeFunction[],\n) =\u003e { iterator: AsyncIterableIterator\u003cany\u003e };\n```\n\n## Transforms\n\nFunctions to modify the stream of data\n\nAll transforms return a `PipeFunction`\n\n```ts\ntype PipeFunction = (\n  d: AsyncIterableIterator\u003cany\u003e,\n) =\u003e AsyncIterableIterator\u003cany\u003e;\n```\n\n### map\n\n```ts\ntype map = \u003cA = any, B = any\u003e(func: (d: A, i: number) =\u003e B) =\u003e PipeFunction;\n```\n\n### filter\n\n```ts\ntype filter = \u003cT = any\u003e(func: (d: T, i: number) =\u003e boolean) =\u003e PipeFunction;\n```\n\n### offset\n\n```ts\ntype offset = (n: number) =\u003e PipeFunction;\n```\n\n### limit\n\n```ts\ntype limit = (n: number) =\u003e PipeFunction;\n```\n\n## Output\n\n### toArray\n\n```ts\ntype toArray = \u003cT = any\u003e({ iterator: AsyncIterableIterator\u003cT\u003e, rid?: number }) =\u003e Promise\u003cT[]\u003e;\n```\n\n### find\n\n```ts\ntype find = \u003cT\u003e(\n  func: (d: T) =\u003e boolean,\n) =\u003e ({ iterator: AsyncIterableIterator\u003cT\u003e, rid?: number }) =\u003e\n  Promise\u003cT | undefined\u003e;\n```\n\n### reduce\n\n```ts\ntype reduce = \u003cA = any, B = any\u003e(\n  func: (r: B, d: A, i: number) =\u003e B,\n  start: B,\n) =\u003e ({\n  iterator: AsyncIterableIterator\u003cA\u003e,\n  rid?: number,\n}) =\u003e Promise\u003cB\u003e;\n```\n\n### toNdjsonStdout\n\n```ts\ntype toNdjsonStdout = \u003cT = any\u003e({\n  iterator: AsyncIterableIterator\u003cT\u003e,\n  rid?: number,\n}) =\u003e Promise\u003cvoid\u003e;\n```\n\n### toDsvStdout\n\n```ts\ntype toDsvStdout = \u003cT = any\u003e(\n  { iterator: AsyncIterableIterator\u003cT\u003e, rid?: number },\n  delimiter?: string,\n) =\u003e Promise\u003cvoid\u003e;\n```\n\n### toNdjsonFile\n\n```ts\ntype toNdjsonFile = \u003cT = any\u003e(\n  { iterator: AsyncIterableIterator\u003cT\u003e, rid?: number },\n  path: string,\n) =\u003e Promise\u003cvoid\u003e;\n```\n\n### toDsvFile\n\n```ts\ntype toDsvFile = \u003cT = any\u003e(\n  { iterator: AsyncIterableIterator\u003cT\u003e, rid?: number },\n  path: string,\n  delimiter?: string,\n) =\u003e Promise\u003cvoid\u003e;\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidris-maps%2Fdatastream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidris-maps%2Fdatastream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidris-maps%2Fdatastream/lists"}