{"id":23272534,"url":"https://github.com/sister-software/spliterator","last_synced_at":"2026-03-11T03:03:53.359Z","repository":{"id":267458284,"uuid":"901259926","full_name":"sister-software/spliterator","owner":"sister-software","description":"High-performance delimited data pipeline tools.","archived":false,"fork":false,"pushed_at":"2025-03-31T00:56:49.000Z","size":1555,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-21T04:46:09.547Z","etag":null,"topics":["data-science","iterators","streams","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sister-software.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-10T10:32:38.000Z","updated_at":"2025-03-31T00:56:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"14fe243b-be3b-4393-9ed6-af3f338cb99e","html_url":"https://github.com/sister-software/spliterator","commit_stats":{"total_commits":7,"total_committers":1,"mean_commits":7.0,"dds":0.0,"last_synced_commit":"4a7184ca174e1040ef8087388dea52082ac288bd"},"previous_names":["sister-software/ribbon","sister-software/spliterator"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/sister-software/spliterator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sister-software%2Fspliterator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sister-software%2Fspliterator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sister-software%2Fspliterator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sister-software%2Fspliterator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sister-software","download_url":"https://codeload.github.com/sister-software/spliterator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sister-software%2Fspliterator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30368634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T21:41:54.280Z","status":"online","status_checked_at":"2026-03-11T02:00:07.027Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","iterators","streams","typescript"],"created_at":"2024-12-19T19:17:48.313Z","updated_at":"2026-03-11T03:03:53.338Z","avatar_url":"https://github.com/sister-software.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spliterator 🎀\n\nSpliterator is a TypeScript library for streaming delimited content such as CSV, TSV and JSONL.\n\nLet's say you have a huge newline-delimited JSON file that can't fit into memory:\n\n```js\n{\"name\": \"Jessie\", \"age\": 30}\n{\"name\": \"Kelly\", \"age\": 40}\n{\"name\": \"Loren\", \"age\": 50}\n// Several hundred thousand more lines...\n```\n\nSpliterator can help you read it line-by-line without loading the entire file into memory:\n\n```ts\nimport { JSONSpliterator } from \"spliterator\"\n\ninterface Person {\n\tname: string\n\tage: number\n}\n\nconst reader = JSONSpliterator.fromAsync(\"example.jsonl\")\n\nfor await (const line of reader) {\n\tconsole.log(line) // {\"name\": \"Alice\", \"age\": 30}, etc.\n}\n```\n\n[![NPM Version](https://img.shields.io/npm/v/spliterator)](https://www.npmjs.com/package/spliterator)\n![NPM License](https://img.shields.io/npm/l/spliterator)\n\n# Installation\n\n```bash\nyarn add spliterator\n# or\nnpm install spliterator\n```\n\n# Usage\n\n## Character-delimited files\n\nWhile Spliterator supports any delimited byte stream, it's particularly useful for character-delimited content such as comma-separated values (CSV), tab-separated values (TSV) – or any other delimiter you can think of.\n\n```csv\nFull Name, Occupation, Age\nMorgan, Developer, 30\nNataly, Designer, 40\nOrlando, Manager, 50\n```\n\n```ts\nimport { CSVSpliterator } from \"spliterator\"\n\nconst reader = CSVSpliterator.fromAsync(\"people.csv\")\n\nfor await (const columns of reader) {\n\tconsole.log(columns) // [\"Full Name\", \"Occupation\", \"Age\"], [\"Morgan\", \"Developer\", 30], etc.\n}\n```\n\nCSV files can also be emitted as objects with headers as keys, with some quality-of-life features, such as normalizing property keys:\n\n```ts\nimport { CSVSpliterator } from \"spliterator\"\n\ninterface Person {\n\tfull_name: string\n\toccupation: string\n\tage: number\n}\n\nconst reader = CSVSpliterator.fromAsync\u003cPerson\u003e(\"people.csv\", { mode: \"object\" })\n\nfor await (const columns of reader) {\n\tconsole.log(columns) // { full_name: \"Morgan\", occupation: \"Developer\", age: 30 }, etc.\n}\n```\n\n## CLI Usage\n\nSpliterator also includes a CLI tool that can be used to stream delimited content from the command line, transform it, filter it, and more.\n\n```bash\nspliterator csv people.csv people.jsonl\n```\n\nThe CLI also supports reading from standard input:\n\n```bash\ncat people.csv | spliterator csv people.jsonl\n```\n\nFor information on all available commands, run `spliterator --help`.\n\n## Advanced Usage\n\nSpliterator includes a collection of low-level classes and interfaces that can be used to create custom generators for any kind of delimited content.\n\nFor more advanced usage, check out our tests in the `test` directory, or our fully-annotated source code.\n\n### Reading from a stream\n\nAll included Spliterators implement the `Generator` and `AsyncGenerator` interfaces, so you can use them in `for...of` and `for await...of` loops, as well the web-native [ReadableStreams](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream), so you can use them in `for await...of` loops, as well as piping them through transformations to avoid nested and partially materialized streams.\n\n```ts\nimport { JSONSpliterator } from \"spliterator\"\n\nconst people = [\n\t{ name: \"Alice\", age: 30 },\n\t{ name: \"Bob\", age: 40 },\n\t{ name: \"Charlie\", age: 50 },\n]\n\nconst generator = JSONSpliterator.from(people.map(JSON.stringify).join(\"\\n\"))\nconst stream = ReadableStream.from(generator)\n\nfor await (const line of stream) {\n\tconsole.log(line) // {\"name\": \"Alice\", \"age\": 30}, etc.\n}\n```\n\n### Custom generators\n\nWhile Spliterator includes premade exports for most use-cases, custom generators can be created via `Spliterator` and `AsyncSpliterator`. This class is a low-level interface that allows you to create your own generators for any kind of delimited content.\n\n# License\n\nSpliterator is licensed under the AGPL-3.0 license. Generally,\nthis means that you can use the software for free, but you must share\nany modifications you make to the software.\n\nFor more information on commercial usage licensing, please contact us at\n`hello@sister.software`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsister-software%2Fspliterator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsister-software%2Fspliterator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsister-software%2Fspliterator/lists"}