{"id":49549125,"url":"https://github.com/faizkhairi/flatfile-js","last_synced_at":"2026-05-02T21:04:28.809Z","repository":{"id":339663092,"uuid":"1162803613","full_name":"faizkhairi/flatfile-js","owner":"faizkhairi","description":"Schema-first parser and generator for pipe, comma, and tab-delimited flat files with type coercion and streaming","archived":false,"fork":false,"pushed_at":"2026-03-21T18:04:27.000Z","size":61,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-07T21:39:00.455Z","etag":null,"topics":["csv","flat-file","npm-package","parser","typescript"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/@faizkhairi/flatfile-js","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/faizkhairi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-20T18:02:35.000Z","updated_at":"2026-03-21T17:58:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/faizkhairi/flatfile-js","commit_stats":null,"previous_names":["faizkhairi/flatfile-js"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/faizkhairi/flatfile-js","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faizkhairi%2Fflatfile-js","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faizkhairi%2Fflatfile-js/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faizkhairi%2Fflatfile-js/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faizkhairi%2Fflatfile-js/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/faizkhairi","download_url":"https://codeload.github.com/faizkhairi/flatfile-js/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faizkhairi%2Fflatfile-js/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32549388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T19:18:06.202Z","status":"ssl_error","status_checked_at":"2026-05-02T19:16:21.335Z","response_time":132,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","flat-file","npm-package","parser","typescript"],"created_at":"2026-05-02T21:04:27.967Z","updated_at":"2026-05-02T21:04:28.797Z","avatar_url":"https://github.com/faizkhairi.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# @faizkhairi/flatfile-js\n\nSchema-first parser and generator for pipe, comma, and tab-delimited flat files — with type coercion, validation, and streaming support.\n\n[![CI](https://github.com/faizkhairi/flatfile-js/actions/workflows/ci.yml/badge.svg)](https://github.com/faizkhairi/flatfile-js/actions/workflows/ci.yml)\n[![npm](https://img.shields.io/npm/v/@faizkhairi/flatfile-js)](https://www.npmjs.com/package/@faizkhairi/flatfile-js)\n\n---\n\n## Why @faizkhairi/flatfile-js?\n\nEnterprise data exchange often uses pipe-delimited flat files (bank statements, government B2B integrations). Parsing them correctly means:\n\n- Trimming whitespace and coercing types\n- Collecting errors per-field without aborting the whole file\n- Streaming multi-GB files without loading them into memory\n\n`@faizkhairi/flatfile-js` handles all of this with a single schema definition.\n\n---\n\n## Install\n\n```bash\nnpm install @faizkhairi/flatfile-js\n```\n\nZero dependencies. Works in Node.js 18+ and modern browsers.\n\n---\n\n## Quick Start\n\n```typescript\nimport { createSchema, parseFlat, stringifyFlat } from '@faizkhairi/flatfile-js'\n\n// 1. Define your schema once\nconst schema = createSchema({\n  delimiter: '|',\n  hasHeader: false,\n  fields: [\n    { name: 'id',     type: 'number',  position: 0 },\n    { name: 'name',   type: 'string',  position: 1, required: true },\n    { name: 'salary', type: 'decimal', position: 2, decimalPlaces: 2 },\n    { name: 'dob',    type: 'date',    position: 3, format: 'YYYYMMDD' },\n    { name: 'active', type: 'boolean', position: 4 },\n  ],\n})\n\n// 2. Parse a flat file string\nconst fileContent = `\n1|Alice Smith|75000.50|19850315|1\n2|Bob Jones|82000.00|19901122|1\n3|Carol White|91500.75|19781005|0\n`.trim()\n\nconst { records, errors } = parseFlat(fileContent, schema)\n\nif (errors.length \u003e 0) {\n  console.error('Parse errors:', errors)\n}\n\nrecords.forEach(record =\u003e {\n  // record.id     → number\n  // record.name   → string\n  // record.salary → number (precision-controlled decimal)\n  // record.dob    → Date object\n  // record.active → boolean\n  console.log(record)\n})\n\n// 3. Generate a flat file from records\nconst output = stringifyFlat(records, schema)\n// → '1|Alice Smith|75000.50|19850315|1\\n2|...'\n```\n\n---\n\n## Streaming Large Files\n\nFor files too large to load into memory, use `parseStream`. It yields one typed record at a time with no error collection (optimized for throughput):\n\n```typescript\nimport { createReadStream } from 'node:fs'\nimport { Readable } from 'node:stream'\nimport { parseStream } from '@faizkhairi/flatfile-js'\n\n// Convert Node.js stream to Web ReadableStream\nconst nodeStream = createReadStream('large-file.dat')\nconst webStream = Readable.toWeb(nodeStream)\n\nfor await (const record of parseStream(webStream, schema)) {\n  await db.insert(record)\n}\n```\n\n\u003e For error diagnostics, use `parseFlat` instead — `parseStream` prioritizes throughput.\n\n---\n\n## API Reference\n\n### `createSchema(config)`\n\nValidates and normalizes a schema at creation time (fail-fast). Sorts fields by position.\n\n```typescript\nconst schema = createSchema({\n  delimiter: '|',       // Required. Common: '|', ',', '\\t'\n  fields: [...],        // Required. At least one field.\n  hasHeader?: false,    // Default: false. If true, first line is skipped on parse.\n  lineEnding?: 'auto',  // 'LF' | 'CRLF' | 'auto'. Default: 'auto'.\n})\n```\n\nThrows if: delimiter is empty, no fields, duplicate positions, or duplicate names.\n\n---\n\n### `parseFlat(content, schema)`\n\nParses a flat file string into typed records. Returns `{ records, errors }`.\n\n- Records with field errors are **still included** — failed fields are set to `null`.\n- Empty optional fields → `null` (no error).\n- Empty required fields → error collected, field set to `null`.\n\n```typescript\nconst { records, errors } = parseFlat(content, schema)\n// records: Record\u003cstring, unknown\u003e[]\n// errors:  ParseError[]\n```\n\n**ParseError shape:**\n```typescript\n{\n  line:     number   // 1-indexed line number\n  field:    string   // field name\n  position: number   // 0-indexed column position\n  message:  string   // human-readable error\n  raw:      string   // the raw string value that caused the error\n}\n```\n\n---\n\n### `stringifyFlat(records, schema)`\n\nSerializes typed records back to a flat file string.\n\n```typescript\nconst output = stringifyFlat(records, schema)\nfs.writeFileSync('output.dat', output)\n```\n\n- `null` / `undefined` → empty string\n- `number` → `Math.round()` (integer)\n- `decimal` → `.toFixed(decimalPlaces)`\n- `date` → formatted per `field.format` (default: ISO 8601)\n- `boolean` → `trueValue ?? '1'` or `falseValue ?? '0'`\n\n---\n\n### `parseStream(stream, schema)`\n\nAsync generator that yields one record per line. No error collection.\n\n```typescript\nfor await (const record of parseStream(webReadableStream, schema)) {\n  process(record)\n}\n```\n\n---\n\n## Field Types\n\n| Type | Input Example | Output |\n|------|--------------|--------|\n| `string` | `' Alice '` | `'Alice'` (trimmed) |\n| `number` | `'1001'` | `1001` |\n| `decimal` | `'75000.50'` | `75000.5` (precision float) |\n| `date` | `'19850315'` | `Date` object (UTC) |\n| `boolean` | `'1'` / `'yes'` | `true` |\n\n### Date Formats\n\n| Format | Example Input |\n|--------|--------------|\n| `YYYYMMDD` | `19850315` |\n| `DD/MM/YYYY` | `15/03/1985` |\n| `MM/DD/YYYY` | `03/15/1985` |\n| `YYYY-MM-DD` | `1985-03-15` |\n| `ISO` (default) | `1985-03-15T00:00:00.000Z` |\n\nAll dates are parsed and stored in **UTC** to avoid timezone drift.\n\n### Boolean Defaults\n\n| True | False |\n|------|-------|\n| `true`, `1`, `y`, `yes` | `false`, `0`, `n`, `no` |\n\nCase-insensitive. Override with `trueValue` / `falseValue` per field.\n\n---\n\n## Error Handling Strategy\n\n`parseFlat` uses a **collect-and-continue** approach designed for enterprise data processing:\n\n```typescript\nconst { records, errors } = parseFlat(content, schema)\n\n// records always has one entry per non-empty line\n// failed fields are null, not omitted\nrecords.forEach((record, i) =\u003e {\n  if (record.salary === null) {\n    console.warn(`Line ${i + 1}: salary could not be parsed`)\n  }\n  db.insert(record) // still processable as partial data\n})\n```\n\nThis matches how enterprise ETL systems work — you want to see *all* the data and *all* the errors together, not halt at the first failure.\n\n---\n\n## License\n\nMIT © [Faiz Khairi](https://github.com/faizkhairi)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffaizkhairi%2Fflatfile-js","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffaizkhairi%2Fflatfile-js","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffaizkhairi%2Fflatfile-js/lists"}