{"id":28996631,"url":"https://github.com/observedobserver/omnidata","last_synced_at":"2025-06-25T05:10:48.162Z","repository":{"id":296033204,"uuid":"992112565","full_name":"ObservedObserver/omnidata","owner":"ObservedObserver","description":"data loaders","archived":false,"fork":false,"pushed_at":"2025-05-28T17:04:24.000Z","size":103,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-05-28T17:47:11.486Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ObservedObserver.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-28T16:29:51.000Z","updated_at":"2025-05-28T17:04:28.000Z","dependencies_parsed_at":"2025-05-28T17:47:18.159Z","dependency_job_id":"ccd463fb-18e8-474c-bdbd-b2717f562749","html_url":"https://github.com/ObservedObserver/omnidata","commit_stats":null,"previous_names":["observedobserver/omnidata"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ObservedObserver/omnidata","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ObservedObserver%2Fomnidata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ObservedObserver%2Fomnidata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ObservedObserver%2Fomnidata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ObservedObserver%2Fomnidata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ObservedObserver","download_url":"https://codeload.github.com/ObservedObserver/omnidata/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ObservedObserver%2Fomnidata/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261808072,"owners_count":23212694,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-25T05:10:47.369Z","updated_at":"2025-06-25T05:10:48.137Z","avatar_url":"https://github.com/ObservedObserver.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# omnidata\n\nData loaders for common file formats. Supports CSV, Parquet, Avro, SQLite and XLSX.\n\n## Installation\n\n```bash\nnpm install omnidata\n```\n\n## Usage\n\n### CSV\n\n```typescript\nimport { parseCSV } from 'omnidata/csv';\n\nconst result = await parseCSV('name,age\\nJohn,30', { headers: true });\nconsole.log(result.rows);\n```\n\n### Parquet\n\n```typescript\nimport { parseParquet } from 'omnidata/parquet';\n\nconst data = await parseParquet('/path/to/file.parquet');\nconsole.log(data.rows);\n```\n\n### Avro\n\n```typescript\nimport { parseAvro } from 'omnidata/avro';\n\nconst data = await parseAvro('/path/to/file.avro');\nconsole.log(data.rows);\n```\n\n### SQLite\n\n```typescript\nimport { parseSQLite } from 'omnidata/sqlite';\n\nconst db = await parseSQLite('/path/to/file.sqlite');\nconsole.log(db.tables);\n```\n\n### XLSX\n\n```typescript\nimport { parseXLSX } from 'omnidata/xlsx';\n\nconst workbook = await parseXLSX('/path/to/file.xlsx');\nconsole.log(workbook.rows);\n```\n\n## API Reference\n\n### CSV\n\n| Function | Description |\n| --- | --- |\n| `parseCSVSimple(input, options?)` | Parse a small CSV file and return all rows |\n| `parseCSV(input, options?)` | Unified parser that auto-detects streaming |\n| `parseCSVStream(input, callbacks, options?)` | Streaming parser for large files |\n\n#### CSV Input\n\n- `string` - CSV text\n- `File` - browser file object\n- `string` - file path in Node.js\n\n#### CSV Options\n\n```ts\ninterface CSVParseOptions {\n  delimiter?: string;\n  quote?: string;\n  escape?: string;\n  skipEmptyLines?: boolean;\n  headers?: boolean | string[];\n  encoding?: string;\n  chunkSize?: number;\n}\n```\n\n#### CSV Callbacks\n\n```ts\ninterface CSVStreamCallbacks {\n  onRow?: (row: string[] | CSVRow, index: number) =\u003e void;\n  onHeaders?: (headers: string[]) =\u003e void;\n  onError?: (error: CSVError) =\u003e void;\n  onEnd?: (result: { totalRows: number; errors: CSVError[] }) =\u003e void;\n}\n```\n\n### Parquet\n\n| Function | Description |\n| --- | --- |\n| `parseParquetSimple(input, options?)` | Parse a Parquet file and return all rows |\n| `parseParquet(input, options?)` | Unified parser (non-streaming or streaming) |\n| `parseParquetStream(input, callbacks, options?)` | Streaming parser |\n\n#### Parquet Input\n\n- `ArrayBuffer` - binary Parquet data\n- `File` - browser file object\n- `string` - file path in Node.js\n\n#### Parquet Options\n\n```ts\ninterface ParquetParseOptions {\n  chunkSize?: number;\n  encoding?: string;\n}\n```\n\n#### Parquet Callbacks\n\n```ts\ninterface ParquetStreamCallbacks {\n  onRow?: (row: ParquetRow, index: number) =\u003e void;\n  onSchema?: (schema: ParquetSchema) =\u003e void;\n  onError?: (error: ParquetError) =\u003e void;\n  onEnd?: (result: { totalRows: number; errors: ParquetError[] }) =\u003e void;\n}\n```\n\n### Avro\n\n| Function | Description |\n| --- | --- |\n| `parseAvroSimple(input, options?)` | Parse an Avro file and return all rows |\n| `parseAvro(input, options?)` | Unified parser |\n| `parseAvroStream(input, callbacks, options?)` | Streaming parser |\n\n#### Avro Input\n\n- `ArrayBuffer` - binary Avro data\n- `File` - browser file object\n- `string` - file path in Node.js\n\n#### Avro Options\n\n```ts\ninterface AvroParseOptions {\n  chunkSize?: number;\n  encoding?: string;\n}\n```\n\n#### Avro Callbacks\n\n```ts\ninterface AvroStreamCallbacks {\n  onRow?: (row: AvroRow, index: number) =\u003e void;\n  onSchema?: (schema: AvroSchema) =\u003e void;\n  onError?: (error: AvroError) =\u003e void;\n  onEnd?: (result: { totalRows: number; errors: AvroError[] }) =\u003e void;\n}\n```\n\n### SQLite\n\n| Function | Description |\n| --- | --- |\n| `parseSQLiteSimple(input, options?)` | Parse an SQLite file and return all tables |\n| `parseSQLite(input, options?)` | Unified parser |\n| `parseSQLiteStream(input, callbacks, options?)` | Streaming parser |\n\n#### SQLite Input\n\n- `ArrayBuffer` - raw SQLite database bytes\n- `File` - browser file object\n- `string` - file path in Node.js\n\n#### SQLite Options\n\n```ts\ninterface SQLiteParseOptions {\n  encoding?: string;\n}\n```\n\n#### SQLite Callbacks\n\n```ts\ninterface SQLiteStreamCallbacks {\n  onTable?: (table: SQLiteTable, index: number) =\u003e void;\n  onError?: (error: SQLiteError) =\u003e void;\n  onEnd?: (result: { totalTables: number; errors: SQLiteError[] }) =\u003e void;\n}\n```\n\n### XLSX\n\n| Function | Description |\n| --- | --- |\n| `parseXLSXSimple(input, options?)` | Parse an XLSX file and return all rows |\n| `parseXLSX(input, options?)` | Unified parser |\n| `parseXLSXStream(input, callbacks, options?)` | Streaming parser |\n\n#### XLSX Input\n\n- `ArrayBuffer` - XLSX binary data\n- `File` - browser file object\n- `string` - file path in Node.js\n\n#### XLSX Options\n\n```ts\ninterface XLSXParseOptions {\n  sheet?: number | string;\n}\n```\n\n#### XLSX Callbacks\n\n```ts\ninterface XLSXStreamCallbacks {\n  onRow?: (row: XLSXRow, index: number) =\u003e void;\n  onError?: (error: XLSXError) =\u003e void;\n  onEnd?: (result: { totalRows: number; errors: XLSXError[] }) =\u003e void;\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobservedobserver%2Fomnidata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fobservedobserver%2Fomnidata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobservedobserver%2Fomnidata/lists"}