{"id":50891284,"url":"https://github.com/imirfanul/csv-processor","last_synced_at":"2026-06-15T21:01:38.565Z","repository":{"id":246840424,"uuid":"822954919","full_name":"imirfanul/csv-processor","owner":"imirfanul","description":"A Node.js package for processing large CSV files in chunks, designed for modular and flexible use with any database or data handling system.","archived":false,"fork":false,"pushed_at":"2024-07-09T07:58:59.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-02T02:26:00.375Z","etag":null,"topics":["csv","csv-file-process","csv-parser"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/@imirfanul/csv-processor","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/imirfanul.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-02T06:44:58.000Z","updated_at":"2024-07-09T07:59:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"1c7b977d-dcae-4d2a-ab50-dfaf1f0f07cd","html_url":"https://github.com/imirfanul/csv-processor","commit_stats":null,"previous_names":["imirfanul/csv-processor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/imirfanul/csv-processor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imirfanul%2Fcsv-processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imirfanul%2Fcsv-processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imirfanul%2Fcsv-processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imirfanul%2Fcsv-processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/imirfanul","download_url":"https://codeload.github.com/imirfanul/csv-processor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imirfanul%2Fcsv-processor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34379915,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-15T02:00:07.085Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","csv-file-process","csv-parser"],"created_at":"2026-06-15T21:01:37.866Z","updated_at":"2026-06-15T21:01:38.560Z","avatar_url":"https://github.com/imirfanul.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CSV Processor\n\nA Node.js package for processing large CSV files in chunks, designed for modular and flexible use with any database or data handling system.\n\n## Installation\n\nInstall the package via npm:\n\n```bash\nnpm install @imirfanul/csv-processor\n```\n\n## Usage\n\n### Importing the Package\n\nFirst, import the necessary classes from the package:\n\n```typescript\nimport { CSVProcessor, CSVProcessorConfig, ChunkHandler } from '@imirfanul/csv-processor';\n```\n\n### Configuration\n\nConfigure the CSVProcessor with the desired chunk size and CSV file path:\n\n```typescript\nconst csvProcessorConfig: CSVProcessorConfig = {\n  chunkSize: 100000,\n  csvFilePath: 'path/to/your/large_file.csv',\n};\n\nconst csvProcessor = new CSVProcessor(csvProcessorConfig);\n```\n\n### Chunk Handler\n\nDefine a `chunkHandler` function that processes each chunk of data. This can be customized to perform any operation, such as inserting data into a database.\n\nExample with PostgreSQL:\n\n```typescript\nimport { Pool } from 'pg';\n\nconst dbConfig = {\n  user: 'username',\n  host: 'localhost',\n  database: 'your_database',\n  password: 'password',\n  port: 5432,\n};\n\nconst pool = new Pool(dbConfig);\n\nconst chunkHandler: ChunkHandler = async (chunk) =\u003e {\n  const client = await pool.connect();\n  try {\n    const values = chunk.map(row =\u003e `(${Object.values(row).map(val =\u003e `'${val}'`).join(',')})`).join(',');\n    const columns = Object.keys(chunk[0]).map(col =\u003e `\"${col}\"`).join(',');\n\n    const query = `INSERT INTO your_table (${columns}) VALUES ${values}`;\n    await client.query(query);\n    console.log(`Inserted ${chunk.length} rows`);\n  } catch (error) {\n    console.error('Error inserting chunk:', error);\n  } finally {\n    client.release();\n  }\n};\n```\n\n### Processing the CSV\n\nCall the `processCSV` method with the `chunkHandler`:\n\n```typescript\ncsvProcessor.processCSV(chunkHandler).then(() =\u003e {\n  console.log('CSV processing complete.');\n}).catch(error =\u003e {\n  console.error('Error during CSV processing:', error);\n});\n```\n\n## Example Project\n\nHere is a full example combining all the pieces together:\n\n```typescript\nimport { CSVProcessor, CSVProcessorConfig, ChunkHandler } from 'your-package-name';\nimport { Pool } from 'pg';\n\nconst dbConfig = {\n  user: 'username',\n  host: 'localhost',\n  database: 'your_database',\n  password: 'password',\n  port: 5432,\n};\n\nconst pool = new Pool(dbConfig);\n\nconst csvProcessorConfig: CSVProcessorConfig = {\n  chunkSize: 100000,\n  csvFilePath: 'path/to/your/large_file.csv',\n};\n\nconst csvProcessor = new CSVProcessor(csvProcessorConfig);\n\nconst chunkHandler: ChunkHandler = async (chunk) =\u003e {\n  const client = await pool.connect();\n  try {\n    const values = chunk.map(row =\u003e `(${Object.values(row).map(val =\u003e `'${val}'`).join(',')})`).join(',');\n    const columns = Object.keys(chunk[0]).map(col =\u003e `\"${col}\"`).join(',');\n\n    const query = `INSERT INTO your_table (${columns}) VALUES ${values}`;\n    await client.query(query);\n    console.log(`Inserted ${chunk.length} rows`);\n  } catch (error) {\n    console.error('Error inserting chunk:', error);\n  } finally {\n    client.release();\n  }\n};\n\ncsvProcessor.processCSV(chunkHandler).then(() =\u003e {\n  console.log('CSV processing complete.');\n}).catch(error =\u003e {\n  console.error('Error during CSV processing:', error);\n});\n```\n\n## License\n\nThis package is licensed under the MIT License. See the [LICENSE](LICENSE) file for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimirfanul%2Fcsv-processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimirfanul%2Fcsv-processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimirfanul%2Fcsv-processor/lists"}