{"id":15029431,"url":"https://github.com/keyang/node-csvtojson","last_synced_at":"2025-05-13T20:23:18.647Z","repository":{"id":8994324,"uuid":"10744004","full_name":"Keyang/node-csvtojson","owner":"Keyang","description":"Blazing fast and Comprehensive CSV Parser for Node.JS / Browser / Command Line. ","archived":false,"fork":false,"pushed_at":"2023-07-18T21:57:32.000Z","size":6156,"stargazers_count":2024,"open_issues_count":126,"forks_count":270,"subscribers_count":34,"default_branch":"master","last_synced_at":"2025-04-28T11:58:07.714Z","etag":null,"topics":["csv","csv-columns","csv-data","csv-row","csv-stream","csvparser","csvtojson","javascript","nodejs","parse"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Keyang.png","metadata":{"files":{"readme":"readme-old.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2013-06-17T18:29:56.000Z","updated_at":"2025-03-28T14:33:12.000Z","dependencies_parsed_at":"2023-01-13T15:06:26.041Z","dependency_job_id":"4baaad97-4200-4298-b448-4f8956aed55f","html_url":"https://github.com/Keyang/node-csvtojson","commit_stats":{"total_commits":314,"total_committers":60,"mean_commits":5.233333333333333,"dds":"0.37898089171974525","last_synced_commit":"89a9a36cf1be8fb2da93359b2a4d1d6765a7adf7"},"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Keyang%2Fnode-csvtojson","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Keyang%2Fnode-csvtojson/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Keyang%2Fnode-csvtojson/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Keyang%2Fnode-csvtojson/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Keyang","download_url":"https://codeload.github.com/Keyang/node-csvtojson/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251311332,"owners_count":21569008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","csv-columns","csv-data","csv-row","csv-stream","csvparser","csvtojson","javascript","nodejs","parse"],"created_at":"2024-09-24T20:10:39.428Z","updated_at":"2025-04-28T11:58:13.710Z","avatar_url":"https://github.com/Keyang.png","language":"TypeScript","readme":"[![Build Status](https://travis-ci.org/Keyang/node-csvtojson.svg?branch=master)](https://travis-ci.org/Keyang/node-csvtojson)\n\n# CSVTOJSON\nAll you need nodejs csv to json converter.\n* Large CSV data\n* Command Line Tool and Node.JS Lib\n* Complex/nested JSON\n* Easy Customised Parser\n* Stream based\n* multi CPU core support\n* Easy Usage\n* more!\n\n# Demo\n\n[Here](http://keyangxiang.com/csvtojson/) is a free online csv to json service ultilising latest csvtojson module.\n\n## Menu\n* [Installation](#installation)\n* [Usage](#usage)\n  * [Library](#library)\n    * [Convert from a file](#from-file)\n    * [Convert from a web resource / Readable stream](#from-web)\n    * [Convert from CSV string](#from-string)\n* [Parameters](#params)\n* [Result Transform](#result-transform)\n  * [Synchronouse Transformer](#synchronouse-transformer)\n  * [Asynchronouse Transformer](#asynchronouse-transformer)\n  * [Convert to other data type](#convert-to-other-data-type)\n* [Hooks](#hooks)\n* [Events](#events)\n* [Flags](#flags)\n* [Big CSV File Streaming](#big-csv-file)\n* [Process Big CSV File in CLI](#convert-big-csv-file-with-command-line-tool)\n* [Parse String](#parse-string)\n* [Empowered JSON Parser](#empowered-json-parser)\n* [Field Type](#field-type)\n* [Multi-Core / Fork Process](#multi-cpu-core)\n* [Header Configuration](#header-configuration)\n* [Error Handling](#error-handling)\n* [Customised Parser](#parser)\n* [Stream Options](#stream-options)\n* [Change Log](#change-log)\n\nGitHub: https://github.com/Keyang/node-csvtojson\n\n## Installation\n\n\u003enpm install -g csvtojson\n\n\u003enpm install csvtojson --save\n\n## Usage\n\n### library\n\n#### From File\n\nYou can use File stream\n\n```js\n//Converter Class\nvar Converter = require(\"csvtojson\").Converter;\nvar converter = new Converter({});\n\n//end_parsed will be emitted once parsing finished\nconverter.on(\"end_parsed\", function (jsonArray) {\n   console.log(jsonArray); //here is your result jsonarray\n});\n\n//read from file\nrequire(\"fs\").createReadStream(\"./file.csv\").pipe(converter);\n```\n\nOr use fromFile convenient function\n\n```js\n//Converter Class\nvar Converter = require(\"csvtojson\").Converter;\nvar converter = new Converter({});\nconverter.fromFile(\"./file.csv\",function(err,result){\n\n});\n```\n\n#### From Web\n\nTo convert any CSV data from readable stream just simply pipe in the data.\n\n```js\n//Converter Class\nvar Converter = require(\"csvtojson\").Converter;\nvar converter = new Converter({constructResult:false}); //for big csv data\n\n//record_parsed will be emitted each csv row being processed\nconverter.on(\"record_parsed\", function (jsonObj) {\n   console.log(jsonObj); //here is your result json object\n});\n\nrequire(\"request\").get(\"http://csvwebserver\").pipe(converter);\n\n```\n\n#### From String\n\n```js\nvar Converter = require(\"csvtojson\").Converter;\nvar converter = new Converter({});\nconverter.fromString(csvString, function(err,result){\n  //your code here\n});\n```\n\n### Command Line Tools\n\n\u003ecsvtojson \u003ccsv file path\u003e\n\nExample\n\n\u003ecsvtojson ./myCSVFile \u003coption1=value\u003e\n\nOr use pipe:\n\n\u003ecat myCSVFile | csvtojson\n\nCheck current version:\n\n\u003ecsvtojson version\n\nAdvanced usage with parameters support, check help:\n\n\u003ecsvtojson --help\n\n# Params\n\nThe constructor of csv Converter allows parameters:\n\n```js\nvar converter=new require(\"csvtojson\").Converter({\n  constructResult:false,\n  workerNum:4,\n  noheader:true\n});\n```\n\nFollowing parameters are supported:\n\n* **constructResult**: true/false. Whether to construct final json object in memory which will be populated in \"end_parsed\" event. Set to false if deal with huge csv data. default: true.\n* **delimiter**: delimiter used for seperating columns. Use \"auto\" if delimiter is unknown in advance, in this case, delimiter will be auto-detected (by best attempt). Use an array to give a list of potential delimiters e.g. [\",\",\"|\",\"$\"]. default: \",\"\n* **quote**: If a column contains delimiter, it is able to use quote character to surround the column content. e.g. \"hello, world\" wont be split into two columns while parsing. Set to \"off\" will ignore all quotes. default: \" (double quote)\n* **trim**: Indicate if parser trim off spaces surrounding column content. e.g. \"  content  \" will be trimmed to \"content\". Default: true\n* **checkType**: This parameter turns on and off weather check field type. default is true. See [Field type](#field-type)\n* **toArrayString**: Stringify the stream output to JSON array. This is useful when pipe output to a file which expects stringified JSON array. default is false and only stringified JSON (without []) will be pushed to downstream.\n* **ignoreEmpty**: Ignore the empty value in CSV columns. If a column value is not giving, set this to true to skip them. Defalut: false.\n* **workerNum**: Number of worker processes. The worker process will use multi-cores to help process CSV data. Set to number of Core to improve the performance of processing large csv file. Keep 1 for small csv files. Default 1.\n* **fork(Deprecated, same as workerNum=2)**: Use another CPU core to process the CSV stream.\n* **noheader**:Indicating csv data has no header row and first row is data row. Default is false. See [header configuration](#header-configuration)\n* **headers**: An array to specify the headers of CSV data. If --noheader is false, this value will override CSV header row. Default: null. Example: [\"my field\",\"name\"]. See [header configuration](#header-configuration)\n* **flatKeys**: Don't interpret dots (.) and square brackets in header fields as nested object or array identifiers at all (treat them like regular characters for JSON field identifiers). Default: false.\n* **maxRowLength**: the max character a csv row could have. 0 means infinite. If max number exceeded, parser will emit \"error\" of \"row_exceed\". if a possibly corrupted csv data provided, give it a number like 65535 so the parser wont consume memory. default: 0\n* **checkColumn**: whether check column number of a row is the same as headers. If column number mismatched headers number, an error of \"mismatched_column\" will be emitted.. default: false\n* **eol**: End of line character. If omitted, parser will attempt retrieve it from first chunk of CSV data. If no valid eol found, then operation system eol will be used.\n* **escape**: escape character used in quoted column. Default is double quote (\") according to RFC4108. Change to back slash (\\) or other chars for your own case.\n\nAll parameters can be used in Command Line tool. see\n\n```\ncsvtojson --help\n```\n\n# Result Transform\n\nTo transform JSON result, (e.g. change value of one column), just simply add 'transform handler'.\n\n## Synchronouse transformer\n\n```js\nvar Converter=require(\"csvtojson\").Converter;\nvar csvConverter=new Converter({});\ncsvConverter.transform=function(json,row,index){\n    json[\"rowIndex\"]=index;\n    /* some other examples:\n    delete json[\"myfield\"]; //remove a field\n    json[\"dateOfBirth\"]=new Date(json[\"dateOfBirth\"]); // convert a field type\n    */\n};\ncsvConverter.fromString(csvString,function(err,result){\n  //all result rows will add a field 'rowIndex' indicating the row number of the csv data:\n  /*\n  [{\n    field1:value1,\n    rowIndex: 0\n }]\n  */\n});\n```\n\nAs shown in example above, it is able to apply any changes to the result json which will be pushed to down stream and \"record_parsed\" event.\n\n## Asynchronouse Transformer\n\nAsynchronouse transformation can be achieve either through \"record_parsed\" event or creating a Writable stream.\n\n### Use record_parsed\n\nTo transform data asynchronously, it is suggested to use csvtojson with [Async Queue](https://github.com/caolan/async#queue).\n\nThis mainly is used when transformation of each csv row needs be mashed with data retrieved from external such as database / server / file system.\n\nHowever this approach will **not** change the json result pushed to downstream.\n\nHere is an example:\n\n```js\nvar Conv=require(\"csvtojson\").Converter;\nvar async=require(\"async\");\nvar rs=require(\"fs\").createReadStream(\"path/to/csv\"); // or any readable stream to csv data.\nvar q=async.queue(function(json,callback){\n  //process the json asynchronously.\n  require(\"request\").get(\"http://myserver/user/\"+json.userId,function(err,user){\n    //do the data mash here\n    json.user=user;\n    callback();\n  });\n},10);//10 concurrent worker same time\nq.saturated=function(){\n  rs.pause(); //if queue is full, it is suggested to pause the readstream so csvtojson will suspend populating json data. It is ok to not to do so if CSV data is not very large.\n}\nq.empty=function(){\n  rs.resume();//Resume the paused readable stream. you may need check if the readable stream isPaused() (this is since node 0.12) or finished.\n}\nvar conv=new Conv({construct:false});\nconv.transform=function(json){\n  q.push(json);\n};\nconv.on(\"end_parsed\",function(){\n  q.drain=function(){\n    //code when Queue process finished.\n  }\n})\nrs.pipe(conv);\n```\n\nIn example above, the transformation will happen if one csv rown being processed. The related user info will be pulled from a web server and mashed into json result.\n\nThere will be at most 10 data transformation woker working concurrently with the help of Async Queue.\n\n### Use Stream\n\nIt is able to create a Writable stream (or Transform) which process data asynchronously. See [Here](https://nodejs.org/dist/latest-v4.x/docs/api/stream.html#stream_class_stream_transform) for more details.\n\n## Convert to other data type\n\nBelow is an example of result tranformation which converts csv data to a column array rather than a JSON.\n\n```js\nvar Converter=require(\"csvtojson\").Converter;\nvar columArrData=__dirname+\"/data/columnArray\";\nvar rs=fs.createReadStream(columArrData);\nvar result = {}\nvar csvConverter=new Converter();\n//end_parsed will be emitted once parsing finished\ncsvConverter.on(\"end_parsed\", function(jsonObj) {\n    console.log(result);\n    console.log(\"Finished parsing\");\n    done();\n});\n\n//record_parsed will be emitted each time a row has been parsed.\ncsvConverter.on(\"record_parsed\", function(resultRow, rawRow, rowIndex) {\n\n    for (var key in resultRow) {\n        if (!result[key] || !result[key] instanceof Array) {\n            result[key] = [];\n        }\n        result[key][rowIndex] = resultRow[key];\n    }\n\n});\nrs.pipe(csvConverter);\n```\n\nHere is an example:\n\n```csv\n    TIMESTAMP,UPDATE,UID,BYTES SENT,BYTES RCVED\n    1395426422,n,10028,1213,5461\n    1395426422,n,10013,9954,13560\n    1395426422,n,10109,221391500,141836\n    1395426422,n,10007,53448,308549\n    1395426422,n,10022,15506,72125\n```\n\nIt will be converted to:\n\n```json\n{\n  \"TIMESTAMP\": [\"1395426422\", \"1395426422\", \"1395426422\", \"1395426422\", \"1395426422\"],\n  \"UPDATE\": [\"n\", \"n\", \"n\", \"n\", \"n\"],\n  \"UID\": [\"10028\", \"10013\", \"10109\", \"10007\", \"10022\"],\n  \"BYTES SENT\": [\"1213\", \"9954\", \"221391500\", \"53448\", \"15506\"],\n  \"BYTES RCVED\": [\"5461\", \"13560\", \"141836\", \"308549\", \"72125\"]\n}\n```\n\n# Hooks\n## preProcessRaw\nThis hook is called when parser received any data from upper stream and allow developers to change it. e.g.\n```js\n/*\nCSV data:\na,b,c,d,e\n12,e3,fb,w2,dd\n*/\n\nvar conv=new Converter();\nconv.preProcessRaw=function(data,cb){\n    //change all 12 to 23\n    cb(data.replace(\"12\",\"23\"));\n}\nconv.fromString(csv,function(err,json){\n  //json:{a:23 ....}\n})\n```\nBy default, the preProcessRaw just returns the data from the source\n```js\nConverter.prototype.preProcessRaw=function(data,cb){\n  cb(data);\n}\n```\nIt is also very good to sanitise/prepare the CSV data stream.\n```js\nvar headWhiteSpaceRemoved=false;\nconv.preProcessRaw=function(data,cb){\n    if (!headWhiteSpaceRemoved){\n      data=data.replace(/^\\s+/,\"\");\n      cb(data);\n    }else{\n      cb(data);\n    }\n}\n```\n## preProcessLine\nthis hook is called when a file line is emitted. It is called with two parameters `fileLineData,lineNumber`. The `lineNumber` is starting from 1.\n```js\n/*\nCSV data:\na,b,c,d,e\n12,e3,fb,w2,dd\n*/\n\nvar conv=new Converter();\nconv.preProcessLine=function(line,lineNumber){\n    //only change 12 to 23 for line 2\n    if (lineNumber === 2){\n      line=line.replace(\"12\",\"23\");\n    }\n    return line;\n}\nconv.fromString(csv,function(err,json){\n  //json:{a:23 ....}\n})\n```\nNotice that preProcessLine does not support async changes not like preProcessRaw hook.\n\n\n# Events\n\nFollowing events are used for Converter class:\n\n* end_parsed: It is emitted when parsing finished. the callback function will contain the JSON object if constructResult is set to true.\n* record_parsed: it is emitted each time a row has been parsed. The callback function has following parameters: result row JSON object reference, Original row array object reference, row index of current row in csv (header row does not count, first row content will start from 0)\n\nTo subscribe the event:\n\n```js\n//Converter Class\nvar Converter=require(\"csvtojson\").Converter;\n\n//end_parsed will be emitted once parsing finished\ncsvConverter.on(\"end_parsed\",function(jsonObj){\n    console.log(jsonObj); //here is your result json object\n});\n\n//record_parsed will be emitted each time a row has been parsed.\ncsvConverter.on(\"record_parsed\",function(resultRow,rawRow,rowIndex){\n   console.log(resultRow); //here is your result json object\n});\n```\n\n# Flags\n\nThere are flags in the library:\n\n\\*omit\\*: Omit a column. The values in the column will not be built into JSON result.\n\n\\*flat\\*: Mark a head column as is the key of its JSON result.\n\nExample:\n\n```csv\n*flat*user.name, user.age, *omit*user.gender\nJoe , 40, Male\n```\n\nIt will be converted to:\n\n```js\n[{\n  \"user.name\":\"Joe\",\n  \"user\":{\n    \"age\":40\n  }\n}]\n```\n\n# Big CSV File\ncsvtojson library was designed to accept big csv file converting. To avoid memory consumption, it is recommending to use read stream and write stream.\n\n```js\nvar Converter=require(\"csvtojson\").Converter;\nvar csvConverter=new Converter({constructResult:false}); // The parameter false will turn off final result construction. It can avoid huge memory consumption while parsing. The trade off is final result will not be populated to end_parsed event.\n\nvar readStream=require(\"fs\").createReadStream(\"inputData.csv\");\n\nvar writeStream=require(\"fs\").createWriteStream(\"outpuData.json\");\n\nreadStream.pipe(csvConverter).pipe(writeStream);\n```\n\nThe constructResult:false will tell the constructor not to combine the final result which would drain the memory as progressing. The output is piped directly to writeStream.\n\n# Convert Big CSV File with Command line tool\ncsvtojson command line tool supports streaming in big csv file and stream out json file.\n\nIt is very convenient to process any kind of big csv file. It's proved having no issue to proceed csv files over 3,000,000 lines (over 500MB) with memory usage under 30MB.\n\nOnce you have installed [csvtojson](#installation), you could use the tool with command:\n\n```\ncsvtojson [path to bigcsvdata] \u003e converted.json\n```\n\nOr if you prefer streaming data in from another application:\n\n```\ncat [path to bigcsvdata] | csvtojson \u003e converted.json\n```\n\nThey will do the same job.\n\n\n\n# Parse String\nTo parse a string, simply call fromString(csvString,callback) method. The callback parameter is optional.\n\nFor example:\n\n```js\nvar testData=__dirname+\"/data/testData\";\nvar data=fs.readFileSync(testData).toString();\nvar csvConverter=new CSVConverter();\n\n//end_parsed will be emitted once parsing finished\ncsvConverter.on(\"end_parsed\", function(jsonObj) {\n    //final result poped here as normal.\n});\ncsvConverter.fromString(data,function(err,jsonObj){\n    if (err){\n      //err handle\n    }\n    console.log(jsonObj);\n});\n\n```\n\n# Empowered JSON Parser\n\n*Note: If you want to maintain the original CSV data header values as JSON keys \"as is\" without being\ninterpreted as (complex) JSON structures you can set the option `--flatKeys=true`.*\n\nSince version 0.3.8, csvtojson now can replicate any complex JSON structure.\nAs we know, JSON object represents a graph while CSV is only 2-dimension data structure (table).\nTo make JSON and CSV containing same amount information, we need \"flatten\" some information in JSON.\n\nHere is an example. Original CSV:\n\n```csv\nfieldA.title, fieldA.children[0].name, fieldA.children[0].id,fieldA.children[1].name, fieldA.children[1].employee[].name,fieldA.children[1].employee[].name, fieldA.address[],fieldA.address[], description\nFood Factory, Oscar, 0023, Tikka, Tim, Joe, 3 Lame Road, Grantstown, A fresh new food factory\nKindom Garden, Ceil, 54, Pillow, Amst, Tom, 24 Shaker Street, HelloTown, Awesome castle\n\n```\nThe data above contains nested JSON including nested array of JSON objects and plain texts.\n\nUsing csvtojson to convert, the result would be like:\n\n```json\n[{\n    \"fieldA\": {\n        \"title\": \"Food Factory\",\n        \"children\": [{\n            \"name\": \"Oscar\",\n            \"id\": \"0023\"\n        }, {\n            \"name\": \"Tikka\",\n            \"employee\": [{\n                \"name\": \"Tim\"\n            }, {\n                \"name\": \"Joe\"\n            }]\n        }],\n        \"address\": [\"3 Lame Road\", \"Grantstown\"]\n    },\n    \"description\": \"A fresh new food factory\"\n}, {\n    \"fieldA\": {\n        \"title\": \"Kindom Garden\",\n        \"children\": [{\n            \"name\": \"Ceil\",\n            \"id\": \"54\"\n        }, {\n            \"name\": \"Pillow\",\n            \"employee\": [{\n                \"name\": \"Amst\"\n            }, {\n                \"name\": \"Tom\"\n            }]\n        }],\n        \"address\": [\"24 Shaker Street\", \"HelloTown\"]\n    },\n    \"description\": \"Awesome castle\"\n}]\n```\n\nHere is the rule for CSV data headers:\n\n* Use dot(.) to represent nested JSON. e.g. field1.field2.field3 will be converted to {field1:{field2:{field3:\u003c value \u003e}}}\n* Use square brackets([]) to represent an Array. e.g. field1.field2[\u003c index \u003e] will be converted to {field1:{field2:[\u003c values \u003e]}}. Different column with same header name will be added to same array.\n* Array could contain nested JSON object. e.g. field1.field2[\u003c index \u003e].name will be converted to {field1:{field2:[{name:\u003c value \u003e}]}}\n* The index could be omitted in some situation. However it causes information lost. Therefore Index should **NOT** be omitted if array contains JSON objects with more than 1 field (See example above fieldA.children[1].employee field, it is still ok if child JSON contains only 1 field).\n\nSince 0.3.8, JSON parser is the default parser. It does not need to add \"\\*json\\*\" to column titles. Theoretically, the JSON parser now should have functionality of \"Array\" parser, \"JSONArray\" parser, and old \"JSON\" parser.\n\nThis mainly purposes on the next few versions where csvtojson could convert a JSON object back to CSV format without losing information.\nIt can be used to process JSON data exported from no-sql database like MongoDB.\n\n# Field Type\n\nFrom version 0.3.14, type of fields are supported by csvtojson.\nThe parameter checkType is used to whether to check and convert the field type.\nSee [here](#params) for the parameter usage.\n\nThank all who have contributed to ticket [#20](https://github.com/Keyang/node-csvtojson/issues/20).\n\n## Implict Type\n\nWhen checkType is turned on, parser will try to convert value to its implicit type if it is not explicitly specified.\n\nFor example, csv data:\n```csv\nname, age, married, msg\nTom, 12, false, {\"hello\":\"world\",\"total\":23}\n\n```\nWill be converted into:\n\n```json\n{\n  \"name\":\"Tom\",\n  \"age\":12,\n  \"married\":false,\n  \"msg\":{\n    \"hello\":\"world\",\n    \"total\":\"23\"\n  }\n}\n```\nIf checkType is turned **OFF**, it will be converted to:\n\n```json\n{\n  \"name\":\"Tom\",\n  \"age\":\"12\",\n  \"married\":\"false\",\n  \"msg\":\"{\\\"hello\\\":\\\"world\\\",\\\"total\\\":23}\"\n}\n```\n\n\n## Explicit Type\nCSV header column can explicitly define the type of the field.\nSimply add type before column name with a hash and exclaimation (#!).\n\n### Supported types:\n* string\n* number\n\n### Define Type\nTo define the field type, see following example\n\n```csv\nstring#!appNumber, string#!finished, *flat*string#!user.msg, unknown#!msg\n201401010002, true, {\"hello\":\"world\",\"total\":23},a message\n```\nThe data will be converted to:\n\n```json\n{\n  \"appNumber\":\"201401010002\",\n  \"finished\":\"true\",\n  \"user.msg\":\"{\\\"hello\\\":\\\"world\\\",\\\"total\\\":23}\"\n}\n```\n\n## Multi-CPU (Core)\nSince version 0.4.0, csvtojson supports multiple CPU cores to process large csv files.\nThe implementation and benchmark result can be found [here](http://keyangxiang.com/2015/06/11/node-js-multi-core-programming-pracitse/).\n\nTo enable multi-core, just pass the worker number as parameter of constructor:\n\n```js\n  var Converter=require(\"csvtojson\").Converter;\n  var converter=new Converter({\n      workerNum:2 //use two cores\n  });\n```\nThe minimum worker number is 1. When worker number is larger than 1, the parser will balance the job load among workers.\n\nFor command line, to use worker just use ```--workerNum``` argument:\n\n```\ncsvtojson --workerNum=3 ./myfile.csv\n```\n\nIt is worth to mention that for small size of CSV file it actually costs more time to create processes and keep the communication between them. Therefore, use less workers for small CSV files.\n\n### Fork Process (Deprecated since 0.5.0)\n*Node.JS is running on single thread. You will not want to convert a large csv file on the same process where your node.js webserver is running. csvtojson gives an option to fork the whole conversion process to a new system process while the origin process will only pipe the input and result in and out. It very simple to enable this feature:\n\n```js\nvar Converter=require(\"csvtojson\").Converter;\n  var converter=new Converter({\n      fork:true //use child process to convert\n  });\n```\nSame as multi-workers, fork a new process will cause extra cost on process communication and life cycle management. Use it wisely.*\n\nSince 0.5.0, fork=true is the same as workerNum=2.\n\n### Header configuration\n\nCSV header row can be configured programmatically.\n\nthe *noheader* parameter indicate if first row of csv is header row or not. e.g. CSV data:\n\n```\nCC102-PDMI-001,eClass_5.1.3,10/3/2014,12,40,green,40\nCC200-009-001,eClass_5.1.3,11/3/2014,5,3,blue,38,extra field!\n```\n\nWith noheader=true\n\n```\ncsvtojson  ./test/data/noheadercsv  --noheader=true\n```\n\nwe can get following result:\n\n```json\n[\n{\"field1\":\"CC102-PDMI-001\",\"field2\":\"eClass_5.1.3\",\"field3\":\"10/3/2014\",\"field4\":\"12\",\"field5\":\"40\",\"field6\":\"green\",\"field7\":\"40\"},\n{\"field1\":\"CC200-009-001\",\"field2\":\"eClass_5.1.3\",\"field3\":\"11/3/2014\",\"field4\":\"5\",\"field5\":\"3\",\"field6\":\"blue\",\"field7\":\"38\",\"field8\":\"extra field!\"}\n]\n```\n\nor we can use it in code:\n\n```js\nvar converter=new require(\"csvtojson\").Converter({noheader:true});\n```\n\nthe *headers* parameter specify the header row in an array. If *noheader* is false, this value will override csv header row. With csv data above, run command:\n\n```\ncsvtojson  ./test/data/noheadercsv  --noheader=true --headers='[\"hell\",\"csv\"]'\n```\n\nwe get following results:\n\n```json\n[\n  {\"hell\":\"CC102-PDMI-001\",\"csv\":\"eClass_5.1.3\",\"field3\":\"10/3/2014\",\"field4\":\"12\",\"field5\":\"40\",\"field6\":\"green\",\"field7\":\"40\"},\n  {\"hell\":\"CC200-009-001\",\"csv\":\"eClass_5.1.3\",\"field3\":\"11/3/2014\",\"field4\":\"5\",\"field5\":\"3\",\"field6\":\"blue\",\"field7\":\"38\",\"field8\":\"extra field!\"}\n]\n```\n\nIf length of headers array is smaller than the column of csv, converter will automatically fill the column with \"field*\". where * is current column index starting from 1.\n\nAlso we can use it in code:\n\n```js\nvar converter=new require(\"csvtojson\").Converter({headers:[\"my header1\",\"hello world\"]});\n```\n\n# Error handling\n\nSince version 0.4.4, parser detects CSV data corruption. It is important to catch those erros if CSV data is not guranteed correct. Just simply register a listener to error event:\n\n```js\n  var converter=new require(\"csvtojson\").Converter();\n  converter.on(\"error\",function(errMsg,errData){\n    //do error handling here\n  });\n```\n\nOnce an error is emitted, the parser will continously parse csv data if up stream is still populating data. Therefore, a general practise is to close / destroy up stream once error is captured.\n\nHere are built-in error messages and corresponding error data:\n\n* unclosed_quote: If quote in csv is not closed, this error will be populated. The error data is a string which contains un-closed csv row.\n* row_exceed: If maxRowLength is given a number larger than 0 and a row is longer than the value, this error will be populated. The error data is a string which contains the csv row exceeding the length.\n* row_process: Any error happened while parser processing a csv row will populate this error message. The error data is detailed error message (e.g. checkColumn is true and column size of a row does not match that of header).\n\n\n# Parser\n\n** Parser will be replaced by [Result Transform](#result-transform) and [Flags](#flags) **\n\nThis feature will be disabled in future.\n\nCSVTOJSON allows adding customised parsers which concentrating on what to parse and how to parse.\nIt is the main power of the tool that developer only needs to concentrate on how to deal with the data and other concerns like streaming, memory, web, cli etc are done automatically.\n\nHow to add a customised parser:\n\n```js\n//Parser Manager\nvar parserMgr=require(\"csvtojson\").parserMgr;\n\nparserMgr.addParser(\"myParserName\",/^\\*parserRegExp\\*/,function (params){\n   var columnTitle=params.head; //params.head be like: *parserRegExp*ColumnName;\n   var fieldName=columnTitle.replace(this.regExp, \"\"); //this.regExp is the regular expression above.\n   params.resultRow[fieldName]=\"Hello my parser\"+params.item;\n});\n```\n\nparserMgr's addParser function take three parameters:\n\n1. parser name: the name of your parser. It should be unique.\n\n2. Regular Expression: It is used to test if a column of CSV data is using this parser. In the example above any column's first row starting with *parserRegExp* will be using it.\n\n3. Parse function call back: It is where the parse happens. The converter works row by row and therefore the function will be called each time needs to parse a cell in CSV data.\n\nThe parameter of Parse function is a JSON object. It contains following fields:\n\n**head**: The column's first row's data. It generally contains field information. e.g. *array*items\n\n**item**: The data inside current cell.  e.g. item1\n\n**itemIndex**: the index of current cell of a row. e.g. 0\n\n**rawRow**: the reference of current row in array format. e.g. [\"item1\", 23 ,\"hello\"]\n\n**resultRow**: the reference of result row in JSON format. e.g. {\"name\":\"Joe\"}\n\n**rowIndex**: the index of current row in CSV data. start from 1 since 0 is the head. e.g. 1\n\n**resultObject**: the reference of result object in JSON format. It always has a field called csvRows which is in Array format. It changes as parsing going on. e.g.\n\n```json\n{\n   \"csvRows\":[\n      {\n          \"itemName\":\"item1\",\n          \"number\":10\n      },\n      {\n         \"itemName\":\"item2\",\n         \"number\":4\n      }\n   ]\n}\n```\n#Stream Options\nSince version 1.0.0, the Converter constructor takes stream options as second parameter.\n\n```js\nconst conv=new Converter(params,{\n  objectMode:true, // stream down JSON object instead of JSON array\n  highWaterMark:65535 //Buffer level\n})\n\n```\n\nSee more detailed information [here](https://nodejs.org/api/stream.html#stream_class_stream_transform).\n\n\n#Change Log\n\n## 1.1.0\n\n* Remove support of `new Converter(true)`\n\n## 1.0.2\n* supported ndjson format as per #113 and #87\n* issue: #120\n\n## 1.0.0\n* Add [Stream Options](#stream-options)\n* Change version syntax to follow x.y.z\n\n## 0.5.12\n* Added support for scientific notation number support (#100)\n* Added \"off\" option to quote parameter\n\n## 0.5.4\n* Added new feature: accept special delimiter \"auto\" and array\n\n## 0.5.2\n\n* Changed type separator from # to #!\n* Fixed bugs\n\n## 0.5.0\n\n* Fixed some bugs\n* Performance improvement\n* **Implicity type for numbers now use RegExp:/^[-+]?[0-9]*\\.?[0-9]+$/. Previously 00131 is a string now will be recognised as number type**\n* **If a column has no head, now it will use current column index as column name: 'field*'. previously parser uses a fixed index starting from 1. e.g. csv data: 'aa,bb,cc' with head 'a,b'. previously it will convert to {'a':'aa','b':'bb','field1':'cc'} and now it is {'a':'aa','b':'bb','field3':'cc'}**\n\n## 0.4.7\n* ignoreEmpty now ignores empty rows as well\n* optimised performance\n* added fromFile method\n\n## 0.4.4\n* Add error handling for corrupted CSV data\n* Exposed \"eol\" param\n\n## 0.4.3\n* Added header configuration\n* Refactored worker code\n* **Number type field now returns 0 if parseFloat returns NaN with the value of the field. Previously it returns original value in string.**\n\n## 0.4.0\n* Added Multi-core CPU support to increase performance\n* Added \"fork\" option to delegate csv converting work to another process.\n* Refactoring general flow\n\n## 0.3.21\n* Refactored Command Line Tool.\n* Added ignoreEmpty parameter.\n\n## 0.3.18\n* Fixed double qoute parse as per CSV standard.\n\n## 0.3.14\n* Added field type support\n* Fixed some minor bugs\n\n## 0.3.8\n* Empowered built-in JSON parser.\n* Change: Use JSON parser as default parser.\n* Added parameter trim in constructor. default: true. trim will trim content spaces.\n\n## 0.3.5\n* Added fromString method to support direct string input\n\n## 0.3.4\n* Added more parameters to command line tool.\n\n## 0.3.2\n* Added quote in parameter to support quoted column content containing delimiters\n* Changed row index starting from 0 instead of 1 when populated from record_parsed event\n\n## 0.3\n* Removed all dependencies\n* Deprecated applyWebServer\n* Added construct parameter for Converter Class\n* Converter Class now works as a proper stream object\n\n# IMPORTANT!!\nSince version 0.3, the core class of csvtojson has been inheriting from stream.Transform class. Therefore, it will behave like a normal Stream object and CSV features will not be available any more. Now the usage is like:\n```js\n//Converter Class\nvar fs = require(\"fs\");\nvar Converter = require(\"csvtojson\").Converter;\nvar fileStream = fs.createReadStream(\"./file.csv\");\n//new converter instance\nvar converter = new Converter({constructResult:true});\n//end_parsed will be emitted once parsing finished\nconverter.on(\"end_parsed\", function (jsonObj) {\n   console.log(jsonObj); //here is your result json object\n});\n//read from file\nfileStream.pipe(converter);\n```\n\nTo convert from a string, previously the code was:\n```js\ncsvConverter.from(csvString);\n```\n\nNow it is:\n```js\ncsvConverter.fromString(csvString, callback);\n```\n\nThe callback function above is optional. see [Parse String](#parse-string).\n\nAfter version 0.3, csvtojson requires node 0.10 and above.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeyang%2Fnode-csvtojson","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkeyang%2Fnode-csvtojson","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeyang%2Fnode-csvtojson/lists"}