{"id":33185522,"url":"https://github.com/wiluite/csvsuite","last_synced_at":"2025-11-17T10:01:36.687Z","repository":{"id":266488174,"uuid":"898416099","full_name":"wiluite/csvsuite","owner":"wiluite","description":"Written in C++ Python's csvkit ","archived":false,"fork":false,"pushed_at":"2025-10-17T17:10:08.000Z","size":9232,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-10-18T19:46:43.711Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wiluite.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-04T10:59:59.000Z","updated_at":"2025-10-17T17:10:12.000Z","dependencies_parsed_at":"2024-12-18T19:32:38.558Z","dependency_job_id":"fd45d325-db44-4c4d-bd6b-9d69bf5ad6db","html_url":"https://github.com/wiluite/csvsuite","commit_stats":null,"previous_names":["wiluite/csvsuite"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/wiluite/csvsuite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wiluite%2Fcsvsuite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wiluite%2Fcsvsuite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wiluite%2Fcsvsuite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wiluite%2Fcsvsuite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wiluite","download_url":"https://codeload.github.com/wiluite/csvsuite/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wiluite%2Fcsvsuite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284861038,"owners_count":27075155,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-17T02:00:06.431Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-16T05:00:20.093Z","updated_at":"2025-11-17T10:01:36.676Z","avatar_url":"https://github.com/wiluite.png","language":"C++","funding_links":[],"categories":["\u003ca name=\"data-management-tabular\"\u003e\u003c/a\u003eData management - Tabular data"],"sub_categories":[],"readme":"# csvsuite\n## The same as [csvkit](https://csvkit.readthedocs.io/en/latest/), but written in C++.\n\n*    [About](#about)\n*    [Sharing](#sharing)\n*    [Restrictions](#restrictions)\n*    [Tutorial](#tutorial)\n*    [Statistics performance](#statistics-performance)\n*    [Sorting performance](#sorting-performance)\n*    [SQL performance](#sql-performance)\n*    [Joining performance](#joining-performance)\n*    [Build All](#build-all)\n*    [Testing](#testing)\n*    [Installation](#installation)\n*    [Reference](#reference)\n\n### About\n_csvsuite_ is written to dramatically increase the speed of working with large amounts of data by taking advantage of\nthe high-performance compiled programming language C++.\n\nIt tries to reproduce the functionality of the [csvkit](https://csvkit.readthedocs.io/en/latest/) whenever possible.\nIt is written on top of the [csv_co](https://github.com/wiluite/CSV_co) CSV reader and, where needed, with the help of\nthe task-based parallelism library [transwarp](https://github.com/bloomen/transwarp) and the thread pool-based library\n[poolSTL](https://github.com/alugowski/poolSTL). Interaction with SQL databases is carried out on the basis of the\n[SOCI](https://github.com/SOCI/soci) and [OCILIB](https://vrogier.github.io/ocilib/) libraries. More than a dozen other\nlibraries support the rest of the product's functionality.\n\nThe goals for the reproduction were: to find out the complexity and limitations of the applicability of the C++\necosystem for broad universal tasks, where Python is good with its rich environment for data processing, text encoding,\nnumeric format localization, SQL databases and so on. It was also interesting to see the performance benefits of C++\napplications in non-traditional areas. These utilities (of those 14) seem to be almost fully operational at the moment:\n1) csvClean (ala [csvclean](https://csvkit.readthedocs.io/en/latest/scripts/csvclean.html))\n2) csvCut (ala [csvcut](https://csvkit.readthedocs.io/en/latest/scripts/csvcut.html))\n3) csvGrep (ala [csvgrep](https://csvkit.readthedocs.io/en/latest/scripts/csvgrep.html))\n4) csvJoin (ala [csvjoin](https://csvkit.readthedocs.io/en/latest/scripts/csvjoin.html))\n5) csvJson (ala [csvjson](https://csvkit.readthedocs.io/en/latest/scripts/csvjson.html))\n6) csvLook (ala [csvlook](https://csvkit.readthedocs.io/en/latest/scripts/csvlook.html))\n7) csvSort (ala [csvsort](https://csvkit.readthedocs.io/en/latest/scripts/csvsort.html))\n8) csvStack (ala [csvstack](https://csvkit.readthedocs.io/en/latest/scripts/csvstack.html))\n9) csvStat (ala [csvstat](https://csvkit.readthedocs.io/en/latest/scripts/csvstat.html))\n10) csvSql (ala [csvsql](https://csvkit.readthedocs.io/en/latest/scripts/csvsql.html))\n11) Sql2csv (ala [sql2csv](https://csvkit.readthedocs.io/en/latest/scripts/sql2csv.html))\n12) In2csv (ala [in2csv](https://csvkit.readthedocs.io/en/latest/scripts/in2csv.html))\n\nThere are binary packages for these Linux distributions: Debian 11 Bullseye (amd64), Debian 12 Bookworm (amd64),\nUbuntu 22.04 (amd64), Ubuntu 24.04 (amd64). And there are two binary packages for Windows x64: MSVC 2022 Community,\nMinGW. Otherwise, please build it yourself using one of the methods listed in the [Build All](#build-all) section.\n\n### Sharing\nThe _csvsuite_ is not meant to replace the _csvkit_, it has its limitations. Moreover, data import ([In2csv](#in2csv))\nis less complete and generally less tested in terms of data diversity. But as you gain more confidence in the tool,\nyou could use it in a streaming bundle to essentially speed up the processing of larger documents.\nFor example (on case-sensitive systems):\n  \n    in2csv ... | csvCut ... | csvGrep ... | csvSort ... | ... | csvformat\n\nIf you are used to working with the _csvkit_, and other tools don't support your data types (6 types of data: Boolean,\nNumber, TimeDelta, DateTime, Date, Text) or are just not comfortable for you, give the _csvsuite_ a chance! Bug reports,\nscenarios that failed and so on are very welcome. I can keep this in active development. Also, see performance\nstatistics after the next two sections.\n\n### Restrictions\nYour CSV sources must be [RFC-4180](https://en.wikipedia.org/wiki/Comma-separated_values)-compliant. Fortunately, the\noverwhelming percentage of documents in the world adhere to this rule. If not, you can/should always resort to the\n[csvClean](#csvclean) (or even a more powerful one from the original package:\n[csvclean](https://csvkit.readthedocs.io/en/latest/scripts/csvclean.html)), to fix your document. In any case, such\ndocument just obviously needs to be fixed. When running, any utility tries to quickly check the strong tabular shape of\nyour documents to match [RFC-4180] and whistles if this is not the case.\n\nDue to the fact the _csvsuite_ will work with RFC-4180-compliant only, the following options are unsupported:\n\n| unsupported options                      |\n|------------------------------------------|\n| -d DELIMITER, --delimiter DELIMITER      |\n| -t, --tabs                               |\n| -q QUOTECHAR, --quotechar QUOTECHAR      |\n| -u {0,1,2,3}, --quoting {0,1,2,3}        |\n| -b, --no-doublequote                     |\n| -y SNIFF_LIMIT, --snifflimit SNIFF_LIMIT |\n\nWhen handling date and datetime data types and their localization, the _csvkit_ relies on the rich Python datetime\nlibrary. It also allows you to work with time representations such as 'yesterday', 'today', 'tomorrow', and so on.\nThe _csvsuite_ , though, is tightly bound to the `--date-format` and `--datetime-format` options and works well only on\nthose platforms where this is supported by the environment/compiler/runtime. And the `--date-lib-parser` option engages\nthe special [date](https://github.com/HowardHinnant/date) library to improve the situation and ensure consistency\neverywhere (on most platforms). For more info see tests located in the\n[csvsuite_core_test.cpp](https://github.com/wiluite/csvsuite/blob/main/suite/test/csvsuite_core_test.cpp) module. For\ncomplete info see [formatting section](https://howardhinnant.github.io/date/date.html#from_stream_formatting) of the\ndocumentation.\n\nLocale-formatted support for numbers is provided out of the box, that is, by the development tool. If there is no\nsuch support somewhere (for example MinGW/Windows), you will not be able to work with locale-formatted numbers.\n\nOther restrictions and some substitutions are presented in section [Reference](#reference), describing utilities.\n\n### Tutorial\n### 1. Getting started\n#### 1.1. About this tutorial\nThis tutorial should be almost exactly the same as the original\n[tutorial](https://csvkit.readthedocs.io/en/latest/tutorial.html).\n\n#### 1.2. Installing csvsuite\nThe best way to install the tool is to simply download a required binary archive from the\n[release](https://github.com/wiluite/csvsuite/releases) page and unpack it. Then add the path to the unpacked directory\nroot to the list of directories in which to search for commands, according to the rules for doing this for this\nparticular operating system. See [Installation](#installation) section for details. \n\nSee also [Build All](#build-all) section.\n\n#### 1.3. Remaining steps\nJust repeat the lessons from the original training:\n[1.3 - 1.8](https://csvkit.readthedocs.io/en/latest/tutorial/1_getting_started.html),\n[2.1 - 2.4](https://csvkit.readthedocs.io/en/latest/tutorial/2_examining_the_data.html),\n[3.1 - 3.4](https://csvkit.readthedocs.io/en/latest/tutorial/3_power_tools.html),\n[4.1](https://csvkit.readthedocs.io/en/latest/tutorial/4_going_elsewhere.html), and make sure everything works,\nbut first pay attention to the following notes for changes for you to do.  \n\n\u003e Parts of utility names that reflect their purpose are capitalized to avoid confusion between the\n_csvsuite_ and the _csvkit_ on case-sensitive systems. Thus, you must type their names correctly. See their names in the\n[About](#about) section.  \n\n\u003e In paragraph 1.4, note that in the resulting data.csv document in the 10th column (ship_date), there is a\nnumber, not a date. If this is too important for you right now, then to improve it, run the following command instead of\nthe one suggested:\n\u003e \n\u003e    **In2csv ne_1033_data.xlsx --d-excel ship_date --is1904=0 \u003e data.csv**\n\u003e \n\u003e\u003e The reason the _csvkit_ can detect the date automatically is because it relies on the heuristic capabilities of\npackages like [xlrd](https://xlrd.readthedocs.io/en/latest/) and [openpyxl](https://openpyxl.readthedocs.io/en/stable/),\nwhich do not guarantee that Dates/DateTimes are correctly recognized, since Excel documents themselves do not have a\ndate or datetime storage type. So you are facing the necessity to always specify which numeric columns and using which\nera you want to convert to dates or datetimes.  \n\n\u003e  In paragraphs where [_csvLook_](#csvlook) is used, you will not see (by default) separators in the numbers displayed\non the screen, unlike [_csvStat_](#csvstat) which displays number separator according to the current global locale.\nThis is because in the _csvkit_ there is a difference between the locales according to which numbers are output in the\ntwo utilities. To overcome this contradiction and still see separators in numbers, simply specify the locale in which\nyou want to see them. For example:\n\u003e \n\u003e **csvCut -c acquisition_cost data.csv | csvLook data.csv -G en_US**\n\u003e \n\u003e Or:  \n\u003e\t\n\u003e **csvCut -c acquisition_cost data.csv | csvLook data.csv -G en_US.utf-8**\n\u003e \n\u003e where `-G` option is a \"Superseded global locale\".  \n\n\u003e In the _csvStat_ utility, \"Most common values\" with the same number of repetitions may not be the same as in the\noriginal utility, due to different sorting algorithms. To display more data, use the `--freq-count` option.\n\n\u003e In paragraph 3.3 you must use: **--db sqlite3://leso.db** instead of **--db sqlite:///leso.db**. For more details, see\nthe description of the `--db` option in the utilities [_csvSql_](#csvsql) and [_Sql2csv_](#sql2csv).\n\n\n### Statistics performance\nThere were measured the performances of three tools: [csvkit(2.0.1)'s csvstat](https://pypi.org/project/csvkit/), \n[xsv(0.13.0)' stats](https://github.com/BurntSushi/xsv/releases/tag/0.13.0) and [_csvStat_](#csvstat) at files: \ncrime.csv(102M), worldcitiespop.csv(145M), flights.csv(565M) and question_tags.csv(844M) with (or with not) a limited\nnumber of columns (so as not to break up screenshots). These files may be available among\n[Kaggle open datasets](https://www.kaggle.com/datasets), and one of them (worldcitiespop.csv) can be found\n[here](https://burntsushi.net/stuff/worldcitiespop.csv).\n\nYou can also download all the files in packaged form from [this link](https://disk.yandex.ru/d/HqpSmLX9OuopRg).\n\nAs to all tests. It should be said that for the _csvsuite_ to successfully compete with the _xsv_, it would be good to\nhave a fast external storage device, since when processing data through mentioned CSV parser, memory-mapped file\ntechnology is used.\n\nHere are the competition screenshots:  \n\n**crime.csv/csvkit**\n\n\u003cimg alt=\"image info\" height=\"794\" src=\"./img/stat_crime_kit.png\" width=\"471\"/\u003e\n\n**crime.csv/xsv**\n\n\u003cimg alt=\"image info\" height=\"144\" src=\"./img/stat_crime_xsv.png\" width=\"1054\"/\u003e\n\n**crime.csv/csvsuite**\n\n\u003cimg alt=\"image info\" height=\"767\" src=\"./img/stat_crime_suite.png\" width=\"480\"/\u003e\n\nHere, the _xsv_ is the winner. It produces results in less than a second. (Note, however, slight difference at StDev in\nthe first case).\n\n---\n\n**worldcitiespop.csv/csvkit**  \n\n\u003cimg alt=\"image info\" height=\"846\" src=\"./img/stat_worldcitiespop_kit.png\" width=\"456\"/\u003e\n\n**worldcitiespop.csv/xsv**\n\n\u003cimg alt=\"image info\" height=\"143\" src=\"./img/stat_worldcitiespop_xsv.png\" width=\"912\"/\u003e\n\n**worldcitiespop.csv/csvsuite**\n\n\u003cimg alt=\"image info\" height=\"807\" src=\"./img/stat_worldcitiespop_suite.png\" width=\"539\"/\u003e\n\nHere, the _csvsuite_ is the winner. Since the _xsv_ does not calculate 'most decimal places' in its statistics, unlike\n_csvkit_ , we have disabled this option at ours (`--no-mdp` option).\n\n---\n\n**flights.csv/csvkit**\n\n\u003cimg alt=\"image info\" height=\"105\" src=\"./img/stat_flights_kit.png\" width=\"457\"/\u003e\n\n**flights.csv/xsv**\n\n\u003cimg alt=\"image info\" height=\"233\" src=\"./img/stat_flights_xsv.png\" width=\"827\"/\u003e\n\n**flights.csv/csvsuite**\n\n\u003cimg alt=\"image info\" height=\"1053\" src=\"./img/stat_flights_suite.png\" width=\"407\"/\u003e\n\nHere we again beat the _xsv_ by more than 2 times by doing statistics on the first 10 columns. If we did full statistics\non all the columns, the _xsv_ would run out of memory on our current machine (we had 12 GB RAM), as in the next test.\n\nThe _csvkit_ is out of the competition.\n\n---\n\n**question_tags.csv/xsv**\n\n\u003cimg alt=\"image info\" height=\"103\" src=\"./img/stat_qtags_xsv.png\" width=\"629\"/\u003e\n\n**question_tags.csv/csvsuite**\n\n\u003cimg alt=\"image info\" height=\"521\" src=\"./img/stat_qtags_suite.png\" width=\"465\"/\u003e\n\nOur tool gives the result in about 42 seconds. Here we could not wait for the result from _csvkit_ within a reasonable\ntime. Thus, both the _csvkit_ and the _xsv_ are unable to produce their results where for out tool the reason why this\nis not possible is not the case. This is a subject for further research.\n\nUPDATE:\n\nWe've managed to get results from the _xsv_ in Windows:\n\n**question_tags.csv/xsv/Windows**\n\n\u003cimg alt=\"image info\" height=\"115\" src=\"./img/stat_qtags_xsv_windows.png\" width=\"1257\"/\u003e\n\n**question_tags.csv/suite/Windows**\n\n\u003cimg alt=\"image info\" height=\"706\" src=\"./img/stat_qtags_suite_windows.png\" width=\"587\"/\u003e\n\nWe see, the performance of both utilities is most likely lower than in Linux. Between themselves they are relatively\nequal.\n\n\n### Sorting performance\nIf we talk about sorting by columns of string types, then the _xsv_ is unrivaled, far ahead of other means in time,\nbecause it obviously uses efficient algorithms for sorting strings, and without the use of parallelism. However, let's\nsee how effective it is to sort a group of columns where there is one numeric type (the -N option at the _xsv_ is\nrequired, otherwise the results will be incorrect). We only need about 6 seconds versus 22 at the _xsv_. The _csvsort_\nis more than a minute behind us.\n\n\u003cimg alt=\"image info\" height=\"612\" src=\"./img/sort.png\" width=\"593\"/\u003e\n\n\n### SQL performance\nC++ is said to outperform Python in general, non-specialized areas by about 3 times. In light of the impossibility of\nparallelizing the filling of the SQL database table using language tools. Here it turned out, we are 8 times faster,\nprobably because the _csvsql_ spends a significant part of the time determining column types, unlike our tool.\n\n\u003cimg alt=\"image info\" height=\"547\" src=\"./img/sql.png\" width=\"761\"/\u003e\n\n### Joining performance\nNow we will try to display the first 9 results corresponding to the conditions when the City and AccentCity fields in\nthe worldcitiespop.csv file are equal. Here the _csvJoin_ outperforms the _csvjoin_ by a factor of 8 as well.\n\n\u003cimg alt=\"image info\" height=\"660\" src=\"./img/join.png\" width=\"739\"/\u003e\n\nDespite the _xsv_'s awesome performance in all previous tests (in many tests not shown, the _xsv_ simply outperforms\nthe _csvsuite_ by many-many times), it doesn't do everything right. Let's criticize it a little more.\n\nSuppose we have got two CSV files: a.csv and b.csv:\n\n\u003cimg src=\"./img/a_and_b.png\" alt=\"image info\" height=\"143\" width=\"367\" /\u003e\n\nCorrect join results for each of the pairwise columns:\n\n\u003cimg alt=\"image info\" height=\"157\" src=\"./img/join_checks.png\" width=\"470\"/\u003e\n\nAnd not such correct results in the case of the _xsv_:\n\n\u003cimg alt=\"image info\" height=\"130\" src=\"./img/join_checks_xsv.png\" width=\"473\"/\u003e\n\nIn the first case, the _xsv_ ignores the spaces to the left and right of the significant part of the fields, resulting\nin the equality of two unequal values.\n\nIn the second case, it does not detect the numeric values 20 and 2e1 as equal.\n\nIn the third case, it considers the pairwise values of “1” to be identical, ignoring the type of the third\ncolumn of the second file as text. When it becomes erroneous to compare numeric values with the numeric values of the\nthird column of the first file.\n\nThus, despite its undeniable speed and possible practical benefits, in some cases this product clearly simplifies its\ntask, which affects the reliability of the results.\n\n\n### Build All\n\n\u003e For now, you need Python installed to configure several libraries used. Over time, this dependence will be removed.\n\n#### _PREPARATION_\n\nInstall the necessary SQL database servers and client libraries for them into the system, if not already done. This is\nnecessary for the build system to create the appropriate libraries for the [_csvSql_](#csvsql) and [_Sql2csv_](#sql2csv)\nutilities. Thanks to [SOCI - The C++ Database Access Library](https://github.com/SOCI/soci) and\n[OCILIB - Driver for Oracle](https://vrogier.github.io/ocilib/) the _csvsuite_ supports the following SQL databases:\n\n- Linux: SQLite3, MySQL, MariaDB, PostgreSQL, Firebird, Oracle (not tested).\n- Windows: SQLite3, MySQL, MariaDB, PostgreSQL, Firebird, Oracle.\n\n\u003e Please, refer to the SOCI documentation to find out which versions of these databases are supported.  \n\u003e Please, refer to the OCILIB documentation to find out which versions of ORACLE databases are supported.  \n\u003e Access to other databases within the _csvsuite_ is disabled due to lack of verification capabilities.  \n\n* **Linux**\n\nIt seems that no additional preparations required. Except for, you may need to create ORACLE_HOME environment\n  variable. Example:\n\u003e export ORACLE_HOME=~/product/21c/dbhomeXE\n\nin your user's  ~/.profile, and reboot.\n\n* **Windows**\n\nAdd to the PATH environment variable the access paths to the bin and lib directories of your databases (except for\nSQLite3). Examples:\n\u003e C:\\Program Files\\MySQL\\MySQL Server 8.0\\bin;\u003cbr\u003e\n\u003e C:\\Program Files\\MySQL\\MySQL Server 8.0\\lib;\u003cbr\u003e\n\u003e C:\\Program Files\\Firebird\\Firebird_5_0;\n\nCreate ORACLE_HOME environment variable. Example:\n\u003e ORACLE_HOME=C:\\app\\youruser\\product\\21c\\dbhomeXE\n  \n\n#### _BUILD VARIANTS_\n\n_Conventional (Linux/GCC, Windows/MinGW):_\n```bash\nmkdir build \u0026\u0026 cd build\ncmake ..\nmake -j 6\n```\n\n_Alternative 1 (Linux, with Clang \u003e= 15):_\n```bash\nmkdir build \u0026\u0026 cd build\ncmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -D_STDLIB_LIBCPP=OFF ..\nmake -j 6\n```\n\u003e You may have to specify an explicit compiler version (example: clang++-16 instead of clang++, clang-16 instead of\nclang). \n \n_Alternative 2 (Linux, with Clang, libc++-dev, libc++abi-dev packages installed):_\n```bash\nmkdir build \u0026\u0026 cd build\ncmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -D_STDLIB_LIBCPP=ON ..\nmake -j 6\n```\n\u003e You may have to specify an explicit compiler version (example: clang++-16, clang-16). \n\n_Test build for memory safety (Linux, with Clang, libc++-dev, libc++abi-dev, and Clang sanitizers packages installed):_\n```bash\nmkdir build \u0026\u0026 cd build\ncmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -D_STDLIB_LIBCPP=ON -D_SANITY_CHECK=ON -DCMAKE_BUILD_TYPE=Debug ..\nmake -j 4\n```\n\u003e You may have to specify an explicit compiler version (example: clang++-16, clang-16). \n\n_MSVC (Windows, in x64 Native Tools Command Prompt):_\n```bash\nmkdir build \u0026\u0026 cd build\ncmake -G \"Visual Studio 17 2022\" -DCMAKE_BUILD_TYPE=Release ..\nmsbuild /property:Configuration=Release csvsuite.sln\n```\n\n_Docker container (Linux):_\n\nGo to the _csvsuite_ directory, then build a necessary docker image:\n```bash\ndocker build -t foo/your_image_name -f ./a_dockerfile .\n```\nRun a docker container from the new docker image:\n```bash\ndocker run -it --rm --name=random_name --mount type=bind,source=${PWD},target=/src foo/your_image_name\n```\nInside the docker container build all:\n```bash\ncd src \u0026\u0026 mkdir build \u0026\u0026 cd build \u0026\u0026 cmake .. \u0026\u0026 make -j 4 all\n```\n\n### Testing\nThis assumes that you have successfully [built](#build-all) the product.  \nCreate necessary testing SQL databases, and the following environment variables with your own corresponding values, to\nsuccessfully test all branches of the [_csvSql_](#csvsql) utility. This step is not necessary if you've built the\nproduct in a Docker container, where the environment is completely customized.\n\n* **Linux**\n\n\u003e SOCI_DB_SQLITE3=\"sqlite3://db=test.sqlite3\" \u003cbr\u003e\n\u003e SOCI_DB_MYSQL=\"mysql://db=your_db user=your_user password=your_password host=127.0.0.1 port=3306\" \u003cbr\u003e\n\u003e SOCI_DB_MARIADB=\"mariadb://db=your_db user=your_user password=your_password host=127.0.0.1 port=3307\" \u003cbr\u003e\n\u003e SOCI_DB_POSTGRESQL=\"postgresql://dbname=your_db user=your_user password=your_password\" \u003cbr\u003e\n\u003e SOCI_DB_FIREBIRD=\"firebird://service=/path_to/your_db.fdb user=SYSDBA password=masterkey\"\n\n* **Windows**\n\n\u003e SOCI_DB_SQLITE3=sqlite3://db=test.sqlite3 timeout=2 share-cache=true \u003cbr\u003e\n\u003e SOCI_DB_MYSQL=mysql://db=your_db user=your_user password=your_password host=127.0.0.1 port=3306 \u003cbr\u003e\n\u003e SOCI_DB_MARIADB=mysql://db=your_db user=your_user password=your_password host=127.0.0.1 port=3307 \u003cbr\u003e\n\u003e SOCI_DB_POSTGRESQL=postgresql://dbname=your_db user=your_user password=your_password \u003cbr\u003e\n\u003e SOCI_DB_FIREBIRD=firebird://service=d:\\\\your_directory_path\\\\your_db.fdb user=SYSDBA password=masterkey \u003cbr\u003e\n\u003e SOCI_DB_ORACLE=\"oracle://service=//127.0.0.1:1521/xepdb1 user=hr password=hr\"\n\nGo to your build/suite/test directory and run all the unit tests:\n\n```bash\nctest -j 1 --repeat until-fail:10 --stop-on-failure\n```\n  \u003e You could run tests in parallel as well (ctest -j 6), but keep in mind csvSql_test and Sql2csv_test executables may\n    have non-shared states.\n  \n### Installation\nIn general, the _csvsuite_ should not be installed into system directories. It is enough to leave binary build results\nin place. Or, unpack a ready-made binary archive from the releases page into any directory. In both cases, you only need\nto establish global paths.\n\n* **Linux**\n\n  - If you've built it, please modify your ~/.profile to have:\n    \u003e  export PATH=$PATH:/path/to/csvsuite/suite/build/\n\n    and reboot.\n\n  - If you've downloaded it as a binary release, please modify your ~/.profile to have:\n    \u003e  export PATH=$PATH:/path/to/unpacked/archive \u003cbr\u003e\n       export LD_LIBRARY_PATH=/path/to/unpacked/archive/lib:$LD_LIBRARY_PATH\n\n    If not installed, please install C++ standard library package too:\n    \u003e  sudo apt-get update  \n       sudo apt-get install libstdc++6\n\n      and reboot.\n  \n  \n* **Windows**\n\n  - If you've built it, add the path to your\\build\\suite directory to the Path environment variable.\n  - If you've downloaded it as a binary release, add the path to the suite directory of the unpacked archive to the Path\n    environment variable.\n  - If you've downloaded a MSVC binary archive, please install the redistributable package too:\n    [VC_redist.x64.exe](https://github.com/wiluite/csvsuite/releases/download/v0.2.0/VC_redist.x64.exe)\n\n---\n\n### Reference\n\n*    [Input](#input)\n*    [Processing](#processing)\n*    [Output and Analysis](#output-and-analysis)\n\n---\n\n#### Input\n* [In2csv](#in2csv)\n* [Sql2csv](#sql2csv)\n\n---\n\n#### In2csv\n##### Description\nConverts various tabular data formats into CSV.\n\nConverting fixed width requires that you provide a schema file with the `-s` option. The schema file should have the\nfollowing format:\n\n    column,start,length\n    name,0,30\n    birthday,30,10\n    age,40,3\n\nThe header line is required though the columns may be in any order:\n\n    Usage: In2csv arg_0  [options...]  \n    arg_0 : The file of a specified format to operate on. If omitted, will accept input as piped data via STDIN. [default: ]\n\nOptions:\n\n    --help : print help [implicit: \"true\", default: false]\n    -f,--format : The format {csv,dbf,fixed,geojson,json,ndjson,xls,xlsx} of the input file. If not specified will be inferred from the file type. [default: ]  \n    -s,--schema : Specify a CSV-formatted schema file for converting fixed-width files. See In2csv_test as example. [default: ]  \n    -k,--key : Specify a top-level key to look within for a list of objects to be converted when processing JSON. [default: ]  \n    --non-flat : Specify, whether to navigate through nested structures when processing JSON. [implicit: \"true\", default: false]\n    -n,--names : Display sheet names from the input Excel file. [implicit: \"true\", default: false]  \n    --sheet : The name of the Excel sheet to operate on. [default: ]  \n    --write-sheets : The names of the Excel sheets to write to files, or \"-\" to write all sheets. [default: ]  \n    --use-sheet-names : Use the sheet names as file names when --write-sheets is set. [implicit: \"true\", default: false]  \n    --encoding-xls : Specify the encoding of the input XLS file. [default: UTF-8]  \n    --d-excel : A comma-separated list of numeric columns of the input XLS/XLSX/CSV source, considered as dates, e.g. \"1,id,3-5\". [default: none]  \n    --dt-excel : A comma-separated list of numeric columns of the input XLS/XLSX/CSV source, considered as datetimes, e.g. \"1,id,3-5\". [default: none]  \n    --is1904 : Epoch based on the 1900/1904 datemode for input XLSX source, or for the input CSV source, converted from XLS/XLSX. [implicit: \"true\", default: true]  \n    -I,--no-inference : Disable type inference (and --locale, --date-format, --datetime-format, --no-leading-zeroes) when parsing the input. [implicit: \"true\", default: false]\n\nSome command-line flags only pertain to specific input formats.\n\nSee also: [Arguments common to all tools](#arguments-common-to-all-tools).\n\n**Examples**\n\nConvert the 2000 census geo headers file from fixed-width to CSV and from latin-1 encoding to utf8:\n\n    In2csv -e iso-8859-1 -f fixed -s examples/realdata/census_2000/census2000_geo_schema.csv examples/realdata/census_2000/usgeo_excerpt.upl\n\nConvert an Excel .xls file:\n\n    In2csv examples/test.xls  \n\nStandardize the formatting of a CSV file (quoting, line endings, etc.):\n\n    In2csv examples/realdata/FY09_EDU_Recipients_by_State.csv -L en_US.utf-8  \n\n\u003e Unlike the _csvkit_ , which defaults to en_US as the locale for any formatted numbers, here you must specify this\n\u003e locale explicitly, since the utility uses the C/Posix locale by default.\n\nFetch csvkit’s open issues from the GitHub API, convert the JSON response into a CSV and write it to a file:  \n\n    Not supported some complex JSONs. To be implemented.\n\nConvert a JSON with nested structures into a CSV:\n\n    In2csv examples/testjson_nested.json --non-flat\n\nConvert a DBase DBF file to an equivalent CSV:\n\n    In2csv examples/testdbf.dbf  \n\n**Troubleshooting**\n\nIf an error like the following occurs when providing an input file in one of the formats:  \n\n    The document has 1 column at NNN row...  \n\nThen the input file might have initial rows before the header and data rows. You can skip such rows with `--skip-lines\n(-K)`:\n\n    In2csv -K 3 examples/test_skip_lines.csv\n\n---\n\n#### Sql2csv\n##### Description\nExecutes arbitrary commands against a SQL database and outputs the results as a CSV.\n##### Requirements\n    \n\n    Usage: Sql2csv arg_0  [options...]  \n    arg_0 : The FILE to use as SQL query. If it and --query are omitted, the query is piped data via STDIN. [default: ]  \n\nOptions:\n\n    --db : If present, a 'soci' connection string to use to directly execute generated SQL on a database. [default: ]\n    --query : The SQL query to execute. Overrides FILE and STDIN. [default: ]\n    -e,--encoding : Specify the encoding of the input query file. [default: UTF-8]\n    -H,--no-header-row : Do not output column names. [implicit: \"true\", default: false]\n    --help : print help [implicit: \"true\", default: false]\n\n**Examples**\n\nLoad sample data into a table using [csvSql](#csvsql) and then query it using _Sql2csv_:\n\n    csvSql --db \"sqlite3://dummy.db\" --tables \"test\" --insert examples/dummy.csv\n    Sql2csv --db \"sqlite3://dummy.db\" --query \"select * from test\"\n\nalternatively (too verbose):\n\n    csvSql --db \"sqlite3://dbname=dummy.db\" --tables \"test\" --insert examples/dummy.csv\n    Sql2csv --db \"sqlite3://dbname=dummy.db\" --query \"select * from test\"\n\nTo access databases, the _csvsuite_ uses 2 libraries: the [ocilib](https://github.com/vrogier/ocilib) for accessing\nOracle and the [soci](https://github.com/SOCI/soci) for the rest. In this particular case, you must specify the value\nfor the `--db` option as the library expects it: see [connections](https://soci.sourceforge.net/doc/master/connections/).  \n\n\u003e NOTE: Two next examples assume that you have already created your test database.\n\nLoad data about financial aid recipients into PostgreSQL. \n\n    csvSql -L en_US --db \"postgresql://dbname=database user=name password=pass\" --tables \"fy09\" --insert examples/realdata/FY09_EDU_Recipients_by_State.csv\n\n\u003e Again, you must specify `en_US` (or `en_US.utf-8`) numeric locale option value explicitly, since the utility uses the\n\u003e C/Posix locale by default. Otherwise, the numeric columns will not be recognized as such and the database table will\n\u003e end up with text values.\n\nThen find the three states that received the most, while also filtering out empty rows:  \n\n    Sql2csv --db \"postgresql://dbname=database user=name password=pass\" --query \"select * from fy09 where \\\"State Name\\\" != '' order by fy09.\\\"TOTAL\\\" limit 3\"\n\nYou can even use it as a simple SQL calculator (in this example an in-memory SQLite database is used as the default):\n\n    Sql2csv --query \"select 300 * 47 % 14 * 27 + 7000\"\n---\n\n#### Processing\n* [csvClean](#csvclean)\n* [csvCut](#csvcut)\n* [csvGrep](#csvgrep)\n* [csvJoin](#csvjoin)\n* [csvSort](#csvsort)\n* [csvStack](#csvstack)\n\n---\n\n#### csvClean\n##### Description\nReports and fixes common errors in a CSV file.\n\n    Usage: csvClean arg_0  [options...]\n    arg_0 : The CSV file to operate on. If omitted, will accept input as piped data via STDIN. [default: ]\n\nOptions:\n\n    -n,--dry-run : Do not create output files. Information about what would have been done will be printed to STDERR. [implicit: \"true\", default: false]\n    --help : print help [implicit: \"true\", default: false]\n\nSee also: [Arguments common to all tools](#arguments-common-to-all-tools).\n\nThis utility currently has very basic functionality.\nSee [changelog](https://csvkit.readthedocs.io/en/latest/changelog.html) for what was done for original csvclean 2.0.0,\nand what you will not see here. Please use original\n[csvclean](https://csvkit.readthedocs.io/en/latest/scripts/csvclean.html#) utility to fix sophisticated problems in your\ndocuments, until csvClean gains similar functionality.\n\n**Examples**\n\nReport rows that have a different number of columns than the header row:\n\n    $ csvClean examples/bad.csv -n\n    Line 1: Expected 3 columns, found 4 columns\n    Line 2: Expected 3 columns, found 2 columns\n\nFix this document:\n\n    $ csvClean examples/bad.csv\n    2 errors logged to bad_err.csv\n\n    $ cat bad_out.csv\n    column_a,column_b,column_c\n    0,mixed types.... uh oh,17\n\n    $ cat bad_err.csv\n    line_number,msg,column_a,column_b,column_c\n    1,\"Expected 3 columns, found 4 columns\",1,27,,I'm too long!\n    2,\"Expected 3 columns, found 2 columns\",,I'm too short!\n\n---\n\n#### csvCut\n##### Description\nFilters and truncates CSV files. Like the Unix “cut” command, but for tabular data:\n\n    Usage: csvCut arg_0  [options...]\n    arg_0 : The CSV file to operate on. If omitted, will accept input as piped data via STDIN. [default: ]\n\nOptions:\n\n    --help : print help [implicit: \"true\", default: false]\n    -n,--names : Display column names and indices from the input CSV and exit. [implicit: \"true\", default: false]\n    -c,--columns : A comma-separated list of column indices, names or ranges to be extracted, e.g. \"1,id,3-5\". [default: all columns]\n    -C,--not-columns : A comma-separated list of column indices, names or ranges to be excluded, e.g. \"1,id,3-5\". Ignores unknown columns. [default: no columns]\n    -x,--delete-empty-rows : After cutting delete rows which are completely empty. [implicit: \"true\", default: false]\n\nSee also: [Arguments common to all tools](#arguments-common-to-all-tools).\n\n**Examples**\n\nPrint the indices and names of all columns:\n\n    $ csvCut -n examples/realdata/FY09_EDU_Recipients_by_State.csv\n    1: State Name\n    2: State Abbreviate\n    3: Code\n    4: Montgomery GI Bill-Active Duty\n    5: Montgomery GI Bill- Selective Reserve\n    6: Dependents' Educational Assistance\n    7: Reserve Educational Assistance Program\n    8: Post-Vietnam Era Veteran's Educational Assistance Program\n    9: TOTAL\n    10:\n\nPrint only the names of all columns, by removing the indices with the _cut_ command:\n\n    $ csvCut -n examples/realdata/FY09_EDU_Recipients_by_State.csv | cut -c6-\n    State Name\n    State Abbreviate\n    Code\n    Montgomery GI Bill-Active Duty\n    Montgomery GI Bill- Selective Reserve\n    Dependents' Educational Assistance\n    Reserve Educational Assistance Program\n    Post-Vietnam Era Veteran's Educational Assistance Program\n    TOTAL\n\nExtract the first and third columns:\n\n    csvCut -c 1,3 examples/realdata/FY09_EDU_Recipients_by_State.csv\n\nExtract columns named “TOTAL” and “State Name” (in that order):\n\n    csvCut -c TOTAL,\"State Name\" examples/realdata/FY09_EDU_Recipients_by_State.csv\n\nAdd line numbers to a file, making no other changes:\n\n    csvCut -l examples/realdata/FY09_EDU_Recipients_by_State.csv\n\nExtract a column that may not exist in all files:\n\n    echo d, | csvJoin examples/dummy.csv _ | csvCut -c d\n\n    echo d, | csvJoin examples/join_no_header_row.csv _ | csvCut -c d\n\n\u003e NOTE: _csvsuite_ uses `_` (instead of `-`) as a placeholder for piped source.\n\nDisplay a column’s unique values:\n\n    csvCut -c 1 examples/realdata/FY09_EDU_Recipients_by_State.csv | sed 1d | sort | uniq\n\nOr:\n\n    csvCut -c 1 examples/realdata/FY09_EDU_Recipients_by_State.csv | csvSql --query \"SELECT DISTINCT(\\\"State Name\\\") FROM stdin\"\n\n---\n\n#### csvGrep\n##### Description\nFilter tabular data to only those rows where certain columns contain a given value or match a regular expression:\n\n    Usage: csvGrep arg_0  [options...]\n    arg_0 : The CSV file to operate on. If omitted, will accept input as piped data via STDIN. [default: ]\n\nOptions:\n\n    --help : print help [implicit: \"true\", default: false]\n    -n,--names : Display column names and indices from the input CSV and exit. [implicit: \"true\", default: false]\n    -c,--columns : A comma-separated list of column indices, names or ranges to be searched, e.g. \"1,id,3-5\". [default: none]\n    -m,--match : A string to search for. [default: ]\n    -r,--regex : A regular expression to match. [default: ]\n    --r-icase : Character matching should be performed without regard to case. [implicit: \"true\", default: false]\n    -f,--file : A path to a file. For each row, if any line in the file (stripped of line separators) is an exact match of the cell value, the row matches. [default: ]\n    -i,--invert-match : Select non-matching rows, instead of matching rows. [implicit: \"true\", default: false]\n    -a,--any-match : Select rows in which any column matches, instead of all columns. [implicit: \"true\", default: false]\n\nSee also: [Arguments common to all tools](#arguments-common-to-all-tools).  \n\n\u003e NOTE: Even though ‘-m’, ‘-r’, and ‘-f’ are listed as “optional” arguments, you must specify one of them.  \n\n\u003e NOTE: the C++ standard only requires conformance to the POSIX regular expression syntax(which does not include Perl\nextensions like this one) and conformance to the ECMAScript regular expression specification (with minor exceptions, per\nISO 14882-2011§28.13), which is described in ECMA-262, §15.10.2. ECMAScript's regular expression grammar **does not**\ninclude the use of modifiers in the form of the (?) syntax.\nThis is why there is the `--r-icase` option if you need the case-insensitive comparison.\n\n**Examples**\n\nSearch for the row relating to Illinois:\n\n    csvGrep -c 1 -m ILLINOIS examples/realdata/FY09_EDU_Recipients_by_State.csv\n\nSearch for rows relating to states with names beginning with the letter “I”:\n\n    csvGrep -c 1 -r \"^I\" examples/realdata/FY09_EDU_Recipients_by_State.csv\n\nSearch for rows that do not contain an empty state cell:\n\n    csvGrep -c 1 -r \"^$\" -i examples/realdata/FY09_EDU_Recipients_by_State.csv\n\nPerform a case-insensitive search:\n\n    csvGrep -c 1 -r \"^illinois\" --r-icase examples/realdata/FY09_EDU_Recipients_by_State.csv\n\nRemove comment rows:\n\n    **This example can not be demonstrated due to the _csvsuite_ does not support non-tabular forms.**\n\nGet the indices of the columns that contain matching text (\\x1e is the Record Separator (RS) character):\n\n    **This example can not be demonstrated due to the _csvsuite_ does not support the csvformat utility.**\n\n---\n\n#### csvJoin\n##### Description\nMerges two or more CSV tables together using a method analogous to SQL JOIN operation. By default, it performs an inner\njoin, but full outer, left outer, and right outer are also available via flags. Key columns are specified with the -c\nflag (either a single column which exists in all tables, or a comma-separated list of columns with one corresponding to\neach). If the columns flag is not provided then the tables will be merged “sequentially”, that is they will be merged in\nrow order with no filtering:\n\n    Usage: csvJoin arg_0  [options...]\n    arg_0 : The CSV files to operate on. [default: unknown]\n\nOptions:\n\n    --help : print help [implicit: \"true\", default: false]\n    -c,--columns : The column name(s) on which to join. Should be either one name (or index) or a comma-separated list with one name (or index) for each file, in the same order that the files were specified. May also be left unspecified, in which case the two files will be joined sequentially without performing any matching. [default: ]\n    --outer : Perform a full outer join, rather than the default inner join. [implicit: \"true\", default: false]\n    --honest-outer : Typify outer joins result before printing. [implicit: \"true\", default: false]\n    --left : Perform a left outer join, rather than the default inner join. If more than two files are provided this will be executed as a sequence of left outer joins, starting at the left. [implicit: \"true\", default: false]\n    --right : Perform a right outer join, rather than the default inner join. If more than two files are provided this will be executed as a sequence of right outer joins, starting at the right. [implicit: \"true\", default: false]\n    -I,--no-inference : Disable type inference (and --locale, --date-format, --datetime-format, --no-leading-zeroes) when parsing the input. [implicit: \"true\", default: false]\n\nSee also: [Arguments common to all tools](#arguments-common-to-all-tools).  \n\n\u003e NOTE: There has been introduced the `--honest-outer` option here. Well, the _csvkit_ does not recalculate types after the\nlast join, which is necessary in some cases.\n\n**Examples**\n\n    csvJoin -c 1 examples/join_a.csv examples/join_b.csv\n\nAdd two empty noname columns to the right of a CSV:  \n\n    echo , | csvJoin examples/dummy.csv _\n\nAdd two named columns (a,b) (as a matter of fact a2 and b2) to the right of a CSV:\n\n    echo a,b | csvJoin examples/dummy.csv _\n\nAdd a single column to the right of a CSV:  \n\n    echo \"new-column\" | csvJoin examples/dummy.csv _\n\n\u003e NOTE: _csvsuite_ uses `_` (instead of `-`) as a placeholder for piped source.\n\n---\n\n#### csvSort\n##### Description\nSort CSV files. Like the Unix “sort” command, but for tabular data:\n\n    Usage: csvSort arg_0  [options...]\n    arg_0 : The CSV file to operate on. If omitted, will accept input as piped data via STDIN. [default: ]\n\nOptions:\n\n    --help : print help [implicit: \"true\", default: false]\n    -n,--names : Display column names and indices from the input CSV and exit. [implicit: \"true\", default: false]\n    -c,--columns : A comma-separated list of column indices, names or ranges to sort by, e.g. \"1,id,3-5\". [default: all columns]\n    -r,--reverse : Sort in descending order. [implicit: \"true\", default: false]\n    -i,--ignore-case : Perform case-independent sorting. [implicit: \"true\", default: false]\n    -I,--no-inference : Disable type inference (and --locale, --date-format, --datetime-format, --no-leading-zeroes) when parsing the input. [implicit: \"true\", default: false]\n    -p,--parallel-sort : Use parallel sort. [implicit: \"true\", default: true]\n\nSee also: [Arguments common to all tools](#arguments-common-to-all-tools).\n\n\u003e NOTE: There has been introduced the `--parallel-sort` option to speed up the operation. It is ON by default now.\n\n**Examples**\n\nSort the veteran’s education benefits table by the “TOTAL” column (don't forget to specify the numeric locale):\n\n    csvSort -c 9 -L en_US examples/realdata/FY09_EDU_Recipients_by_State.csv\n\nView the five states with the most individuals claiming veteran’s education benefits (don't forget to specify the\nnumeric locale):\n\n    csvCut -c 1,9 examples/realdata/FY09_EDU_Recipients_by_State.csv | csvSort -r -c 2 -L en_US | head -n 5\n\n---\n\n#### csvStack\n##### Description\nStack up the rows from multiple CSV files, optionally adding a grouping value to each row:\n\n    Usage: csvStack arg_0  [options...]\n    arg_0 : The CSV files to operate on. [default: unknown]\n\nOptions:\n\n    --help : print help [implicit: \"true\", default: false]\n    -g,--groups : A comma-separated list of values to add as \"grouping factors\", one per CSV being stacked. These are added to the output as a new column. You may specify a name for the new column using the -n flag. [default: empty]\n    -n,--group-name : A name for the grouping column, e.g. \"year\". Only used when also specifying -g. [default: ]\n    --filenames : Use the filename of each input file as its grouping value. When specified, -g will be ignored. [implicit: \"true\", default: false]\n\n**Examples**\n\nJoin a set of files for different years:\n\n    csvStack -g 2009,2010 examples/realdata/FY09_EDU_Recipients_by_State.csv examples/realdata/Datagov_FY10_EDU_recp_by_State.csv\n\nAdd a single column to the left of a CSV:  \n\n    csvStack -n NEWCOL -g \"\" examples/dummy.csv\n\n---\n\n#### Output and Analysis\n* [csvJson](#csvjson)\n* [csvLook](#csvlook)\n* [csvSql](#csvsql)\n* [csvStat](#csvstat)\n---\n\n#### csvJson\n##### Description\nConverts a CSV file into JSON or GeoJSON (depending on flags):\n\n    Usage: csvJson arg_0  [options...]\n    arg_0 : The CSV file to operate on. If omitted, will accept input as piped data via STDIN. [default: ]\n\nOptions:\n\n    --help : print help [implicit: \"true\", default: false]\n    -i,--indent : Indent the output JSON this many spaces. Disabled by default. [default: -2147483648]\n    -k,--key : Output JSON as an object keyed by a given column, KEY, rather than as an array. All column values must be unique. If --lat and --lon are specified, this column is used as the GeoJSON Feature ID. [default: ]\n    --lat : A column index or name containing a latitude. Output will be GeoJSON instead of JSON. Requires --lon. [default: ]\n    --lon : A column index or name containing a longitude. Output will be GeoJSON instead of JSON. Requires --lat. [default: ]\n    --type : A column index or name containing a GeoJSON type. Output will be GeoJSON instead of JSON. Requires --lat and --lon. [default: ]\n    --geometry : A column index or name containing a GeoJSON geometry. Output will be GeoJSON instead of JSON. Requires --lat and --lon. [default: ]\n    --crs : A coordinate reference system string to be included with GeoJSON output. Requires --lat and --lon. [default: ]\n    --no-bbox : Disable the calculation of a bounding box. [implicit: \"true\", default: false]\n    --stream : Output JSON as a stream of newline-separated objects, rather than an as an array. [implicit: \"true\", default: false]\n    -I,--no-inference : Disable type inference (and --locale, --date-format, --datetime-format) when parsing the input. [implicit: \"true\", default: false]\n\nSee also: [Arguments common to all tools](#arguments-common-to-all-tools).\n\n\u003e NOTE: `--geometry` option for now is not supported.\n\n**Examples**  \n\nConvert veteran’s education dataset to JSON keyed by state abbreviation (again, do not forget to specify the numeric\nlocale):\n\n    $ csvJson -k \"State Abbreviate\" -i 4 examples/realdata/FY09_EDU_Recipients_by_State.csv -L en_US\n    {\n        \"AL\": {\n            \"State Name\": \"ALABAMA\",\n            \"State Abbreviate\": \"AL\",\n            \"Code\": 1.0,\n            \"Montgomery GI Bill-Active Duty\": 6718.0,\n            \"Montgomery GI Bill- Selective Reserve\": 1728.0,\n            \"Dependents' Educational Assistance\": 2703.0,\n            \"Reserve Educational Assistance Program\": 1269.0,\n            \"Post-Vietnam Era Veteran's Educational Assistance Program\": 8.0,\n            \"TOTAL\": 12426.0,\n            \"\": null\n        },\n        \"...\": {\n            \"...\": \"...\"\n        }\n    }\n\nConvert locations of public art into GeoJSON:\n\n    $ csvJson --date-format=%m/%d/%y --lat latitude --lon longitude --k slug --crs EPSG:4269 -i 4 examples/test_geo.csv\n    {\n        \"type\": \"FeatureCollection\",\n        \"bbox\": [\n            -95.334619,\n            32.299076986939205,\n            -95.250699,\n            32.351434\n        ],\n        \"features\": [\n            {\n                \"type\": \"Feature\",\n                \"properties\": {\n                    \"title\": \"Downtown Coffee Lounge\",\n                    \"description\": \"In addition to being the only coffee shop in downtown Tyler, DCL also features regular exhibitions of work by local artists.\",\n                    \"address\": \"200 West Erwin Street\",\n                    \"type\": \"Gallery\",\n                    \"last_seen_date\": \"2012-03-30\"\n                },\n                \"id\": \"dcl\",\n                \"geometry\": {\n                    \"type\": \"Point\",\n                    \"coordinates\": [\n                    -95.30181,\n                    32.35066\n                    ]\n                }\n            },\n            {\n                ...\n            },\n        ],\n        \"crs\": {\n            \"type\": \"name\",\n            \"properties\": {\n                \"name\": \"EPSG:4269\"\n            }\n        }\n    }\n\n---\n\n#### csvLook\n##### Description\nRenders a CSV to the command line in a Markdown-compatible, fixed-width format:\n\n    Usage: csvLook arg_0  [options...]\n    arg_0 : The CSV file to operate on. If omitted, will accept input as piped data via STDIN. [default: ]\n\nOptions:\n\n    --max-rows : The maximum number of rows to display before truncating the data. [default: 4294967295]\n    --max-columns : The maximum number of columns to display before truncating the data. [default: 4294967295]\n    --max-column-width : Truncate all columns to at most this width. The remainder will be replaced with ellipsis. [default: 4294967295]\n    --max-precision : The maximum number of decimal places to display. The remainder will be replaced with ellipsis. [default: 3]\n    --no-number-ellipsis : Disable the ellipsis if --max-precision is exceeded. [implicit: \"true\", default: false]\n    -I,--no-inference : Disable type inference (and --locale, --date-format, --datetime-format, --no-leading-zeroes) when parsing the input. [implicit: \"true\", default: false]\n    -G,--glob-locale : Superseded global locale. [default: C]\n\nSee also: [Arguments common to all tools](#arguments-common-to-all-tools).\n\n\u003e NOTE: Unlike the _csvkit_ , where this utility can ignore extra or missing cells in the absence of a strictly tabular\nform, the _csvLook_, like all other utilities in the _csvsuite_ , will simply report this by default (see the \n[-Q, --quick_check option](#arguments-common-to-all-tools)), and you need to anyway use the _csvClean_ utility or its\noriginal.  \n\n\u003e NOTE: There has been introduced `-G,--glob-locale` option, superseded global locale for numerics, for you to still see\nseparator signs in your numbers. Do not mix it with still existent `-L` option, which is the \"input\" locale for numerics.\n\n**Examples**\n\nBasic use:\n\n    csvLook examples/testfixed_converted.csv\n\nServe as the final operation when piping through other tools:\n\n    csvCut -c 9,1 examples/realdata/FY09_EDU_Recipients_by_State.csv | csvLook\n\nTo ignore extra cells:  \n\n    _not supported, see notation above._\n\nTo ignore missing cells:\n\n    _not supported, see notation above._\n\n---\n\n#### csvSql\n##### Description\nGenerate SQL statements for a CSV file or execute those statements directly on a database. In the latter case supports\nboth creating tables and inserting data.\n\n    Usage: csvSql arg_0  [options...]\n    arg_0 : The CSV files to operate on. [default: unknown]\n\nOptions:\n\n    --help : print help [implicit: \"true\", default: false]\n    -i,--dialect : Dialect of SQL {mysql,postgresql,sqlite,firebird,oracle} to generate. Cannot be used with --db or --query.  [default: ]\n    --db : If present, a 'soci' connection string to use to directly execute generated SQL on a database. [default: ]\n    --query : Execute one or more SQL queries delimited by --sql-delimiter, and output the result of the last query as CSV. QUERY may be a filename. --query may be specified multiple times. [default: ]\n    --insert : Insert the data into the table. Requires --db. [implicit: \"true\", default: false]\n    --prefix : Add an expression following the INSERT keyword, like OR IGNORE or OR REPLACE. [default: ]\n    --before-insert : Before the INSERT command, execute one or more SQL queries delimited by --sql-delimiter. Requires --insert. [default: ]\n    --after-insert : After the INSERT command, execute one or more SQL queries delimited by --sql-delimiter. Requires --insert. [default: ]\n    --sql-delimiter : Delimiter separating SQL queries in --query, --before-insert, and --after-insert. [default: ;]\n    --tables : A comma-separated list of names of tables to be created. By default, the tables will be named after the filenames without extensions or \"stdin\". [default: ]\n    --no-constraints : Generate a schema without length limits or null checks. Useful when sampling big tables. [implicit: \"true\", default: false]\n    --unique-constraint : A comma-separated list of names of columns to include in a UNIQUE constraint [default: ]\n    --no-create : Skip creating the table. Requires --insert. [implicit: \"true\", default: false]\n    --create-if-not-exists : Create the table if it does not exist, otherwise keep going. Requires --insert. [implicit: \"true\", default: false]\n    --overwrite : Drop the table if it already exists. Requires --insert. Cannot be used with --no-create. [implicit: \"true\", default: false]\n    -I,--no-inference : Disable type inference (and --locale, --date-format, --datetime-format, --no-leading-zeroes) when parsing the input. [implicit: \"true\", default: false]\n    --chunk-size : Chunk size for batch insert into the table. Requires --insert. [default: 0]\n\nSee also: [Arguments common to all tools](#arguments-common-to-all-tools).\n\n\u003e [--engine-option](https://csvkit.readthedocs.io/en/latest/scripts/csvsql.html) option is not supported because the\n\u003e utility is [SOCI](https://github.com/SOCI/soci) driven, not SQLAlchemy driven.  \n\u003e [--min-col-len and --col-len-multiplier](https://csvkit.readthedocs.io/en/latest/scripts/csvsql.html) options are\n\u003e not supported as well. See the utility [source](https://github.com/wiluite/csvsuite/blob/main/suite/csvSql.cpp) to\n\u003e know about constant lengths of text columns. This will definitely be fixed soon.  \n\u003e [--db-schema](https://csvkit.readthedocs.io/en/latest/scripts/csvsql.html) option is not supported as well.\n\n**Examples**  \n\n***Generate SQL statements***\n\nGenerate a statement in the PostgreSQL dialect:\n\n    csvSql -L en_US -i postgresql examples/realdata/FY09_EDU_Recipients_by_State.csv\n\n\u003e NOTE: Each further example assumes that you have already created your test database.\n\n***Interact with a SQL database***\n\nCreate a table and import data from the CSV directly into PostgreSQL:\n\n    csvSql -L en_US --db \"postgresql://dbname=database user=name password=pass\" --tables \"fy09\" --insert examples/realdata/FY09_EDU_Recipients_by_State.csv\n\nFor large tables it may not be practical to process the entire table. One solution to this is to analyze a sample of the\ntable. In this case it can be useful to turn off length limits and null checks with the `--no-constraints` option:\n\n    head -n 20 examples/realdata/FY09_EDU_Recipients_by_State.csv | csvSql -L en_US --no-constraints --tables fy09\n\nCreate tables for an entire directory of CSVs and import data from those files directly into PostgreSQL:\n\n    csvSql -L en_US --db \"postgresql://dbname=database user=name password=pass\" --insert examples/*_converted.csv\n\nIf those CSVs have identical headers, you can import them into the same table by using the _csvStack_ first:\n\n    csvStack examples/dummy?.csv | csvSql --db \"postgresql://dbname=database user=name password=pass\" --insert\n\n\u003e NOTE: in this case you will have a table named stdin in your database.\n\n***Query and output CSV files using SQL***\n\nYou can use csvSql to “directly” query one or more CSV files. Please note that this will create an in-memory SQLite\ndatabase, so it won’t be very fast:\n\n    csvSql --query \"SELECT m.usda_id, avg(i.sepal_length) AS mean_sepal_length FROM iris AS i JOIN irismeta AS m ON (i.species = m.species) GROUP BY m.species\" examples/iris.csv examples/irismeta.csv\n\nGroup rows by one column:\n\n    csvSql --query \"SELECT * FROM 'dummy3' GROUP BY a\" examples/dummy3.csv\n\nConcatenate two columns:\n\n    csvSql --query \"SELECT a || b FROM 'dummy3'\" --no-inference examples/dummy3.csv\n\nIf a column contains null values, you must COALESCE the column:\n\n    csvSql --query \"SELECT a || COALESCE(b, '') as RESULT FROM 'sort_ints_nulls'\" --no-inference examples/sort_ints_nulls.csv\n\nThe UPDATE SQL statement produces no output. Remember to SELECT the columns and rows you want:\n\n    csvSql --query \"UPDATE 'dummy3' SET a = '0'; SELECT * FROM 'dummy3'\" examples/dummy3.csv\n    csvSql --query \"UPDATE 'dummy3' SET a = '1'; SELECT * FROM 'dummy3'\" examples/dummy3.csv\n\n\u003e In this particular case, column 'a' should still only accept values of its specific type. Assigning other values\nas [here](https://csvkit.readthedocs.io/en/latest/scripts/csvsql.html) is possible, but will not appear in the selection.\n\n---\n\n#### csvStat\n##### Description\nPrints descriptive statistics for all columns in a CSV file. Will intelligently determine the type of each column and\nthen print analysis relevant to that type (ranges for dates, mean and median for integers, etc.):\n\n    Usage: csvStat arg_0  [options...]\n    arg_0 : The CSV file to operate on. If omitted, will accept input as piped data via STDIN. [default: ]\n\nOptions:\n\n    --help : print help [implicit: \"true\", default: false]\n    --csv : Output results as a CSV table, rather than plain text. [implicit: \"true\", default: false]\n    --json : Output results as a JSON test, rather than plain text. [implicit: \"true\", default: false]\n    -i,--indent : Indent the output JSON this many spaces. Disabled by default. [default: -2147483648]\n    -n,--names : Display column names and indices from the input CSV and exit. [implicit: \"true\", default: false]\n    -c,--columns : A comma-separated list of column indices, names or ranges to be examined, e.g. \"1,id,3-5\". [default: all columns]\n    --type : Only output data type. [implicit: \"true\", default: false]\n    --nulls : Only output whether columns contain nulls. [implicit: \"true\", default: false]\n    --non-nulls : Only output counts of non-null values. [implicit: \"true\", default: false]\n    --unique : Only output counts of unique values. [implicit: \"true\", default: false]\n    --min : Only output smallest values. [implicit: \"true\", default: false]\n    --max : Only output largest values. [implicit: \"true\", default: false]\n    --sum : Only output sums. [implicit: \"true\", default: false]\n    --mean : Only output means. [implicit: \"true\", default: false]\n    --median : Only output medians. [implicit: \"true\", default: false]\n    --stdev : Only output standard deviations. [implicit: \"true\", default: false]\n    --len : Only output the length of the longest values. [implicit: \"true\", default: false]\n    --max-precision : Only output the most decimal places. [implicit: \"true\", default: false]\n    --freq : Only output lists of frequent values. [implicit: \"true\", default: false]\n    --freq-count : The maximum number of frequent values to display. [default: 5]\n    --count : Only output total row count. [implicit: \"true\", default: false]\n    --decimal-format : %-format specification for printing decimal numbers. Defaults to locale-specific formatting with \"%.3f\" [default: %.3f]\n    -G,--no-grouping-separator : Do not use grouping separators in decimal numbers [implicit: \"true\", default: false]\n    -I,--no-inference : Disable type inference (and --locale, --date-format, --datetime-format, --no-leading-zeroes) when parsing the input. [implicit: \"true\", default: false]\n    --no-mdp,--no-max-precision : Do not calculate most decimal places. [implicit: \"true\", default: false]\n\nSee also: [Arguments common to all tools](#arguments-common-to-all-tools).\n\n\u003e There has been introduced `--no-mdp,--no-max-precision` option to turn off the most decimal places calculation if\nit is not necessary for you right now: if this calculation is performed through the boost::multiprecision library, then\nthis greatly slows down the work of the main purpose of the utility - other statistics. You can find out whether most\ndecimal places are calculated using the quick method (by default) or via boost::multiprecision library by looking at the\n[source](https://github.com/wiluite/csvsuite/blob/main/suite/include/reader-bridge-impl.hpp). In any case, you can build\nthe utility with one or another method of this calculation.\n\n**Examples**\n\nBasic use:\n\n    csvStat -L en_US examples/realdata/FY09_EDU_Recipients_by_State.csv\n\nWhen a statistic name is passed, only that stat will be printed:\n\n     $ csvStat -L en_US --min examples/realdata/FY09_EDU_Recipients_by_State.csv\n     1. State Name: None\n     2. State Abbreviate: None\n     3. Code: 1,\n     4. Montgomery GI Bill-Active Duty: 435,\n     5. Montgomery GI Bill- Selective Reserve: 48,\n     6. Dependents' Educational Assistance: 118,\n     7. Reserve Educational Assistance Program: 60,\n     8. Post-Vietnam Era Veteran's Educational Assistance Program: 1,\n     9. TOTAL: 768,\n    10. j: None\n\n\u003e In all cases of numeric output, the numeric locale is the global locale of your system environment (again, do not\nmix with the `-L` option which is the \"input\" locale for numerics).\n\nIf a single stat and a single column are requested, only a value will be returned:\n\n    $ csvStat -L en_US -c 4 --mean examples/realdata/FY09_EDU_Recipients_by_State.csv\n    6 263,904\n\n---\n\n### Arguments common to all tools\n\n    -z,--maxfieldsize : Maximum length of a single field in the input CSV file. [default: 4294967295]\n    -e,--encoding : Specify the encoding of the input CSV file. [default: UTF-8]\n    -S,--skipinitialspace : Ignore whitespace immediately following the delimiter. [implicit: \"true\", default: false]\n    -H,--no-header-row : Specify that the input CSV file has no header row. Will create default headers (a,b,c,...). [implicit: \"true\", default: false]\n    -K,--skip-lines : Specify the number of initial lines to skip before the header row (e.g. comments, copyright notices, empty rows). [default: 0]\n    -l,--linenumbers : Insert a column of line numbers at the front of the output. Useful when piping to grep or as a simple primary key. [implicit: \"true\", default: false]\n    --zero : When interpreting or displaying column numbers, use zero-based numbering instead of the default 1-based numbering. [implicit: \"true\", default: false]\n    -Q,--quick-check : Quickly check the CSV source for matrix shape [implicit: \"true\", default: true]\n    -L,--locale : Specify the locale (\"C\") of any formatted numbers. [default: C]\n    --blanks : Do not convert \"\", \"na\", \"n/a\", \"none\", \"null\", \".\" to NULL. [implicit: \"true\", default: false]\n    --null-value : Convert this value to NULL. --null-value can be specified multiple times. [default: unknown]\n    --date-format : Specify a strptime date format string like \"%m/%d/%Y\". [default: %m/%d/%Y]\n    --datetime-format : Specify a strptime datetime format string like \"%m/%d/%Y %I:%M %p\". [default: %m/%d/%Y %I:%M %p]\n    --no-leading-zeroes : Do not convert a numeric value with leading zeroes to a number. [implicit: \"true\", default: false]\n    --date-lib-parser : Use date library as Dates and DateTimes parser backend instead compiler-supported [implicit: \"true\", default: true]\n    --ASAP : Print result output stream as soon as possible. [implicit: \"true\", default: true]\n\nThere are 3 NEW options (compared to the _csvkit_) here:\n\n* `-Q,--quick-check` to make sure that the sources are in matrix form. Default is ON.\n* `--date-lib-parser` to turn on a professional date and date-time parser for cross-platform use. Default is ON.\n* `--ASAP` to emit the results of work without waiting for the source to be fully processed. Default is ON.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwiluite%2Fcsvsuite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwiluite%2Fcsvsuite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwiluite%2Fcsvsuite/lists"}