{"id":23379003,"url":"https://github.com/santisq/pscsvutil","last_synced_at":"2025-04-08T05:49:37.975Z","repository":{"id":62673382,"uuid":"552599156","full_name":"santisq/PSCsvUtil","owner":"santisq","description":"tiny module to split and merge large Csv files","archived":false,"fork":false,"pushed_at":"2023-05-21T17:55:34.000Z","size":14,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-14T03:55:29.658Z","etag":null,"topics":["csv","merge","powershell","split","streamreader","streamwriter"],"latest_commit_sha":null,"homepage":"https://www.powershellgallery.com/packages/PSCsvUtil","language":"PowerShell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/santisq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-16T23:35:06.000Z","updated_at":"2023-05-12T13:48:53.000Z","dependencies_parsed_at":"2022-11-04T09:45:57.240Z","dependency_job_id":null,"html_url":"https://github.com/santisq/PSCsvUtil","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santisq%2FPSCsvUtil","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santisq%2FPSCsvUtil/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santisq%2FPSCsvUtil/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santisq%2FPSCsvUtil/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/santisq","download_url":"https://codeload.github.com/santisq/PSCsvUtil/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247785927,"owners_count":20995644,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","merge","powershell","split","streamreader","streamwriter"],"created_at":"2024-12-21T19:16:09.283Z","updated_at":"2025-04-08T05:49:37.941Z","avatar_url":"https://github.com/santisq.png","language":"PowerShell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003ePSCsvUtil\u003c/h1\u003e\n\nPSCsvUtil is a tiny PowerShell Module composed of two functions to efficiently __split and merge__ big Csv files using [`StreamReader`](https://learn.microsoft.com/en-us/dotnet/api/system.io.streamreader.-ctor?view=net-7.0) and [`StreamWriter`](https://learn.microsoft.com/en-us/dotnet/api/system.io.streamwriter?view=net-7.0) .NET Classes.\n\n\n## Installation\n\n```powershell\nInstall-Module PSCsvUtil -Scope CurrentUser\n```\n\n## Compatibility\n\nTested and compatible with Windows PowerShell 5.1 and [PowerShell Core 7+](https://github.com/PowerShell/PowerShell).\n\n## Examples\n\n### Split a Csv into 5 chunks\n\nThis example demonstrates how to split a Csv using the `-Chunks` parameter from [`Split-Csv`](/public/Split-Csv.ps1).\n\n```powershell\nPS ..\\PS-CsvUtil\u003e gpstree .\\tests\\\n\nMode     Hierarchy                                   Size\n----     ---------                                   ----\nd----    tests                                       1.78 GB\n-a---    └── bigcsv.csv                              1.78 GB\n\nPS ..\\PS-CsvUtil\u003e Split-Csv .\\tests\\bigcsv.csv -DestinationFolder .\\tests\\by5chunks -PassThru -Chunks 5\n\nMode                 LastWriteTime         Length Name\n----                 -------------         ------ ----\n-a---          10/16/2022  8:12 PM      382955825 bigcsv - Part 0.csv\n-a---          10/16/2022  8:12 PM      382955849 bigcsv - Part 1.csv\n-a---          10/16/2022  8:13 PM      382955817 bigcsv - Part 2.csv\n-a---          10/16/2022  8:13 PM      382955855 bigcsv - Part 3.csv\n-a---          10/16/2022  8:14 PM      382954454 bigcsv - Part 4.csv\n\nPS ..\\PS-CsvUtil\u003e gpstree .\\tests\\\n\nMode     Hierarchy                                   Size\n----     ---------                                   ----\nd----    tests                                       1.78 GB\n-a---    ├── bigcsv.csv                              1.78 GB\nd----    └── by5chunks                               1.78 GB\n-a---        ├── bigcsv - Part 0.csv                 365.22 MB\n-a---        ├── bigcsv - Part 1.csv                 365.22 MB\n-a---        ├── bigcsv - Part 2.csv                 365.22 MB\n-a---        ├── bigcsv - Part 3.csv                 365.22 MB\n-a---        └── bigcsv - Part 4.csv                 365.21 MB\n\nPS ..\\PS-CsvUtil\u003e Import-Csv '.\\tests\\by5chunks\\bigcsv - Part 3.csv' | Select-Object -First 3 | FT -Auto\n\nColumn0    Column1    Column2    Column3    Column4    Column5\n-------    -------    -------    -------    -------    -------\n2051598092 444770706  545010177  699773973  933288705  2135983609\n649944517  1548086840 1011873614 1927411868 1902850446 240759930\n182136426  1642338384 983474548  38656729   167556228  569658758\n```\n\n### Split a Csv into chunks of 250Mb\n\nThis example demonstrates how to split a Csv using the `-Size` parameter from [`Split-Csv`](/public/Split-Csv.ps1).\n\n```powershell\nPS ..\\PS-CsvUtil\u003e Split-Csv .\\tests\\bigcsv.csv -DestinationFolder .\\tests\\by50mbchunks -Size 50mb\nPS ..\\PS-CsvUtil\u003e gpstree .\\tests\\by50mbchunks\\\n\nMode     Hierarchy                                                        Size\n----     ---------                                                        ----\nd----    by50mbchunks                                                     1.78 GB\n-a---    ├── bigcsv - Part 00.csv                                         50 MB\n-a---    ├── bigcsv - Part 01.csv                                         50 MB\n-a---    ├── bigcsv - Part 02.csv                                         50 MB\n-a---    ├── bigcsv - Part 03.csv                                         50 MB\n....\n....\n....\n-a---    ├── bigcsv - Part 33.csv                                         50 MB\n-a---    ├── bigcsv - Part 34.csv                                         50 MB\n-a---    ├── bigcsv - Part 35.csv                                         50 MB\n-a---    └── bigcsv - Part 36.csv                                         26.08 MB\n\nPS ..\\PS-CsvUtil\u003e Import-Csv '.\\tests\\by50mbchunks\\bigcsv - Part 28.csv' | Select-Object -First 3 | FT -Auto\n\nColumn0    Column1   Column2    Column3    Column4    Column5\n-------    -------   -------    -------    -------    -------\n1241863732 224211646 1035291147 1555398003 1777828960 416276892\n1148375056 433051937 1367055122 556502602  38341193   113835498\n2056137503 825665841 1921526568 2102926379 1093669865 2030084321\n```\n\n### Merging a split Csv\n\nThis example demonstrates how to merge the Csv files generated with the examples above using [`Merge-Csv`](/public/Merge-Csv.ps1).\n\n```powershell\nPS ..\\PS-CsvUtil\u003e Get-ChildItem .\\tests\\by50mbchunks\\ -Filter *Part*.csv | Merge-Csv -DestinationPath mergedCsv50mbchunks.csv -PassThru\n\n    Directory: ..\\pwsh\\PS-CsvUtil\n\nMode                 LastWriteTime         Length Name\n----                 -------------         ------ ----\n-a---          10/16/2022  8:32 PM     1914777556 mergedCsv50mbchunks.csv\n\nPS ..\\PS-CsvUtil\u003e (Get-FileHash .\\mergedCsv50mbchunks.csv -Algorithm MD5).Hash\n0452ACEC1664A93137C41D131FA4C3A9\n\nPS ..\\PS-CsvUtil\u003e (Get-FileHash .\\tests\\bigcsv.csv -Algorithm MD5).Hash\n0452ACEC1664A93137C41D131FA4C3A9\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsantisq%2Fpscsvutil","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsantisq%2Fpscsvutil","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsantisq%2Fpscsvutil/lists"}