https://github.com/adrianh-za/netcore-process-large-csv

App for sorting a very large csv by using chunking. Just for fun.
https://github.com/adrianh-za/netcore-process-large-csv

csv dotnet parallel

Last synced: 8 months ago
JSON representation

App for sorting a very large csv by using chunking. Just for fun.

• Host: GitHub
• URL: https://github.com/adrianh-za/netcore-process-large-csv
• Owner: adrianh-za
• Created: 2024-11-27T09:57:06.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-11-28T14:03:47.000Z (over 1 year ago)
• Last Synced: 2025-04-04T05:43:23.172Z (11 months ago)
• Topics: csv, dotnet, parallel
• Language: C#
• Homepage:
• Size: 13.7 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
This was done for fun!

The purpose of the app is to sort a large unsorted (ten million records - 500MB) csv file without loading the entire file into memory and sorting,

The app will chunk the large CSV file into smaller files.  Each file is then sorted in parallel. Once all the chunked files are sorted, they are then merged into a single sorted CSV file.

The solution consists of two consoles apps, and a single class library

 - GenerateLargeCSV

	 - This is for generating the large, unsorted CSV.

 - ProcessLargeCSV

	 - This is for sorting the large, unsorted CSV.

 - LibLargeCSV

	 - The lib containing the CSV utils and data type.