https://github.com/agenty/filereader
C# library to read extremely large text, csv, tsv files efficiently
https://github.com/agenty/filereader
csharp csharp-code csharp-library dot-net dotnet dotnet-framework nuget
Last synced: 3 months ago
JSON representation
C# library to read extremely large text, csv, tsv files efficiently
- Host: GitHub
- URL: https://github.com/agenty/filereader
- Owner: Agenty
- License: mit
- Created: 2017-06-20T04:31:17.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2018-06-08T04:27:48.000Z (about 7 years ago)
- Last Synced: 2025-04-09T23:14:55.773Z (3 months ago)
- Topics: csharp, csharp-code, csharp-library, dot-net, dotnet, dotnet-framework, nuget
- Language: C#
- Homepage:
- Size: 41 KB
- Stars: 20
- Watchers: 3
- Forks: 9
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# FileReader
FileReader is a c# library open sourced by [Agenty](https://www.agenty.com) used to read and process very large text files with pagination by setting ``limit`` and ``offset`` parameters.
Because loading the whole text file in memory will cause objects to grow, and .net will throw OOM exceptions if it cannot allocate enough contiguous memory for an object.
So FileReader architecture is designed to stream the file with pagination instead reading the entire content in one go. Which prevents the ``out of memory execption`` when you want to read big TXT files, which is in size around 500 MB or more
## Delimiter
- Tab (``\t``)
- Comma (``,``)
## Basic Example
```
string Path = @"C:\sample.txt";
var table = Path.FileToTable(heading: true, delimiter: '\t');
// All your processing here
table.TableToFile(@"C:\output.txt");
```
## Pagination Example
```
int Offset = 0;
int Limit = 100000
string Path = @"C:\sample.txt";
var table = Path.FileToTable(heading: true, delimiter: '\t', offset : Offset, limit: Limit);
// Do all your processing here and with limit and offset and save to drive in append mode
// The append mode will write the output in same file for each processed batch.
table.TableToFile(@"C:\output.txt");
```