{"id":15035883,"url":"https://github.com/agenty/filereader","last_synced_at":"2025-04-09T23:15:01.388Z","repository":{"id":115857641,"uuid":"94849909","full_name":"Agenty/FileReader","owner":"Agenty","description":"C# library to read extremely large text, csv, tsv files efficiently","archived":false,"fork":false,"pushed_at":"2018-06-08T04:27:48.000Z","size":42,"stargazers_count":20,"open_issues_count":0,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-09T23:14:55.773Z","etag":null,"topics":["csharp","csharp-code","csharp-library","dot-net","dotnet","dotnet-framework","nuget"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Agenty.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-20T04:31:17.000Z","updated_at":"2024-10-01T12:28:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"2a193842-f473-4053-b280-94f0c7a646b3","html_url":"https://github.com/Agenty/FileReader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agenty%2FFileReader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agenty%2FFileReader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agenty%2FFileReader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agenty%2FFileReader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Agenty","download_url":"https://codeload.github.com/Agenty/FileReader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248125591,"owners_count":21051770,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csharp","csharp-code","csharp-library","dot-net","dotnet","dotnet-framework","nuget"],"created_at":"2024-09-24T20:29:41.278Z","updated_at":"2025-04-09T23:15:01.349Z","avatar_url":"https://github.com/Agenty.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FileReader\n\nFileReader is a c# library open sourced by [Agenty](https://www.agenty.com) used to read and process very large text files with pagination by setting ``limit`` and ``offset`` parameters.\n\nBecause loading the whole text file in memory will cause objects to grow, and .net will throw OOM exceptions if it cannot allocate enough contiguous memory for an object.\n\nSo FileReader architecture is designed to stream the file with pagination instead reading the entire content in one go. Which prevents the ``out of memory execption`` when you want to read big TXT files, which is in size around 500 MB or more\n\n## Delimiter\n  - Tab (``\\t``)\n  - Comma (``,``)\n\n## Basic Example \n\n```\nstring Path = @\"C:\\sample.txt\";\nvar table = Path.FileToTable(heading: true, delimiter: '\\t');\n\n// All your processing here\n\ntable.TableToFile(@\"C:\\output.txt\");\n```\n\n## Pagination Example \n\n```\nint Offset = 0;\nint Limit = 100000\nstring Path = @\"C:\\sample.txt\";\nvar table = Path.FileToTable(heading: true, delimiter: '\\t', offset : Offset, limit: Limit);\n\n// Do all your processing here and with limit and offset and save to drive in append mode\n// The append mode will write the output in same file for each processed batch.\n\ntable.TableToFile(@\"C:\\output.txt\");\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagenty%2Ffilereader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagenty%2Ffilereader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagenty%2Ffilereader/lists"}