{"id":26657574,"url":"https://github.com/ghuls/polars_streaming_csv_decompression","last_synced_at":"2026-01-17T07:32:50.503Z","repository":{"id":277401621,"uuid":"932315834","full_name":"ghuls/polars_streaming_csv_decompression","owner":"ghuls","description":"Polars IO plugin for reading compressed CSV/TSV files in a streaming fashion","archived":false,"fork":false,"pushed_at":"2025-02-24T15:10:38.000Z","size":11,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-06T21:41:49.126Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ghuls.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-13T18:03:49.000Z","updated_at":"2025-02-24T15:10:42.000Z","dependencies_parsed_at":"2025-02-14T14:02:29.269Z","dependency_job_id":null,"html_url":"https://github.com/ghuls/polars_streaming_csv_decompression","commit_stats":null,"previous_names":["ghuls/polars_streaming_csv_decompression"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghuls%2Fpolars_streaming_csv_decompression","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghuls%2Fpolars_streaming_csv_decompression/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghuls%2Fpolars_streaming_csv_decompression/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghuls%2Fpolars_streaming_csv_decompression/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ghuls","download_url":"https://codeload.github.com/ghuls/polars_streaming_csv_decompression/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245431716,"owners_count":20614182,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-25T09:01:31.724Z","updated_at":"2026-01-17T07:32:50.490Z","avatar_url":"https://github.com/ghuls.png","language":"Python","funding_links":[],"categories":["Libraries/Packages/Scripts"],"sub_categories":["Polars plugins"],"readme":"# Polars IO plugin for reading compressed CSV/TSV files in a streaming fashion\n\nThis plugin provides a way to read compressed CSV/TSV files in a streaming fashion for usage with [Polars](https://pola.rs/).\n\nCurrently, [Polars](https://pola.rs/) decompresses compressed CSV/TSV files completely in memory\n(when using `pl.read_csv(\"file.csv.gz\")` or `pl.scan_csv(\"file.csv.gz\")`) before trying to parse them, which results in\na lot of memory usage when reading large compressed CSV/TSV files (several GBs to 100s of GBs) as common in e.g. bioinformatics.\n\nThis plugin provides a way to read compressed CSV/TSV files in a streaming fashion, where the file is decompressed and\nparsed in chunks. This results in a much lower overall memory usage when reading large compressed CSV/TSV files.\n\nAs it is mainly intended for reading large compressed CSV/TSV files produced by bioinformatics tools, records are\nassumed to be separated by `eol_char` (=`\"\\n\"` by default) and embedded `eol_char` in fields are not expected. The last\nrecord also should end in `eol_char`. If those conditions are not met, reading such files could give corrupt data.\n\nIt can also be used for decoding CSV files with a different character encoding than `utf8` and/or for decoding CSV files for\nwhich not all bytes can be decoded in that encoding. Compared with `read_csv`, the decoding will require a lower amount of\ntotal memory.\n\nStreaming decompression is handled by [xopen](https://github.com/pycompression/xopen/), which supports the following compression\nformats and backends and automatically selects the best backend available on the system:\n  - gzip (`.gz`):\n    - [python-isal](https://github.com/pycompression/python-isal)\n    - [python-zlib-ng](https://github.com/pycompression/python-zlib-ng)\n    - [pigz](https://zlib.net/pigz/) (a parallel version of gzip)\n    - [gzip](https://www.gnu.org/software/gzip/)\n  - bzip2 (`.bz2`):\n    - [pbzip2](http://compression.great-site.net/pbzip2/) (parallel bzip2)\n  - xz (`.xz`):\n    - [xz](https://github.com/tukaani-project/xz)\n  - Zstandard (`.zst`) (optional)\":\n    - [zstd](https://github.com/facebook/zstd)\n    - [zstdandard](https://github.com/indygreg/python-zstandard): Install with `pip install xopen[zstd]`\n  - fallback to Python’s built-in functions (`gzip.open`, `lzma.open`, `bz2.open`) if none of the other methods can be\n    used.\n\n\n## Installation\n\n```bash\npip install git+https://github.com/ghuls/polars_streaming_csv_decompression.git\n```\n\n## Usage\n\n```python\nimport polars as pl\nimport polars_streaming_csv_decompression\n\n# Read compressed CSV file in a streaming fashion.\n(\n    polars_streaming_csv_decompression.streaming_csv(\n        \"my_big_file.csv.gz\"\n    )  # lazy, doesn't do a thing\n    .select(\n        [\"a\", \"c\"]\n    )  # select only 2 columns (other columns will not be read)\n    .filter(\n        pl.col(\"a\") \u003e 10\n    )  # the filter is pushed down the scan, so less data is read into memory\n    .head(100)  # constrain number of returned results to 100\n)\n\n\n# Read CSV file with non-utf8 encoding in a streaming fashion.\n(\n    polars_streaming_csv_decompression.streaming_csv(\n        \"file_encoded_in_windows-1252.csv\",\n        encoding=\"windows-1252\",\n    )\n    .head()\n)\n\n# Read CSV file with non-utf8 encoding where not all bytes can be decoded in a streaming fashion.\n(\n    polars_streaming_csv_decompression.streaming_csv(\n        \"file_encoded_in_windows-1252_but_not_all_bytes_can_be_decoded.csv\",\n        encoding=\"windows-1252-lossy\",\n    )\n    .head()\n)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fghuls%2Fpolars_streaming_csv_decompression","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fghuls%2Fpolars_streaming_csv_decompression","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fghuls%2Fpolars_streaming_csv_decompression/lists"}