{"id":34838966,"url":"https://github.com/sumeshi/qsv-rs","last_synced_at":"2026-05-25T04:34:27.692Z","repository":{"id":295590422,"uuid":"977277550","full_name":"sumeshi/qsv-rs","owner":"sumeshi","description":"A fast, flexible, and memory-efficient command-line tool written in Rust for processing large CSV files.","archived":false,"fork":false,"pushed_at":"2025-07-01T03:17:42.000Z","size":287,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-01T04:26:42.520Z","etag":null,"topics":["csv","dfir","forensics","loganalysis","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sumeshi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-03T20:33:20.000Z","updated_at":"2025-07-01T03:17:46.000Z","dependencies_parsed_at":"2025-06-14T03:15:15.927Z","dependency_job_id":null,"html_url":"https://github.com/sumeshi/qsv-rs","commit_stats":null,"previous_names":["sumeshi/qsv-rs"],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/sumeshi/qsv-rs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumeshi%2Fqsv-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumeshi%2Fqsv-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumeshi%2Fqsv-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumeshi%2Fqsv-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sumeshi","download_url":"https://codeload.github.com/sumeshi/qsv-rs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumeshi%2Fqsv-rs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33461089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-25T02:24:28.008Z","status":"ssl_error","status_checked_at":"2026-05-25T02:23:23.339Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","dfir","forensics","loganalysis","rust"],"created_at":"2025-12-25T16:58:09.866Z","updated_at":"2026-05-25T04:34:27.685Z","avatar_url":"https://github.com/sumeshi.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Quilter-CSV\n[![MIT License](http://img.shields.io/badge/license-MIT-blue.svg?style=flat)](LICENSE)\n[![CI/CD Pipeline](https://github.com/sumeshi/qsv-rs/actions/workflows/release.yml/badge.svg?branch=main)](https://github.com/sumeshi/qsv-rs/actions/workflows/release.yml)\n\n![qsv-rs](https://gist.githubusercontent.com/sumeshi/c2f430d352ae763273faadf9616a29e5/raw/8484142e88948ecc0c8887db8f3bbb5be0dbe51e/qsv-rs.svg)\n\nA fast, flexible, and memory-efficient command-line tool written in Rust for processing large CSV files. Inspired by [xsv](https://github.com/BurntSushi/xsv) and built on [Polars](https://www.pola.rs/), it's designed for handling tens or hundreds of gigabytes of CSV data efficiently in workflows like log analysis and digital forensics.\n\n\u003e [!NOTE]\n\u003e The original version of this project was implemented in Python and can be found at [sumeshi/quilter-csv](https://github.com/sumeshi/quilter-csv). This Rust version is a complete rewrite.\n\n## Features\n\n- **Pipeline-style command chaining**: Chain multiple commands in a single line for fast and efficient data processing\n- **Flexible filtering and transformation**: Perform operations like select, filter, sort, deduplicate, and timezone conversion\n- **YAML-based batch processing (Quilt)**: Automate complex workflows using YAML configuration files\n\n## Usage\n![](https://gist.githubusercontent.com/sumeshi/644af27c8960a9b6be6c7470fe4dca59/raw/2a19fafd4f4075723c731e4a8c8d21c174cf0ffb/qsv.svg)\n\n### Getting Help\n\nTo see available commands and options, run `qsv` without any arguments:\n\n```bash\n$ qsv -h\n```\n\n### Example\n\nHere's an example of reading a CSV file, extracting rows that contain 4624 in the 'Event ID' column, and displaying the top 3 rows sorted by the 'Date and Time' column:\n\n```bash\n$ qsv load Security.csv - isin 'Event ID' 4624 - sort 'Date and Time' - head 3 - showtable\n```\n\nThis command:\n1. Loads `Security.csv`\n2. Filters rows where `Event ID` is 4624\n3. Sorts by `Date and Time`\n4. Shows the first 3 rows as a table\n\n### Command Structure\n\nqsv commands are composed of three types of steps:\n\n- **Initializer**: Loads data (e.g., `load`)\n- **Chainable**: Transforms or filters data (e.g., `select`, `grep`, `sort`, etc.)\n- **Finalizer**: Outputs or summarizes data (e.g., `show`, `showtable`, `headers`, etc.)\n\nEach step is separated by a hyphen (`-`):\n\n```bash\n$ qsv \u003cINITIALIZER\u003e \u003cargs\u003e - \u003cCHAINABLE\u003e \u003cargs\u003e - \u003cFINALIZER\u003e \u003cargs\u003e\n```\n\n### Command Separator `-`\n\nThe `-` token (a single hyphen surrounded by spaces) is the command separator. A standalone `-` is never treated as data.\n\n- To separate commands: `qsv load file.csv - select col1 - head 5`\n- If you need `-` as an option value, use an attached form such as `--separator=-` or `-s-`.\n\nA standalone `-` positional value, including stdin-style usage, is not currently supported.\n\n**Note:** If no finalizer is explicitly specified, default builds automatically use `showtable`, making it easy to quickly view results:\n\n```bash\n$ qsv load data.csv - select col1,col2 - head 5\n# Equivalent to:\n$ qsv load data.csv - select col1,col2 - head 5 - showtable\n```\n\nBuilds compiled without the optional `table` feature fall back to `show` instead, and the `showtable` command prints a rebuild hint.\n\n## Command Reference\n\n### Initializers\n\n#### `load`\nLoad one or more CSV or Parquet files.\n\n**Supported formats:**\n- CSV files (.csv, .tsv, .txt)\n- Gzipped CSV files (.csv.gz)\n- Parquet files (.parquet) - high performance, preserves data types\n\n| Parameter     | Type        | Default | Description                                      |\n|---------------|-------------|---------|--------------------------------------------------|\n| path          | list[str] |         | One or more paths to CSV or Parquet files. Quoted glob patterns such as `\"logs/*.tsv\"` are supported. Cannot mix CSV and Parquet files in the same command. |\n| -s, --separator | str       | `,`     | Field separator character (CSV files only).     |\n| --low-memory  | flag    | `false` | Enable low-memory mode for very large files (CSV files only). |\n| --no-headers  | flag    | `false` | Treat the first row as data, not headers (CSV files only). When enabled, columns will be named automatically (`column_1`, `column_2`, etc.). |\n| --chunk-size  | int     | (auto)  | Number of rows to read per chunk (CSV files only). Controls memory usage during file processing. |\n\n**Environment Variables:**\n- `QSV_CHUNK_SIZE`: Default chunk size for CSV processing (overrides auto-detection, can be overridden by --chunk-size)\n- `QSV_MEMORY_LIMIT_MB`: Memory limit for gzip decompression and streaming operations (default: 1024MB, range: 512-4096MB)\n\nExample:\n```bash\n$ qsv load data.csv\n$ qsv load data.csv.gz\n$ qsv load data1.csv data2.csv data3.csv\n$ qsv load \"logs/*.tsv\" -s $'\\t'\n$ qsv load \"logs/*.tsv\" --separator=$'\\t'\n$ qsv load data.csv --low-memory\n$ qsv load data.csv --no-headers\n$ qsv load data.csv --chunk-size 50000\n$ qsv load cache.parquet                              # Load from parquet cache\n$ qsv load cache1.parquet cache2.parquet              # Load multiple parquet files\n```\n\n### Chainable Functions\n\n#### `select`\nSelect columns by name, numeric index, or range notation.\n\n| Parameter | Type                | Default | Description                                                                                                |\n|-----------|---------------------|---------|------------------------------------------------------------------------------------------------------------|\n| colnames  | str/list/range      |         | Column name(s) or indices. Supports multiple formats (see examples below). This is a required argument. |\n\n**Column Selection Formats:**\n- **Individual columns**: `col1,col3` - Select specific columns by name\n- **Numeric indices**: `1,3` - Select columns by position (1-based indexing)  \n- **Range notation (hyphen)**: `col1-col3` - Select range using hyphen\n- **Range notation (colon)**: `col1:col3` - Select range using colon\n- **Numeric range**: `2:4` - Select 2nd through 4th columns (e.g., col1, col2, col3)\n- **Quoted colon notation**: `\"col:1\":\"col:3\"` - For column names containing colons\n- **Mixed formats**: `1,col2,4:6` - Combine different selection methods\n\n**Disambiguation rule:** If an exact column name matching `col1-col3` exists, it is selected as-is. Range expansion only occurs when no exact match is found.\n\n```bash\n$ qsv load data.csv - select datetime                       # Select single column by name\n$ qsv load data.csv - select col1,col3                      # Select specific columns by name\n$ qsv load data.csv - select col1-col3                      # Select range using hyphen\n$ qsv load data.csv - select col1:col3                      # Select range using colon\n$ qsv load data.csv - select 1                              # Select 1st column (datetime)\n$ qsv load data.csv - select 2:4                            # Select 2nd-4th columns (col1, col2, col3)\n$ qsv load data.csv - select 2,4                            # Select 2nd and 4th columns (col1, col3)\n$ qsv load data.csv - select \"col:1\":\"col:3\"                # For columns with colons in names\n$ qsv load data.csv - select 1,datetime,3:5                 # Mixed selection methods\n```\n\n#### `isin`\nFilter rows where a column matches any of the given values.\n\n| Parameter | Type   | Default | Description                                                                          |\n|-----------|--------|---------|--------------------------------------------------------------------------------------|\n| colname   | str    |         | Column name to filter. Required.                                                     |\n| values    | list   |         | Comma-separated values. Filters rows where the column matches any of these values (OR condition). Required. |\n\n```bash\n$ qsv load data.csv - isin col1 1\n$ qsv load data.csv - isin col1 1,4\n```\n\n#### `contains`\nFilter rows where a column contains a specific literal substring.\n\n| Parameter   | Type   | Default | Description                                 |\n|-------------|--------|---------|---------------------------------------------|\n| colname     | str    |         | Column name to search. Required.            |\n| substring   | str    |         | The literal substring to search for. Required. |\n| -i, --ignorecase | flag | `false` | Perform case-insensitive matching.          |\n\n```bash\n$ qsv load data.csv - contains str ba\n$ qsv load data.csv - contains str BA -i\n$ qsv load data.csv - contains str BA --ignorecase\n```\n\n#### `sed`\nReplace values in column(s) using a Regex pattern.\n\n| Parameter   | Type   | Default | Description                                 |\n|-------------|--------|---------|---------------------------------------------|\n| pattern     | str    |         | Regex pattern to search for. Required.      |\n| replacement | str    |         | Replacement string. Required.               |\n| --column    | str    | (all)   | Apply replacement to specific column only. If not specified, applies to all columns. |\n| -i, --ignorecase | flag | `false` | Perform case-insensitive matching.          |\n\n\u003e **Warning:** When `--column` is omitted, `sed` replaces across **all columns**. In log/DFIR data this can silently modify timestamps, EventIDs, file paths, and usernames. Always specify `--column` unless you intend a full-dataset replacement.\n\n```bash\n$ qsv load data.csv - sed foo foooooo                       # Replace 'foo' with 'foooooo' in all columns\n$ qsv load data.csv - sed foo foooooo --column str          # Replace 'foo' with 'foooooo' in 'str' column only\n$ qsv load data.csv - sed FOO foooooo -i                    # Case-insensitive replacement in all columns\n$ qsv load data.csv - sed \".*o.*\" foooooo --column str      # Regex replacement in specific column\n```\n\n#### `grep`\nFilter rows where any column matches a regex pattern.\n\n| Parameter | Type | Default | Description |\n|---|---|---|---|\n| pattern | str |         | Regex pattern to search for in any column. Required. |\n| --column | str | (all columns) | Restrict search to specific column(s). Comma-separated for multiple. |\n| -i, --ignore-case | flag | `false` | Perform case-insensitive matching. |\n| -v, --invert-match | flag | `false` | Invert the sense of matching, to select non-matching lines. |\n\nExample:\n```bash\n$ qsv load data.csv - grep foo \n$ qsv load data.csv - grep \"^FOO\" -i                        # Case-insensitive search\n$ qsv load data.csv - grep \"^FOO\" --ignore-case              # Long form case-insensitive\n$ qsv load data.csv - grep \"^FOO\" -i -v                     # Case-insensitive inverted match\n$ qsv load data.csv - grep \"^FOO\" --ignore-case --invert-match  # Long form inverted match\n$ qsv load logs.csv - grep \"FAILED\" --column EventData\n$ qsv load logs.csv - grep \"192\\\\.168\\\\.\" --column src_ip,dst_ip\n```\n\n#### `head`\nDisplays the first N rows of the dataset.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| number | int  | 5       | Number of rows to display. Can be specified as positional argument or with -n/--number option. |\n| -n, --number | int | | Alternative way to specify number of rows. |\n\n```bash\n$ qsv load data.csv - head 3\n$ qsv load data.csv - head 10\n$ qsv load data.csv - head -n 3\n$ qsv load data.csv - head --number 10\n```\n\n#### `tail`\nDisplays the last N rows of the dataset.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| number | int  | 5       | Number of rows to display. Can be specified as positional argument or with -n/--number option. |\n| -n, --number | int | | Alternative way to specify number of rows. |\n\n```bash\n$ qsv load data.csv - tail 3\n$ qsv load data.csv - tail 10\n$ qsv load data.csv - tail -n 3\n$ qsv load data.csv - tail --number 10\n```\n\n#### `sort`\nSorts the dataset based on the specified column(s).\n\n\u003e ⚠️ **Memory:** This command materializes the full dataset into memory.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| colnames  | str/list |         | Column name(s) to sort by. Comma-separated for multiple columns (e.g., `col1,col3`) or a single column name. Required. |\n| -d, --desc    | flag | `false` | Sort in descending order. Applies to all specified columns. |\n\n```bash\n$ qsv load data.csv - sort str\n$ qsv load data.csv - sort str -d\n$ qsv load data.csv - sort str --desc\n$ qsv load data.csv - sort col1,col2,col3 --desc\n```\n\n#### `count`\nCount duplicate rows, grouping by all columns by default. Results are automatically sorted by count in descending order.\n\n\u003e ⚠️ **Memory:** This command materializes the full dataset into memory.\n\n| Parameter | Type | Default | Description |\n|---|---|---|---|\n| columns   | str | (all columns) | Optional positional column list. Use `col1` or `col1,col2` to group by specific columns only. |\n\n```bash\n$ qsv load Security.csv - count EventID          # Count by one column\n$ qsv load proxy.csv - count src_ip,dst_ip       # Count by multiple columns\n$ qsv load data.csv - count                       # Count all unique rows (original behavior)\n$ qsv load data.csv - count - sort col1          # Count and then sort by col1 instead\n```\n\n#### `uniq`\nFilters unique rows, removing duplicates based on all columns.\n\n\u003e ⚠️ **Memory:** This command materializes the full dataset into memory.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| (None)    |      |         | Takes no arguments. Removes duplicate rows based on all columns. |\n\n```bash\n$ qsv load data.csv - uniq\n```\n\n#### `changetz`\nChanges the timezone of a datetime column.\n\n| Parameter | Type | Default | Description |\n|---|---|---|---|\n| colname | str |         | Name of the datetime column. Required. |\n| --from-tz | str |         | Source timezone (e.g., `UTC`, `America/New_York`, `local`). Required. |\n| --to-tz | str |         | Target timezone (e.g., `Asia/Tokyo`). Required. |\n| --input-format | str | `auto` | Input datetime format string (e.g., `%Y-%m-%d %H:%M:%S%.f`). `auto` uses intelligent parsing similar to Python's dateutil.parser, supporting fuzzy parsing and automatic format detection. |\n| --output-format | str | `auto` | Output datetime format string (e.g., `%Y/%m/%d %H:%M:%S`). `auto` uses ISO8601 format `%Y-%m-%dT%H:%M:%S%.6f%:z` (microsecond precision). |\n| --ambiguous | str | `earliest` | Strategy for ambiguous times during DST transitions: `earliest` (first occurrence) or `latest` (second occurrence). |\n\n**Understanding `--ambiguous` option:**\n\nDuring Daylight Saving Time (DST) transitions in autumn, clocks \"fall back\" creating duplicate hours. For example, 2:30 AM occurs twice:\n- First time: 2:30 AM DST (before transition)  \n- Second time: 2:30 AM Standard Time (after transition)\n\nWhen encountering such ambiguous times:\n- `earliest`: Uses the first occurrence (DST time)\n- `latest`: Uses the second occurrence (Standard time)\n\nExample:\n```bash\n$ qsv load data.csv - changetz datetime --from-tz UTC --to-tz Asia/Tokyo\n# Output: 2023-01-01T09:00:00.123456+09:00 (ISO8601 with microsecond precision)\n\n$ qsv load data.csv - changetz datetime --from-tz UTC --to-tz America/New_York --input-format \"%Y/%m/%d %H:%M\" --output-format \"%Y-%m-%d %H:%M:%S\"\n# Custom output format\n\n$ qsv load data.csv - changetz datetime --from-tz America/New_York --to-tz UTC --ambiguous latest\n# Handle ambiguous DST times\n\n# Automatic format detection (similar to Python dateutil.parser):\n$ qsv load logs.csv - changetz timestamp --from-tz local --to-tz UTC\n# Handles: \"Jan 15, 2023 2:30 PM\", \"2023/01/15 14:30\", \"15-Jan-2023 14:30:00\", etc.\n\n# Fuzzy parsing with embedded text:\n$ qsv load events.csv - changetz event_time --from-tz EST --to-tz UTC  \n# Handles: \"Meeting on January 15th, 2023 at 2:30 PM\", \"Call scheduled for Jan 15 2023\"\n```\n\n**TODO:** Upgrade to 7-digit sub-second precision (100-nanosecond precision for Windows FILETIME compatibility) when chrono-tz library supports it. Current `auto` output uses microsecond precision.\n\n#### `renamecol`\nRenames a specific column.\n\n| Parameter   | Type | Default | Description             |\n|-------------|------|---------|-------------------------|\n| old_name    | str  |         | The current column name. Required. |\n| new_name    | str  |         | The new column name. Required.   |\n\n```bash\n$ qsv load data.csv - renamecol current_name new_name\n```\n\n#### `convert`\nConverts data formats between JSON, YAML, and XML. Also supports formatting/prettifying data in the same format.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| colname | str |         | Column name containing the data to convert. Required. |\n| --from | str |         | Source format: `json`, `yaml`, or `xml`. Required. |\n| --to | str |         | Target format: `json`, `yaml`, or `xml`. Required. |\n\n**Supported conversions:**\n- Cross-format: `json ↔ yaml`, `json ↔ xml`, `yaml ↔ xml`\n- Same-format (formatting): `json → json`, `yaml → yaml`, `xml → xml`\n\n**Features:**\n- Automatically handles malformed JSON with extra quotes\n- Prettifies and formats data for better readability\n- Preserves data structure during conversion\n\nExample:\n```bash\n$ qsv load data.csv - convert json_col --from json --to yaml\n$ qsv load data.csv - convert config --from yaml --to json\n$ qsv load data.csv - convert data --from json --to xml\n$ qsv load data.csv - convert messy_json --from json --to json  # Format/prettify JSON\n$ qsv load data.csv - convert compact_yaml --from yaml --to yaml  # Format YAML\n```\n\n#### `timeline`\nAggregates data by time intervals, creating time-based summaries.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| time_column | str |         | Name of the datetime column to use for time bucketing. Required. |\n| --interval | str |         | Time interval for aggregation (e.g., `1h`, `30m`, `5s`, `1d`). Required. |\n| --sum | str | | Column name to sum within each time bucket. Optional. |\n| --avg | str | | Column name to average within each time bucket. Optional. |\n| --min | str | | Column name to find minimum within each time bucket. Optional. |\n| --max | str | | Column name to find maximum within each time bucket. Optional. |\n| --std | str | | Column name to calculate standard deviation within each time bucket. Optional. |\n\n**Features:**\n- Creates a time bucket column named `timeline_{interval}` (e.g., `timeline_1h`, `timeline_30m`)\n- If no aggregation column is specified, only row counts are provided for each time bucket\n- Supports various time interval formats: hours (`1h`), minutes (`30m`), seconds (`5s`), days (`1d`)\n\n**CLI and Quilt YAML mapping:**\n\n| CLI option | YAML key |\n|-----------|----------|\n| `--interval 1h` | `interval: 1h` |\n| `--avg cpu_usage` | `agg_type: avg` + `agg_column: cpu_usage` |\n| `--sum value` | `agg_type: sum` + `agg_column: value` |\n\nExample:\n```bash\n$ qsv load access.log - timeline timestamp --interval 1h\n# Creates column: timeline_1h\n\n$ qsv load metrics.csv - timeline time --interval 5m --avg cpu_usage\n# Creates columns: timeline_5m, count, avg_cpu_usage\n\n$ qsv load sales.csv - timeline date --interval 1d --sum amount\n# Creates columns: timeline_1d, count, sum_amount\n\n$ qsv load server.log - timeline timestamp --interval 30s --max response_time\n# Creates columns: timeline_30s, count, max_response_time\n```\n\n#### `timeslice`\nFilters data based on time ranges, extracting records within specified time boundaries.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| time_column | str |         | Name of the datetime column to filter on. Required. |\n| --start | str | | Start time (inclusive). Optional. |\n| --end | str | | End time (inclusive). Optional. |\n\nAt least one of `--start` or `--end` must be specified. Both boundaries are inclusive (`[start, end]`). Supports various datetime formats including ISO8601, timestamps, and common log formats.\n\nExample:\n```bash\n$ qsv load data.csv - timeslice timestamp --start \"2023-01-01 00:00:00\"\n$ qsv load data.csv - timeslice timestamp --end \"2023-12-31 23:59:59\"\n$ qsv load data.csv - timeslice timestamp --start \"2023-06-01\" --end \"2023-06-30\"\n$ qsv load access.log - timeslice timestamp --start \"2023-01-01T10:00:00\"\n```\n\n#### `pivot`\nCreates grouped aggregations over row and column keys.\n\n\u003e **Note:** `pivot` currently performs a **grouped aggregation** (long-form output), not an Excel-style wide cross-tabulation. If you need wide output, use `select` + `count` or wait for the planned `--wide` flag. Alternatives that better describe this operation: `groupby`, `aggregate`, `summarize`.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| --rows | str |         | Comma-separated list of columns for row grouping. Optional. |\n| --cols | str |         | Comma-separated list of columns for column grouping. Optional. |\n| --values | str |         | Column to aggregate values from. Required. |\n| --agg | str |         | Aggregation function: `sum`, `mean`, `count`, `min`, `max`, `median`, `std`. Optional (default: `sum`). |\n\nAt least one of `--rows` or `--cols` must be specified.\n\nExample:\n```bash\n$ qsv load sales.csv - pivot --rows region --cols product --values sales_amount --agg sum\n$ qsv load data.csv - pivot --rows category --cols year --values revenue --agg mean\n$ qsv load logs.csv - pivot --rows date --cols error_type --values count --agg count\n$ qsv load metrics.csv - pivot --rows department --values performance --agg median\n```\n\n#### `timeround`\nRounds datetime values to specified time units, creating a new rounded column while preserving the original.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| colname | str |         | Name of the datetime column to round. Required. |\n| --unit | str |         | Time unit for rounding: `y`/`year`, `M`/`month`, `d`/`day`, `h`/`hour`, `m`/`minute`, `s`/`second`. Required. |\n| --output | str | `{column}_rounded` | Output column name. Defaults to `{column}_rounded`, preserving the original column. |\n\n**Features:**\n- Rounds datetime values down to the nearest specified time unit boundary\n- Useful for time-based grouping and analysis\n- Supports both short (`h`, `d`) and long (`hour`, `day`) unit names\n- Output format automatically adjusts to the specified unit (clean, minimal format)\n\n**Output formats by unit:**\n- **year (y)**: `2023`\n- **month (M)**: `2023-01`\n- **day (d)**: `2023-01-01`\n- **hour (h)**: `2023-01-01 12`\n- **minute (m)**: `2023-01-01 12:34`\n- **second (s)**: `2023-01-01 12:34:56`\n\nExample:\n```bash\n$ qsv load data.csv - timeround timestamp --unit d --output date_only\n# Input:  2023-01-01 12:34:56\n# Output: 2023-01-01\n\n$ qsv load data.csv - timeround timestamp --unit h --output hour_rounded\n# Input:  2023-01-01 12:34:56\n# Output: 2023-01-01 12\n\n$ qsv load logs.csv - timeround timestamp --unit m\n# Rounds to minute boundary, writes to timestamp_rounded\n\n$ qsv load metrics.csv - timeround created_at --unit year --output created_year\n# Input:  2023-01-01 12:34:56\n# Output: 2023\n```\n\n### Finalizers\n\nFinalizers are used to output or summarize the processed data. They are typically the last command in a chain.\n\n#### `partition`\nSplits data into separate CSV files based on unique values in a specified column. Each unique value creates its own file.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| colname | str |         | Column name to partition by. Required. |\n| output_directory | str | `./partitions/` | Directory to save partitioned files. Optional - if not specified, creates a `./partitions/` directory. |\n\nThe output directory will be created if it doesn't exist. Each file is named after the unique value in the partition column (with invalid filename characters replaced by underscores).\n\nExample:\n```bash\n$ qsv load data.csv - partition category                    # Uses default ./partitions/ directory\n$ qsv load data.csv - partition category ./partitions/      # Explicit directory\n$ qsv load sales.csv - partition region ./by_region/\n$ qsv load logs.csv - partition date ./daily_logs/\n$ qsv load data.csv - select col1,col2 - partition col1 ./numeric_partitions/\n```\n\n#### `headers`\nDisplays the column headers of the current dataset.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| -p, --plain   | flag | `false` | Display headers as plain text, one per line, instead of a formatted table. |\n\nExample:\n```bash\n$ qsv load data.csv - headers\n$ qsv load data.csv - headers -p\n$ qsv load data.csv - headers --plain\n```\n\n#### `stats`\nDisplays summary statistics for each column in the dataset (e.g., count, null_count, mean, std, min, max).\n\n\u003e [!WARNING]\n\u003e This command loads the entire dataset into memory to compute statistics. It may fail or cause performance issues with very large files (e.g., 10GB+). For large datasets, consider using `head` or other filters to reduce the data size before running `stats`.\n\nThis command does not take any arguments or options.\n\nExample:\n```bash\n$ qsv load data.csv - stats\n```\n\n#### `showquery`\nDisplays the Polars LazyFrame query plan. This is useful for debugging and understanding the operations being performed.\n\nThis command does not take any arguments or options.\n\nExample:\n```bash\n$ qsv load data.csv - select col1 - showquery\n```\n\n#### `show`\nDisplays the resulting data as CSV to standard output. Header is included by default.\n\n| Parameter | Type | Default | Description |\n|---|---|---|---|\n| --batch-size | str | `1GB` | Memory batch size for streaming large datasets (e.g., `512MB`, `2GB`). Range: 1MB-10GB. |\n\nExample:\n```bash\n$ qsv load data.csv - head 5 - show\n$ qsv load huge.csv - show --batch-size 2GB                 # Streaming mode for large files\n$ qsv load data.csv - select col1,col2 - show --batch-size 512MB\n```\n\n#### `showtable`\nDisplays the resulting data in a formatted table to standard output. Shows table dimensions and intelligently truncates large datasets.\n\n**Features:**\n- Displays table size information (rows × columns) like Python Polars\n- For datasets with 9+ rows: shows the first 8 rows and a truncation indicator (`⋮`)\n- For datasets with 8 or fewer rows: shows all rows without truncation\n- Automatically used as default finalizer when no explicit finalizer is specified\n\nThis command does not take any arguments or options.\nThis command is controlled by the optional cargo feature `table`, which is enabled in the default build.\n\n\u003e **Tip for large files:** Pipe through `head N` before `showtable`, or use `show` instead. `showtable` with implicit finalization collects all rows by default.\n\nExample:\n```bash\n$ qsv load data.csv - select col1,col2 - head 3 - showtable\n# Output includes: shape: (3, 2) followed by formatted table\n\n$ qsv load large_data.csv - select col1,col2\n# Automatically calls showtable if no finalizer specified\n```\n\nTo build a smaller binary without table rendering support:\n\n```bash\n$ cargo build --release --no-default-features\n```\n\nIn that build, `showtable` exits with a clear rebuild message and implicit finalization falls back to `show`.\n\n#### `dump`\nOutputs the processing results to a CSV file.\n\n| Parameter | Type | Default | Description |\n|---|---|---|---|\n| -o, --output | str | `dump_\u003ctimestamp\u003e.csv` | File path to save the CSV data. Optional - if not specified, a default timestamped filename is automatically generated. |\n| -s, --separator | char | `,` | Field separator character for the output CSV file. |\n| --batch-size | str | `1GB` | Memory batch size for streaming large datasets (e.g., `512MB`, `2GB`). Range: 1MB-10GB. |\n\nExample:\n```bash\n$ qsv load data.csv - dump                                  # Saves to dump_\u003ctimestamp\u003e.csv\n$ qsv load data.csv - head 100 - dump -o results.csv\n$ qsv load data.csv - head 100 - dump --output results.csv\n$ qsv load data.csv - head 100 - dump -o results.csv -s ';'\n$ qsv load huge.csv - dump -o output.csv --batch-size 2GB   # Streaming mode for large files\n```\n\n#### `dumpcache`\nSaves the processing results as a Parquet cache file for fast reloading.\n\n**Features:**\n- Saves DataFrame as compressed Parquet format\n- Preserves data types (unlike CSV)\n- High-performance for large datasets\n- Can be loaded back using the `load` command\n\n| Parameter | Type | Default | Description |\n|---|---|---|---|\n| -o, --output | str | `cache_\u003ctimestamp\u003e.parquet` | Output file path (optional). Extension will be changed to .parquet if not specified. |\n\nExample:\n```bash\n$ qsv load data.csv - head 100 - dumpcache                 # Auto-named cache file\n$ qsv load data.csv - select col1,col2 - dumpcache -o cache.parquet\n$ qsv load data.csv - sort col1 - dumpcache --output processed_data\n\n# Load from cache for fast access\n$ qsv load cache.parquet - show\n```\n\n### Quilt (YAML Workflows)\n\nQuilt allows you to define complex data processing workflows in YAML configuration files. This is useful for automating repetitive tasks or creating reusable data processing pipelines.\n\n#### Usage\nThe `quilt` command itself takes the path to a YAML configuration file. Input data sources and other parameters are typically defined within the YAML file.\n\n```bash\n$ qsv quilt \u003cconfig_file_path.yaml\u003e [options]\n```\n| Parameter | Type | Description |\n|---|---|---|\n| config_file_path.yaml | str | Path to the YAML configuration file defining the pipeline stages. Required. |\n| -o, --output | str | Overrides the output path defined in the YAML config for the final dump operation (if any). |\n\n\n#### Example: Running a Quilt File\n```bash\n$ qsv quilt rules/my_workflow.yaml\n$ qsv quilt rules/my_analysis.yaml -o custom_output.csv\n```\n\nThe YAML configuration file (e.g., `rules/my_workflow.yaml`) defines the stages and steps. For example, the `Sample YAML (rules/test.yaml)` below defines a pipeline that:\n1. Loads data (implicitly or explicitly via a `load` step in a `process` stage).\n2. Performs selections and a join operation across different stages.\n3. Displays the final result as a table.\n\n#### Pipeline Operations in YAML\nWithin a Quilt YAML file, stages can be of different types to orchestrate the flow.\n\n| Operation Type | Description                                                | Key Parameters                                                                                                                                    |\n| -------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |\n| `process`      | Executes a series of qsv operations on a dataset.          | `steps`: Dictionary of operations (e.g., `load`, `select`, `head`, `showtable`). Each key is a qsv command, and its value contains arguments/options. \u003cbr\u003e `source` (optional): Specifies the output of a previous stage as input. |\n| `concat`       | Concatenates multiple datasets (stages).                   | `sources`: List of stage names whose outputs to concatenate. \u003cbr\u003e`params.how` (optional): Method for concatenation, `vertical` (default). Note: `horizontal` concatenation is not yet implemented. |\n| `join`         | Joins datasets from multiple stages based on keys.         | `sources`: List of two stage names whose outputs to join. \u003cbr\u003e`params.left_on`/`params.right_on` or `params.on`: Column(s) for joining. \u003cbr\u003e`params.how` (optional): Join type (`inner`, `left`, `outer`, `cross`). |\n| `where` step   | Filters rows using a SQL `WHERE` clause embedded in a process step. | `sql`: Full SQL statement such as `SELECT * FROM logs WHERE ...`. \u003cbr\u003e`field_map` (optional): Inline mapping of SQL field names to CSV column names. \u003cbr\u003e`annotate` (optional): Add `sigma_title`, `sigma_id`, `sigma_level`, `sigma_tags` columns. |\n\n#### `sigma2quilt`\n\n`sigma2quilt` converts Zircolite JSON rules into a regular quilt YAML file. The generated quilt uses normal `process`, `concat`, and `output` stages, and each rule becomes a `where` step over `${input}`.\n\n- A single-rule JSON file defaults to `quilt-\u003crule-title\u003e.yaml`\n- Rule titles are converted to lowercase hyphen-joined filenames\n- `--separate` writes one quilt file per rule\n- Each conversion also writes one `_mapping.json` template for the whole input ruleset\n- `--annotate` is mainly useful when multiple rules are kept in one generated quilt and you want the matched rows to retain per-rule metadata\n- For `rules_dir/` input, `-o \u003cdir\u003e` is required\n\nSupported SQL in the generated `where` step:\n1. `=`\n2. `LIKE ... ESCAPE '\\'`\n3. `NOT (...)`\n4. `AND`\n5. `OR`\n\nField resolution is:\n1. Explicit mapping via `qsv quilt --mapping \u003cfile\u003e` or `params.field_map`\n2. Exact CSV column match\n3. Case-insensitive CSV column match\n4. Otherwise warn and skip that condition\n\n```yaml\ntitle: Sigma JSON Conversion: rules_windows_generic\n\nstages:\n  load_stage:\n    type: process\n    steps:\n      load:\n        path: ${input}\n\n  detect_1_suspicious_high_integritylevel_conhost_legacy_option:\n    type: process\n    source: load_stage\n    steps:\n      where:\n        sql: \"SELECT * FROM logs WHERE Channel='Security' AND EventID=4688\"\n        annotate: true\n        sigma_title: \"Suspicious High IntegrityLevel Conhost Legacy Option\"\n        sigma_id: \"3037d961-21e9-4732-b27a-637bcc7bf539\"\n        sigma_level: \"informational\"\n        sigma_tags: \"attack.defense-evasion,attack.t1202\"\n\n  output_stage:\n    type: output\n    source: detect_1_suspicious_high_integritylevel_conhost_legacy_option\n    steps:\n      dump:\n        output: ${output}\n```\n\nGenerated mapping template example:\n\n```json\n{\n  \"Channel\": \"\",\n  \"CommandLine\": \"\",\n  \"EventID\": \"\"\n}\n```\n\nRecommended flow:\n\n1. Run `sigma2quilt` to generate both the quilt YAML and `_mapping.json`\n2. Fill in the CSV column names inside `_mapping.json`\n3. Run `quilt --mapping \u003cgenerated_mapping.json\u003e`\n\nExamples:\n\n```bash\n$ qsv sigma2quilt rules_windows_generic.json\n$ qsv sigma2quilt rules_windows_generic.json -o custom.yaml\n$ qsv sigma2quilt rules_dir/ -o generated_quilts/\n$ qsv sigma2quilt rules_windows_generic.json --annotate\n$ qsv sigma2quilt rules_windows_generic.json --separate -o generated_quilts/\n$ qsv quilt quilt-rules_windows_generic.yaml --mapping quilt-rules_windows_generic_mapping.json --var input=events.csv --var output=alerts.csv\n```\n\nTimeline steps in Quilt use explicit aggregation keys:\n\n```yaml\nstages:\n  hourly_metrics:\n    type: process\n    steps:\n      load:\n        path: metrics.csv\n      timeline:\n        time_column: timestamp\n        interval: 1h\n        agg_type: avg\n        agg_column: cpu_usage\n      show:\n```\n\n## Huge File Processing\n\nqsv-rs supports streaming processing for huge files without loading them entirely into memory.\n\n### Memory Behavior by Command\n\nNot all commands stream. Before running a pipeline on a large file, check the memory behavior of each operation:\n\n| Mode | Commands | Notes |\n|------|----------|-------|\n| **Streaming** (safe for huge files) | `show`, `dump`, `head`, `tail` | Row-by-row; constant memory |\n| **Lazy / Polars-optimized** | `select`, `isin`, `contains`, `grep`, `sed` | Pushdown; usually safe |\n| **Materializing** ⚠️ | `sort`, `uniq`, `count`, `stats`, `pivot`, `timeline` | Loads all rows into memory |\n\n\u003e **Warning:** Running a materializing command on a multi-GB file may exhaust memory. Use `head`, `timeslice`, or `isin` to reduce the dataset first.\n\n### Usage Examples\n\n```bash\n# Stream display huge files (1GB batches by default)\n$ qsv load huge.csv - show\n\n# Custom memory usage - 512MB batches\n$ qsv load huge.csv - show --batch-size 512MB\n\n# High-memory server - 2GB batches for maximum performance\n$ qsv load huge.csv - show --batch-size 2GB\n\n# Stream save large results to file with custom batch size\n$ qsv load huge.csv - select important,columns - dump -o output.csv --batch-size 2GB\n```\n\n### Memory Configuration\n\n```bash\n# Configure batch size for your system\n--batch-size 512MB    # Low memory systems\n--batch-size 1GB      # Default (balanced)\n--batch-size 2GB      # High memory systems (2GB+)\n\n# Configure gzip decompression memory (environment variable)\nexport QSV_MEMORY_LIMIT_MB=512   # Low memory systems\nexport QSV_MEMORY_LIMIT_MB=1024  # Default (1GB)\nexport QSV_MEMORY_LIMIT_MB=2048  # High memory systems (2GB+)\n```\n\n### Gzip File Processing\n\n```bash\n# Process large gzip files with different memory settings\n$ QSV_MEMORY_LIMIT_MB=2048 qsv load huge.csv.gz - show\n$ QSV_MEMORY_LIMIT_MB=512 qsv load huge.csv.gz - head 1000 - show  # Low memory\n```\n\n### Parquet Cache for Performance\n\nFor repeated processing of large CSV files, convert to Parquet format for significantly faster loading.\n\n**Performance Benefits:**\n- Faster loading compared to CSV format\n- Better compression (smaller file sizes)\n- Preserves data types (no re-parsing needed)\n\n```bash\n# One-time conversion: CSV to Parquet cache\n$ qsv load huge.csv - dumpcache -o huge.parquet\n\n# Subsequent processing: Load from Parquet (much faster)\n$ qsv load huge.parquet - select col1,col2 - show\n$ qsv load huge.parquet - isin category \"important\" - dump -o result.csv\n```\n\n## Installation\n\n### Pre-built Binaries\nDownload the latest release from [GitHub Releases](https://github.com/sumeshi/qsv-rs/releases).\n\n### Build from Source\n```bash\n$ git clone https://github.com/sumeshi/qsv-rs.git\n$ cd qsv-rs\n$ cargo build --release\n```\n\n## Contributing\nContributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n## License\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\nInspired by [xsv](https://github.com/BurntSushi/xsv).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsumeshi%2Fqsv-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsumeshi%2Fqsv-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsumeshi%2Fqsv-rs/lists"}