{"id":40168768,"url":"https://github.com/jqnatividad/csv-nose","last_synced_at":"2026-06-14T02:05:48.770Z","repository":{"id":333363287,"uuid":"1137017320","full_name":"jqnatividad/csv-nose","owner":"jqnatividad","description":"Detecting CSV file dialects by Table Uniformity Measurement and Data Type Inference","archived":false,"fork":false,"pushed_at":"2026-06-14T00:07:53.000Z","size":358,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-14T00:13:06.527Z","etag":null,"topics":["cli","csv","library","sniffer"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jqnatividad.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-18T19:07:30.000Z","updated_at":"2026-06-14T00:07:56.000Z","dependencies_parsed_at":"2026-01-21T19:01:49.868Z","dependency_job_id":null,"html_url":"https://github.com/jqnatividad/csv-nose","commit_stats":null,"previous_names":["jqnatividad/csv-nose"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/jqnatividad/csv-nose","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jqnatividad%2Fcsv-nose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jqnatividad%2Fcsv-nose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jqnatividad%2Fcsv-nose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jqnatividad%2Fcsv-nose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jqnatividad","download_url":"https://codeload.github.com/jqnatividad/csv-nose/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jqnatividad%2Fcsv-nose/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34306774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-14T02:00:07.365Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","csv","library","sniffer"],"created_at":"2026-01-19T17:01:29.072Z","updated_at":"2026-06-14T02:05:48.764Z","avatar_url":"https://github.com/jqnatividad.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Crates.io](https://img.shields.io/crates/v/csv-nose.svg?logo=crates.io)](https://crates.io/crates/csv-nose)\n[![Docs.rs](https://docs.rs/csv-nose/badge.svg)](https://docs.rs/crate/csv-nose)\n![License: MIT OR Apache-2.0](https://img.shields.io/crates/l/csv-nose.svg)\n[![DOI](https://zenodo.org/badge/1137017320.svg)](https://doi.org/10.5281/zenodo.18303093)\n\n# csv-nose\n\nA Rust port of the [Table Uniformity Method](https://github.com/ws-garcia/CSVsniffer) for CSV dialect detection.\n\n## Background\n\nThis crate [implements](docs/IMPLEMENTATION.md) the algorithm from [\"Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference\"](https://doi.org/10.3233/DS-240062)[^1] by [W. García](https://github.com/ws-garcia).\n\nBenchmarked against the standard CSVsniffer test suites, **csv-nose is the most accurate and robust of the dialect sniffers compared**; it ranks first or second on every dataset on both success ratio and F1, leads outright on POLLOCK, W3C-CSVW and CSV Wrangling; the **only tool to have 100% quote detection accuracy on the W3C-CSVW and POLLOCK test suites**; and the **only tool that never errors on a single file**.[^bench]\n\nThis implementation of the Table Uniformity Method achieves 99.55%[^2] accuracy on the [W3C-CSVW test suite](https://github.com/w3c/csvw) by:\n\n1. Testing multiple potential dialects (delimiter × quote × line terminator combinations)\n2. Scoring each dialect based on table uniformity (consistent field counts)\n3. Scoring based on type detection (consistent data types within columns)\n4. Selecting the dialect with the highest combined gamma score\n\n[^1]: García W. Detecting CSV file dialects by table uniformity measurement and data type inference. Data Science. 2024;7(2):55-72. [doi:10.3233/DS-240062](https://doi.org/10.3233/DS-240062)\n\n[^bench]: Across the five CSVsniffer benchmark suites — POLLOCK, W3C-CSVW, and the CSV Wrangling set with its CODEC and MESSY subsets — csv-nose ranks first or second among the six tools compared on both success ratio and F1, leads outright on POLLOCK, W3C-CSVW, and CSV Wrangling, and is the only tool with a 0% error rate on every suite. The sole exception is the filtered CODEC and MESSY subsets, where DuckDB's `sniff_csv` ties csv-nose on CODEC and narrowly edges ahead on MESSY (while placing last on POLLOCK). All F1 figures are derived with a single consistent methodology from each tool's success and error ratios. See the [Accuracy Benchmarks](#accuracy-benchmarks) tables for full per-tool figures.\n\n## Installation\n\n### As a library\n\n```bash\ncargo add csv-nose\n```\n\n### As a CLI tool\n\n```bash\ncargo install csv-nose\n```\n\n### With HTTP support (for remote URLs)\n\n```bash\ncargo install csv-nose --features http\n```\n\n## Library Usage\n\n```rust\nuse csv_nose::{Sniffer, SampleSize};\n\nlet mut sniffer = Sniffer::new();\nsniffer.sample_size(SampleSize::Records(100));\n\nlet metadata = sniffer.sniff_path(\"data.csv\").unwrap();\n\nprintln!(\"Delimiter: {}\", metadata.dialect.delimiter as char);\nprintln!(\"Has header: {}\", metadata.dialect.header.has_header_row);\nprintln!(\"Fields: {:?}\", metadata.fields);\nprintln!(\"Types: {:?}\", metadata.types);\n```\n\n## CLI Usage\n\n```bash\ncsv-nose data.csv                    # Sniff a single file\ncsv-nose *.csv                       # Sniff multiple files\ncsv-nose -f json data.csv            # Output as JSON\ncsv-nose --delimiter-only data.csv   # Output only the delimiter\ncsv-nose -v data.csv                 # Verbose output with field types\ncsv-nose https://example.com/data.csv  # Sniff remote CSV (requires http feature)\ncsv-nose local.csv https://example.com/remote.csv  # Mix local and remote\n```\n\n```bash\ncsv-nose -v /tmp/NYC_311_SR_2010-2020-sample-1M.csv\nFile: /tmp/NYC_311_SR_2010-2020-sample-1M.csv\n  Delimiter: ','\n  Quote: '\"'\n  Has header: true\n  Preamble rows: 0\n  Flexible: false\n  UTF-8: true\n  Fields: 41\n  Avg record length: 547 bytes\n  Field details:\n    1: Unique Key (Unsigned)\n    2: Created Date (DateTime)\n    3: Closed Date (DateTime)\n    4: Agency (Text)\n    5: Agency Name (Text)\n    6: Complaint Type (Text)\n    7: Descriptor (Text)\n    8: Location Type (Text)\n    9: Incident Zip (Unsigned)\n    10: Incident Address (Text)\n    11: Street Name (Text)\n    12: Cross Street 1 (Text)\n    13: Cross Street 2 (Text)\n    14: Intersection Street 1 (Text)\n    15: Intersection Street 2 (Text)\n    16: Address Type (Text)\n    17: City (Text)\n    18: Landmark (Text)\n    19: Facility Type (Text)\n    20: Status (Text)\n    21: Due Date (DateTime)\n    22: Resolution Description (Text)\n    23: Resolution Action Updated Date (DateTime)\n    24: Community Board (Text)\n    25: BBL (Unsigned)\n    26: Borough (Text)\n    27: X Coordinate (State Plane) (Unsigned)\n    28: Y Coordinate (State Plane) (Unsigned)\n    29: Open Data Channel Type (Text)\n    30: Park Facility Name (Text)\n    31: Park Borough (Text)\n    32: Vehicle Type (NULL)\n    33: Taxi Company Borough (NULL)\n    34: Taxi Pick Up Location (Text)\n    35: Bridge Highway Name (NULL)\n    36: Bridge Highway Direction (NULL)\n    37: Road Ramp (NULL)\n    38: Bridge Highway Segment (NULL)\n    39: Latitude (Float)\n    40: Longitude (Float)\n    41: Location (Text)\n```\n\n## Remote URL Support\n\nWhen built with the `http` feature, csv-nose can sniff remote CSV files directly from URLs:\n\n```bash\n# Build with HTTP support\ncargo build --release --features http\n\n# Sniff remote CSV\ncsv-nose https://raw.githubusercontent.com/datasets/gdp/main/data/gdp.csv\n\n# Limit bytes fetched (useful for large remote files)\ncsv-nose -b 8192 https://example.com/large.csv\n```\n\nThe HTTP feature uses Range requests when supported by the server to minimize data transfer. If the server doesn't support Range requests, it falls back to downloading and truncating at the sample size limit.\n\n## API Compatibility\n\nThis library is designed as a drop-in replacement for [qsv-sniffer](https://github.com/jqnatividad/qsv-sniffer) used by [qsv](https://github.com/dathere/qsv). The public API mirrors qsv-sniffer for easy migration:\n\n```rust\nuse csv_nose::{Sniffer, Metadata, Dialect, Header, Quote, Type, SampleSize, DatePreference};\n\nlet mut sniffer = Sniffer::new();\nsniffer\n    .sample_size(SampleSize::Records(50))\n    .date_preference(DatePreference::MdyFormat)\n    .delimiter(b',')\n    .quote(Quote::Some(b'\"'));\n```\n\n## Accuracy Benchmarks\n\ncsv-nose is benchmarked against the [same test datasets](docs/BENCHMARK_DATASETS_INFO.md) used by [CSVsniffer](https://github.com/ws-garcia/CSVsniffer), enabling direct accuracy comparison with other CSV dialect detection tools.\n\n### Success Ratio\n\nThe table below shows the dialect detection success ratio. Accuracy is measured using only files that do not produce errors during dialect inference.\n\n| Data set | `csv-nose` | `CSVsniffer MADSE` | `CSVsniffer` | `CleverCSV` | `csv.Sniffer` | DuckDB `sniff_csv` |\n|:---------|:-----------|:-------------------|:-------------|:------------|:--------------|:-------------------|\n| POLLOCK  | **97.97%** | 95.27%             | 96.55%       | 95.17%      | 96.35%        | 84.14%             |\n| W3C-CSVW[^2] | **99.55%** | 94.52%             | 95.39%       | 61.11%      | 97.69%        | 99.08%             |\n| CSV Wrangling | **93.30%** | 90.50%          | 89.94%       | 87.99%      | 84.26%        | 91.62%             |\n| CSV Wrangling CODEC | **92.25%** | 90.14%    | 90.14%       | 89.44%      | 84.18%        | 92.25%             |\n| CSV Wrangling MESSY | **91.27%** | 89.60%    | 89.60%       | 89.60%      | 83.06%        | 91.94%             |\n\n[^2]: csv-nose is optimized for the [W3C CSV on the Web Test Suite](https://w3c.github.io/csvw/tests/) - reaching 99.55% accuracy.\n\n### Failure Ratio\n\nThe table below shows the failure ratio (errors during dialect detection) for each tool.\n\n\u003e **Note:** \"Errors\" are files that caused crashes or exceptions during processing (e.g., encoding issues, malformed data). This is distinct from \"failures\" where a file was successfully processed but the wrong dialect was detected. A 0% error rate means all files were processed without crashes, even if some detections were incorrect.\n\n| Data set             | `csv-nose` | `CSVsniffer MADSE` | `CSVsniffer` | `CleverCSV` | `csv.Sniffer` | DuckDB `sniff_csv` |\n|:---------------------|:-----------|:-------------------|:-------------|:------------|:--------------|:-------------------|\n| POLLOCK [148 files]  | **0.00%**  | 0.00%              | 2.03%        | 2.03%       | 7.43%         | 2.03%              |\n| W3C-CSVW [221 files] | **0.00%**  | 0.91%              | 1.81%        | 2.26%       | 41.18%        | 1.81%              |\n| CSV Wrangling [179 files] | **0.00%** | 0.00%         | 0.56%        | 0.56%       | 39.66%        | 0.00%              |\n| CSV Wrangling CODEC [142 files] | **0.00%** | 0.00%   | 0.00%        | 0.00%       | 38.03%        | 0.00%              |\n| CSV Wrangling MESSY [126 files] | **0.00%** | 0.79%   | 0.79%        | 0.79%       | 42.06%        | 0.79%              |\n\n### F1 Score\n\nThe F1 score is the harmonic mean of precision and recall. To compare every tool on equal footing, F1 is derived consistently from the success and error ratios above[^3]:\n\n- **precision** = correct detections ÷ files processed without error\n- **recall** = correct detections ÷ *all* files (files that error count as missed detections)\n\nThis penalizes a tool for both wrong detections and processing errors, so F1 equals the success ratio for any tool with a 0% error rate (e.g. `csv-nose` on every dataset).\n\n| Data set | `csv-nose` | `CSVsniffer MADSE` | `CSVsniffer` | `CleverCSV` | `csv.Sniffer` | DuckDB `sniff_csv` |\n|:---------|:-----------|:-------------------|:-------------|:------------|:--------------|:-------------------|\n| POLLOCK  | **0.980**  | 0.953              | 0.956        | 0.942       | 0.926         | 0.833              |\n| W3C-CSVW | **0.995**  | 0.941              | 0.945        | 0.604       | 0.724         | 0.982              |\n| CSV Wrangling | **0.933** | 0.905             | 0.897        | 0.877       | 0.634         | 0.916              |\n| CSV Wrangling CODEC | **0.923** | 0.901       | 0.901        | 0.894       | 0.644         | 0.923              |\n| CSV Wrangling MESSY | **0.913** | 0.892       | 0.892        | 0.892       | 0.609         | 0.916              |\n\n[^3]: F1 = 2·P·R/(P+R) with the precision/recall definitions above, i.e. F1 = 2·s·(1−e)/(2−e) where *s* is the success ratio and *e* the error ratio. Competitor F1 values are derived from their reported success/error ratios (sourced from the [CSVsniffer](https://github.com/ws-garcia/CSVsniffer) benchmark) using this same formula, rather than reproduced from a separate measurement, so all columns are directly comparable.\n\n### Component Accuracy\n\ncsv-nose's delimiter and quote detection accuracy on each dataset:\n\n| Data set | Delimiter Accuracy | Quote Accuracy |\n|:---------|:-------------------|:---------------|\n| POLLOCK  | 97.97%             | 100.00%        |\n| W3C-CSVW | 99.55%             | 100.00%        |\n| CSV Wrangling | 93.85%         | 99.44%         |\n| CSV Wrangling CODEC | 92.96%   | 99.30%         |\n| CSV Wrangling MESSY | 92.06%   | 99.21%         |\n\n\u003e NOTE: See [ACCURACY.md](docs/ACCURACY.md) for details on accuracy breakdowns and known limitations.\n\n### Benchmark Setup\n\nThe benchmark test files are not included in this repository. To run benchmarks, first clone [CSVsniffer](https://github.com/ws-garcia/CSVsniffer) and copy the test files:\n\n```bash\n# Clone CSVsniffer (if not already available)\ngit clone https://github.com/ws-garcia/CSVsniffer.git /path/to/CSVsniffer\n\n# Copy test files to csv-nose\ncp -r /path/to/CSVsniffer/CSV/* tests/data/pollock/\ncp -r /path/to/CSVsniffer/W3C-CSVW/* tests/data/w3c-csvw/\ncp -r \"/path/to/CSVsniffer/CSV_Wrangling/data/github/Curated files/\"* tests/data/csv-wrangling/\n```\n\n### Running Benchmarks\n\nOnce the test files are in place:\n\n```bash\n# Run benchmark on POLLOCK dataset\ncargo run --release -- --benchmark tests/data/pollock\n\n# Run benchmark on W3C-CSVW dataset\ncargo run --release -- --benchmark tests/data/w3c-csvw\n\n# Run benchmark on CSV Wrangling dataset (all 179 files)\ncargo run --release -- --benchmark tests/data/csv-wrangling\n\n# Run benchmark on CSV Wrangling filtered CODEC (142 files)\ncargo run --release -- --benchmark tests/data/csv-wrangling --annotations tests/data/annotations/csv-wrangling-codec.txt\n\n# Run benchmark on CSV Wrangling MESSY (126 non-normal files)\ncargo run --release -- --benchmark tests/data/csv-wrangling --annotations tests/data/annotations/csv-wrangling-messy.txt\n\n# Run integration tests with detailed output\ncargo test --test benchmark_accuracy -- --nocapture\n```\n\n## License\n\nMIT OR Apache-2.0\n\n## Naming\n\nThe name \"csv-nose\" is a play on words, combining \"CSV\" (Comma-Separated Values) with \"nose,\" suggesting the tool's ability to \"sniff out\" the correct CSV dialect. \"Nose\" also sounds like \"knows,\" implying expertise in CSV dialect detection.\n\n## AI Contributions\n\nClaude Code using Opus 4.5 was used to assist in code generation and documentation. All AI-generated content has been reviewed and edited by human contributors to ensure accuracy and quality.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjqnatividad%2Fcsv-nose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjqnatividad%2Fcsv-nose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjqnatividad%2Fcsv-nose/lists"}