{"id":44924969,"url":"https://github.com/albertxli/ambers","last_synced_at":"2026-02-24T06:00:40.625Z","repository":{"id":339079215,"uuid":"1160385154","full_name":"albertxli/ambers","owner":"albertxli","description":"Pure Rust SPSS .sav/.zsav reader with Arrow output and Python bindings","archived":false,"fork":false,"pushed_at":"2026-02-21T16:56:02.000Z","size":484,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-22T09:49:34.693Z","etag":null,"topics":["data-io","data-science","polars","polars-dataframe","python","rust","spss","spss-reader","spss-sav"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/albertxli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-17T21:56:09.000Z","updated_at":"2026-02-21T16:54:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/albertxli/ambers","commit_stats":null,"previous_names":["albertxli/ambers"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/albertxli/ambers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albertxli%2Fambers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albertxli%2Fambers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albertxli%2Fambers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albertxli%2Fambers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/albertxli","download_url":"https://codeload.github.com/albertxli/ambers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albertxli%2Fambers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29738083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-23T04:51:08.365Z","status":"ssl_error","status_checked_at":"2026-02-23T04:49:15.865Z","response_time":90,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-io","data-science","polars","polars-dataframe","python","rust","spss","spss-reader","spss-sav"],"created_at":"2026-02-18T04:05:08.491Z","updated_at":"2026-02-23T05:01:20.951Z","avatar_url":"https://github.com/albertxli.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ambers\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/albertxli/ambers/main/images/ambers-banner-v2.svg\" alt=\"ambers banner\" width=\"900\"\u003e\n\u003c/p\u003e\n\n[![Crates.io](https://img.shields.io/crates/v/ambers)](https://crates.io/crates/ambers)\n[![PyPI](https://img.shields.io/pypi/v/ambers?color=blue)](https://pypi.org/project/ambers/)\n[![License: MIT](https://img.shields.io/badge/license-MIT-grey.svg)](LICENSE)\n\nPure Rust SPSS `.sav`/`.zsav` reader and writer — Arrow-native, zero C dependencies.\n\n## Features\n\n- Blazing fast read and write for SPSS `.sav` (bytecode) and `.zsav` (zlib) files\n- Rich metadata: variable labels, value labels, missing values, MR sets, measure levels, and more\n- Lazy reader via `scan_sav()` — Polars LazyFrame with projection and row limit pushdown\n- Pure Rust with a native Python API — native Arrow integration, no C dependencies\n- Benchmarked up to 3–10x faster reads and 4–20x faster writes compared to current popular SPSS I/O libraries\n\n## Installation\n\n**Python:**\n\n```bash\nuv add ambers\n```\n\n**Rust:**\n\n```bash\ncargo add ambers\n```\n\n## Python\n\n```python\nimport ambers as am\nimport polars as pl\n\n# Eager read — data + metadata\ndf, meta = am.read_sav(\"survey.sav\")\n\n# Lazy read — returns Polars LazyFrame\nlf, meta = am.scan_sav(\"survey.sav\")\ndf = lf.select([\"Q1\", \"Q2\", \"age\"]).head(1000).collect()\n\n# Explore metadata\nmeta.summary()\nmeta.describe(\"Q1\")\nmeta.value(\"Q1\")\n\n# Read metadata only (fast, skips data)\nmeta = am.read_sav_metadata(\"survey.sav\")\n\n# Write back — roundtrip with full metadata\ndf = df.filter(pl.col(\"age\") \u003e 18)\nam.write_sav(df, \"filtered.sav\", meta=meta)\n\n# Write as .zsav (zlib compressed)\nam.write_sav(df, \"compressed.zsav\", meta=meta)\n\n# From scratch — metadata is optional, inferred from DataFrame schema\nam.write_sav(df, \"new.sav\")\n```\n\nUse `.sav` for bytecode compression (default), `.zsav` for zlib compression. Pass `meta=` to preserve all metadata from a prior `read_sav()`, or omit it to infer formats from the DataFrame. Individual writable fields (e.g., `variable_labels`, `variable_value_labels`) can also be passed directly as keyword arguments for fine-grained control.\n\n## Rust\n\n```rust\nuse ambers::{read_sav, read_sav_metadata};\n\n// Read data + metadata\nlet (batch, meta) = read_sav(\"survey.sav\")?;\nprintln!(\"{} rows, {} cols\", batch.num_rows(), meta.number_columns);\n\n// Read metadata only\nlet meta = read_sav_metadata(\"survey.sav\")?;\nprintln!(\"{}\", meta.label(\"Q1\").unwrap_or(\"(no label)\"));\n```\n\n## Metadata API (Python)\n\n| Method | Description |\n|--------|-------------|\n| `meta.summary()` | Formatted overview: file info, type distribution, annotations |\n| `meta.describe(\"Q1\")` | Deep-dive into a single variable (or list of variables) |\n| `meta.diff(other)` | Compare two metadata objects, returns `MetaDiff` |\n| `meta.label(\"Q1\")` | Variable label |\n| `meta.value(\"Q1\")` | Value labels dict |\n| `meta.format(\"Q1\")` | SPSS format string (e.g. `\"F8.2\"`, `\"A50\"`) |\n| `meta.measure(\"Q1\")` | Measurement level (`\"nominal\"`, `\"ordinal\"`, `\"scale\"`) |\n| `meta.schema` | Full metadata as a nested Python dict |\n\nAll variable-name methods raise `KeyError` for unknown variables.\n\n### Metadata Fields\n\nAll fields returned by the reader. Fields marked **Write** are preserved when passed via `meta=` to `write_sav()`. Read-only fields are set automatically (encoding, timestamps, row/column counts, etc.).\n\n\u003e **Note:** This is a first pass — field names and behavior may change without warning in future releases.\n\n| Field | Read | Write | Type |\n|-------|:----:|:-----:|------|\n| `variable_names` | yes | yes | `list[str]` |\n| `variable_labels` | yes | yes | `dict[str, str]` |\n| `variable_value_labels` | yes | yes | `dict[str, dict[float\\|str, str]]` |\n| `variable_measure` | yes | yes | `dict[str, str]` |\n| `variable_alignment` | yes | yes | `dict[str, str]` |\n| `variable_display_width` | yes | yes | `dict[str, int]` |\n| `variable_storage_width` | yes | yes | `dict[str, int]` |\n| `variable_missing` | yes | yes | `dict[str, list[dict]]` |\n| `spss_variable_types` | yes | yes | `dict[str, str]` |\n| `rust_variable_types` | yes | — | `dict[str, str]` |\n| `weight_variable` | yes | yes | `str \\| None` |\n| `mr_sets` | yes | yes | `dict[str, dict]` |\n| `file_label` | yes | yes | `str` |\n| `file_format` | yes | — | `str` |\n| `file_encoding` | yes | — | `str` |\n| `creation_time` | yes | — | `str` |\n| `modification_time` | yes | — | `str` |\n| `number_rows` | yes | — | `int \\| None` |\n| `number_columns` | yes | — | `int` |\n| `compression` | yes | — | `str` |\n| `notes` | yes | yes | `list[str]` |\n\n## Streaming Reader (Rust)\n\n```rust\nlet mut scanner = ambers::scan_sav(\"survey.sav\")?;\nscanner.select(\u0026[\"age\", \"gender\"])?;\nscanner.limit(1000);\n\nwhile let Some(batch) = scanner.next_batch()? {\n    println!(\"Batch: {} rows\", batch.num_rows());\n}\n```\n\n## Performance\n\n### Eager Read\n\nAll results return a Polars DataFrame. Best of 3–5 runs (with warmup) on Windows 11, Python 3.13, 24-core machine.\n\n| File | Size | Rows | Cols | ambers | polars_readstat | pyreadstat | vs prs | vs pyreadstat |\n|------|------|-----:|-----:|-------:|----------------:|-----------:|-------:|--------------:|\n| test_1 (bytecode) | 0.2 MB | 1,500 | 75 | \u003c 0.01s | \u003c 0.01s | 0.011s | — | — |\n| test_2 (bytecode) | 147 MB | 22,070 | 677 | **0.286s** | 0.897s | 3.524s | **3.1x** | **12x** |\n| test_3 (uncompressed) | 1.1 GB | 79,066 | 915 | **0.322s** | 1.150s | 4.918s | **3.6x** | **15x** |\n| test_4 (uncompressed) | 0.6 MB | 201 | 158 | **0.002s** | 0.003s | 0.012s | **1.5x** | **6x** |\n| test_5 (uncompressed) | 0.6 MB | 203 | 136 | **0.002s** | 0.003s | 0.016s | **1.5x** | **8x** |\n| test_6 (uncompressed) | 5.4 GB | 395,330 | 916 | **1.600s** | 1.752s | 25.214s | **1.1x** | **16x** |\n\n- **Faster than polars_readstat on all tested files** — 1.1–3.6x faster\n- **6–16x faster than pyreadstat** across all file sizes\n- No PyArrow dependency — uses Arrow PyCapsule Interface for zero-copy transfer\n\n### Lazy Read with Pushdown\n\n`scan_sav()` returns a Polars LazyFrame. Unlike eager reads, it only reads the data you ask for:\n\n| File (size) | Full collect | Select 5 cols | Head 1000 rows | Select 5 + head 1000 |\n|-------------|------------:|-------------:|--------------:|--------------------:|\n| test_2 (147 MB, 22K × 677) | 0.903s | 0.363s (2.5x) | 0.181s (5.0x) | **0.157s (5.7x)** |\n| test_3 (1.1 GB, 79K × 915) | 0.700s | 0.554s (1.3x) | 0.020s (35x) | **0.012s (58x)** |\n| test_6 (5.4 GB, 395K × 916) | 3.062s | 2.343s (1.3x) | 0.022s (139x) | **0.013s (236x)** |\n\nOn the 5.4 GB file, selecting 5 columns and 1000 rows completes in **13ms** — 236x faster than reading the full dataset.\n\n### Write\n\n`write_sav()` writes a Polars DataFrame + metadata back to `.sav` (bytecode) or `.zsav` (zlib). Best of 5 runs on the same machine.\n\n| File | Size | Rows | Cols | Mode | ambers | pyreadstat | Speedup |\n|------|------|-----:|-----:|------|-------:|-----------:|--------:|\n| test_1 (bytecode) | 0.2 MB | 1,500 | 75 | .sav | **0.001s** | 0.019s | **13x** |\n| | | | | .zsav | **0.004s** | 0.026s | **7x** |\n| test_2 (bytecode) | 147 MB | 22,070 | 677 | .sav | **0.567s** | 3.849s | **7x** |\n| | | | | .zsav | **1.088s** | 4.415s | **4x** |\n| test_3 (uncompressed) | 1.1 GB | 79,066 | 915 | .sav | **0.950s** | 16.152s | **17x** |\n| | | | | .zsav | **1.774s** | 17.362s | **10x** |\n| test_6 (uncompressed) | 5.4 GB | 395,330 | 916 | .sav | **5.700s** | 79.999s | **14x** |\n| | | | | .zsav | **8.193s** | 85.491s | **10x** |\n\n- **4–20x faster than pyreadstat** on writes across all files and compression modes\n- Full metadata roundtrip: variable labels, value labels, missing values, MR sets, display properties\n- Bytecode (.sav) and zlib (.zsav) compression\n\n## Roadmap\n\n- Continued I/O performance optimization\n- Expanded SPSS metadata field coverage\n- Rich metadata manipulation — add, update, merge, and remove metadata programmatically\n- Individual metadata field overrides in `write_sav()` — pass `variable_labels=`, `variable_value_labels=`, etc. alongside `meta=` to selectively override fields\n- Currently supports read and write with Polars DataFrames (eager and lazy) — extending to pandas, Narwhals, DuckDB, and others\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falbertxli%2Fambers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falbertxli%2Fambers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falbertxli%2Fambers/lists"}