{"id":37228846,"url":"https://github.com/rannd1nt/phaeton","last_synced_at":"2026-01-15T03:29:53.039Z","repository":{"id":331020959,"uuid":"1123551171","full_name":"rannd1nt/phaeton","owner":"rannd1nt","description":"A high-performance preprocessing and ETL engine for sanitizing raw data streams, accelerated by Rust.","archived":false,"fork":false,"pushed_at":"2026-01-11T19:49:44.000Z","size":19545,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-11T23:38:28.294Z","etag":null,"topics":["data-cleaning","data-preprocessing","etl","information-analysis","pyo3-rust-bindings","python","python-library","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rannd1nt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-27T05:35:04.000Z","updated_at":"2026-01-11T19:49:36.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rannd1nt/phaeton","commit_stats":null,"previous_names":["rannd1nt/phaeton"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/rannd1nt/phaeton","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rannd1nt%2Fphaeton","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rannd1nt%2Fphaeton/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rannd1nt%2Fphaeton/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rannd1nt%2Fphaeton/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rannd1nt","download_url":"https://codeload.github.com/rannd1nt/phaeton/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rannd1nt%2Fphaeton/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28442282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-15T00:55:22.719Z","status":"online","status_checked_at":"2026-01-15T02:00:08.019Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-cleaning","data-preprocessing","etl","information-analysis","pyo3-rust-bindings","python","python-library","rust"],"created_at":"2026-01-15T03:29:52.544Z","updated_at":"2026-01-15T03:29:53.021Z","avatar_url":"https://github.com/rannd1nt.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Phaeton\n\n[![PyPI version](https://badge.fury.io/py/phaeton.svg)](https://badge.fury.io/py/phaeton)\n[![Python Versions](https://img.shields.io/pypi/pyversions/phaeton.svg)](https://pypi.org/project/phaeton/)\n[![Rust](https://img.shields.io/badge/built%20with-Rust-orange)](https://www.rust-lang.org/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n\u003e ⚠️ **Project Status:** Phaeton is currently in **Stable Beta (v0.3.0)**.\n\u003e The core streaming engine is fully functional. However, please note that some auxiliary methods (marked in docs) are currently placeholders and will be implemented in future versions.\n\n\n**Phaeton** is a specialized, Rust-powered preprocessing and ETL engine designed to sanitize raw data streams before they reach your analytical environment.\n\nIt acts as the strictly typed **\"Gatekeeper\"** of your data pipeline. Unlike traditional DataFrame libraries that attempt to load entire datasets into RAM, Phaeton employs a **zero-copy streaming architecture**. It processes data chunk-by-chunk filtering noise, fixing encodings, and standardizing formats ensuring **O(1) memory complexity** relative to file size.\n\nThis allows you to process massive datasets on standard hardware without memory spikes, delivering clean, high-quality data to downstream tools like Pandas, Polars, or ML models.\n\n\u003e **The Philosophy:** Don't waste memory loading garbage. Clean the stream first, then analyze the gold.\n\n---\n\n## Key Features\n\n* **Streaming Architecture:** Processes files chunk-by-chunk. Memory usage remains stable regardless of whether the file is 100MB or 100GB.\n* **Parallel Execution:** Utilizes all CPU cores via **Rust Rayon** to handle heavy lifting (Regex, Fuzzy Matching) without blocking Python.\n* **Strict Quarantine:** Bad data isn't just dropped silently; it's quarantined into a separate file with a generated `_phaeton_reason` column for auditing.\n* **Smart Casting:** Automatically handles messy formats (e.g., `\"Rp 5.250.000,00\"` → `5250000` int) without complex manual parsing.\n* **Privacy \u0026 Security:** Built-in email masking and SHA-256 hashing for PII compliance.\n* **Configurable Engine:** Full control over `batch_size` and worker threads to tune performance for low-memory devices or high-end servers.\n\n\n---\n\n##  Performance Benchmark\n\nPhaeton is optimized for \"Dirty Data\" scenarios involving heavy string parsing, regex filtering, and fuzzy matching.\n\n\n**Test Scenario:**\n* **Dataset:** 1 Million Rows of generated mixed dirty data.\n* **Operations:** Trim whitespace, Currency scrubbing (`$ 50.000,00` -\u003e `50000`), Type casting, Fuzzy Alignment (Typo correction for City names), and Filtering.\n* **Hardware:** Entry-level Laptop (Intel Core i3-1220P, 16GB RAM).\n\n**Results:**\n\n| OS Environment | Speed (Rows/sec) | Duration (1M Rows) | Throughput |\n| :--- | :--- | :--- | :--- |\n| **Windows 11** | **~820,000 rows/s** | **1.21s** | **~70 MB/s** |\n| **Linux (Arch)** | ~575,000 rows/s | 1.73s | ~49 MB/s |\n\n\u003cbr\u003e\n\n\u003e ⚠️ Note on I/O Bottleneck: The performance difference above is due to hardware configuration during testing.\n\u003e * Windows: Ran on internal NVMe SSD (High I/O speed).\n\u003e * Linux: Ran on External SSD via USB 3.2 enclosure (I/O Bottleneck).\n\nIn an equal hardware environment, Phaeton performs identically on Linux and Windows. The engine is heavily I/O bound; faster disk = faster processing.\n\n---\n##  Usage Example\nBased on the features available in the current version.\n\n```python\nimport phaeton\n\n# 1. Initialize Engine\n# 'strict=True' enables schema validation before execution starts.\neng = phaeton.Engine(workers=0, batch_size=25_000, strict=True)\n\n# 2. Define Base Pipeline (Shared Logic)\nbase = (\n    eng.ingest(\"dirty_data.csv\")\n        # Critical Data Filter: Drop row if 'email' OR 'username' is missing\n        .prune(['email', 'username'])\n        \n        # Deduplication: Ensure email uniqueness across the dataset\n        .dedupe('email')\n        \n        # Cleaning \u0026 Normalization\n        .scrub('username', mode='trim')             # Remove whitespace\n        .scrub('salary', mode='currency')           # Normalize format (\"$ 5,000\" -\u003e \"5000\")\n        \n        # Type Enforcement: Validate data is integer, strip noise if needed\n        .cast('salary', dtype='int', clean=True)\n        \n        # Imputation: Fill missing status with a default value\n        .fill('status', value='UNKNOWN')\n        \n        # Correction: Fix typos using Jaro-Winkler distance\n        .fuzzyalign('city',\n            ref=['Jakarta', 'Minnesota'],\n            threshold=0.85 \n        ) # e.g., \"Jkarta\" -\u003e \"Jakarta\"\n)\n\n# 3. Pipeline Branching using .fork()\n\n# Pipeline 1: Secure \u0026 Clean Active Users\np1 = (\n    base.fork('Active Users')\n        .keep('status', match='ACTIVE', mode='exact')\n        .hash('email', salt='s3cret')               # Anonymize PII (SHA-256)\n        .dump('clean_active.csv')\n)\n\n# Pipeline 2: Audit Banned Users\np2 = (\n    base.fork('Banned Analysis')\n        .keep('status', match='BANNED', mode='exact')\n        .quarantine('quarantine_banned.csv')        # Isolate bad rows for review\n        .dump('clean_banned.csv')\n)\n\n# 4. Execute Pipelines in Parallel\n# Returns a list of result statistics\nstats = eng.exec([p1, p2])\n\nprint(f\"Pipeline 1 (Active) | Processed: {stats[0].processed}, Saved: {stats[0].saved}\")\nprint(f\"Pipeline 2 (Banned) | Processed: {stats[1].processed}, Saved: {stats[1].saved}\")\n```\n\n---\n\n## Installation\n\nPhaeton provides **Universal Wheels (ABI3)**. No Rust compiler needed.\n```bash\npip install phaeton\n```\n\u003e **Supported:** Python 3.8+ on Windows, Linux, and macOS (Intel \u0026 Apple Silicon).\n\n---\n\n## API Reference\n\n### 1. Engine \u0026 Diagnostics \u003cbr\u003e\n| Method | Description |\n| :--- | :--- | \n| `phaeton.probe(path)` | Detects encoding and delimiter automatically. |\n| `eng.ingest(source)` | Creates a new pipeline builder. |\n| `eng.exec(pipelines)` | Executes pipelines in parallel threads. |\n| `eng.validate(pipelines)` | Runs a schema dry-run check without executing data processing. | \n\n\n### 2. Pipeline: Cleaning \u0026 Transformation \u003cbr\u003e\nMethods to sanitize data content.\n\n| Method | Description |\n| :--- | :--- |\n| `.decode(encoding)` | Fixes file encoding (e.g., `latin-1` or `cp1252`). **Mandatory** as the first step if encoding is broken. |\n| `.scrub(col, mode)` | Basic string cleaning. \u003cbr\u003e **Modes:** `'trim'`, `'lower'`, `'upper'`, `'currency'`, `'html'`, `numeric_only`, `email (masking)` . |\n|`.fill(col, val, method)`|**Methods:** `fixed` (constant value) or `ffill` (forward fill).|\n|`.dedupe(col)`|Removes duplicates. `col` can be `None` (full row), `str` (single col), or `list` (composite key).|\n| `.fuzzyalign(col, ref, threshold)` | Fixes typos using Jaro-Winkler distance against a reference list. |\n| `.cast(col, dtype, clean)` | **Smart Cast.** Converts types (`int`/`float`/`bool`). \u003cbr\u003e Set `clean=True` to strip non-numeric chars before casting. |\n\n### 3. Pipeline: Structure \u0026 Security\nMethods to manage columns and privacy.\n\n| Method | Description |\n| :--- | :--- |\n| `.headers(style)` | Standardizes header casing. \u003cbr\u003e **Styles:** `'snake'`, `'camel'`, `'pascal'`, `'kebab', 'constant`. |\n| `.rename(mapping)` | Renames specific columns using a dictionary mapping `({'old': 'new'})`. |\n| `.hash(col, salt)` | Applies hashing (SHA-256) to specific columns for PII anonymization. |\n|`.map(col, mapping)`| Maps values using a dictionary lookup (VLOOKUP style).|\n\n\n### 4. Pipeline: Output \u0026 Flow\n\nMethods to save the final results or handle rejected data.\n\n| Method | Description |\n| :--- | :--- |\n| `.quarantine(path)` | Saves rejected rows (with reasons) to a separate CSV file. |\n| `.dump(path, format)` | Saves clean data to `.csv`. |\n|`.fork(tag)`|Creates a branch of the pipeline.|\n|`.peek(n, col)`| Runs a dry-run preview. `n`: rows limit. `col`: specific column(s) to inspect (optional). |\n\n\u003cbr\u003e\n\n\u003e ⚠️ **Placeholder Methods (Coming Soon)**\n\u003e\n\u003e These methods are present in the API for compatibility but do not perform operations yet in v0.3.0.\n\u003e * `reformat(col, ...)`: Date parsing/reformatting.\n\u003e * `split(col, ...)`: Splitting columns.\n\u003e * `combine(cols, ...)`: Merging columns.\n---\n\n## Roadmap\n\nPhaeton is currently in **Stable Beta (v0.3.0)**. Here is the status of our development:\n\n| Feature | Status | Implementation Notes |\n| :--- | :---: | :--- |\n| **Parallel Streaming Engine** | ✅ Ready | Powered by Rust Rayon (Multi-core) |\n| **Filter Logic \u0026 Regex** | ✅ Ready | `keep`, `discard`, `prune` implemented |\n| **Text Scrubbing** | ✅ Ready | HTML, Currency, Email Masking, etc. |\n| **Type Enforcement** | ✅ Ready | Validates data types \u0026 scrubs noise for clean CSV output |\n| **Fuzzy Alignment** | ✅ Ready | Jaro-Winkler for typo correction |\n| **Quarantine System** | ✅ Ready | Full audit trail for rejected rows |\n| **Deduplication** | ✅ Ready | Row-level \u0026 Column-level dedupe |\n| **Hashing \u0026 Anonymization** | ✅ Ready | SHA-256 for PII data |\n| **Header Normalization** | ✅ Ready  | `snake_case`, `camelCase` conversions |\n|**Strict Schema Validation**| ✅ Ready | `Engine(strict=True)`|\n| **Inspector Engine** | 📝 Planned | Dedicated stream for data profiling (Read-Only) |\n| **Date Normalization** | 📝 Planned | Auto-detect \u0026 reformat dates |\n| **Parquet/Arrow Support** | 📝 Planned | Native output integration |\n\n---\n\n## Contributing\n\nThis project is built with **Maturin** (PyO3 + Rust). Interested in contributing?\n\n1.  **Clone** this repository.\n2.  Ensure **Rust \u0026 Cargo** are installed.\n3.  Set up the environment and build:\n\n```bash\n# Setup virtual environment (optional)\npython -m venv .venv\nsource .venv/bin/activate\n\n# Build \u0026 Install package in development mode\nmaturin develop --release\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frannd1nt%2Fphaeton","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frannd1nt%2Fphaeton","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frannd1nt%2Fphaeton/lists"}