{"id":46369718,"url":"https://github.com/mrmcmullan/flycatcher","last_synced_at":"2026-03-05T03:34:07.409Z","repository":{"id":324977521,"uuid":"1099278407","full_name":"mrmcmullan/flycatcher","owner":"mrmcmullan","description":"Define your schema once \u0026 for all — built for DataFrames, powered across Pydantic, Polars, and SQLAlchemy.","archived":false,"fork":false,"pushed_at":"2025-12-10T15:58:08.000Z","size":408,"stargazers_count":3,"open_issues_count":12,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-16T11:53:30.840Z","etag":null,"topics":["data-engineering","data-validation","dataframe","etl","orm","polars","pydantic","python","python3","schema","sqlalchemy","type-checking","validation"],"latest_commit_sha":null,"homepage":"https://mrmcmullan.github.io/flycatcher/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrmcmullan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-18T19:53:13.000Z","updated_at":"2026-01-16T08:46:07.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mrmcmullan/flycatcher","commit_stats":null,"previous_names":["mrmcmullan/flycatcher"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/mrmcmullan/flycatcher","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrmcmullan%2Fflycatcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrmcmullan%2Fflycatcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrmcmullan%2Fflycatcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrmcmullan%2Fflycatcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrmcmullan","download_url":"https://codeload.github.com/mrmcmullan/flycatcher/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrmcmullan%2Fflycatcher/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30108668,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T03:32:43.378Z","status":"ssl_error","status_checked_at":"2026-03-05T03:32:22.667Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-validation","dataframe","etl","orm","polars","pydantic","python","python3","schema","sqlalchemy","type-checking","validation"],"created_at":"2026-03-05T03:34:06.721Z","updated_at":"2026-03-05T03:34:07.396Z","avatar_url":"https://github.com/mrmcmullan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\" style=\"line-height: 1.2;\"\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/mrmcmullan/flycatcher/main/docs/assets/logo.png\" alt=\"Flycatcher Logo\" width=\"400\" style=\"margin-bottom: 0.5em;\"/\u003e\n\n\u003c!-- \u003ch1 style=\"margin: 0.3em 0; font-family: system-ui, -apple-system, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif; font-weight: 600; color: #000;\"\u003e🐦 Flycatcher\u003c/h1\u003e --\u003e\n\u003cp style=\"margin: 0.2em 0; font-size: 1.3em;\"\u003e\u003cstrong\u003eDefine your schema once. Validate at scale. Stay columnar.\u003c/strong\u003e\u003c/p\u003e\n\u003cp style=\"margin: 0.2em 0;\"\u003e\u003cem\u003eBuilt for DataFrames, powered across Pydantic, Polars, and SQLAlchemy.\u003c/em\u003e\u003c/p\u003e\n\n\u003cp\u003e\n  \u003ca href=\"https://github.com/mrmcmullan/flycatcher/actions/workflows/ci.yml\" title=\"CI Status\"\u003e\n    \u003cimg src=\"https://github.com/mrmcmullan/flycatcher/actions/workflows/ci.yml/badge.svg\" alt=\"CI\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://codecov.io/gh/mrmcmullan/flycatcher\" title=\"Codecov\"\u003e\n    \u003cimg src=\"https://codecov.io/gh/mrmcmullan/flycatcher/branch/main/graph/badge.svg\" alt=\"codecov\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://badge.fury.io/py/flycatcher\" title=\"PyPI Version\"\u003e\n    \u003cimg src=\"https://badge.fury.io/py/flycatcher.svg\" alt=\"PyPI version\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.python.org/downloads/\" title=\"Python 3.12+\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/python-3.12+-blue.svg\" alt=\"Python 3.12+\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://opensource.org/licenses/MIT\" title=\"License: MIT\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" alt=\"License: MIT\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://mrmcmullan.github.io/flycatcher\" title=\"Documentation\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/docs-mkdocs-blue.svg\" alt=\"Documentation\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n---\n\nFlycatcher is a **DataFrame-native schema layer** for Python. Define your data model once and generate optimized representations for every part of your stack:\n\n- 🎯 **Pydantic models** for API validation \u0026 serialization\n- ⚡ **Polars validators** for blazing-fast bulk validation\n- 🗄️ **SQLAlchemy tables** for typed database access\n\n**Built for modern data workflows:** Validate millions of rows at high speed, keep schema drift at zero, and stay columnar end-to-end.\n\n## ❓ Why Flycatcher?\n\nModern Python data projects need **row-level validation** (Pydantic), **efficient bulk operations** (Polars), and **typed database queries** (SQLAlchemy). But maintaining multiple schemas across this stack can lead to duplication, drift, and manually juggling row-oriented and columnar paradigms.\n\n**Flycatcher solves this:** One schema definition → three optimized outputs.\n\n```python\nfrom flycatcher import Schema, Field, col, model_validator\n\nclass ProductSchema(Schema):\n    id: int = Field(primary_key=True)\n    name: str = Field(min_length=3, max_length=100)\n    price: float = Field(gt=0)\n    discount_price: float | None = Field(default=None, gt=0, nullable=True)\n\n    @model_validator\n    def check_discount():\n        # Cross-field validation with DSL\n        return (\n            col('discount_price') \u003c col('price'),\n            \"Discount price must be less than regular price\"\n        )\n\n# Generate three optimized representations\nProductModel = ProductSchema.to_pydantic()         # → Pydantic BaseModel\nProductValidator = ProductSchema.to_polars_validator() # → Polars DataFrame validator\nProductTable = ProductSchema.to_sqlalchemy()       # → SQLAlchemy Table\n```\n\n**Flycatcher lets you stay DataFrame-native without giving up the speed of Polars, the ergonomic validation of Pydantic, or the Pythonic power of SQLAlchemy**.\n\n---\n\n## 🚀 Quick Start\n\n### Installation\n\n```bash\npip install flycatcher\n# or\nuv add flycatcher\n```\n\n### Define Your Schema\n\n```python\nfrom datetime import datetime\nfrom flycatcher import Schema, Field\n\nclass UserSchema(Schema):\n    id: int = Field(primary_key=True)\n    username: str = Field(min_length=3, max_length=50, unique=True)\n    email: str = Field(pattern=r'^[^@]+@[^@]+\\.[^@]+$', unique=True, index=True)\n    age: int = Field(ge=13, le=120)\n    is_active: bool = Field(default=True)\n    created_at: datetime\n```\n\n### Use Pydantic for Row-Level Validation\n\nPerfect for APIs, forms, and single-record validation:\n\n```python\nfrom datetime import datetime\n\nUser = UserSchema.to_pydantic()\n\n# Validates constraints automatically via Pydantic\nuser = User(\n    id=1,\n    username=\"alice\",\n    email=\"alice@example.com\",\n    age=25,\n    created_at=datetime.utcnow()\n)\n\n# Serialize to JSON/dict\nprint(user.model_dump_json())\n```\n\n### Use Polars for Bulk Validation\n\nPerfect for DataFrame-level validation:\n\n```python\nimport polars as pl\n\nUserValidator = UserSchema.to_polars_validator()\n\n# Validate 1M+ rows with blazing speed\ndf = pl.read_csv(\"users.csv\")\nvalidated_df = UserValidator.validate(df, strict=True)\n\nvalidated_df.write_parquet(\"validated_users.parquet\")\n```\n\n### Use SQLAlchemy for Database Operations\n\nPerfect for typed queries and database interactions:\n\n```python\nfrom sqlalchemy import create_engine\n\nUserTable = UserSchema.to_sqlalchemy(table_name=\"users\")\n\nengine = create_engine(\"postgresql://localhost/mydb\")\n\n# Type-safe queries\nwith engine.connect() as conn:\n    result = conn.execute(\n        UserTable.select()\n        .where(UserTable.c.is_active == True)\n        .where(UserTable.c.age \u003e= 18)\n    )\n    for row in result:\n        print(row)\n```\n\n---\n\n## ✨ Key Features\n\n### Rich Field Types \u0026 Constraints\n\nUse standard Python types with `Field(...)` constraints:\n\n| Python Type | Constraints | Example |\n|-------------|-------------|---------|\n| `int` | `ge`, `gt`, `le`, `lt`, `multiple_of` | `age: int = Field(ge=0, le=120)` |\n| `float` | `ge`, `gt`, `le`, `lt` | `price: float = Field(gt=0)` |\n| `str` | `min_length`, `max_length`, `pattern` | `email: str = Field(pattern=r'^[^@]+@...')` |\n| `bool` | - | `is_active: bool = Field(default=True)` |\n| `datetime` | `ge`, `gt`, `le`, `lt` | `created_at: datetime = Field(ge=datetime(2020, 1, 1))` |\n| `date` | `ge`, `gt`, `le`, `lt` | `birth_date: date` |\n\n**All fields support (validation):** `nullable`, `default`, `description`\n\n**SQLAlchemy-specific:** `primary_key`, `unique`, `index`, `autoincrement`\n\n### Custom \u0026 Cross-Field Validation\n\nUse the `col()` DSL for powerful field-level and cross-field validation that works across both Pydantic and Polars:\n\n```python\nfrom datetime import datetime\nfrom flycatcher import Schema, Field, col, model_validator\n\nclass BookingSchema(Schema):\n    email: str\n    phone: str\n    check_in: datetime = Field(ge=datetime(2024, 1, 1))\n    check_out: datetime = Field(ge=datetime(2024, 1, 1))\n    nights: int = Field(ge=1)\n\n    @model_validator\n    def check_dates():\n        return (\n            col('check_out') \u003e col('check_in'),\n            \"Check-out must be after check-in\"\n        )\n\n    @model_validator\n    def check_phone_format():\n        cleaned = col('phone').str.replace(r'[^\\d]', '')\n        return (cleaned.str.len_chars() == 10, \"Phone must have 10 digits\")\n\n    @model_validator\n    def check_minimum_stay():\n        # For operations not yet in DSL (like .is_in()), use explicit Polars format\n        # Note: .dt.month() is available in DSL, but .is_in() is not yet supported\n        import polars as pl\n        return {\n            'polars': (\n                (~pl.col('check_in').dt.month().is_in([7, 8])) | (pl.col('nights') \u003e= 3),\n                \"Minimum stay in July and August is 3 nights\"\n            ),\n            'pydantic': lambda v: (\n                v.check_in.month not in [7, 8] or v.nights \u003e= 3,\n                \"Minimum stay in July and August is 3 nights\"\n            )\n        }\n\n```\n\n### Validation Modes\n\nPolars validation supports flexible error handling:\n\n```python\n# Strict mode: Raise on validation errors (default)\nvalidated_df = UserValidator.validate(df, strict=True)\n\n# Non-strict mode: Filter out invalid rows\nvalid_df = UserValidator.validate(df, strict=False)\n\n# Show violations for debugging\nvalidated_df = UserValidator.validate(df, strict=True, show_violations=True)\n```\n\n---\n\n## 🎯 Complete Example: ETL Pipeline\n\n```python\nimport polars as pl\nfrom datetime import datetime\nfrom flycatcher import Schema, Field, col, model_validator\nfrom sqlalchemy import create_engine, MetaData\n\n# 1. Define schema once\nclass OrderSchema(Schema):\n    order_id: int = Field(primary_key=True)\n    customer_email: str = Field(pattern=r'^[^@]+@[^@]+\\.[^@]+$', index=True)\n    amount: float = Field(gt=0)\n    tax: float = Field(ge=0)\n    total: float = Field(gt=0)\n    created_at: datetime\n\n    @model_validator\n    def check_total():\n        return (\n            col('total') == col('amount') + col('tax'),\n            \"Total must equal amount + tax\"\n        )\n\n# 2. Extract \u0026 Validate with Polars (handles millions of rows)\nOrderValidator = OrderSchema.to_polars_validator()\ndf = pl.read_csv(\"orders.csv\")\nvalidated_df = OrderValidator.validate(df, strict=True)\n\n# 3. Load to database with SQLAlchemy\nOrderTable = OrderSchema.to_sqlalchemy(table_name=\"orders\")\nengine = create_engine(\"postgresql://localhost/analytics\")\n\nwith engine.connect() as conn:\n    conn.execute(OrderTable.insert(), validated_df.to_dicts())\n    conn.commit()\n```\n\n✅ **Result:** Validated millions of rows, enforced business rules, and loaded to database — all from one schema definition.\n\n---\n\n## 🏗️ Design Philosophy\n\n**One schema, three representations. Each optimized for its use case.**\n\n```\n        Schema Definition\n               ↓\n    ┌──────────┼──────────┐\n    ↓          ↓          ↓\nPydantic    Polars    SQLAlchemy\n   ↓          ↓          ↓\n APIs       ETL      Database\n```\n\n### What Flycatcher Does\n\n✅ Single source of truth for schema definitions\n\u003cbr\u003e\n✅ Generate optimized representations for different use cases\n\u003cbr\u003e\n✅ Keep runtimes separate (no ORM ↔ DataFrame conversions)\n\u003cbr\u003e\n✅ Use stable public APIs (Pydantic, Polars, SQLAlchemy)\n\u003cbr\u003e\n\n### What Flycatcher Doesn't Do\n\n❌ Mix row-oriented and columnar paradigms\n\u003cbr\u003e\n❌ Create a \"unified runtime\" (that would be slow)\n\u003cbr\u003e\n❌ Reinvent validation logic (delegates to proven libraries when possible)\n\u003cbr\u003e\n❌ Depend on internal APIs\n\n---\n\n## ⚠️ Current Limitations (v0.1.0)\n\nFlycatcher v0.1.0 is an **alpha release**. The core functionality works perfectly, but some advanced features are planned for future versions:\n\n### Polars DSL\n\nThe `col()` DSL supports **basic operations** (`\u003e`, `\u003c`, `==`, `+`, etc.),\n**numeric math operations** (`.abs()`, `.round()`, `.floor()`, `.ceil()`, `.sqrt()`, `.pow()`),\n**limited string operations** (`.str.contains()`, `.str.starts_with()`, `.str.len_chars()`, etc.),\n and a **limited datetime accessor** (`.dt.year()`, `.dt.month()`, `.dt.total_days(other)`, etc.).\n\nThe `col()` DSL does not support the full range of Polars operations. However, additional\noperations will be added in future versions to better support the full functionality of Polars.\n\n**Workaround**: Use the explicit format in `@model_validator`:\n\n```python\n@model_validator\ndef check():\n    return {\n        'polars': (pl.col('field').is_null(), \"Message\"),\n        'pydantic': lambda v: (v.field is None, \"Message\")\n    }\n```\n\n### Pydantic Features\n\n- ❌ `@field_validator` - Only `@model_validator` is supported (coming in v0.2.0)\n- ❌ Field aliases and computed fields (coming in v0.2.0+)\n- ❌ Custom serialization options (coming in v0.2.0+)\n\n**Workaround**: Use `@model_validator` for all validation needs.\n\n### SQLAlchemy Features\n\n- ❌ Foreign key relationships - Must be added manually after table generation (coming in v0.3.0+)\n- ❌ Composite primary keys - Only single-field primary keys supported (coming in v0.3.0+)\n- ❌ Function-based defaults (e.g., `default=func.now()`) - Only literal defaults supported\n\n**Workaround**: Add relationships and composite keys manually in SQLAlchemy after table generation.\n\n### Field Types\n\n- ❌ Enum, UUID, JSON, Array field types (coming in v0.3.0+)\n- ❌ Numeric/Decimal field type (coming in v0.3.0+)\n\n**Workaround**: Use `String` with pattern validation or manual handling.\n\n\u003c!-- **See [Limitations Guide](docs/dev/MISSING-FUNCTIONALITY.md) for details and workarounds.** --\u003e\n\n---\n\n## 📊 Comparison\n\n| Feature | Flycatcher | SQLModel | Patito |\n|---------|-----------|----------|--------|\n| Pydantic support | ✅ | ✅ | ✅ |\n| Polars support | ✅ | ❌ | ✅ |\n| SQLAlchemy support | ✅ | ✅ | ❌ |\n| DataFrame-level DB ops | 🚧 (v0.2) | ❌ | ❌ |\n| Cross-field validation | ✅ | ⚠️ (Pydantic only) | ⚠️ (Polars only) |\n| Single schema definition | ✅ | ⚠️ (Pydantic + ORM hybrid) | ⚠️ (Pydantic + Polars hybrid) |\n\n**Flycatcher** is the only library that generates optimized representations for all three systems while keeping them properly separated.\n\n---\n\n## 📚 Documentation\n\n- **[Getting Started](https://mrmcmullan.github.io/flycatcher/)** - Installation and basics\n- **[Tutorials](https://mrmcmullan.github.io/flycatcher/tutorials/)** - Step-by-step guides\n- **[How-To Guides](https://mrmcmullan.github.io/flycatcher/how-to/)** - Solve specific problems\n- **[API Reference](https://mrmcmullan.github.io/flycatcher/api/)** - Complete API documentation\n- **[Explanations](https://mrmcmullan.github.io/flycatcher/explanations/)** - Deep dives and concepts\n\n---\n\n## 🛣️ Roadmap\n\n### v0.1.0 (Released) 🚀\n\n- [x] Core schema definition with metaclass\n- [x] Field types with constraints (Integer, String, Float, Boolean, Datetime, Date)\n- [x] Pydantic model generator\n- [x] Polars DataFrame validator with bulk validation\n- [x] SQLAlchemy table generator\n- [x] Cross-field validators with DSL (`col()`)\n- [x] Test suite with 70%+ coverage\n- [x] Complete documentation site\n- [x] PyPI publication\n\n### v0.2.0 (In Progress) 🚧\n\n**Theme:** Enhanced validation and database operations\n\n- [ ] `@field_validator` support in addition to existing `@model_validator`\n- [x] Enhanced Polars DSL: `.is_null()`, `.is_not_null()`, `.str.contains()`, `.str.startswith()`, `.dt.month`, `.dt.year`, `.is_in([...])`, `.is_between()`\n- [ ] Pydantic enhancements: field aliases, computed fields, custom serialization\n- [ ] Enable inheritance of `Schema` to create subclasses with different fields\n- [ ] For more details, see the [GitHub Milestone for v0.2.0](https://github.com/mrmcmullan/flycatcher/milestone/2)\n\n### v0.3.0 (Planned)\n\n- [ ] DataFrame-level queries (`Schema.query()`)\n- [ ] Bulk write operations (`Schema.insert()`, `Schema.update()`, `Schema.upsert()`)\n- [ ] Complete ETL loop staying columnar end-to-end\n- [ ] Add PascalCase metaclass\n- [ ] Additional Pydantic validation modes (`mode='before'`, `mode='wrap'`)\n- [ ] For more details, see the [GitHub Milestone for v0.3.0](https://github.com/mrmcmullan/flycatcher/milestone/3)\n\n### v0.4.0+ (Future)\n\n**Theme:** Advanced field types and relationships\n\n- [ ] Additional field types: Enum, UUID, JSON, Array, Numeric/Decimal, Time, Binary, Interval\n- [ ] SQLAlchemy relationships: Foreign keys, composite primary keys\n- [ ] SQLAlchemy function-based defaults (e.g., `default=func.now()`)\n- [ ] JOIN support in queries\n- [ ] Aggregations (GROUP BY, COUNT, SUM)\n- [ ] Schema migrations helper\n\n\u003c!-- See our [full roadmap](docs/dev/ROADMAP.md) for details. --\u003e\n\n## 🤝 Contributing\n\nContributions are welcome! Please see our [Contributing Guide]\u003c!--(CONTRIBUTING.md) --\u003e for details.\n\n---\n\n## 📄 License\n\nMIT License - see [LICENSE]([LICENSE](https://github.com/mrmcmullan/flycatcher?tab=MIT-1-ov-file)) for details.\n\n---\n\n## 💬 Community\n\n- **[GitHub Issues](https://github.com/mrmcmullan/flycatcher/issues)** - Bug reports and feature requests\n- **[GitHub Discussions](https://github.com/mrmcmullan/flycatcher/discussions)** - Questions and community discussion\n- **[Documentation](https://mrmcmullan.github.io/flycatcher)** - Full guides and API reference\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n\n\n\u003cp\u003e\u003cstrong\u003eBuilt with ❤️ for the DataFrame generation\u003c/strong\u003e\u003c/p\u003e\n\n\u003cp\u003e\n  \u003ca href=\"https://github.com/mrmcmullan/flycatcher\"\u003e⭐ Star us on GitHub\u003c/a\u003e\n  \u0026nbsp;|\u0026nbsp;\n  \u003ca href=\"https://mrmcmullan.github.io/flycatcher\"\u003e📖 Read the docs\u003c/a\u003e\n  \u0026nbsp;|\u0026nbsp;\n  \u003ca href=\"https://github.com/mrmcmullan/flycatcher/issues\"\u003e🐛 Report a bug\u003c/a\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrmcmullan%2Fflycatcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrmcmullan%2Fflycatcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrmcmullan%2Fflycatcher/lists"}