{"id":30687616,"url":"https://github.com/query-farm/python-sql-scan-planning","last_synced_at":"2026-04-14T18:34:11.703Z","repository":{"id":303329925,"uuid":"1015110703","full_name":"Query-farm/python-sql-scan-planning","owner":"Query-farm","description":"A Python library for intelligent file filtering using SQL expressions and metadata-based scan planning. This library enables efficient data lake query optimization by determining which files need to be scanned based on their statistical metadata.","archived":false,"fork":false,"pushed_at":"2025-07-07T03:31:54.000Z","size":0,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-07T03:49:49.880Z","etag":null,"topics":["duckdb","predicate-pushdown","predicates","query-processing","scan-planning","sql","sql-engines"],"latest_commit_sha":null,"homepage":"https://query.farm","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Query-farm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-07T02:53:50.000Z","updated_at":"2025-07-07T03:31:57.000Z","dependencies_parsed_at":"2025-07-07T03:49:54.314Z","dependency_job_id":"dc4fd7bb-fe14-4054-9455-68e5090c9197","html_url":"https://github.com/Query-farm/python-sql-scan-planning","commit_stats":null,"previous_names":["query-farm/python-sql-scan-planning"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Query-farm/python-sql-scan-planning","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Query-farm%2Fpython-sql-scan-planning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Query-farm%2Fpython-sql-scan-planning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Query-farm%2Fpython-sql-scan-planning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Query-farm%2Fpython-sql-scan-planning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Query-farm","download_url":"https://codeload.github.com/Query-farm/python-sql-scan-planning/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Query-farm%2Fpython-sql-scan-planning/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273208777,"owners_count":25064204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","predicate-pushdown","predicates","query-processing","scan-planning","sql","sql-engines"],"created_at":"2025-09-02T00:03:34.324Z","updated_at":"2026-04-14T18:34:11.667Z","avatar_url":"https://github.com/Query-farm.png","language":"Python","readme":"# [Query.Farm](https://query.farm) SQL Scan Planning\n\nA Python library for intelligent file filtering using SQL expressions and metadata-based scan planning. This library enables efficient data lake query optimization by determining which files need to be scanned based on their statistical metadata.\n\n## Overview\n\nThis module provides predicate pushdown capabilities for file-based data storage systems. By maintaining metadata about file contents (min/max values, value sets, null presence), the library can quickly determine which files contain data that could satisfy a given `SQL WHERE` clause, significantly reducing query execution time.\n\n## Features\n\n- **SQL Expression Parsing**: Parse and evaluate complex `SQL WHERE` clauses using [SQLGlot](https://github.com/tobymao/sqlglot)\n- **Metadata-Based Filtering**: Support for both range-based (min/max) and set-based field metadata\n- **Null Handling**: Comprehensive support for `NULL` value semantics in SQL expressions\n- **Complex Predicates**: Handle `AND`, `OR`, `XOR`, `NOT`, `IN`, `BETWEEN`, `CASE` statements, and more\n- **Multiple Data Types**: Support for integers, floats, strings, decimals, and `NULL` values.  Support for casting between Arrow scalar types.\n- **Dialect Support**: Configurable SQL dialect support (default: DuckDB)\n\n## Installation\n\n```bash\npip install query-farm-sql-scan-planning\n```\n\nOr using rye:\n\n```bash\nrye add query-farm-sql-scan-planning\n```\n\n## Quick Start\n\n```python\nfrom query_farm_sql_scan_planning import Planner, RangeFieldInfo, SetFieldInfo\nimport pyarrow as pa\n\n# Define file metadata\nfiles = [\n    (\n        \"data_2023_q1.parquet\",\n        {\n            \"sales_amount\": RangeFieldInfo(\n                min_value=pa.scalar(100), max_value=pa.scalar(50000),\n                has_nulls=False, has_non_nulls=True\n            ),\n            \"region\": SetFieldInfo(\n                values={pa.scalar(\"US\"), pa.scalar(\"CA\"), pa.scalar(\"MX\")},\n                has_nulls=False, has_non_nulls=True\n            ),\n        }\n    ),\n    (\n        \"data_2023_q2.parquet\",\n        {\n            \"sales_amount\": RangeFieldInfo(\n                min_value=pa.scalar(200), max_value=pa.scalar(75000),\n                has_nulls=False, has_non_nulls=True\n            ),\n            \"region\": SetFieldInfo(\n                values={pa.scalar(\"US\"), pa.scalar(\"EU\"), pa.scalar(\"UK\")},\n                has_nulls=False, has_non_nulls=True\n            ),\n        }\n    ),\n]\n\n# Create planner\nplanner = Planner(files)\n\n# Filter files based on SQL expressions\nmatching_files = set(planner.files(\"sales_amount \u003e 40000 AND region = 'US'\"))\nprint(matching_files)  # {'data_2023_q1.parquet', 'data_2023_q2.parquet'}\n\n# More complex queries\nmatching_files = set(planner.files(\"region IN ('EU', 'UK')\"))\nprint(matching_files)  # {'data_2023_q2.parquet'}\n```\n\n## Field Information Types\n\n### `RangeFieldInfo`\n\nFor fields with known minimum and maximum values:\n\n```python\nRangeFieldInfo(\n    min_value=pa.scalar(0),\n    max_value=pa.scalar(100),\n    has_nulls=False,      # Whether the field contains NULL values\n    has_non_nulls=True    # Whether the field contains non-NULL values\n)\n```\n\n### `SetFieldInfo`\n\nFor fields with a known set of possible values (useful for categorical data):\n\n```python\nSetFieldInfo(\n    values={pa.scalar(\"apple\"), pa.scalar(\"banana\"), pa.scalar(\"cherry\")},\n    has_nulls=False,\n    has_non_nulls=True\n)\n```\n\n**Note**: `SetFieldInfo` can produce false positives - if a value is in the set, the file *might* contain it, but the file could contain additional values not in the set.\n\n## Supported SQL Operations\n\n### Comparison Operators\n- `=`, `!=`, `\u003c\u003e` (equality and inequality)\n- `\u003c`, `\u003c=`, `\u003e`, `\u003e=` (range comparisons)\n- `IS NULL`, `IS NOT NULL` (null checks)\n- `IS DISTINCT FROM`, `IS NOT DISTINCT FROM` (null-safe comparisons)\n\n### Logical Operators\n- `AND`, `OR`, `XOR` (logical connectors)\n- `NOT` (negation)\n\n### Set Operations\n- `IN`, `NOT IN` (membership tests)\n- `BETWEEN`, `NOT BETWEEN` (range tests)\n\n### Control Flow\n- `CASE WHEN ... THEN ... ELSE ... END` (conditional expressions)\n\n### Data Types\n- `CAST` (type casting)\n\n### Literals\n- Numeric literals: `123`, `45.67`\n- String literals: `'hello'`\n- Boolean literals: `TRUE`, `FALSE`\n- NULL literal: `NULL`\n\n## Examples\n\n### Range Queries\n```python\n# Files with sales between 1000 and 5000\nplanner.files(\"sales_amount BETWEEN 1000 AND 5000\")\n\n# Files with any sales over 10000\nplanner.files(\"sales_amount \u003e 10000\")\n```\n\n### Set Membership\n```python\n# Files containing specific regions\nplanner.files(\"region IN ('US', 'CA')\")\n\n# Files not containing specific regions\nplanner.files(\"region NOT IN ('UNKNOWN', 'TEST')\")\n```\n\n### Complex Conditions\n```python\n# Combination of range and set conditions\nplanner.files(\n    \"sales_amount \u003e 5000 AND region IN ('US', 'EU') AND customer_id IS NOT NULL\"\n)\n\n# Case expressions\nplanner.files(\n    \"CASE WHEN region = 'US' THEN sales_amount \u003e 1000 ELSE sales_amount \u003e 500 END\"\n)\n```\n\n### Null Handling\n```python\n# Files that might contain null values in sales_amount\nplanner.files(\"sales_amount IS NULL\")\n\n# Files with non-null sales amounts over 1000\nplanner.files(\"sales_amount IS NOT NULL AND sales_amount \u003e 1000\")\n```\n\n## Performance Considerations\n\n- **Metadata Quality**: More accurate metadata (tighter ranges, complete value sets) leads to better filtering\n- **Expression Complexity**: Simple expressions evaluate faster than complex nested conditions\n- **False Positives**: The library errs on the side of including files that might match rather than risk excluding files that do match\n\n## Use Cases\n\n- **Data Lake Query Optimization**: Skip irrelevant files in distributed query engines\n- **ETL Pipeline Optimization**: Process only files containing relevant data\n- **Data Catalog Integration**: Enhance metadata catalogs with query planning capabilities\n- **Columnar Storage**: Optimize scans of Parquet, ORC, or similar formats\n\n## Development\n\n### Setup\n```bash\ngit clone https://github.com/query-farm/python-sql-scan-planning.git\ncd python-sql-scan-planning\nrye sync\n```\n\n### Running Tests\n```bash\nrye run pytest\n```\n\n### Code Quality\n```bash\nrye run ruff check\nrye run pytest --mypy\n```\n\n## Dependencies\n\n- **sqlglot**: SQL parsing and AST manipulation\n- **Python 3.12+**: Required for modern type hints and pattern matching\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Add tests for new functionality\n4. Ensure all tests pass\n5. Submit a pull request\n\n## Author\n\nThis Python module was created by [Query.Farm](https://query.farm).\n\n# License\n\nMIT Licensed.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquery-farm%2Fpython-sql-scan-planning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquery-farm%2Fpython-sql-scan-planning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquery-farm%2Fpython-sql-scan-planning/lists"}