{"id":31829920,"url":"https://github.com/legout/pydala2","last_synced_at":"2025-10-11T20:58:47.867Z","repository":{"id":186094940,"uuid":"674613895","full_name":"legout/pydala2","owner":"legout","description":"poor man´s data lake - Simple api to efficiently query your parquet datasets using Duckdb or polars","archived":false,"fork":false,"pushed_at":"2025-09-24T16:10:28.000Z","size":7408,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-10T03:27:11.957Z","etag":null,"topics":["duckdb","fsspec","local","localcache","object-storage","pandas","polars","pyarrow","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/legout.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-08-04T11:19:46.000Z","updated_at":"2025-09-24T16:10:31.000Z","dependencies_parsed_at":"2023-12-22T02:55:40.604Z","dependency_job_id":"4222c9a3-3c86-4570-bab6-f3f759e254d9","html_url":"https://github.com/legout/pydala2","commit_stats":null,"previous_names":["legout/pydala2"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/legout/pydala2","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/legout%2Fpydala2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/legout%2Fpydala2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/legout%2Fpydala2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/legout%2Fpydala2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/legout","download_url":"https://codeload.github.com/legout/pydala2/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/legout%2Fpydala2/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279008635,"owners_count":26084480,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","fsspec","local","localcache","object-storage","pandas","polars","pyarrow","python"],"created_at":"2025-10-11T20:58:46.377Z","updated_at":"2025-10-11T20:58:47.861Z","avatar_url":"https://github.com/legout.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PyDala2\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"logo.jpeg\" width=\"400\" alt=\"PyDala2\"\u003e\n\u003c/p\u003e\n\n[![PyPI version](https://badge.fury.io/py/pydala2.svg)](https://badge.fury.io/py/pydala2)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://pydala2.readthedocs.io)\n\n## Overview 📖\n\nPyDala2 is a high-performance Python library for managing Parquet datasets with advanced metadata capabilities. Built on Apache Arrow, it provides efficient management of Parquet datasets with features including:\n\n- Smart dataset management with metadata optimization\n- Multi-format support (Parquet, CSV, JSON)\n- Multi-backend integration (Polars, PyArrow, DuckDB, Pandas)\n- Advanced querying with predicate pushdown\n- Schema management with automatic validation\n- Performance optimization with caching and partitioning\n- Catalog system for centralized dataset management\n\n## ✨ Key Features\n\n- **🚀 High Performance**: Built on Apache Arrow with optimized memory usage and processing speed\n- **📊 Smart Dataset Management**: Efficient Parquet handling with metadata optimization and caching\n- **🔄 Multi-backend Support**: Seamlessly switch between Polars, PyArrow, DuckDB, and Pandas\n- **🔍 Advanced Querying**: SQL-like filtering with predicate pushdown for maximum efficiency\n- **📋 Schema Management**: Automatic validation, evolution, and tracking of data schemas\n- **⚡ Performance Optimization**: Built-in caching, compression, and intelligent partitioning\n- **🛡️ Type Safety**: Comprehensive validation and error handling throughout the library\n- **🏗️ Catalog System**: Centralized dataset management across namespaces\n\n## 🚀 Quick Start\n\n### Installation\n\n```bash\n# Install PyDala2\npip install pydala2\n\n# Install with all optional dependencies\npip install pydala2[all]\n\n# Install with specific backends\npip install pydala2[polars,duckdb]\n```\n\n### Basic Usage\n\n```python\nfrom pydala import ParquetDataset\nimport pandas as pd\n\n# Create a dataset\ndataset = ParquetDataset(\"data/my_dataset\")\n\n# Write data\ndata = pd.DataFrame({\n    'id': range(100),\n    'category': ['A', 'B', 'C'] * 33 + ['A'],\n    'value': [i * 2 for i in range(100)]\n})\ndataset.write_to_dataset(\n    data=data,\n    partition_cols=['category']\n)\n\n# Read with filtering - automatic backend selection\nresult = dataset.filter(\"category IN ('A', 'B') AND value \u003e 50\")\n\n# Export to different formats\ndf_polars = result.table.to_polars()  # or use shortcut: result.t.pl\ndf_pandas = result.table.df           # or result.t.df\nduckdb_rel = result.table.ddb         # or result.t.ddb\n```\n\n### Using Different Backends\n\n```python\n# PyDala2 provides automatic backend selection\n# Just access data in your preferred format:\n\n# Polars LazyFrame (recommended for performance)\nlazy_df = dataset.table.pl  # or dataset.t.pl\nresult = (\n    lazy_df\n    .filter(pl.col(\"value\") \u003e 100)\n    .group_by(\"category\")\n    .agg(pl.mean(\"value\"))\n    .collect()\n)\n\n# DuckDB (for SQL queries)\nresult = dataset.ddb_con.sql(\"\"\"\n    SELECT category, AVG(value) as avg_value\n    FROM dataset\n    GROUP BY category\n\"\"\").to_arrow()\n\n# PyArrow Table (for columnar operations)\ntable = dataset.table.arrow  # or dataset.t.arrow\n\n# Pandas DataFrame (for compatibility)\ndf_pandas = dataset.table.df  # or dataset.t.df\n\n# Direct export methods\ndf_polars = dataset.table.to_polars(lazy=False)\ntable = dataset.table.to_arrow()\ndf_pandas = dataset.table.to_pandas()\n```\n\n### Catalog Management\n\n```python\nfrom pydala import Catalog\n\n# Create catalog from YAML configuration\ncatalog = Catalog(\"catalog.yaml\")\n\n# YAML configuration example:\n# tables:\n#   sales_2023:\n#     path: \"/data/sales/2023\"\n#     filesystem: \"local\"\n#   customers:\n#     path: \"/data/customers\"\n#     filesystem: \"local\"\n\n# Query across datasets using automatic table loading\nresult = catalog.query(\"\"\"\n    SELECT\n        s.*,\n        c.customer_name,\n        c.segment\n    FROM sales_2023 s\n    JOIN customers c ON s.customer_id = c.id\n    WHERE s.date \u003e= '2023-01-01'\n\"\"\")\n\n# Or access datasets directly\nsales_dataset = catalog.get_dataset(\"sales_2023\")\nfiltered_sales = sales_dataset.filter(\"amount \u003e 1000\")\n```\n\n## 📚 Documentation\n\nComprehensive documentation is available at [pydala2.readthedocs.io](https://pydala2.readthedocs.io):\n\n### Getting Started\n- [Installation Guide](https://pydala2.readthedocs.io/getting-started)\n- [Quick Start Tutorial](https://pydala2.readthedocs.io/quick-start)\n\n### User Guide\n- [Basic Usage](https://pydala2.readthedocs.io/user-guide/basic-usage)\n- [Data Operations](https://pydala2.readthedocs.io/user-guide/data-operations)\n- [Performance Optimization](https://pydala2.readthedocs.io/user-guide/performance)\n- [Catalog Management](https://pydala2.readthedocs.io/user-guide/catalog-management)\n- [Schema Management](https://pydala2.readthedocs.io/user-guide/schema-management)\n\n### API Reference\n- [Core Classes](https://pydala2.readthedocs.io/api/core)\n- [Dataset Classes](https://pydala2.readthedocs.io/api/datasets)\n- [Table Operations](https://pydala2.readthedocs.io/api/table)\n- [Metadata Management](https://pydala2.readthedocs.io/api/metadata)\n- [Catalog System](https://pydala2.readthedocs.io/api/catalog)\n- [Filesystem](https://pydala2.readthedocs.io/api/filesystem)\n- [Utilities](https://pydala2.readthedocs.io/api/utilities)\n\n### Advanced Topics\n- [Performance Tuning](https://pydala2.readthedocs.io/advanced/performance-tuning)\n- [Integration Patterns](https://pydala2.readthedocs.io/advanced/integration)\n- [Deployment Guide](https://pydala2.readthedocs.io/advanced/deployment)\n- [Troubleshooting](https://pydala2.readthedocs.io/advanced/troubleshooting)\n\n## 🤝 Contributing\n\nContributions are welcome! Please see our [Contributing Guide](https://pydala2.readthedocs.io/contributing) for details.\n\n## 📝 License\n\n[MIT License](LICENSE)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flegout%2Fpydala2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flegout%2Fpydala2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flegout%2Fpydala2/lists"}