{"id":17972328,"url":"https://github.com/apache/datafusion-python","last_synced_at":"2025-05-14T09:06:55.061Z","repository":{"id":47898243,"uuid":"515951203","full_name":"apache/datafusion-python","owner":"apache","description":"Apache DataFusion Python Bindings","archived":false,"fork":false,"pushed_at":"2025-05-09T20:29:29.000Z","size":8017,"stargazers_count":450,"open_issues_count":99,"forks_count":109,"subscribers_count":34,"default_branch":"main","last_synced_at":"2025-05-10T17:16:23.701Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://datafusion.apache.org/python","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-07-20T11:19:28.000Z","updated_at":"2025-05-09T06:49:50.000Z","dependencies_parsed_at":"2023-09-24T02:18:28.012Z","dependency_job_id":"fbf7c2d9-6245-43be-a923-7a3f3a5f4641","html_url":"https://github.com/apache/datafusion-python","commit_stats":{"total_commits":297,"total_committers":37,"mean_commits":8.027027027027026,"dds":0.6902356902356903,"last_synced_commit":"1f49d4615ebc464394d38b2dea0fa130749ea55f"},"previous_names":["apache/datafusion-python","apache/arrow-datafusion-python"],"tags_count":72,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fdatafusion-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fdatafusion-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fdatafusion-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fdatafusion-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/datafusion-python/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253450236,"owners_count":21910514,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T16:12:16.184Z","updated_at":"2025-05-14T09:06:55.035Z","avatar_url":"https://github.com/apache.png","language":"Python","funding_links":[],"categories":["Data Processing \u0026 DataFrames","Python"],"sub_categories":[],"readme":"\u003c!---\n  Licensed to the Apache Software Foundation (ASF) under one\n  or more contributor license agreements.  See the NOTICE file\n  distributed with this work for additional information\n  regarding copyright ownership.  The ASF licenses this file\n  to you under the Apache License, Version 2.0 (the\n  \"License\"); you may not use this file except in compliance\n  with the License.  You may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\n  Unless required by applicable law or agreed to in writing,\n  software distributed under the License is distributed on an\n  \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n  KIND, either express or implied.  See the License for the\n  specific language governing permissions and limitations\n  under the License.\n--\u003e\n\n# DataFusion in Python\n\n[![Python test](https://github.com/apache/datafusion-python/actions/workflows/test.yaml/badge.svg)](https://github.com/apache/datafusion-python/actions/workflows/test.yaml)\n[![Python Release Build](https://github.com/apache/datafusion-python/actions/workflows/build.yml/badge.svg)](https://github.com/apache/datafusion-python/actions/workflows/build.yml)\n\nThis is a Python library that binds to [Apache Arrow](https://arrow.apache.org/) in-memory query engine [DataFusion](https://github.com/apache/datafusion).\n\nDataFusion's Python bindings can be used as a foundation for building new data systems in Python. Here are some examples:\n\n- [Dask SQL](https://github.com/dask-contrib/dask-sql) uses DataFusion's Python bindings for SQL parsing, query\n  planning, and logical plan optimizations, and then transpiles the logical plan to Dask operations for execution.\n- [DataFusion Ballista](https://github.com/apache/datafusion-ballista) is a distributed SQL query engine that extends\n  DataFusion's Python bindings for distributed use cases.\n- [DataFusion Ray](https://github.com/apache/datafusion-ray) is another distributed query engine that uses\n  DataFusion's Python bindings.\n\n## Features\n\n- Execute queries using SQL or DataFrames against CSV, Parquet, and JSON data sources.\n- Queries are optimized using DataFusion's query optimizer.\n- Execute user-defined Python code from SQL.\n- Exchange data with Pandas and other DataFrame libraries that support PyArrow.\n- Serialize and deserialize query plans in Substrait format.\n- Experimental support for transpiling SQL queries to DataFrame calls with Polars, Pandas, and cuDF.\n\n## Example Usage\n\nThe following example demonstrates running a SQL query against a Parquet file using DataFusion, storing the results\nin a Pandas DataFrame, and then plotting a chart.\n\nThe Parquet file used in this example can be downloaded from the following page:\n\n- https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page\n\n```python\nfrom datafusion import SessionContext\n\n# Create a DataFusion context\nctx = SessionContext()\n\n# Register table with context\nctx.register_parquet('taxi', 'yellow_tripdata_2021-01.parquet')\n\n# Execute SQL\ndf = ctx.sql(\"select passenger_count, count(*) \"\n             \"from taxi \"\n             \"where passenger_count is not null \"\n             \"group by passenger_count \"\n             \"order by passenger_count\")\n\n# convert to Pandas\npandas_df = df.to_pandas()\n\n# create a chart\nfig = pandas_df.plot(kind=\"bar\", title=\"Trip Count by Number of Passengers\").get_figure()\nfig.savefig('chart.png')\n```\n\nThis produces the following chart:\n\n![Chart](examples/chart.png)\n\n## Registering a DataFrame as a View\n\nYou can use SessionContext's `register_view` method to convert a DataFrame into a view and register it with the context.\n\n```python\nfrom datafusion import SessionContext, col, literal\n\n# Create a DataFusion context\nctx = SessionContext()\n\n# Create sample data\ndata = {\"a\": [1, 2, 3, 4, 5], \"b\": [10, 20, 30, 40, 50]}\n\n# Create a DataFrame from the dictionary\ndf = ctx.from_pydict(data, \"my_table\")\n\n# Filter the DataFrame (for example, keep rows where a \u003e 2)\ndf_filtered = df.filter(col(\"a\") \u003e literal(2))\n\n# Register the dataframe as a view with the context\nctx.register_view(\"view1\", df_filtered)\n\n# Now run a SQL query against the registered view\ndf_view = ctx.sql(\"SELECT * FROM view1\")\n\n# Collect the results\nresults = df_view.collect()\n\n# Convert results to a list of dictionaries for display\nresult_dicts = [batch.to_pydict() for batch in results]\n\nprint(result_dicts)\n```\n\nThis will output:\n\n```python\n[{'a': [3, 4, 5], 'b': [30, 40, 50]}]\n```\n\n## Configuration\n\nIt is possible to configure runtime (memory and disk settings) and configuration settings when creating a context.\n\n```python\nruntime = (\n    RuntimeEnvBuilder()\n    .with_disk_manager_os()\n    .with_fair_spill_pool(10000000)\n)\nconfig = (\n    SessionConfig()\n    .with_create_default_catalog_and_schema(True)\n    .with_default_catalog_and_schema(\"foo\", \"bar\")\n    .with_target_partitions(8)\n    .with_information_schema(True)\n    .with_repartition_joins(False)\n    .with_repartition_aggregations(False)\n    .with_repartition_windows(False)\n    .with_parquet_pruning(False)\n    .set(\"datafusion.execution.parquet.pushdown_filters\", \"true\")\n)\nctx = SessionContext(config, runtime)\n```\n\nRefer to the [API documentation](https://arrow.apache.org/datafusion-python/#api-reference) for more information.\n\nPrinting the context will show the current configuration settings.\n\n```python\nprint(ctx)\n```\n\n## Extensions\n\nFor information about how to extend DataFusion Python, please see the extensions page of the\n[online documentation](https://datafusion.apache.org/python/).\n\n## More Examples\n\nSee [examples](examples/README.md) for more information.\n\n### Executing Queries with DataFusion\n\n- [Query a Parquet file using SQL](https://github.com/apache/datafusion-python/blob/main/examples/sql-parquet.py)\n- [Query a Parquet file using the DataFrame API](https://github.com/apache/datafusion-python/blob/main/examples/dataframe-parquet.py)\n- [Run a SQL query and store the results in a Pandas DataFrame](https://github.com/apache/datafusion-python/blob/main/examples/sql-to-pandas.py)\n- [Run a SQL query with a Python user-defined function (UDF)](https://github.com/apache/datafusion-python/blob/main/examples/sql-using-python-udf.py)\n- [Run a SQL query with a Python user-defined aggregation function (UDAF)](https://github.com/apache/datafusion-python/blob/main/examples/sql-using-python-udaf.py)\n- [Query PyArrow Data](https://github.com/apache/datafusion-python/blob/main/examples/query-pyarrow-data.py)\n- [Create dataframe](https://github.com/apache/datafusion-python/blob/main/examples/import.py)\n- [Export dataframe](https://github.com/apache/datafusion-python/blob/main/examples/export.py)\n\n### Running User-Defined Python Code\n\n- [Register a Python UDF with DataFusion](https://github.com/apache/datafusion-python/blob/main/examples/python-udf.py)\n- [Register a Python UDAF with DataFusion](https://github.com/apache/datafusion-python/blob/main/examples/python-udaf.py)\n\n### Substrait Support\n\n- [Serialize query plans using Substrait](https://github.com/apache/datafusion-python/blob/main/examples/substrait.py)\n\n## How to install\n\n### uv\n\n```bash\nuv add datafusion\n```\n\n### Pip\n\n```bash\npip install datafusion\n# or\npython -m pip install datafusion\n```\n\n### Conda\n\n```bash\nconda install -c conda-forge datafusion\n```\n\nYou can verify the installation by running:\n\n```python\n\u003e\u003e\u003e import datafusion\n\u003e\u003e\u003e datafusion.__version__\n'0.6.0'\n```\n\n## How to develop\n\nThis assumes that you have rust and cargo installed. We use the workflow recommended by [pyo3](https://github.com/PyO3/pyo3) and [maturin](https://github.com/PyO3/maturin). The Maturin tools used in this workflow can be installed either via `uv` or `pip`. Both approaches should offer the same experience. It is recommended to use `uv` since it has significant performance improvements\nover `pip`.\n\nBootstrap (`uv`):\n\nBy default `uv` will attempt to build the datafusion python package. For our development we prefer to build manually. This means\nthat when creating your virtual environment using `uv sync` you need to pass in the additional `--no-install-package datafusion`\nand for `uv run` commands the additional parameter `--no-project`\n\n```bash\n# fetch this repo\ngit clone git@github.com:apache/datafusion-python.git\n# create the virtual enviornment\nuv sync --dev --no-install-package datafusion\n# activate the environment\nsource .venv/bin/activate\n```\n\nBootstrap (`pip`):\n\n```bash\n# fetch this repo\ngit clone git@github.com:apache/datafusion-python.git\n# prepare development environment (used to build wheel / install in development)\npython3 -m venv .venv\n# activate the venv\nsource .venv/bin/activate\n# update pip itself if necessary\npython -m pip install -U pip\n# install dependencies\npython -m pip install -r pyproject.toml\n```\n\nThe tests rely on test data in git submodules.\n\n```bash\ngit submodule update --init\n```\n\nWhenever rust code changes (your changes or via `git pull`):\n\n```bash\n# make sure you activate the venv using \"source venv/bin/activate\" first\nmaturin develop --uv\npython -m pytest\n```\n\nAlternatively if you are using `uv` you can do the following without\nneeding to activate the virtual environment:\n\n```bash\nuv run --no-project maturin develop --uv\nuv --no-project pytest .\n```\n\n### Running \u0026 Installing pre-commit hooks\n\n`datafusion-python` takes advantage of [pre-commit](https://pre-commit.com/) to assist developers with code linting to help reduce\nthe number of commits that ultimately fail in CI due to linter errors. Using the pre-commit hooks is optional for the\ndeveloper but certainly helpful for keeping PRs clean and concise.\n\nOur pre-commit hooks can be installed by running `pre-commit install`, which will install the configurations in\nyour DATAFUSION_PYTHON_ROOT/.github directory and run each time you perform a commit, failing to complete\nthe commit if an offending lint is found allowing you to make changes locally before pushing.\n\nThe pre-commit hooks can also be run adhoc without installing them by simply running `pre-commit run --all-files`\n\n## Running linters without using pre-commit\n\nThere are scripts in `ci/scripts` for running Rust and Python linters.\n\n```shell\n./ci/scripts/python_lint.sh\n./ci/scripts/rust_clippy.sh\n./ci/scripts/rust_fmt.sh\n./ci/scripts/rust_toml_fmt.sh\n```\n\n## How to update dependencies\n\nTo change test dependencies, change the `pyproject.toml` and run\n\n```bash\nuv sync --dev --no-install-package datafusion\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fdatafusion-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fdatafusion-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fdatafusion-python/lists"}