Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/apache/datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://github.com/apache/datafusion-comet

arrow datafusion rust spark

Last synced: 1 day ago
JSON representation

Apache DataFusion Comet Spark Accelerator

Awesome Lists containing this project

README 

        



# Apache DataFusion Comet



[![Apache licensed][license-badge]][license-url]

[![Discord chat][discord-badge]][discord-url]



[license-badge]: https://img.shields.io/badge/license-Apache%20v2-blue.svg

[license-url]: https://github.com/apache/datafusion-comet/blob/main/LICENSE.txt

[discord-badge]: https://img.shields.io/discord/885562378132000778.svg?logo=discord&style=flat-square

[discord-url]: https://discord.gg/3EAr4ZX6JK



logo



Apache DataFusion Comet is a high-performance accelerator for Apache Spark, built on top of the powerful

[Apache DataFusion] query engine. Comet is designed to significantly enhance the

performance of Apache Spark workloads while leveraging commodity hardware and seamlessly integrating with the

Spark ecosystem without requiring any code changes.



[Apache DataFusion]: https://datafusion.apache.org



# Benefits of Using Comet



## Run Spark Queries at DataFusion Speeds



Comet delivers a performance speedup for many queries, enabling faster data processing and shorter time-to-insights.



The following chart shows the time it takes to run the 22 TPC-H queries against 100 GB of data in Parquet format 

using a single executor with 8 cores. See the [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html)

for details of the environment used for these benchmarks.



When using Comet, the overall run time is reduced from 640 seconds to 331 seconds, very close to a 2x speedup.



![](docs/source/_static/images/benchmark-results/0.5.0/tpch_allqueries.png)



Here is a breakdown showing relative performance of Spark and Comet for each TPC-H query.



![](docs/source/_static/images/benchmark-results/0.5.0/tpch_queries_compare.png)



The following charts shows how much Comet currently accelerates each query from the benchmark.



### Relative speedup



![](docs/source/_static/images/benchmark-results/0.5.0/tpch_queries_speedup_rel.png)



### Absolute speedup



![](docs/source/_static/images/benchmark-results/0.5.0/tpch_queries_speedup_abs.png)



These benchmarks can be reproduced in any environment using the documentation in the 

[Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html). We encourage 

you to run your own benchmarks.



Results for our benchmark derived from TPC-DS are available in the [benchmarking guide](https://datafusion.apache.org/comet/contributor-guide/benchmark-results/tpc-ds.html).



## Use Commodity Hardware



Comet leverages commodity hardware, eliminating the need for costly hardware upgrades or

specialized hardware accelerators, such as GPUs or FPGA. By maximizing the utilization of commodity hardware, Comet 

ensures cost-effectiveness and scalability for your Spark deployments.



## Spark Compatibility



Comet aims for 100% compatibility with all supported versions of Apache Spark, allowing you to integrate Comet into

your existing Spark deployments and workflows seamlessly. With no code changes required, you can immediately harness

the benefits of Comet's acceleration capabilities without disrupting your Spark applications.



## Tight Integration with Apache DataFusion



Comet tightly integrates with the core Apache DataFusion project, leveraging its powerful execution engine. With

seamless interoperability between Comet and DataFusion, you can achieve optimal performance and efficiency in your

Spark workloads.



## Active Community



Comet boasts a vibrant and active community of developers, contributors, and users dedicated to advancing the

capabilities of Apache DataFusion and accelerating the performance of Apache Spark.



## Getting Started



To get started with Apache DataFusion Comet, follow the

[installation instructions](https://datafusion.apache.org/comet/user-guide/installation.html). Join the

[DataFusion Slack and Discord channels](https://datafusion.apache.org/contributor-guide/communication.html) to connect

with other users, ask questions, and share your experiences with Comet.



Follow [Apache DataFusion Comet Overview](https://datafusion.apache.org/comet/user-guide/overview.html) to get more detailed information 



## Contributing



We welcome contributions from the community to help improve and enhance Apache DataFusion Comet. Whether it's fixing

bugs, adding new features, writing documentation, or optimizing performance, your contributions are invaluable in

shaping the future of Comet. Check out our

[contributor guide](https://datafusion.apache.org/comet/contributor-guide/contributing.html) to get started.



## License



Apache DataFusion Comet is licensed under the Apache License 2.0. See the [LICENSE.txt](LICENSE.txt) file for details.



## Acknowledgments



We would like to express our gratitude to the Apache DataFusion community for their support and contributions to

Comet. Together, we're building a faster, more efficient future for big data processing with Apache Spark.