{"id":30559812,"url":"https://github.com/apache/auron","last_synced_at":"2025-08-28T10:03:17.345Z","repository":{"id":36974961,"uuid":"380944275","full_name":"apache/auron","owner":"apache","description":"The Auron accelerator for  distributed computing framework (e.g., Spark) leverages native vectorized execution to accelerate query processing","archived":false,"fork":false,"pushed_at":"2025-08-27T09:41:44.000Z","size":11681,"stargazers_count":1533,"open_issues_count":18,"forks_count":163,"subscribers_count":26,"default_branch":"master","last_synced_at":"2025-08-27T21:55:53.089Z","etag":null,"topics":["big-data","datafusion","rust-lang","spark"],"latest_commit_sha":null,"homepage":"https://auron-project.github.io/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-06-28T07:29:43.000Z","updated_at":"2025-08-27T13:21:10.000Z","dependencies_parsed_at":"2024-11-08T09:27:13.248Z","dependency_job_id":"4fead363-490d-4cc1-9821-f2e670ed1f97","html_url":"https://github.com/apache/auron","commit_stats":{"total_commits":1002,"total_committers":23,"mean_commits":43.56521739130435,"dds":0.5099800399201597,"last_synced_commit":"f222016e4cc1ceb96467e8b92e2c0d735d8768cf"},"previous_names":["blaze-init/blaze","kwai/auron","apache/auron"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/apache/auron","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fauron","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fauron/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fauron/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fauron/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/auron/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fauron/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272481561,"owners_count":24941757,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","datafusion","rust-lang","spark"],"created_at":"2025-08-28T10:01:47.408Z","updated_at":"2025-08-28T10:03:17.314Z","avatar_url":"https://github.com/apache.png","language":"Rust","funding_links":[],"categories":["HarmonyOS","Rust"],"sub_categories":["Windows Manager"],"readme":"\u003c!--\n- Licensed to the Apache Software Foundation (ASF) under one or more\n- contributor license agreements.  See the NOTICE file distributed with\n- this work for additional information regarding copyright ownership.\n- The ASF licenses this file to You under the Apache License, Version 2.0\n- (the \"License\"); you may not use this file except in compliance with\n- the License.  You may obtain a copy of the License at\n-\n-   http://www.apache.org/licenses/LICENSE-2.0\n-\n- Unless required by applicable law or agreed to in writing, software\n- distributed under the License is distributed on an \"AS IS\" BASIS,\n- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n- See the License for the specific language governing permissions and\n- limitations under the License.\n--\u003e\n\n# AURON\n\n[![TPC-DS](https://github.com/kwai/blaze/actions/workflows/tpcds.yml/badge.svg?branch=master)](https://github.com/kwai/blaze/actions/workflows/tpcds.yml)\n[![master-ce7-builds](https://github.com/kwai/blaze/actions/workflows/build-ce7-releases.yml/badge.svg?branch=master)](https://github.com/kwai/blaze/actions/workflows/build-ce7-releases.yml)\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"./dev/auron-logo.png\" /\u003e\u003c/p\u003e\n\nThe Auron accelerator for big data engine (e.g., Spark, Flink) leverages native vectorized execution to accelerate query processing. It combines\nthe power of the [Apache DataFusion](https://arrow.apache.org/datafusion/) library and the scale of the distributed\ncomputing framework.\n\nAuron takes a fully optimized physical plan from distributed computing framework, mapping it into DataFusion's execution plan, and performs native\nplan computation.\n\nThe key capabilities of Auron include:\n\n- **Native execution**:  Implemented in Rust, eliminating JVM overhead and enabling predictable performance.\n- **Vectorized computation**: Built on Apache Arrow's columnar format, fully leveraging SIMD instructions for batch processing.\n- **Pluggable architecture:**: Seamlessly integrates with Apache Spark while designed for future extensibility to other engines.\n- **Production-hardened optimizations:** Multi-level memory management, compacted shuffle formats, and adaptive execution strategies developed through large-scale deployment.\n\nBased on the inherent well-defined extensibility of DataFusion, Auron can be easily extended to support:\n\n- Various object stores.\n- Operators.\n- Simple and Aggregate functions.\n- File formats.\n\nWe encourage you to [extend DataFusion](https://github.com/apache/arrow-datafusion) capability directly and add the\nsupports in Auron with simple modifications in plan-serde and extension translation.\n\n## Build from source\n\nTo build Auron, please follow the steps below:\n\n1. Install Rust\n\nThe native execution lib is written in Rust. So you're required to install Rust (nightly) first for\ncompilation. We recommend you to use [rustup](https://rustup.rs/).\n\n2. Install JDK\n\nAuron has been well tested on jdk8/11/17.\n\n3. Check out the source code.\n\n4. Build the project.\n\nuse `./auron-build.sh` for building the project. execute `./auron-build.sh --help` for help.\n\nAfter the build is finished, a fat Jar package that contains all the dependencies will be generated in the `target`\ndirectory.\n\n## Build with docker\n\nYou can use the following command to build a centos-7 compatible release:\n```shell\nSHIM=spark-3.3 MODE=release JAVA_VERSION=8 SCALA_VERSION=2.12 ./release-docker.sh\n```\n\n## Run Spark Job with Auron Accelerator\n\nThis section describes how to submit and configure a Spark Job with Auron support.\n\n1. Move the Auron JAR to the Spark client classpath (normally spark-xx.xx.xx/jars/).\n\n2. Add the following configs to spark configuration in `spark-xx.xx.xx/conf/spark-default.conf`:\n\n```properties\nspark.auron.enable true\nspark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension\nspark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager\nspark.memory.offHeap.enabled false\n\n# suggested executor memory configuration\nspark.executor.memory 4g\nspark.executor.memoryOverhead 4096\n```\n\n3. submit a query with spark-sql, or other tools like spark-thriftserver:\n```shell\nspark-sql -f tpcds/q01.sql\n```\n\n## Performance\n\nTPC-DS 1TB Benchmark (for details, see https://auron-project.github.io/documents/benchmarks.html):\n\n![tpcds-benchmark-echarts.png](./benchmark-results/tpcds-benchmark-echarts.png)\n\nWe also encourage you to benchmark Auron and share the results with us. 🤗\n\n## Community\n\n### Subscribe Mailing Lists\n\nMail List is the most recognized form of communication in the Apache community.\nContact us through the following mailing list.\n\n| Name                                                       | Scope                           |                                                          |                                                               | \n|:-----------------------------------------------------------|:--------------------------------|:---------------------------------------------------------|:--------------------------------------------------------------|\n| [dev@auron.apache.org](mailto:dev@auron.apache.org)  | Development-related discussions | [Subscribe](mailto:dev-subscribe@auron.apache.org)    | [Unsubscribe](mailto:dev-unsubscribe@auron.apache.org)     |\n\n\n### Report Issues or Submit Pull Request\n\nIf you meet any questions, connect us and fix it by submitting a 🔗[Pull Request](https://github.com/apache/auron/pulls).\n\n## License\n\nAuron is licensed under the Apache 2.0 License. A copy of the license\n[can be found here.](LICENSE.txt)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fauron","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fauron","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fauron/lists"}