{"id":14982402,"url":"https://github.com/apache/incubator-wayang","last_synced_at":"2026-01-11T16:58:10.242Z","repository":{"id":36993226,"uuid":"321981396","full_name":"apache/incubator-wayang","owner":"apache","description":"Apache Wayang(incubating) is the first cross-platform data processing system.","archived":false,"fork":false,"pushed_at":"2025-05-09T14:27:38.000Z","size":19706,"stargazers_count":220,"open_issues_count":117,"forks_count":97,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-05-09T15:40:34.639Z","etag":null,"topics":["apache","big-data","cross-platform","data-management-platform","data-processing","distributed-system","hadoop","java","jdbc","middleware","open-source","performance","scala","spark"],"latest_commit_sha":null,"homepage":"https://wayang.incubator.apache.org/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-12-16T12:51:54.000Z","updated_at":"2025-05-09T14:27:43.000Z","dependencies_parsed_at":"2023-10-14T18:15:03.584Z","dependency_job_id":"3afc9cda-cd00-416f-a250-89bfafd0a550","html_url":"https://github.com/apache/incubator-wayang","commit_stats":{"total_commits":1892,"total_committers":48,"mean_commits":"39.416666666666664","dds":0.5364693446088795,"last_synced_commit":"886938e64644ce3baa92cca1e5273f04a17a509c"},"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-wayang","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-wayang/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-wayang/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-wayang/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/incubator-wayang/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253857429,"owners_count":21974733,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache","big-data","cross-platform","data-management-platform","data-processing","distributed-system","hadoop","java","jdbc","middleware","open-source","performance","scala","spark"],"created_at":"2024-09-24T14:05:21.159Z","updated_at":"2026-01-11T16:58:10.229Z","avatar_url":"https://github.com/apache.png","language":"Java","funding_links":[],"categories":["大数据"],"sub_categories":[],"readme":"\u003c!--\n  - Licensed to the Apache Software Foundation (ASF) under one\n  - or more contributor license agreements.  See the NOTICE file\n  - distributed with this work for additional information\n  - regarding copyright ownership.  The ASF licenses this file\n  - to you under the Apache License, Version 2.0 (the\n  - \"License\"); you may not use this file except in compliance\n  - with the License.  You may obtain a copy of the License at\n  -\n  -   http://www.apache.org/licenses/LICENSE-2.0\n  -\n  - Unless required by applicable law or agreed to in writing,\n  - software distributed under the License is distributed on an\n  - \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n  - KIND, either express or implied.  See the License for the\n  - specific language governing permissions and limitations\n  - under the License.\n  --\u003e\n\n# Apache Wayang™ \u003cimg align=\"right\" width=\"128px\" src=\"https://wayang.apache.org/img/wayang.png\" alt=\"Wayang Logo\"\u003e\n\n## The first open-source cross-platform data processing system\n\n[![Maven central](https://img.shields.io/maven-central/v/org.apache.wayang/wayang-core.svg?style=for-the-badge)](https://img.shields.io/maven-central/v/org.apache.wayang/wayang-core.svg)\n[![License](https://img.shields.io/github/license/apache/incubator-wayang.svg?style=for-the-badge)](http://www.apache.org/licenses/LICENSE-2.0)\n[![Last commit](https://img.shields.io/github/last-commit/apache/incubator-wayang.svg?style=for-the-badge)]()\n![GitHub commit activity (branch)](https://img.shields.io/github/commit-activity/m/apache/incubator-wayang?style=for-the-badge)\n![GitHub forks](https://img.shields.io/github/forks/apache/incubator-wayang?style=for-the-badge)\n![GitHub Repo stars](https://img.shields.io/github/stars/apache/incubator-wayang?style=for-the-badge)\n\n[![Tweet](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/intent/tweet?text=Apache%20Wayang%20enables%20cross%20platform%20data%20processing,%20star%20it%20via:%20\u0026url=https://github.com/apache/incubator-wayang\u0026via=apachewayang\u0026hashtags=dataprocessing,bigdata,analytics,hybridcloud,developers) [![Subreddit subscribers](https://img.shields.io/reddit/subreddit-subscribers/ApacheWayang?style=social)](https://www.reddit.com/r/ApacheWayang/)\n## Table of contents\n  * [Description](#description)\n  * [Quick Guide for Running Wayang](#quick-guide-for-running-wayang)\n  * [Quick Guide for Developing with Wayang](#quick-guide-for-developing-with-wayang)\n  * [Installing Wayang](#installing-wayang)\n    + [Requirements at Runtime](#requirements-at-runtime)\n    + [Validating the installation](#validating-the-installation)\n  * [Getting Started](#getting-started)\n    + [Prerequisites](#prerequisites)\n    + [Building](#building)\n  * [Running the tests](#running-the-tests)\n  * [Example Applications](#example-applications)\n  * [Built With](#built-with)\n  * [Contributing](#contributing)\n  * [Authors](#authors)\n  * [License](#license)\n\n## Description\n\nIn contrast to traditional data processing systems that provide one dedicated execution engine, Apache Wayang can transparently and seamlessly integrate multiple execution engines and use them to perform a single task. We call this *cross-platform data processing*. In Wayang, users can specify any data processing application using one of Wayang's APIs and then Wayang can choose the data processing platform(s), e.g., Postgres or Apache Spark, that best fits the application. Finally, Wayang will orchestrate the execution, thereby hiding the different platform-specific APIs and coordinating inter-platform communication.\n\nApache Wayang aims at freeing data engineers and software developers from the burden of learning all different data processing systems, their APIs, strengths and weaknesses; the intricacies of coordinating and integrating different processing platforms; and the inflexibility when trying a fixed set of processing platforms. As of now, Wayang has built-in support for the following processing platforms:\n- [Java Streams](https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html)\n- [Apache Spark](https://spark.apache.org/)\n- [Apache Flink](https://flink.apache.org/)\n- [Apache Giraph](https://giraph.apache.org/)\n- [GraphChi](https://github.com/GraphChi/graphchi-java)\n- [Postgres](http://www.postgresql.org)\n- [SQLite](https://www.sqlite.org/)\n- [Apache Kafka](https://kafka.apache.org)\n- [Tensorflow](https://www.tensorflow.org/)\n\nApache Wayang (incubating) can be used via the following APIs:\n- Java scala-like\n- Scala\n- SQL\n- Java native (recommended only for low level development)\n\nApache Wayang provides a flexible architecture which enables easy addition of new operators and data processing platforms without requiring any change of the internals of the system. For details on how to add new operators, see [here](https://wayang.apache.org/docs/guide/adding-operators).\n\n## Quick Guide for Running Wayang\n\nFor a quick guide on how to run WordCount see [here](guides/tutorial.md).\n\n### Spark Dataset / DataFrame pipelines\n\nWayang’s Spark platform can now execute end-to-end pipelines on Spark `Dataset[Row]` (aka DataFrames). This is particularly useful when working with lakehouse-style storage (Parquet/Delta) or when you want to plug Spark ML stages into a Wayang plan without repeatedly falling back to RDDs.\n\nTo build a Dataset-backed pipeline:\n\n1. **Use the Dataset-aware plan builder APIs.**\n   - `PlanBuilder.readParquet(..., preferDataset = true)` (or `JavaPlanBuilder.readParquet(..., ..., true)`) reads Parquet files directly into a Dataset channel.\n   - `DataQuanta.writeParquet(..., preferDataset = true)` writes a Dataset channel without converting it back to an RDD.\n2. **Keep operators dataset-compatible.** Most operators continue to work unchanged; if an operator explicitly prefers RDDs, Wayang will insert the necessary conversions automatically (at an additional cost). Custom operators can expose `DatasetChannel` descriptors to stay in the dataframe world.\n3. **Let the optimizer do the rest.** The optimizer now assigns a higher cost to Dataset↔RDD conversions, so once you opt into Dataset sources/sinks the plan will stay in Dataset form by default.\n\nNo extra flags are required—just opt into the Dataset-based APIs where you want dataframe semantics. If you see unexpected conversions in your execution plan, check that the upstream/downstream operators you use can consume `DatasetChannel`s; otherwise Wayang will insert a conversion operator for you.\n\n## Quick Guide for Developing with Wayang\n\nFor a quick guide on how to use Wayang in your Java/Scala project see [here](guides/develop-with-Wayang.md).\n\n## Installing Wayang\n\nYou first have to build the binaries as shown [here](guides/tutorial.md).\nOnce you have the binaries built, follow these steps to install Wayang:\n\n```shell\ntar -xvf wayang-1.0.1-SNAPSHOT.tar.gz\ncd wayang-1.0.1-SNAPSHOT\n```\n\nIn linux\n```shell\necho \"export WAYANG_HOME=$(pwd)\" \u003e\u003e ~/.bashrc\necho \"export PATH=${PATH}:${WAYANG_HOME}/bin\" \u003e\u003e ~/.bashrc\nsource ~/.bashrc\n```\nIn MacOS\n```shell\necho \"export WAYANG_HOME=$(pwd)\" \u003e\u003e ~/.zshrc\necho \"export PATH=${PATH}:${WAYANG_HOME}/bin\" \u003e\u003e ~/.zshrc\nsource ~/.zshrc\n```\n\n### Requirements at Runtime\n\nApache Wayang (incubating) relies on external execution engines and Java to function correctly. Below are the updated runtime requirements:\n\n- **Java 17**: Make sure `JAVA_HOME` is correctly set to your Java 17 installation.\n- **Apache Spark 3.4.4**: Compatible with Scala 2.12. Set the `SPARK_HOME` environment variable.\n- **Apache Hadoop 3+**: Set the `HADOOP_HOME` environment variable.\n\n\u003e 🛠️ **Note:** When using Java 17, you _must_ add JVM flags to allow Wayang and Spark to access internal Java APIs, or you will encounter `IllegalAccessError`. See below.\n\n### Validating the installation\n\nTo execute your first application with Apache Wayang, you need to execute your program with the 'wayang-submit' command:\n\n```shell\nbin/wayang-submit org.apache.wayang.apps.wordcount.Main java file://$(pwd)/README.md\n```\n\n### ⚙️ Java 17 Compatibility\n\nWhen running Wayang applications using Java 17 (especially with Spark), you must add JVM flags to open specific internal Java modules. These flags resolve access issues with `sun.nio.ch.DirectBuffer` and others.\n\nUpdate your `wayang-submit` (wayang-assembly/target/wayang-1.0.1-SNAPSHOT/bin/wayang-submit) script (or command) with:\n\n```bash\neval \"$RUNNER \\\n  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \\\n  --add-opens=java.base/java.nio=ALL-UNNAMED \\\n  --add-opens=java.base/java.lang=ALL-UNNAMED \\\n  --add-opens=java.base/java.util=ALL-UNNAMED \\\n  --add-opens=java.base/java.io=ALL-UNNAMED \\\n  --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \\\n  --add-opens=java.base/java.util.concurrent=ALL-UNNAMED \\\n  --add-opens=java.base/java.net=ALL-UNNAMED \\\n  --add-opens=java.base/java.lang.invoke=ALL-UNNAMED \\\n  $FLAGS -cp \\\"${WAYANG_CLASSPATH}\\\" $CLASS ${ARGS}\"\n```\n\n## Getting Started\n\nWayang is available via Maven Central. To use it with Maven, include the following code snippet into your POM file:\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eorg.apache.wayang\u003c/groupId\u003e\n  \u003cartifactId\u003ewayang-***\u003c/artifactId\u003e\n  \u003cversion\u003e1.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\nNote the `***`: Wayang ships with multiple modules that can be included in your app, depending on how you want to use it:\n* `wayang-core`: provides core data structures and the optimizer (required)\n* `wayang-basic`: provides common operators and data types for your apps (recommended)\n* `wayang-api-scala-java`: provides an easy-to-use Scala and Java API to assemble Wayang plans (recommended)\n* `wayang-java`, `wayang-spark`, `wayang-graphchi`, `wayang-sqlite3`, `wayang-postgres`: adapters for the various supported processing platforms\n* `wayang-profiler`: provides functionality to learn operator and UDF cost functions from historical execution data\n\n\u003e **NOTE:** The module `wayang-api-scala-java` is intended to be used with Java 11 and Scala 2.12.\n\nFor the sake of version flexibility, you still have to include in the POM file your Hadoop (`hadoop-hdfs` and `hadoop-common`) and Spark (`spark-core` and `spark-graphx`) version of choice.\n\nIn addition, you can obtain the most recent snapshot version of Wayang via Sonatype's snapshot repository. Just include:\n```xml\n\u003crepositories\u003e\n  \u003crepository\u003e\n    \u003cid\u003eapache-snapshots\u003c/id\u003e\n    \u003cname\u003eApache Foundation Snapshot Repository\u003c/name\u003e\n    \u003curl\u003ehttps://repository.apache.org/content/repositories/snapshots\u003c/url\u003e\n  \u003c/repository\u003e\n\u003c/repositories\u003e\n```\n\n### Prerequisites\nApache Wayang (incubating) is built with Java 17 and Scala 2.12. However, to run Apache Wayang it is sufficient to have just Java 17 installed. Please also consider that processing platforms employed by Wayang might have further requirements.\n```\nJava 17\nScala 2.12.17\nSpark 3.4.4, Compatible with Scala 2.12.\nMaven\n```\n\n\u003e **NOTE:** In windows, you need to define the variable `HADOOP_HOME` with the winutils.exe, an not official option to obtain [this repository](https://github.com/steveloughran/winutils), or you can generate your winutils.exe following the instructions in the repository. Also, you may need to install [msvcr100.dll](https://www.microsoft.com/en-us/download/details.aspx?id=26999)\n\n\u003e **NOTE:** Make sure that the JAVA_HOME environment variable is set correctly to Java 17 as the prerequisite checker script currently supports up to Java 17 and checks the latest version of Java if you have higher version installed. In Linux, it is preferably to use the export JAVA_HOME method inside the project folder. It is also recommended running './mvnw clean install' before opening the project using IntelliJ.\n\n\n### Building\n\nIf you need to rebuild Wayang, e.g., to use a different Scala version, you can simply do so via Maven:\n\n1. Adapt the version variables (e.g., `spark.version`) in the main `pom.xml` file.\n2. Build Wayang with the adapted versions.\n    ```shell\n   git clone https://github.com/apache/incubator-wayang.git\n   cd incubator-wayang\n   ./mvnw clean install -DskipTests\n    ```\n\u003e **NOTE:** If you receive an error about not finding `MathExBaseVisitor`, then the problem might be that you are trying to build from IntelliJ, without Maven. MathExBaseVisitor is generated code, and a Maven build should generate it automatically.\n\n\u003e **NOTE:**: In the current Maven setup, Wayang supports Java 17. The default Scala version is 2.12.17, which is compatible with Java 17. Ensure that your Spark distribution is also built with Scala 2.12 (e.g., `spark-3.4.4-bin-hadoop3-scala2.12`).\n\n\u003e **NOTE:** For compiling and testing the code it is required to have Hadoop installed on your machine.\n\n\u003e **NOTE:**  the `standalone` profile to fix Hadoop and Spark versions, so that Wayang apps do not explicitly need to declare the corresponding dependencies.\n\n\u003e **NOTE**: When running applications (e.g., WordCount) with Java 17, you must pass additional flags to allow internal module access:\n\n\u003e--add-exports=java.base/sun.nio.ch=ALL-UNNAMED \\\n--add-opens=java.base/java.nio=ALL-UNNAMED \\\n--add-opens=java.base/java.lang=ALL-UNNAMED \\\n--add-opens=java.base/java.util=ALL-UNNAMED \\\n--add-opens=java.base/java.io=ALL-UNNAMED \\\n--add-opens=java.base/java.lang.reflect=ALL-UNNAMED \\\n--add-opens=java.base/java.util.concurrent=ALL-UNNAMED \\\n--add-opens=java.base/java.net=ALL-UNNAMED \\\n--add-opens=java.base/java.lang.invoke=ALL-UNNAMED \\\n\n\u003e\n\u003e Also, note the `distro` profile, which assembles a binary Wayang distribution.\nTo activate these profiles, you need to specify them when running maven, i.e.,\n\n```shell\n./mvnw clean install -DskipTests -P\u003cprofile name\u003e\n```\n\n## Running the tests\nIn the incubator-wayang root folder run:\n```shell\n./mvnw test\n```\n\n## Example Applications\nYou can see examples on how to start using Wayang [here](guides/wayang-examples.md)\n\n## Built With\n\n* [Java 17](https://www.oracle.com/java/technologies/javase/17-0-14-relnotes.html)\n* [Scala 2.12.17](https://www.scala-lang.org/download/2.12.17.html)\n* [Maven](https://maven.apache.org/)\n\n## Contributing\nBefore submitting a PR, please take a look on how to contribute with Apache Wayang contributing guidelines [here](CONTRIBUTING.md).\n\nThere is also a guide on how to compile your code [here](guides/develop-in-Wayang.md).\n## Authors\nThe list of [contributors](https://github.com/apache/incubator-wayang/graphs/contributors).\n\n## License\nAll files in this repository are licensed under the Apache Software License 2.0\n\nCopyright 2020 - 2025 The Apache Software Foundation.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n## Acknowledgements\nThe [Logo](https://wayang.apache.org/img/wayang.png) was donated by Brian Vera.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fincubator-wayang","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fincubator-wayang","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fincubator-wayang/lists"}