{"id":13608935,"url":"https://github.com/linkedin/transport","last_synced_at":"2025-09-12T18:33:14.820Z","repository":{"id":33623976,"uuid":"157273136","full_name":"linkedin/transport","owner":"linkedin","description":"A framework for writing performant user-defined functions (UDFs) that are portable across a variety of engines including Apache Spark, Apache Hive, and Presto.","archived":false,"fork":false,"pushed_at":"2024-01-12T21:12:03.000Z","size":1191,"stargazers_count":299,"open_issues_count":34,"forks_count":72,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-04-12T17:43:10.792Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linkedin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-11-12T20:34:12.000Z","updated_at":"2025-01-24T02:41:50.000Z","dependencies_parsed_at":"2024-01-18T09:57:38.500Z","dependency_job_id":"cea0796a-ad89-4fd3-ab64-bff8584167b2","html_url":"https://github.com/linkedin/transport","commit_stats":null,"previous_names":[],"tags_count":58,"template":false,"template_full_name":null,"purl":"pkg:github/linkedin/transport","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2Ftransport","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2Ftransport/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2Ftransport/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2Ftransport/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linkedin","download_url":"https://codeload.github.com/linkedin/transport/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2Ftransport/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270796215,"owners_count":24647319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:01:31.129Z","updated_at":"2025-08-17T01:35:13.574Z","avatar_url":"https://github.com/linkedin.png","language":"Java","funding_links":[],"categories":["Java","Libraries","大数据"],"sub_categories":[],"readme":"![logo](docs/logo.png)\n# Transport UDFs\n\n**Transport** is a framework for writing performant user-defined\nfunctions (UDFs) that are portable across a variety of engines\nincluding [Apache Spark](https://spark.apache.org/), [Apache Hive](https://hive.apache.org/), and\n[Trino](https://trino.io/). Transport UDFs are also\ncapable of directly processing data stored in serialization formats such as\nApache Avro. With Transport, developers only need to implement their UDF\nlogic once using the Transport API. Transport then takes care of\ntranslating the UDF to native UDF version targeted at various engines\nor formats. Currently, Transport is capable of generating\nengine-artifacts for Spark, Hive, and Trino, and format-artifacts for\nAvro. Further details on Transport can be found in this [LinkedIn Engineering blog post](https://engineering.linkedin.com/blog/2018/11/using-translatable-portable-UDFs).\n\n## Documentation\n\n- Project information: [README.md](/README.md)\n  - Transport release notes: [docs/release-notes.md](/docs/release-notes.md)\n  - Contributing: [#contributing](#contributing)\n- User guides\n  - Transport UDFs API: [docs/transport-udfs-api.md](/docs/transport-udfs-api.md)\n  - Authoring Transport UDFs: [docs/authoring-transport-udfs.md](/docs/authoring-transport-udfs.md)\n  - Using Transport UDFs: [docs/using-transport-udfs.md](/docs/using-transport-udfs.md)\n  - Writing Tests for Transport UDFs: [docs/writing-tests.md](/docs/writing-tests.md)\n  - FAQ: [docs/faq.md](/docs/faq.md)\n- Developer guides\n  - How the Transport Plugin works: [TODO]\n  - How the Transport Test Framework works: [TODO]\n\n\n## Example\n\nThis example shows how a portable UDF is written using the Transport APIs.\n\n```java\npublic class MapFromTwoArraysFunction extends StdUDF2\u003cStdArray, StdArray, StdMap\u003e implements TopLevelStdUDF {\n\n  private StdType _mapType;\n\n  @Override\n  public List\u003cString\u003e getInputParameterSignatures() {\n    return ImmutableList.of(\n        \"array(K)\",\n        \"array(V)\"\n    );\n  }\n\n  @Override\n  public String getOutputParameterSignature() {\n    return \"map(K,V)\";\n  }\n\n  @Override\n  public void init(StdFactory stdFactory) {\n    super.init(stdFactory);\n    _mapType = getStdFactory().createStdType(getOutputParameterSignature());\n  }\n\n  @Override\n  public StdMap eval(StdArray a1, StdArray a2) {\n    if (a1.size() != a2.size()) {\n      return null;\n    }\n    StdMap map = getStdFactory().createMap(_mapType);\n    for (int i = 0; i \u003c a1.size(); i++) {\n      map.put(a1.get(i), a2.get(i));\n    }\n    return map;\n  }\n\n  @Override\n  public String getFunctionName() {\n    return \"map_from_two_arrays\";\n  }\n\n  @Override\n  public String getFunctionDescription() {\n    return \"A function to create a map out of two arrays\";\n  }\n}\n```\n\nIn the example above, `StdMap` and `StdArray` are interfaces that\nprovide high-level map and array operations to their\nobjects. Depending on the engine where this UDF is executed, those\ninterfaces are implemented differently to deal with native data types\nused by that engine. `getStdFactory()` is a method used to create\nobjects that conform to a given data type (such as a map whose keys\nare of the type of elements in the first array and values are of the\ntype of elements in the second array). `StdUDF2` is an abstract class\nto express a UDF that takes two parameters. It is parametrized by the\nUDF input types and the UDF output type. Please consult the [Transport UDFs API](/docs/transport-udfs-api.md) for more details and examples.\n\n## How to Build\nClone the repository:\n```bash\ngit clone https://github.com/linkedin/transport.git\n```\nChange directory to `transport`:\n```bash\ncd transport\n```\n\nBuild:\n```bash\n./gradlew build\n```\n\nPlease note that this project requires Java 8 to run.\nEither set `JAVA_HOME` to the home of an appropriate version and then use `./gradlew build` as described above, or set the `org.gradle.java.home` gradle property to the Java home of an appropriate version as below:\n```bash\n./gradlew -Dorg.gradle.java.home=/path/to/java/home build\n```\nThere are known issues with Java `1.8.291`. Kindly refrain from using this subversion. To recover from build issues, do a fresh checkout and build this project using a different Java subversion.\n\n## How to Use\nThe project under the directory [`transportable-udfs-examples`](transportable-udfs-examples) is a standalone Gradle project\nthat shows how to setup a project that uses the Transport UDFs framework to write Transportable UDFs.\nYou can model your project after that standalone project. It implements a number of [example\nUDFs](transportable-udfs-examples/transportable-udfs-example-udfs) to showcase different features and aspects of the API.\n Basically, you need to check out three components:\n\n* [UDF examples code](transportable-udfs-examples/transportable-udfs-example-udfs/src/main/java/com/linkedin/transport/examples)\nto familiarize yourself with the API, and how to write new UDFs. \n\n* [Test code](transportable-udfs-examples/transportable-udfs-example-udfs/src/test/java/com/linkedin/transport/examples)\n to find out how to write UDF tests in a unified testing API, but have the framework test them on multiple platforms.\n\n* Root [`build.gradle`](transportable-udfs-examples/build.gradle) file\nto find out how to apply the `transport` plugin, which enables generating Hive, Spark, and Trino UDFs out of\nthe transportable UDFs you define once you build your project. To see that in action:\n\nChange directory to `transportable-udfs-examples`:\n\n```bash\ncd transportable-udfs-examples\n```\n\nBuild `transportable-udfs-examples`:\n\n```bash\ngradle build\n```\n\nYou will notice that the build process generates some code. This is the platform-specific versions of the UDFs.\nOnce the build succeeds, check out the output artifacts: \n\n```bash\nls transportable-udfs-example-udfs/build/libs/\n```\n\nThe results should be like:\n\n```\ntransportable-udfs-example-udfs-hive.jar\ntransportable-udfs-example-udfs-trino.jar\ntransportable-udfs-example-udfs-spark.jar\ntransportable-udfs-example-udfs.jar\n```\n\nThat is it! While only one version of the UDFs is implemented, multiple jars are produced upon building the project.\nEach of those jars uses native platform APIs and data models to implement the UDFs. So from an execution engine's perspective,\nthere is no data transformation needed for interoperability or portability. Only suitable classes are used for each engine.\n\nTo call those jars from your SQL engine (i.e., Hive, Spark, or Trino), the standard process for deploying UDF jars is followed\nfor each engine. For example, in Hive, you add the jar to the classpath using the `ADD JAR` statement,\n and register the UDF using `CREATE FUNCTION` statement.\nIn Trino, the jar is deployed to the `plugin` directory. However, a small patch is required for the Trino\nengine to recognize the jar as a plugin, since the generated Trino UDFs implement the `SqlScalarFunction` API,\nwhich is currently not part of Trino's SPI architecture. You can find the patch [here](docs/transport-udfs-trino.patch) and apply it\n before deploying your UDFs jar to the Trino engine.\n \n## Contributing\nThe project is under active development and we welcome contributions of different forms:\n\n* Contributing new general-purpose Transport UDFs (e.g., Machine Learning UDFs, Spatial UDFs, Linear Algebra UDFs, etc).\n\n* Contributing new platform support.\n\n* Contributing a framework for new types of UDFs, e.g., aggregate UDFs (UDAFs), or table functions (UDTFs).\n\nPlease take a look at the [Contribution Agreement](CONTRIBUTING.md).\n\n## Questions?\nPlease send any questions or discussion topics to [transport-udfs@googlegroups.com](mailto:transport-udfs@googlegroups.com)\n\n## License\n\n    BSD 2-CLAUSE LICENSE\n\n    Copyright 2018 LinkedIn Corporation.\n    All Rights Reserved.\n    Redistribution and use in source and binary forms, with or without\n    modification, are permitted provided that the following conditions are\n    met:\n\n    1. Redistributions of source code must retain the above copyright\n       notice, this list of conditions and the following disclaimer.\n\n    2. Redistributions in binary form must reproduce the above copyright\n       notice, this list of conditions and the following disclaimer in the\n       documentation and/or other materials provided with the\n       distribution.\n\n    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n    \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n    HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinkedin%2Ftransport","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinkedin%2Ftransport","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinkedin%2Ftransport/lists"}