{"id":15021499,"url":"https://github.com/sjrusso8/spark-connect-rs","last_synced_at":"2025-05-16T16:08:20.906Z","repository":{"id":195593838,"uuid":"693165063","full_name":"sjrusso8/spark-connect-rs","owner":"sjrusso8","description":"Apache Spark Connect Client for Rust","archived":false,"fork":false,"pushed_at":"2025-04-25T20:16:02.000Z","size":4035,"stargazers_count":107,"open_issues_count":9,"forks_count":17,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-25T21:23:17.510Z","etag":null,"topics":["grpc-client","spark","spark-connect","spark-sql"],"latest_commit_sha":null,"homepage":"https://docs.rs/spark-connect-rs","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sjrusso8.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-18T13:32:20.000Z","updated_at":"2025-04-25T20:16:06.000Z","dependencies_parsed_at":"2024-07-22T22:07:30.633Z","dependency_job_id":"56fabbfa-e8da-4ac6-9175-411cf5ba6d6a","html_url":"https://github.com/sjrusso8/spark-connect-rs","commit_stats":{"total_commits":72,"total_committers":4,"mean_commits":18.0,"dds":0.06944444444444442,"last_synced_commit":"70c325541bf0cbf56f9a334fe78f23ae7fe2e4dd"},"previous_names":["sjrusso8/spark-connect-rs"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sjrusso8%2Fspark-connect-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sjrusso8%2Fspark-connect-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sjrusso8%2Fspark-connect-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sjrusso8%2Fspark-connect-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sjrusso8","download_url":"https://codeload.github.com/sjrusso8/spark-connect-rs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254564127,"owners_count":22092122,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grpc-client","spark","spark-connect","spark-sql"],"created_at":"2024-09-24T19:56:38.834Z","updated_at":"2025-05-16T16:08:20.886Z","avatar_url":"https://github.com/sjrusso8.png","language":"Rust","funding_links":[],"categories":["Packages"],"sub_categories":["Language Bindings"],"readme":"# Apache Spark Connect Client for Rust\n\nThis project houses the **experimental** client for [Spark\nConnect](https://spark.apache.org/docs/latest/spark-connect-overview.html) for\n[Apache Spark](https://spark.apache.org/) written in [Rust](https://www.rust-lang.org/)\n\n\n## Current State of the Project\n\nCurrently, the Spark Connect client for Rust is **highly experimental** and **should\nnot be used in any production setting**. This is currently a \"proof of concept\" to identify the methods\nof interacting with Spark cluster from rust.\n\nThe `spark-connect-rs` aims to provide an entrypoint to [Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html), and provide *similar* DataFrame API interactions.\n\n## Project Layout\n\n```\n├── crates          \u003c- crates for the implementation of the client side spark-connect bindings\n│   └─ connect      \u003c- crate for 'spark-connect-rs'\n│      └─ protobuf  \u003c- connect protobuf for apache/spark\n├── examples        \u003c- examples of using different aspects of the crate\n├── datasets        \u003c- sample files from the main spark repo\n```\n\nFuture state would be to have additional crates that allow for easier creation of other language bindings.\n\n## Getting Started\n\nThis section explains how run Spark Connect Rust locally starting from 0.\n\n**Step 1**: Install rust via rustup: https://www.rust-lang.org/tools/install\n\n**Step 2**: Ensure you have a [cmake](https://cmake.org/download/) and [protobuf](https://grpc.io/docs/protoc-installation/) installed on your machine\n\n**Step 3**: Run the following commands to clone the repo\n\n```bash\ngit clone https://github.com/sjrusso8/spark-connect-rs.git\n\ncargo build\n```\n\n**Step 4**: Setup the Spark Driver on localhost either by downloading spark or with [docker](https://docs.docker.com/engine/install/).\n\nWith local spark:\n\n1. [Download Spark distribution](https://spark.apache.org/downloads.html) (3.5.1 recommended), unzip the package.\n\n2. Set your `SPARK_HOME` environment variable to the location where spark was extracted to,\n\n2. Start the Spark Connect server with the following command (make sure to use a package version that matches your Spark distribution):\n\n```bash\n$ $SPARK_HOME/sbin/start-connect-server.sh --packages \"org.apache.spark:spark-connect_2.12:3.5.1,io.delta:delta-spark_2.12:3.0.0\" \\\n      --conf \"spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp\" \\\n      --conf \"spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension\" \\\n      --conf \"spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog\"\n```\n\nWith docker:\n\n1. Start the Spark Connect server by leveraging the created `docker-compose.yml` in this repo. This will start a Spark Connect Server running on port 15002\n\n```bash\n$ docker compose up --build -d\n```\n\n**Step 5**: Run an example from the repo under [/examples](https://github.com/sjrusso8/spark-connect-rs/tree/main/examples/README.md)\n\n## Features\n\nThe following section outlines some of the larger functionality that are not yet working with this Spark Connect implementation.\n\n- ![done] TLS authentication \u0026 Databricks compatability via the feature flag `feature = 'tls'`\n- ![open] UDFs or any type of functionality that takes a closure (foreach, foreachBatch, etc.)\n\n### SparkSession\n\n[Spark Session](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/spark_session.html) type object and its implemented traits\n\n|SparkSession      |API       |Comment                                |\n|------------------|----------|---------------------------------------|\n|active            |![open]   |                                       |\n|addArtifact(s)    |![open]   |                                       |\n|addTag            |![done]   |                                       |\n|clearTags         |![done]   |                                       |\n|copyFromLocalToFs |![open]   |                                       |\n|createDataFrame   |![partial]|Partial. Only works for `RecordBatch`  |\n|getActiveSessions |![open]   |                                       |\n|getTags           |![done]   |                                       |\n|interruptAll      |![done]   |                                       |\n|interruptOperation|![done]   |                                       |\n|interruptTag      |![done]   |                                       |\n|newSession        |![open]   |                                       |\n|range             |![done]   |                                       |\n|removeTag         |![done]   |                                       |\n|sql               |![done]   |                                       |\n|stop              |![open]   |                                       |\n|table             |![done]   |                                       |\n|catalog           |![done]   |[Catalog](#catalog)                    |\n|client            |![done]   |unstable developer api for testing only |\n|conf              |![done]   |[Conf](#runtimeconfig)                 |\n|read              |![done]   |[DataFrameReader](#dataframereader)    |\n|readStream        |![done]   |[DataStreamReader](#datastreamreader)  |\n|streams           |![done]   |[Streams](#streamingquerymanager)      |\n|udf               |![open]   |[Udf](#udfregistration) - may not be possible   |\n|udtf              |![open]   |[Udtf](#udtfregistration) - may not be possible |\n|version           |![done]   |                                       |\n\n### SparkSessionBuilder\n|SparkSessionBuilder|API       |Comment                                |\n|-------------------|----------|---------------------------------------|\n|appName            |![done]   |                                       |\n|config             |![done]   |                                       |\n|master             |![open]   |                                       |\n|remote             |![partial]|Validate using [spark connection string](https://github.com/apache/spark/blob/master/connector/connect/docs/client-connection-string.md)|\n\n\n### RuntimeConfig\n\n|RuntimeConfig     |API       |Comment                                |\n|------------------|----------|---------------------------------------|\n|get               |![done]   |                                       |\n|isModifiable      |![done]   |                                       |\n|set               |![done]   |                                       |\n|unset             |![done]   |                                       |\n\n\n### Catalog\n\n|Catalog           |API       |Comment                                |\n|------------------|----------|---------------------------------------|\n|cacheTable        |![done]   |                                       |\n|clearCache        |![done]   |                                       |\n|createExternalTale|![done]   |                                       |\n|createTable       |![done]   |                                       |\n|currentCatalog    |![done]   |                                       |\n|currentDatabase   |![done]   |                                       |\n|databaseExists    |![done]   |                                       |\n|dropGlobalTempView|![done]   |                                       |\n|dropTempView      |![done]   |                                       |\n|functionExists    |![done]   |                                       |\n|getDatabase       |![done]   |                                       |\n|getFunction       |![done]   |                                       |\n|getTable          |![done]   |                                       |\n|isCached          |![done]   |                                       |\n|listCatalogs      |![done]   |                                       |\n|listDatabases     |![done]   |                                       |\n|listFunctions     |![done]   |                                       |\n|listTables        |![done]   |                                       |\n|recoverPartitions |![done]   |                                       |\n|refreshByPath     |![done]   |                                       |\n|refreshTable      |![done]   |                                       |\n|registerFunction  |![open]   |                                       |\n|setCurrentCatalog |![done]   |                                       |\n|setCurrentDatabase|![done]   |                                       |\n|tableExists       |![done]   |                                       |\n|uncacheTable      |![done]   |                                       |\n\n\n### DataFrameReader\n\n|DataFrameReader  |API       |Comment                                |\n|------------------|----------|---------------------------------------|\n|csv               |![done]   |                                       |\n|format            |![done]   |                                       |\n|json              |![done]   |                                       |\n|load              |![done]   |                                       |\n|option            |![done]   |                                       |\n|options           |![done]   |                                       |\n|orc               |![done]   |                                       |\n|parquet           |![done]   |                                       |\n|schema            |![done]   |                                       |\n|table             |![done]   |                                       |\n|text              |![done]   |                                       |\n\n\n### DataFrameWriter\n\nSpark Connect *should* respect the format as long as your cluster supports the specified type and has the\nrequired jars\n\n|DataFrameWriter   |API       |Comment                                |\n|------------------|----------|---------------------------------------|\n|bucketBy          |![done]   |                                       |\n|csv               |![done]   |                                       |\n|format            |![done]   |                                       |\n|insertInto        |![done]   |                                       |\n|jdbc              |![open]   |                                       |\n|json              |![done]   |                                       |\n|mode              |![done]   |                                       |\n|option            |![done]   |                                       |\n|options           |![done]   |                                       |\n|orc               |![done]   |                                       |\n|parquet           |![done]   |                                       |\n|partitionBy       |![done]   |                                       |\n|save              |![done]   |                                       |\n|saveAsTable       |![done]   |                                       |\n|sortBy            |![done]   |                                       |\n|text              |![done]   |                                       |\n\n\n### DataFrameWriterV2\n\n|DataFrameWriterV2 |API       |Comment                                |\n|------------------|----------|---------------------------------------|\n|append            |![done]   |                                       |\n|create            |![done]   |                                       |\n|createOrReplace   |![done]   |                                       |\n|option            |![done]   |                                       |\n|options           |![done]   |                                       |\n|overwrite         |![done]   |                                       |\n|overwritePartitions|![done]   |                                       |\n|partitionedBy     |![done]   |                                       |\n|replace           |![done]   |                                       |\n|tableProperty     |![done]   |                                       |\n|using             |![done]   |                                       |\n\n\n### DataStreamReader\n\n|DataStreamReader  |API       |Comment                                |\n|------------------|----------|---------------------------------------|\n|csv               |![open]   |                                       |\n|format            |![done]   |                                       |\n|json              |![open]   |                                       |\n|load              |![done]   |                                       |\n|option            |![done]   |                                       |\n|options           |![done]   |                                       |\n|orc               |![open]   |                                       |\n|parquet           |![open]   |                                       |\n|schema            |![done]   |                                       |\n|table             |![open]   |                                       |\n|text              |![open]   |                                       |\n\n\n### DataStreamWriter\n\nStart a streaming job and return a `StreamingQuery` object to handle the stream operations.\n\n|DataStreamWriter  |API       |Comment                                |\n|------------------|----------|---------------------------------------|\n|foreach           |          |                                       |\n|foreachBatch      |          |                                       |\n|format            |![done]   |                                       |\n|option            |![done]   |                                       |\n|options           |![done]   |                                       |\n|outputMode        |![done]   |Uses an Enum for `OutputMode`          |\n|partitionBy       |![done]   |                                       |\n|queryName         |![done]   |                                       |\n|start             |![done]   |                                       |\n|toTable           |![done]   |                                       |\n|trigger           |![done]   |Uses an Enum for `TriggerMode`         |\n\n\n### StreamingQuery\n\n|StreamingQuery    |API       |Comment                               |\n|------------------|----------|---------------------------------------|\n|awaitTermination  |![done]   |                                       |\n|exception         |![done]   |                                       |\n|explain           |![done]   |                                       |\n|processAllAvailable|![done]   |                                       |\n|stop              |![done]   |                                       |\n|id                |![done]   |                                       |\n|isActive          |![done]   |                                       |\n|lastProgress      |![done]   |                                       |\n|name              |![done]   |                                       |\n|recentProgress    |![done]   |                                       |\n|runId             |![done]   |                                       |\n|status            |![done]   |                                       |\n\n\n### StreamingQueryManager\n\n|StreamingQueryManager|API       |Comment                                |\n|---------------------|----------|---------------------------------------|\n|awaitAnyTermination  |![done]   |                                       |\n|get                  |![done]   |                                       |\n|resetTerminated      |![done]   |                                       |\n|active               |![done]   |                                       |\n\n\n### StreamingQueryListener\n\n|StreamingQueryListener|API       |Comment                                |\n|----------------------|----------|---------------------------------------|\n|onQueryIdle           |![open]   |                                       |\n|onQueryProgress       |![open]   |                                       |\n|onQueryStarted        |![open]   |                                       |\n|onQueryTerminated     |![open]   |                                       |\n\n\n### DataFrame\n\nSpark [DataFrame](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html) type object and its implemented traits.\n\n| DataFrame                     | API     | Comment                                                    |\n|-------------------------------|---------|------------------------------------------------------------|\n| agg                           | ![done] |                                                            |\n| alias                         | ![done] |                                                            |\n| approxQuantile                | ![done] |                                                            |\n| cache                         | ![done] |                                                            |\n| checkpoint                    | ![open] | Not part of Spark Connect                                  |\n| coalesce                      | ![done] |                                                            |\n| colRegex                      | ![done] |                                                            |\n| collect                       | ![done] |                                                            |\n| columns                       | ![done] |                                                            |\n| corr                          | ![done] |                                                            |\n| count                         | ![done] |                                                            |\n| cov                           | ![done] |                                                            |\n| createGlobalTempView          | ![done] |                                                            |\n| createOrReplaceGlobalTempView | ![done] |                                                            |\n| createOrReplaceTempView       | ![done] |                                                            |\n| createTempView                | ![done] |                                                            |\n| crossJoin                     | ![done] |                                                            |\n| crosstab                      | ![done] |                                                            |\n| cube                          | ![done] |                                                            |\n| describe                      | ![done] |                                                            |\n| distinct                      | ![done] |                                                            |\n| drop                          | ![done] |                                                            |\n| dropDuplicatesWithinWatermark | ![done] |                                                            |\n| drop_duplicates               | ![done] |                                                            |\n| dropna                        | ![done] |                                                            |\n| dtypes                        | ![done] |                                                            |\n| exceptAll                     | ![done] |                                                            |\n| explain                       | ![done] |                                                            |\n| fillna                        | ![done] |                                                            |\n| filter                        | ![done] |                                                            |\n| first                         | ![done] |                                                            |\n| foreach                       | ![open] |                                                            |\n| foreachPartition              | ![open] |                                                            |\n| freqItems                     | ![done] |                                                            |\n| groupBy                       | ![done] |                                                            |\n| head                          | ![done] |                                                            |\n| hint                          | ![done] |                                                            |\n| inputFiles                    | ![done] |                                                            |\n| intersect                     | ![done] |                                                            |\n| intersectAll                  | ![done] |                                                            |\n| isEmpty                       | ![done] |                                                            |\n| isLocal                       | ![done] |                                                            |\n| isStreaming                   | ![done] |                                                            |\n| join                          | ![done] |                                                            |\n| limit                         | ![done] |                                                            |\n| localCheckpoint               | ![open] | Not part of Spark Connect                                  |\n| mapInPandas                   | ![open] | TBD on this exact implementation                           |\n| mapInArrow                    | ![open] | TBD on this exact implementation                           |\n| melt                          | ![done] |                                                            |\n| na                            | ![done] |                                                            |\n| observe                       | ![open] |                                                            |\n| offset                        | ![done] |                                                            |\n| orderBy                       | ![done] |                                                            |\n| persist                       | ![done] |                                                            |\n| printSchema                   | ![done] |                                                            |\n| randomSplit                   | ![done] |                                                            |\n| registerTempTable             | ![done] |                                                            |\n| repartition                   | ![done] |                                                            |\n| repartitionByRange            | ![done] |                                                            |\n| replace                       | ![done] |                                                            |\n| rollup                        | ![done] |                                                            |\n| sameSemantics                 | ![done] |                                                            |\n| sample                        | ![done] |                                                            |\n| sampleBy                      | ![done] |                                                            |\n| schema                        | ![done] |                                                            |\n| select                        | ![done] |                                                            |\n| selectExpr                    | ![done] |                                                            |\n| semanticHash                  | ![done] |                                                            |\n| show                          | ![done] |                                                            |\n| sort                          | ![done] |                                                            |\n| sortWithinPartitions          | ![done] |                                                            |\n| sparkSession                  | ![done] |                                                            |\n| stat                          | ![done] |                                                            |\n| storageLevel                  | ![done] |                                                            |\n| subtract                      | ![done] |                                                            |\n| summary                       | ![done] |                                                            |\n| tail                          | ![done] |                                                            |\n| take                          | ![done] |                                                            |\n| to                            | ![done] |                                                            |\n| toDF                          | ![done] |                                                            |\n| toJSON                        | ![partial] | Does not return an `RDD` but a long JSON formatted `String` |\n| toLocalIterator               | ![open] |                                                            |\n| ~~toPandas~~ to_polars \u0026 toPolars  | ![partial] | Convert to a `polars::frame::DataFrame`            |\n| **new** to_datafusion \u0026 toDataFusion | ![done] | Convert to a `datafusion::dataframe::DataFrame`     |\n| transform                     | ![done] |                                                            |\n| union                         | ![done] |                                                            |\n| unionAll                      | ![done] |                                                            |\n| unionByName                   | ![done] |                                                            |\n| unpersist                     | ![done] |                                                            |\n| unpivot                       | ![done] |                                                            |\n| where                         | ![done] | use `filter` instead, `where` is a keyword for rust        |\n| withColumn                    | ![done] |                                                            |\n| withColumns                   | ![done] |                                                            |\n| withColumnRenamed             | ![done] |                                                            |\n| withColumnsRenamed            | ![done] |                                                            |\n| withMetadata                  | ![done] |                                                            |\n| withWatermark                 | ![done] |                                                            |\n| write                         | ![done] |                                                            |\n| writeStream                   | ![done] |                                                            |\n| writeTo                       | ![done] |                                                            |\n\n\n### Column\n\nSpark [Column](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/column.html) type object and its implemented traits\n\n\n| Column           | API     | Comment                                                                      |\n|------------------|---------|------------------------------------------------------------------------------|\n| alias            | ![done] |                                                                              |\n| asc              | ![done] |                                                                              |\n| asc_nulls_first  | ![done] |                                                                              |\n| asc_nulls_last   | ![done] |                                                                              |\n| astype           | ![open] |                                                                              |\n| between          | ![open] |                                                                              |\n| cast             | ![done] |                                                                              |\n| contains         | ![done] |                                                                              |\n| desc             | ![done] |                                                                              |\n| desc_nulls_first | ![done] |                                                                              |\n| desc_nulls_last  | ![done] |                                                                              |\n| dropFields       | ![done] |                                                                              |\n| endswith         | ![done] |                                                                              |\n| eqNullSafe       | ![open] |                                                                              |\n| getField         | ![open] | This is depreciated but will need to be implemented                          |\n| getItem          | ![open] | This is depreciated but will need to be implemented                          |\n| ilike            | ![done] |                                                                              |\n| isNotNull        | ![done] |                                                                              |\n| isNull           | ![done] |                                                                              |\n| isin             | ![done] |                                                                              |\n| like             | ![done] |                                                                              |\n| name             | ![done] |                                                                              |\n| otherwise        | ![open] |                                                                              |\n| over             | ![done] | Refer to **Window** for creating window specifications                       |\n| rlike            | ![done] |                                                                              |\n| startswith       | ![done] |                                                                              |\n| substr           | ![done] |                                                                              |\n| when             | ![open] |                                                                              |\n| withField        | ![done] |                                                                              |\n| eq `==`          | ![done] | Rust does not like when you try to overload `==` and return something other than a `bool`. Currently implemented column equality like `col('name').eq(col('id'))`. Not the best, but it works for now                                                                           |\n| addition `+`     | ![done] |                                                                              |\n| subtration `-`   | ![done] |                                                                              |\n| multiplication `*` | ![done] |                                                                            |\n| division `/`     | ![done] |                                                                              |\n| OR  `\\|`         | ![done] |                                                                              |\n| AND `\u0026`          | ![done] |                                                                              |\n| XOR `^`          | ![done] |                                                                              |\n| Negate `~`       | ![done] |                                                                              |\n\n\n### Data Types\n\nData types are used for creating schemas and for casting columns to specific types\n\n| Column                | API     | Comment           |\n|-----------------------|---------|-------------------|\n| ArrayType             | ![done] |                   |\n| BinaryType            | ![done] |                   |\n| BooleanType           | ![done] |                   |\n| ByteType              | ![done] |                   |\n| DateType              | ![done] |                   |\n| DecimalType           | ![done] |                   |\n| DoubleType            | ![done] |                   |\n| FloatType             | ![done] |                   |\n| IntegerType           | ![done] |                   |\n| LongType              | ![done] |                   |\n| MapType               | ![done] |                   |\n| NullType              | ![done] |                   |\n| ShortType             | ![done] |                   |\n| StringType            | ![done] |                   |\n| CharType              | ![done] |                   |\n| VarcharType           | ![done] |                   |\n| StructField           | ![done] |                   |\n| StructType            | ![done] |                   |\n| TimestampType         | ![done] |                   |\n| TimestampNTZType      | ![done] |                   |\n| DayTimeIntervalType   | ![done] |                   |\n| YearMonthIntervalType | ![done] |                   |\n\n\n### Literal Types\n\nCreate Spark literal types from these rust types. E.g. `lit(1_i64)` would be a `LongType()` in the schema.\n\nAn array can be made like `lit([1_i16,2_i16,3_i16])` would result in an `ArrayType(Short)` since all the values of the slice can be translated into literal type.\n\n| Spark Literal Type | Rust Type           | Status  |\n|--------------------|---------------------|---------|\n| Null               |                     | ![open] |\n| Binary             | `\u0026[u8]`             | ![done] |\n| Boolean            | `bool`              | ![done] |\n| Byte               |                     | ![open] |\n| Short              | `i16`               | ![done] |\n| Integer            | `i32`               | ![done] |\n| Long               | `i64`               | ![done] |\n| Float              | `f32`               | ![done] |\n| Double             | `f64`               | ![done] |\n| Decimal            |                     | ![open] |\n| String             | `\u0026str` / `String`   | ![done] |\n| Date               | `chrono::NaiveDate` | ![done] |\n| Timestamp          | `chrono::DateTime\u003cTz\u003e`  | ![done] |\n| TimestampNtz       | `chrono::NaiveDateTime` | ![done] |\n| CalendarInterval   |                     | ![open] |\n| YearMonthInterval  |                     | ![open] |\n| DayTimeInterval    |                     | ![open] |\n| Array              | `slice` / `Vec`     | ![done] |\n| Map                | Create with the function `create_map` | ![done] |\n| Struct             | Create with the function `struct_col` or `named_struct` | ![done] |\n\n\n### Window \u0026 WindowSpec\n\nFor ease of use it's recommended to use `Window` to create the `WindowSpec`.\n\n| Window                  | API     | Comment |\n|-------------------------|---------|---------|\n| currentRow              | ![done] |         |\n| orderBy                 | ![done] |         |\n| partitionBy             | ![done] |         |\n| rangeBetween            | ![done] |         |\n| rowsBetween             | ![done] |         |\n| unboundedFollowing      | ![done] |         |\n| unboundedPreceding      | ![done] |         |\n| WindowSpec.orderBy      | ![done] |         |\n| WindowSpec.partitionBy  | ![done] |         |\n| WindowSpec.rangeBetween | ![done] |         |\n| WindowSpec.rowsBetween  | ![done] |         |\n\n\n### Functions\n\nOnly a few of the functions are covered by unit tests. Functions involving closures or lambdas are not feasible.\n\n| Functions                   | API     | Comments |\n|-----------------------------|---------|----------|\n| abs                         | ![done] |          |\n| acos                        | ![done] |          |\n| acosh                       | ![done] |          |\n| add_months                  | ![done] |          |\n| aes_decrypt                 | ![done] |          |\n| aes_encrypt                 | ![done] |          |\n| aggregate                   | ![open] |          |\n| any_value                   | ![done] |          |\n| approx_count_distinct       | ![done] |          |\n| approx_percentile           | ![open] |          |\n| array                       | ![done] |          |\n| array_agg                   | ![done] |          |\n| array_append                | ![done] |          |\n| array_compact               | ![done] |          |\n| array_contains              | ![done] |          |\n| array_distinct              | ![done] |          |\n| array_except                | ![done] |          |\n| array_insert                | ![done] |          |\n| array_intersect             | ![done] |          |\n| array_join                  | ![done] |          |\n| array_max                   | ![done] |          |\n| array_min                   | ![done] |          |\n| array_position              | ![done] |          |\n| array_prepend               | ![done] |          |\n| array_remove                | ![done] |          |\n| array_repeat                | ![done] |          |\n| array_size                  | ![done] |          |\n| array_sort                  | ![open] |          |\n| array_union                 | ![done] |          |\n| arrays_overlap              | ![done] |          |\n| arrays_zip                  | ![done] |          |\n| asc                         | ![done] |          |\n| asc_nulls_first             | ![done] |          |\n| asc_nulls_last              | ![done] |          |\n| ascii                       | ![done] |          |\n| asin                        | ![done] |          |\n| asinh                       | ![done] |          |\n| assert_true                 | ![done] |          |\n| atan                        | ![done] |          |\n| atan2                       | ![done] |          |\n| atanh                       | ![done] |          |\n| avg                         | ![done] |          |\n| base64                      | ![done] |          |\n| bin                         | ![done] |          |\n| bit_and                     | ![done] |          |\n| bit_count                   | ![done] |          |\n| bit_get                     | ![done] |          |\n| bit_length                  | ![done] |          |\n| bit_or                      | ![done] |          |\n| bit_xor                     | ![done] |          |\n| bitmap_bit_position         | ![done] |          |\n| bitmap_bucket_number        | ![done] |          |\n| bitmap_construct_agg        | ![done] |          |\n| bitmap_count                | ![done] |          |\n| bitmap_or_agg               | ![done] |          |\n| bitwise_not                 | ![done] |          |\n| bool_and                    | ![done] |          |\n| bool_or                     | ![done] |          |\n| broadcast                   | ![done] |          |\n| bround                      | ![done] |          |\n| btrim                       | ![done] |          |\n| bucket                      | ![done] |          |\n| call_function               | ![open] |          |\n| call_udf                    | ![open] |          |\n| cardinality                 | ![done] |          |\n| cbrt                        | ![done] |          |\n| ceil                        | ![done] |          |\n| ceiling                     | ![done] |          |\n| char                        | ![done] |          |\n| char_length                 | ![done] |          |\n| character_length            | ![done] |          |\n| coalesce                    | ![done] |          |\n| col                         | ![done] |          |\n| collect_list                | ![done] |          |\n| collect_set                 | ![done] |          |\n| column                      | ![done] |          |\n| concat                      | ![done] |          |\n| concat_ws                   | ![done] |          |\n| contains                    | ![done] |          |\n| conv                        | ![done] |          |\n| convert_timezone            | ![done] |          |\n| corr                        | ![done] |          |\n| cos                         | ![done] |          |\n| cosh                        | ![done] |          |\n| cot                         | ![done] |          |\n| count                       | ![done] |          |\n| count_distinct              | ![done] |          |\n| count_if                    | ![done] |          |\n| count_min_sketch            | ![done] |          |\n| covar_pop                   | ![done] |          |\n| covar_samp                  | ![done] |          |\n| crc32                       | ![done] |          |\n| create_map                  | ![done] |          |\n| csc                         | ![done] |          |\n| cume_dist                   | ![done] |          |\n| curdate                     | ![done] |          |\n| current_catalog             | ![done] |          |\n| current_database            | ![done] |          |\n| current_date                | ![done] |          |\n| current_schema              | ![done] |          |\n| current_timestamp           | ![done] |          |\n| current_timezone            | ![done] |          |\n| current_user                | ![done] |          |\n| date_add                    | ![done] |          |\n| date_diff                   | ![done] |          |\n| date_format                 | ![done] |          |\n| date_from_unix_date         | ![done] |          |\n| date_part                   | ![done] |          |\n| date_sub                    | ![done] |          |\n| date_trunc                  | ![done] |          |\n| dateadd                     | ![done] |          |\n| datediff                    | ![done] |          |\n| datepart                    | ![open] |          |\n| day                         | ![done] |          |\n| dayofmonth                  | ![done] |          |\n| dayofweek                   | ![done] |          |\n| dayofyear                   | ![done] |          |\n| days                        | ![done] |          |\n| decode                      | ![done] |          |\n| degrees                     | ![done] |          |\n| dense_rank                  | ![done] |          |\n| desc                        | ![done] |          |\n| desc_nulls_first            | ![done] |          |\n| desc_nulls_last             | ![done] |          |\n| e                           | ![done] |          |\n| element_at                  | ![done] |          |\n| elt                         | ![done] |          |\n| encode                      | ![done] |          |\n| endswith                    | ![done] |          |\n| equal_null                  | ![done] |          |\n| every                       | ![done] |          |\n| exists                      | ![open] |          |\n| exp                         | ![done] |          |\n| explode                     | ![done] |          |\n| explode_outer               | ![done] |          |\n| expm1                       | ![done] |          |\n| expr                        | ![done] |          |\n| extract                     | ![done] |          |\n| factorial                   | ![done] |          |\n| filter                      | ![open] |          |\n| find_in_set                 | ![done] |          |\n| first                       | ![done] |          |\n| first_value                 | ![done] |          |\n| flatten                     | ![done] |          |\n| floor                       | ![done] |          |\n| forall                      | ![open] |          |\n| format_number               | ![done] |          |\n| format_string               | ![done] |          |\n| from_csv                    | ![done] |          |\n| from_json                   | ![done] |          |\n| from_unixtime               | ![done] |          |\n| from_utc_timestamp          | ![done] |          |\n| get                         | ![done] |          |\n| get_json_object             | ![done] |          |\n| getbit                      | ![done] |          |\n| greatest                    | ![done] |          |\n| grouping                    | ![done] |          |\n| grouping_id                 | ![done] |          |\n| hash                        | ![done] |          |\n| hex                         | ![done] |          |\n| histogram_numeric           | ![done] |          |\n| hll_sketch_agg              | ![done] |          |\n| hll_sketch_estimate         | ![done] |          |\n| hll_union                   | ![done] |          |\n| hll_union_agg               | ![done] |          |\n| hour                        | ![done] |          |\n| hours                       | ![done] |          |\n| hypot                       | ![done] |          |\n| ifnull                      | ![done] |          |\n| ilike                       | ![done] |          |\n| initcap                     | ![done] |          |\n| inline                      | ![done] |          |\n| inline_outer                | ![done] |          |\n| input_file_block_length     | ![done] |          |\n| input_file_block_start      | ![done] |          |\n| input_file_name             | ![done] |          |\n| instr                       | ![done] |          |\n| isnan                       | ![done] |          |\n| isnotnull                   | ![done] |          |\n| isnull                      | ![done] |          |\n| java_method                 | ![done] |          |\n| json_array_length           | ![done] |          |\n| json_object_keys            | ![done] |          |\n| json_tuple                  | ![done] |          |\n| kurtosis                    | ![done] |          |\n| lag                         | ![done] |          |\n| last                        | ![done] |          |\n| last_day                    | ![done] |          |\n| last_value                  | ![done] |          |\n| lcase                       | ![done] |          |\n| lead                        | ![done] |          |\n| least                       | ![done] |          |\n| left                        | ![done] |          |\n| length                      | ![done] |          |\n| levenshtein                 | ![done] |          |\n| like                        | ![done] |          |\n| lit                         | ![done] |          |\n| ln                          | ![done] |          |\n| localtimestamp              | ![done] |          |\n| locate                      | ![done] |          |\n| log                         | ![done] |          |\n| log10                       | ![done] |          |\n| log1p                       | ![done] |          |\n| log2                        | ![done] |          |\n| lower                       | ![done] |          |\n| lpad                        | ![done] |          |\n| ltrim                       | ![done] |          |\n| make_date                   | ![done] |          |\n| make_dt_interval            | ![done] |          |\n| make_interval               | ![done] |          |\n| make_timestamp              | ![done] |          |\n| make_timestamp_ltz          | ![done] |          |\n| make_timestamp_ntz          | ![done] |          |\n| make_ym_interval            | ![done] |          |\n| map_concat                  | ![done] |          |\n| map_contains_key            | ![done] |          |\n| map_entries                 | ![done] |          |\n| map_filter                  | ![open] |          |\n| map_from_arrays             | ![done] |          |\n| map_from_entries            | ![done] |          |\n| map_keys                    | ![done] |          |\n| map_values                  | ![done] |          |\n| map_zip_with                | ![open] |          |\n| mask                        | ![open] |          |\n| max                         | ![done] |          |\n| max_by                      | ![done] |          |\n| md5                         | ![done] |          |\n| mean                        | ![done] |          |\n| median                      | ![done] |          |\n| min                         | ![done] |          |\n| min_by                      | ![done] |          |\n| minute                      | ![done] |          |\n| mode                        | ![done] |          |\n| monotonically_increasing_id | ![done] |          |\n| month                       | ![done] |          |\n| months                      | ![done] |          |\n| months_between              | ![done] |          |\n| named_struct                | ![done] |          |\n| nanvl                       | ![done] |          |\n| negate                      | ![done] |          |\n| negative                    | ![done] |          |\n| next_day                    | ![done] |          |\n| now                         | ![done] |          |\n| nth_value                   | ![done] |          |\n| ntile                       | ![done] |          |\n| nullif                      | ![done] |          |\n| nvl                         | ![done] |          |\n| nvl2                        | ![done] |          |\n| octet_length                | ![done] |          |\n| overlay                     | ![done] |          |\n| pandas_udf                  | ![open] |          |\n| parse_url                   | ![done] |          |\n| percent_rank                | ![done] |          |\n| percentile                  | ![done] |          |\n| percentile_approx           | ![done] |          |\n| pi                          | ![done] |          |\n| pmod                        | ![done] |          |\n| posexplode                  | ![done] |          |\n| posexplode_outer            | ![done] |          |\n| position                    | ![done] |          |\n| positive                    | ![done] |          |\n| pow                         | ![done] |          |\n| power                       | ![done] |          |\n| printf                      | ![done] |          |\n| product                     | ![done] |          |\n| quarter                     | ![done] |          |\n| radians                     | ![done] |          |\n| raise_error                 | ![done] |          |\n| rand                        | ![done] |          |\n| randn                       | ![done] |          |\n| rank                        | ![done] |          |\n| reduce                      | ![open] |          |\n| reflect                     | ![done] |          |\n| regexp                      | ![done] |          |\n| regexp_count                | ![done] |          |\n| regexp_extract              | ![done] |          |\n| regexp_extract_all          | ![done] |          |\n| regexp_instr                | ![done] |          |\n| regexp_like                 | ![done] |          |\n| regexp_replace              | ![done] |          |\n| regexp_substr               | ![done] |          |\n| regr_avgx                   | ![done] |          |\n| regr_avgy                   | ![done] |          |\n| regr_count                  | ![done] |          |\n| regr_intercept              | ![done] |          |\n| regr_r2                     | ![done] |          |\n| regr_slope                  | ![done] |          |\n| regr_sxx                    | ![done] |          |\n| regr_sxy                    | ![done] |          |\n| regr_syy                    | ![done] |          |\n| repeat                      | ![done] |          |\n| replace                     | ![done] |          |\n| reverse                     | ![done] |          |\n| right                       | ![done] |          |\n| rint                        | ![done] |          |\n| rlike                       | ![done] |          |\n| round                       | ![done] |          |\n| row_number                  | ![done] |          |\n| rpad                        | ![done] |          |\n| rtrim                       | ![done] |          |\n| schema_of_csv               | ![done] |          |\n| schema_of_json              | ![done] |          |\n| sec                         | ![done] |          |\n| second                      | ![done] |          |\n| sentences                   | ![done] |          |\n| sequence                    | ![done] |          |\n| session_window              | ![done] |          |\n| sha                         | ![done] |          |\n| sha1                        | ![done] |          |\n| sha2                        | ![done] |          |\n| shiftleft                   | ![done] |          |\n| shiftright                  | ![done] |          |\n| shiftrightunsigned          | ![done] |          |\n| shuffle                     | ![done] |          |\n| sign                        | ![done] |          |\n| signum                      | ![done] |          |\n| sin                         | ![done] |          |\n| sinh                        | ![done] |          |\n| size                        | ![done] |          |\n| skewness                    | ![done] |          |\n| slice                       | ![done] |          |\n| some                        | ![done] |          |\n| sort_array                  | ![done] |          |\n| soundex                     | ![done] |          |\n| spark_partition_id          | ![done] |          |\n| split                       | ![done] |          |\n| split_part                  | ![done] |          |\n| sqrt                        | ![done] |          |\n| stack                       | ![done] |          |\n| startswith                  | ![done] |          |\n| std                         | ![done] |          |\n| stddev                      | ![done] |          |\n| stddev_pop                  | ![done] |          |\n| stddev_samp                 | ![done] |          |\n| str_to_map                  | ![done] |          |\n| struct                      | ![open] |          |\n| substr                      | ![done] |          |\n| substring                   | ![done] |          |\n| substring_index             | ![done] |          |\n| sum                         | ![done] |          |\n| sum_distinct                | ![done] |          |\n| tan                         | ![done] |          |\n| tanh                        | ![done] |          |\n| timestamp_micros            | ![done] |          |\n| timestamp_millis            | ![done] |          |\n| timestamp_seconds           | ![done] |          |\n| to_binary                   | ![done] |          |\n| to_char                     | ![done] |          |\n| to_csv                      | ![done] |          |\n| to_date                     | ![done] |          |\n| to_json                     | ![done] |          |\n| to_number                   | ![done] |          |\n| to_timestamp                | ![done] |          |\n| to_timestamp_ltz            | ![done] |          |\n| to_timestamp_ntz            | ![done] |          |\n| to_unix_timestamp           | ![done] |          |\n| to_utc_timestamp            | ![done] |          |\n| to_varchar                  | ![done] |          |\n| to_degrees                  | ![done] |          |\n| to_radians                  | ![done] |          |\n| transform                   | ![open] |          |\n| transform_keys              | ![open] |          |\n| transform_values            | ![open] |          |\n| translate                   | ![done] |          |\n| trim                        | ![done] |          |\n| trunc                       | ![done] |          |\n| try_add                     | ![done] |          |\n| try_aes_decrypt             | ![done] |          |\n| try_avg                     | ![done] |          |\n| try_divide                  | ![done] |          |\n| try_element_at              | ![done] |          |\n| try_multiply                | ![done] |          |\n| try_subtract                | ![done] |          |\n| try_sum                     | ![done] |          |\n| try_to_binary               | ![done] |          |\n| try_to_number               | ![done] |          |\n| try_to_timestamp            | ![done] |          |\n| typeof                      | ![open] |          |\n| ucase                       | ![done] |          |\n| udf                         | ![open] |          |\n| udtf                        | ![open] |          |\n| unbase64                    | ![done] |          |\n| unhex                       | ![done] |          |\n| unix_date                   | ![done] |          |\n| unix_micros                 | ![open] |          |\n| unix_millis                 | ![done] |          |\n| unix_seconds                | ![done] |          |\n| unix_timestamp              | ![done] |          |\n| unwrap_udt                  | ![open] |          |\n| upper                       | ![done] |          |\n| url_decode                  | ![done] |          |\n| url_encode                  | ![done] |          |\n| user                        | ![done] |          |\n| var_pop                     | ![done] |          |\n| var_samp                    | ![done] |          |\n| variance                    | ![done] |          |\n| version                     | ![done] |          |\n| weekday                     | ![done] |          |\n| weekofyear                  | ![done] |          |\n| when                        | ![open] |          |\n| width_bucket                | ![done] |          |\n| window                      | ![done] |          |\n| window_time                 | ![done] |          |\n| xpath                       | ![done] |          |\n| xpath_boolean               | ![done] |          |\n| xpath_double                | ![done] |          |\n| xpath_float                 | ![done] |          |\n| xpath_int                   | ![done] |          |\n| xpath_long                  | ![done] |          |\n| xpath_number                | ![done] |          |\n| xpath_short                 | ![done] |          |\n| xpath_string                | ![done] |          |\n| xxhash64                    | ![done] |          |\n| year                        | ![done] |          |\n| years                       | ![done] |          |\n| zip_with                    | ![open] |          |\n\n\n### UdfRegistration (may not be possible)\n\n|UDFRegistration   |API       |Comment                                |\n|------------------|----------|---------------------------------------|\n|register          |![open]   |                                       |\n|registerJavaFunction|![open]   |                                       |\n|registerJavaUDAF  |![open]   |                                       |\n\n### UdtfRegistration (may not be possible)\n\n|UDTFRegistration  |API       |Comment                                |\n|------------------|----------|---------------------------------------|\n|register          |![open]   |                                       |\n\n\n[open]: https://cdn.jsdelivr.net/gh/Readme-Workflows/Readme-Icons@main/icons/octicons/IssueNeutral.svg\n[done]: https://cdn.jsdelivr.net/gh/Readme-Workflows/Readme-Icons@main/icons/octicons/ApprovedChanges.svg\n[partial]: https://cdn.jsdelivr.net/gh/Readme-Workflows/Readme-Icons@main/icons/octicons/IssueDrafted.svg\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsjrusso8%2Fspark-connect-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsjrusso8%2Fspark-connect-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsjrusso8%2Fspark-connect-rs/lists"}