https://github.com/mlverse/pysparklyr

Extension to {sparklyr} that allows you to interact with Spark & Databricks Connect
https://github.com/mlverse/pysparklyr

databricks pyspark r spark spark-connect

Last synced: 2 months ago
JSON representation

Extension to {sparklyr} that allows you to interact with Spark & Databricks Connect

• Host: GitHub
• URL: https://github.com/mlverse/pysparklyr
• Owner: mlverse
• License: other
• Created: 2023-06-22T18:02:09.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-03-26T16:58:01.000Z (10 months ago)
• Last Synced: 2024-03-27T16:47:53.524Z (10 months ago)
• Topics: databricks, pyspark, r, spark, spark-connect
• Language: R
• Homepage: https://spark.rstudio.com/deployment/databricks-spark-connect.html
• Size: 545 KB
• Stars: 12
• Watchers: 7
• Forks: 2
• Open Issues: 7
• Metadata Files:
  • Readme: README.Rmd
  • License: LICENSE

Awesome Lists containing this project

README

        
---

output: github_document

---

```{r, include = FALSE}

knitr::opts_chunk$set(

  collapse = TRUE,

  comment = "#>",

  fig.path = "man/figures/README-",

  out.width = "100%"

)

```

# pysparklyr

[![R-CMD-check](https://github.com/mlverse/pysparklyr/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/mlverse/pysparklyr/actions/workflows/R-CMD-check.yaml)

[![Spark-Connect](https://github.com/mlverse/pysparklyr/actions/workflows/spark-tests.yaml/badge.svg)](https://github.com/mlverse/pysparklyr/actions/workflows/spark-tests.yaml)

[![codecov](https://codecov.io/gh/mlverse/pysparklyr/graph/badge.svg?token=O1N9qPabpF)](https://app.codecov.io/gh/mlverse/pysparklyr)

[![CRAN status](https://www.r-pkg.org/badges/version/pysparklyr)](https://CRAN.R-project.org/package=pysparklyr)

Integrates `sparklyr` with PySpark and Databricks. The main reason of this 

package is because the new Spark and Databricks Connect connection method does

not work with standard `sparklyr` integration.

## Installing

To install the version in CRAN use:

```r

install.packages("pysparklyr")

```

To get the development version from GitHub use:

```r

remotes::install_github("mlverse/pysparklyr")

```

## Using

To learn how to use, please visit the Spark / Databricks Connect article, 

available in the official `sparklyr` website: [Spark Connect, and Databricks Connect v2

](https://spark.posit.co/deployment/databricks-connect.html)