{"id":13794925,"url":"https://github.com/sparklyr/sparklyr","last_synced_at":"2025-05-14T14:10:06.047Z","repository":{"id":37547669,"uuid":"59305491","full_name":"sparklyr/sparklyr","owner":"sparklyr","description":"R interface for Apache Spark","archived":false,"fork":false,"pushed_at":"2025-03-18T14:29:34.000Z","size":101763,"stargazers_count":963,"open_issues_count":348,"forks_count":308,"subscribers_count":70,"default_branch":"main","last_synced_at":"2025-05-07T15:02:09.893Z","etag":null,"topics":["apache-spark","distributed","dplyr","ide","livy","machine-learning","r","remote-clusters","rstats","spark","sparklyr"],"latest_commit_sha":null,"homepage":"https://spark.rstudio.com/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sparklyr.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-05-20T15:28:53.000Z","updated_at":"2025-05-07T08:31:38.000Z","dependencies_parsed_at":"2024-01-03T00:22:01.409Z","dependency_job_id":"0ed93992-943c-4ea8-b821-ec9437b14391","html_url":"https://github.com/sparklyr/sparklyr","commit_stats":{"total_commits":7075,"total_committers":108,"mean_commits":65.50925925925925,"dds":0.5358303886925795,"last_synced_commit":"f3bae8d0d13f88c8dc4f84e144f42e9278216b63"},"previous_names":["rstudio/sparklyr"],"tags_count":54,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sparklyr%2Fsparklyr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sparklyr%2Fsparklyr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sparklyr%2Fsparklyr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sparklyr%2Fsparklyr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sparklyr","download_url":"https://codeload.github.com/sparklyr/sparklyr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254052668,"owners_count":22006716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","distributed","dplyr","ide","livy","machine-learning","r","remote-clusters","rstats","spark","sparklyr"],"created_at":"2024-08-03T23:00:50.185Z","updated_at":"2025-05-14T14:10:01.033Z","avatar_url":"https://github.com/sparklyr.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"---\ntitle: \"sparklyr: R interface for Apache Spark\"\noutput:\n  github_document:\n    fig_width: 9\n    fig_height: 5\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  eval = TRUE, \n  warning = FALSE,\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"tools/readme/\", \n  dev = \"png\",\n  out.width = \"100%\"\n)\n\nSys.setenv(\"SPARK_VERSION\" = \"3.5\")\n\noptions(width = 60)\n\nlibrary(sparklyr)\nlibrary(dplyr)\nlibrary(ggplot2)\n```\n\n\u003c!-- badges: start --\u003e\n[![R-CMD-check](https://github.com/sparklyr/sparklyr/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/sparklyr/sparklyr/actions/workflows/R-CMD-check.yaml)\n[![Spark-Tests](https://github.com/sparklyr/sparklyr/actions/workflows/spark-tests.yaml/badge.svg)](https://github.com/sparklyr/sparklyr/actions/workflows/spark-tests.yaml)\n[![CRAN status](https://www.r-pkg.org/badges/version/sparklyr)](https://CRAN.R-project.org/package=sparklyr)\n[![Codecov test coverage](https://codecov.io/gh/sparklyr/sparklyr/branch/main/graph/badge.svg)](https://app.codecov.io/gh/sparklyr/sparklyr?branch=main)\n\u003c!-- badges: end --\u003e\n\n\u003cimg src=\"tools/readme/sparklyr-diagram.png\" width=\"320\" align=\"right\" style=\"margin-left: 20px; margin-right: 20px\"/\u003e\n\n\n- Install and connect to [Spark](https://spark.apache.org/) using YARN, Mesos,\nLivy or Kubernetes.\n- Use [dplyr](#using-dplyr) to filter and aggregate Spark datasets and \n[streams](https://spark.posit.co/guides/streaming/) then bring them into R \nfor analysis and visualization.\n- Create interoperable machine learning \n[pipelines](https://spark.posit.co/guides/pipelines.html) \n- Create [extensions](#extensions) that call the full Spark API or run\n[distributed R](#distributed-r) code to support new functionality.\n\n## Table of Contents\n\n```{r, eval = FALSE, echo = FALSE}\ntoc \u003c- function() {\n  re \u003c- readLines(\"README.Rmd\")\n  has_title \u003c- as.logical(lapply(re, function(x) substr(x, 1, 2) == \"##\"))\n  only_titles \u003c- re[has_title]\n  titles \u003c- trimws(gsub(\"#\", \"\", only_titles))\n  links \u003c- trimws(gsub(\"`\", \"\", titles))\n  links \u003c- tolower(links)\n  links \u003c- trimws(gsub(\" \", \"-\", links))\n  links \u003c- trimws(gsub(\",\", \"\", links))\n  toc_list \u003c- lapply(\n    seq_along(titles),\n    function(x) {\n      pad \u003c- ifelse(substr(only_titles[x], 1, 3) == \"###\", \"    - \", \"  - \")\n      paste0(pad, \"[\", titles[x], \"](#\",links[x], \")\")\n    }\n  )\n  toc_full \u003c- paste(toc_list[2:length(toc_list)], collapse = \"\\n\") \n  cat(toc_full)\n}\ntoc()\n```\n  - [Installation](#installation)\n  - [Connecting to Spark](#connecting-to-spark)\n  - [Using dplyr](#using-dplyr)\n    - [Window Functions](#window-functions)\n  - [Using SQL](#using-sql)\n  - [Machine Learning](#machine-learning)\n  - [Reading and Writing Data](#reading-and-writing-data)\n  - [Distributed R](#distributed-r)\n  - [Extensions](#extensions)\n  - [Table Utilities](#table-utilities)\n  - [Connection Utilities](#connection-utilities)\n  - [RStudio IDE](#rstudio-ide)\n  - [Using H2O](#using-h2o)\n  - [Connecting through Livy](#connecting-through-livy)\n  - [Connecting through Databricks Connect](#connecting-through-databricks-connect-v2)\n  \n## Installation\n\nYou can install the **sparklyr** package from [CRAN](https://CRAN.r-project.org) as follows:\n\n```{r, eval=FALSE}\ninstall.packages(\"sparklyr\")\n```\n\nYou should also install a local version of Spark for development purposes:\n\n```{r, eval=FALSE}\nlibrary(sparklyr)\nspark_install()\n```\n\nTo upgrade to the latest version of sparklyr, run the following command and restart your r session:\n\n```{r, eval=FALSE}\ninstall.packages(\"devtools\")\ndevtools::install_github(\"sparklyr/sparklyr\")\n```\n\n## Connecting to Spark\n\nYou can connect to both local instances of Spark as well as remote Spark clusters.\nHere we'll connect to a local instance of Spark via the \n[spark_connect](https://spark.posit.co/packages/sparklyr/latest/reference/spark-connections.html) function:\n\n```{r sparklyr-connect, message=FALSE}\nlibrary(sparklyr)\nsc \u003c- spark_connect(master = \"local\")\n```\n\nThe returned Spark connection (`sc`) provides a remote dplyr data source to the Spark cluster.\n\nFor more information on connecting to remote Spark clusters see the \n[Deployment](https://spark.posit.co/deployment.html) section of the sparklyr\nwebsite.\n\n## Using dplyr\n\nWe can now use all of the available dplyr verbs against the tables within the cluster.\n\nWe'll start by copying some datasets from R into the Spark cluster (note that you\nmay need to install the nycflights13 and Lahman packages in order to execute this code):\n\n```{r, eval=FALSE}\ninstall.packages(c(\"nycflights13\", \"Lahman\"))\n```\n\n```{r dplyr-copy, message=FALSE}\nlibrary(dplyr)\niris_tbl \u003c- copy_to(sc, iris, overwrite = TRUE)\nflights_tbl \u003c- copy_to(sc, nycflights13::flights, \"flights\", overwrite = TRUE)\nbatting_tbl \u003c- copy_to(sc, Lahman::Batting, \"batting\", overwrite = TRUE)\nsrc_tbls(sc)\n```\n\nTo start with here's a simple filtering example:\n\n```{r dplyr-filter}\n# filter by departure delay and print the first few records\nflights_tbl %\u003e% filter(dep_delay == 2)\n```\n\n[Introduction to dplyr](https://spark.posit.co/guides/dplyr.html) provides\nadditional `dplyr` examples you can try. For example, consider the last example\nfrom the tutorial which plots data on flight delays:\n\n```{r dplyr-ggplot2}\ndelay \u003c- flights_tbl %\u003e%\n  group_by(tailnum) %\u003e%\n  summarise(count = n(), dist = mean(distance), delay = mean(arr_delay)) %\u003e%\n  filter(count \u003e 20, dist \u003c 2000, !is.na(delay)) %\u003e%\n  collect()\n\n# plot delays\nlibrary(ggplot2)\nggplot(delay, aes(dist, delay)) +\n  geom_point(aes(size = count), alpha = 1/2) +\n  geom_smooth() +\n  scale_size_area(max_size = 2)\n```\n\n\n### Window Functions\n\ndplyr [window functions](https://spark.posit.co/guides/dplyr.html#grouping)\nare also supported, for example:\n\n```{r dplyr-window}\nbatting_tbl %\u003e%\n  select(playerID, yearID, teamID, G, AB:H) %\u003e%\n  arrange(playerID, yearID, teamID) %\u003e%\n  group_by(playerID) %\u003e%\n  filter(min_rank(desc(H)) \u003c= 2 \u0026 H \u003e 0)\n```\n\nFor additional documentation on using dplyr with Spark see the \n[dplyr](https://spark.posit.co/dplyr.html) section of the sparklyr website.\n\n## Using SQL\n\nIt's also possible to execute SQL queries directly against tables within a Spark\ncluster. The `spark_connection` object implements a [DBI](https://github.com/r-dbi/DBI)\ninterface for Spark, so you can use `dbGetQuery()` to execute SQL and return the\nresult as an R data frame:\n\n```{r sql-dbi}\nlibrary(DBI)\niris_preview \u003c- dbGetQuery(sc, \"SELECT * FROM iris LIMIT 10\")\niris_preview\n```\n\n## Machine Learning\n\nYou can orchestrate machine learning algorithms in a Spark cluster via the\n[machine learning](https://spark.apache.org/docs/latest/mllib-guide.html)\nfunctions within **sparklyr**. These functions connect to a set of high-level\nAPIs built on top of DataFrames that help you create and tune machine learning\nworkflows.\n\nHere's an example where we use [ml_linear_regression](https://spark.posit.co/packages/sparklyr/latest/reference/ml_linear_regression/)\nto fit a linear regression model. We'll use the built-in `mtcars` dataset, and\nsee if we can predict a car's fuel consumption (`mpg`) based on its weight (`wt`),\nand the number of cylinders the engine contains (`cyl`). We'll assume in each\ncase that the relationship between `mpg` and each of our features is linear.\n\n```{r}\n# copy mtcars into spark\nmtcars_tbl \u003c- copy_to(sc, mtcars, overwrite = TRUE)\n\n# transform our data set, and then partition into 'training', 'test'\npartitions \u003c- mtcars_tbl %\u003e%\n  filter(hp \u003e= 100) %\u003e%\n  mutate(cyl8 = cyl == 8) %\u003e%\n  sdf_partition(training = 0.5, test = 0.5, seed = 1099)\n\n# fit a linear model to the training dataset\nfit \u003c- partitions$training %\u003e%\n  ml_linear_regression(response = \"mpg\", features = c(\"wt\", \"cyl\"))\nfit\n```\n\nFor linear regression models produced by Spark, we can use `summary()` to learn\na bit more about the quality of our fit, and the statistical significance of\neach of our predictors.\n\n```{r}\nsummary(fit)\n```\n\nSpark machine learning supports a wide array of algorithms and feature \ntransformations and as illustrated above it's easy to chain these functions \ntogether with dplyr pipelines. To learn more see the\n[machine learning](https://spark.posit.co/mlib/) section.\n\n## Reading and Writing Data\n\nYou can read and write data in CSV, JSON, and Parquet formats. Data can be stored in HDFS, S3, or on the local filesystem of cluster nodes.\n\n```{r}\ntemp_csv \u003c- tempfile(fileext = \".csv\")\ntemp_parquet \u003c- tempfile(fileext = \".parquet\")\ntemp_json \u003c- tempfile(fileext = \".json\")\n\nspark_write_csv(iris_tbl, temp_csv)\niris_csv_tbl \u003c- spark_read_csv(sc, \"iris_csv\", temp_csv)\n\nspark_write_parquet(iris_tbl, temp_parquet)\niris_parquet_tbl \u003c- spark_read_parquet(sc, \"iris_parquet\", temp_parquet)\n\nspark_write_json(iris_tbl, temp_json)\niris_json_tbl \u003c- spark_read_json(sc, \"iris_json\", temp_json)\n\nsrc_tbls(sc)\n```\n\n\n## Distributed R\n\nYou can execute arbitrary r code across your cluster using `spark_apply()`. For\nexample, we can apply `rgamma` over `iris` as follows:\n\n```{r}\nspark_apply(iris_tbl, function(data) {\n  data[1:4] + rgamma(1,2)\n})\n```\n\nYou can also group by columns to perform an operation over each group of rows\nand make use of any package within the closure:\n\n```{r}\nspark_apply(\n  iris_tbl,\n  function(e) broom::tidy(lm(Petal_Width ~ Petal_Length, e)),\n  columns = c(\"term\", \"estimate\", \"std.error\", \"statistic\", \"p.value\"),\n  group_by = \"Species\"\n)\n```\n\n## Extensions\n\nThe facilities used internally by sparklyr for its `dplyr` and machine learning \ninterfaces are available to extension packages. Since Spark is a general purpose \ncluster computing system there are many potential applications for extensions (e.g. \ninterfaces to custom machine learning pipelines, interfaces to 3rd party Spark\npackages, etc.).\n\nHere's a simple example that wraps a Spark text file line counting function with \nan R function:\n\n```{r}\n# write a CSV\ntempfile \u003c- tempfile(fileext = \".csv\")\nwrite.csv(nycflights13::flights, tempfile, row.names = FALSE, na = \"\")\n\n# define an R interface to Spark line counting\ncount_lines \u003c- function(sc, path) {\n  spark_context(sc) %\u003e%\n    invoke(\"textFile\", path, 1L) %\u003e%\n      invoke(\"count\")\n}\n\n# call spark to count the lines of the CSV\ncount_lines(sc, tempfile)\n```\n\n\nTo learn more about creating extensions see the \n[Extensions](https://spark.posit.co/guides/extensions.html) section of the \nsparklyr website.\n\n\n## Table Utilities\n\nYou can cache a table into memory with:\n\n```{r, eval=FALSE}\ntbl_cache(sc, \"batting\")\n```\n\nand unload from memory using:\n\n```{r, eval=FALSE}\ntbl_uncache(sc, \"batting\")\n```\n\n\n## Connection Utilities\n\nYou can view the Spark web console using the `spark_web()` function:\n\n```{r, eval=FALSE}\nspark_web(sc)\n```\n\nYou can show the log using the `spark_log()` function:\n\n```{r}\nspark_log(sc, n = 10)\n```\n\nFinally, we disconnect from Spark:\n\n```{r}\n  spark_disconnect(sc)\n```\n\n## RStudio IDE\n\nThe RStudio IDE includes integrated support for Spark and the sparklyr package, \nincluding tools for:\n\n- Creating and managing Spark connections\n- Browsing the tables and columns of Spark DataFrames\n- Previewing the first 1,000 rows of Spark DataFrames\n\nOnce you've installed the sparklyr package, you should find a new **Spark** pane \nwithin the IDE. This pane includes a **New Connection** dialog which can be used \nto make connections to local or remote Spark instances:\n\n\u003cimg src=\"tools/readme/spark-connect.png\" class=\"screenshot\" width=389 /\u003e\n\nOnce you've connected to Spark you'll be able to browse the tables contained \nwithin the Spark cluster and preview Spark DataFrames using the standard RStudio \ndata viewer:\n\n\u003cimg src=\"tools/readme/spark-dataview.png\" class=\"screenshot\" width=639 /\u003e\n\nYou can also connect to Spark through [Livy](https://livy.apache.org/) through \na new connection dialog:\n\n\u003cimg src=\"tools/readme/spark-connect-livy.png\" class=\"screenshot\" width=389 /\u003e\n\n\u003cdiv style=\"margin-bottom: 15px;\"\u003e\u003c/div\u003e\n\n\n## Using H2O\n\n[rsparkling](https://cran.r-project.org/package=rsparkling) is a CRAN package \nfrom [H2O](https://h2o.ai/) that extends [sparklyr](https://spark.posit.co/) \nto provide an interface into [Sparkling Water](https://github.com/h2oai/sparkling-water). \nFor instance, the following example installs, configures and runs\n[h2o.glm](https://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/glm.html):\n\n```{r results='hide', message=FALSE, eval=FALSE}\nlibrary(rsparkling)\nlibrary(sparklyr)\nlibrary(dplyr)\nlibrary(h2o)\n\nsc \u003c- spark_connect(master = \"local\", version = \"2.3.2\")\nmtcars_tbl \u003c- copy_to(sc, mtcars, \"mtcars\", overwrite = TRUE)\n\nmtcars_h2o \u003c- as_h2o_frame(sc, mtcars_tbl, strict_version_check = FALSE)\n\nmtcars_glm \u003c- h2o.glm(x = c(\"wt\", \"cyl\"),\n                      y = \"mpg\",\n                      training_frame = mtcars_h2o,\n                      lambda_search = TRUE)\n```\n\n```{r eval=FALSE}\nmtcars_glm\n```\n```\n#\u003e Model Details:\n#\u003e ==============\n#\u003e\n#\u003e H2ORegressionModel: glm\n#\u003e Model ID:  GLM_model_R_1527265202599_1\n#\u003e GLM Model: summary\n#\u003e     family     link                              regularization\n#\u003e 1 gaussian identity Elastic Net (alpha = 0.5, lambda = 0.1013 )\n#\u003e                                                                lambda_search\n#\u003e 1 nlambda = 100, lambda.max = 10.132, lambda.min = 0.1013, lambda.1se = -1.0\n#\u003e   number_of_predictors_total number_of_active_predictors\n#\u003e 1                          2                           2\n#\u003e   number_of_iterations                                training_frame\n#\u003e 1                  100 frame_rdd_31_ad5c4e88ec97eb8ccedae9475ad34e02\n#\u003e\n#\u003e Coefficients: glm coefficients\n#\u003e       names coefficients standardized_coefficients\n#\u003e 1 Intercept    38.941654                 20.090625\n#\u003e 2       cyl    -1.468783                 -2.623132\n#\u003e 3        wt    -3.034558                 -2.969186\n#\u003e\n#\u003e H2ORegressionMetrics: glm\n#\u003e ** Reported on training data. **\n#\u003e\n#\u003e MSE:  6.017684\n#\u003e RMSE:  2.453097\n#\u003e MAE:  1.940985\n#\u003e RMSLE:  0.1114801\n#\u003e Mean Residual Deviance :  6.017684\n#\u003e R^2 :  0.8289895\n#\u003e Null Deviance :1126.047\n#\u003e Null D.o.F. :31\n#\u003e Residual Deviance :192.5659\n#\u003e Residual D.o.F. :29\n#\u003e AIC :156.2425\n```\n\n```{r eval=FALSE}\nspark_disconnect(sc)\n```\n\n## Connecting through Livy\n\n[Livy](https://github.com/cloudera/livy) enables remote connections to Apache \nSpark clusters. However, please notice that connecting to Spark clusters through \nLivy is much slower than any other connection method.\n\nBefore connecting to Livy, you will need the connection information to an \nexisting service running Livy. Otherwise, to test `livy` in your local \nenvironment, you can install it and run it locally as follows:\n\n```{r eval=FALSE}\nlivy_install()\n```\n\n```{r livy-start, eval = FALSE}\nlivy_service_start()\n```\n\nTo connect, use the Livy service address as `master` and `method = \"livy\"` in \n`spark_connect()`. Once connection completes, use `sparklyr` as usual, for instance:\n\n```{r livy-connect, eval = FALSE}\nsc \u003c- spark_connect(master = \"http://localhost:8998\", method = \"livy\", version = \"3.0.0\")\ncopy_to(sc, iris, overwrite = TRUE)\n\nspark_disconnect(sc)\n```\n\nOnce you are done using `livy` locally, you should stop this service with:\n\n```{r livy-stop, eval = FALSE}\nlivy_service_stop()\n```\n\nTo connect to remote `livy` clusters that support basic authentication connect as:\n\n```{r eval=FALSE}\nconfig \u003c- livy_config(username=\"\u003cusername\u003e\", password=\"\u003cpassword\u003e\")\nsc \u003c- spark_connect(master = \"\u003caddress\u003e\", method = \"livy\", config = config)\nspark_disconnect(sc)\n```\n\n## Connecting through Databricks Connect v2\n\n`sparklyr` is able to interact with [Databricks Connect v2](https://docs.databricks.com/en/dev-tools/databricks-connect/index.html) \nvia a new extension called `pysparklyr`. To learn how to use, and the latest \nupdates on this integration see \n[the article in `sparklyr`'s official website](https://spark.posit.co/deployment/databricks-connect.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsparklyr%2Fsparklyr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsparklyr%2Fsparklyr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsparklyr%2Fsparklyr/lists"}