{"id":20200386,"url":"https://github.com/nickjer/singularity-rstudio-spark","last_synced_at":"2025-04-10T11:21:25.454Z","repository":{"id":70912427,"uuid":"117893403","full_name":"nickjer/singularity-rstudio-spark","owner":"nickjer","description":"Apache Spark with RStudio and the sparklyr package in a Singularity container","archived":false,"fork":false,"pushed_at":"2018-03-07T01:41:35.000Z","size":6,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-24T10:06:57.654Z","etag":null,"topics":["rstudio-server","singularity-image","spark"],"latest_commit_sha":null,"homepage":"https://www.singularity-hub.org/collections/455","language":"Roff","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nickjer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-17T21:14:10.000Z","updated_at":"2020-03-30T14:59:11.000Z","dependencies_parsed_at":null,"dependency_job_id":"4cf61640-4856-4152-b560-cd193c829e3a","html_url":"https://github.com/nickjer/singularity-rstudio-spark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickjer%2Fsingularity-rstudio-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickjer%2Fsingularity-rstudio-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickjer%2Fsingularity-rstudio-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickjer%2Fsingularity-rstudio-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nickjer","download_url":"https://codeload.github.com/nickjer/singularity-rstudio-spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248208603,"owners_count":21065203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["rstudio-server","singularity-image","spark"],"created_at":"2024-11-14T04:43:30.509Z","updated_at":"2025-04-10T11:21:25.445Z","avatar_url":"https://github.com/nickjer.png","language":"Roff","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Singularity Apache Spark w/ RStudio Server\n\n[![Singularity Hub](https://www.singularity-hub.org/static/img/hosted-singularity--hub-%23e32929.svg)](https://singularity-hub.org/collections/455)\n[![GitHub License](https://img.shields.io/badge/license-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\nSingularity image for [Apache Spark] with the [sparklyr] package installed. It\nwas built on top of the base Singularity image [nickjer/singularity-rstudio] in\norder to launch an [RStudio Server] to more easily connect with an Apache Spark\ncluster running in [Standalone Mode].\n\nThis is still a work in progress.\n\n## Build\n\nYou can build a local Singularity image named `singularity-rstudio-spark.simg`\nwith:\n\n```sh\nsudo singularity build singularity-rstudio-spark.simg Singularity\n```\n\n## Deploy\n\nInstead of building it yourself you can download the pre-built image from\n[Singularity Hub](https://www.singularity-hub.org) with:\n\n```sh\nsingularity pull --name singularity-rstudio-spark.simg shub://nickjer/singularity-rstudio-spark\n```\n\n## Run\n\nYou can launch Spark in [Standalone Mode] by first launching a \"master\" process\nwhich will print out a `spark://HOST:PORT` for itself, which you can then use\nto connect \"workers\" to it.\n\n### Spark Master\n\nYou can launch a \"master\" process as a Singularity app with:\n\n```sh\nsingularity run --app spark-master singularity-rstudio-spark.simg\n```\n\n### Worker\n\nYou can launch a \"worker\" process as a Singularity app with:\n\n```sh\nsingularity run --app spark-worker singularity-rstudio-spark.simg\n```\n\n### RStudio Server\n\nSee [nickjer/singularity-rstudio] for more information on how to run `rserver`\nfrom within this Singularity image.\n\n### R and Rscript\n\nSee [nickjer/singularity-r] for more information on how to run `R` and\n`Rscript` from within this Singularity image.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at\nhttps://github.com/nickjer/singularity-rstudio-spark.\n\n## License\n\nThe code is available as open source under the terms of the [MIT License].\n\n[Apache Spark]: https://spark.apache.org/\n[sparklyr]: http://spark.rstudio.com/\n[RStudio Server]: https://www.rstudio.com/products/rstudio/\n[nickjer/singularity-r]: https://github.com/nickjer/singularity-r\n[nickjer/singularity-rstudio]: https://github.com/nickjer/singularity-rstudio\n[Standalone Mode]: https://spark.apache.org/docs/latest/spark-standalone.html\n[MIT License]: http://opensource.org/licenses/MIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickjer%2Fsingularity-rstudio-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnickjer%2Fsingularity-rstudio-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickjer%2Fsingularity-rstudio-spark/lists"}