{"id":18749678,"url":"https://github.com/coditva/spark-pbspro","last_synced_at":"2025-10-25T10:05:28.372Z","repository":{"id":98240261,"uuid":"161750174","full_name":"coditva/Spark-PBSPro","owner":"coditva","description":"Support for PBS Pro in Spark","archived":false,"fork":false,"pushed_at":"2019-04-29T08:56:05.000Z","size":126,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-20T10:45:13.878Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coditva.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-14T07:48:43.000Z","updated_at":"2019-04-29T08:56:07.000Z","dependencies_parsed_at":"2023-05-18T21:45:13.692Z","dependency_job_id":null,"html_url":"https://github.com/coditva/Spark-PBSPro","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/coditva/Spark-PBSPro","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coditva%2FSpark-PBSPro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coditva%2FSpark-PBSPro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coditva%2FSpark-PBSPro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coditva%2FSpark-PBSPro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coditva","download_url":"https://codeload.github.com/coditva/Spark-PBSPro/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coditva%2FSpark-PBSPro/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259360806,"owners_count":22845824,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T17:08:20.669Z","updated_at":"2025-10-25T10:05:23.340Z","avatar_url":"https://github.com/coditva.png","language":"Scala","readme":"# Spark PBSPro Support\nThis adds support for [PBS Professional](https://github.com/pbspro/pbspro)\nHPC resource manager in [Apache Spark](https://github.com/apache/spark).\n\n\n### Status of build with latest Spark\n[![Build Status](https://travis-ci.com/PBSPro/spark-pbspro-connector.svg?branch=master)](https://travis-ci.com/PBSPro/spark-pbspro-connector)\n\n\n## Usage\n\nYou can run Spark on the PBS cluster just by adding \"--master pbs\" while submitting as follows:\n```bash\n# start spark shell. only in client mode\n./bin/spark-shell --master pbs\n\n# submit a spark application in client mode\n./bin/spark-submit --master pbs --deploy-mode client --class org.apache.spark.examples.SparkPi $SPARK_HOME/examples/target/scala-2.12/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar 100\n\n# submit a spark application in cluster mode\n./bin/spark-submit --master pbs --deploy-mode cluster --class org.apache.spark.examples.SparkPi $SPARK_HOME/examples/target/scala-2.12/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar 100\n```\n\nYou can also just append `spark.master pbs` in `conf/spark-defaults.conf` to avoid adding\n`--master pbs` on every submit.\n\n\nTo run Spark UI with PBS cluster:\n```bash\nbin/spark-class org.apache.spark.deploy.pbs.ui.PbsClusterUI\n```\n\n\n\n## Installation\n\nThis expects PBSPro to be installed at `/opt/pbs`.\n\nClone the Spark repository and move to spark folder\n```bash\ngit clone https://github.com/apache/spark.git\ncd spark\n```\n\nIn the spark project root, punch in these commands:\n```bash\n# Clone the repo\ngit clone https://github.com/PBSPro/spark-pbspro-connector resource-managers/pbs\n\n# Apply patch to spark (in the root directory).\ngit am resource-managers/pbs/*.patch\n\n# Build!\nbuild/mvn -DskipTests -Ppbs package\n```\n\nAdd executor home to your configuration:\n```bash\n# in file conf/spark-defaults.conf add line:\nspark.pbs.executor.home \"SPARK INSTALLATION DIRECTORY PATH IN PBS MOMS\"\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoditva%2Fspark-pbspro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoditva%2Fspark-pbspro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoditva%2Fspark-pbspro/lists"}