{"id":20165086,"url":"https://github.com/mengyou658/spark-job-rest","last_synced_at":"2025-07-24T08:40:23.373Z","repository":{"id":36503843,"uuid":"40809523","full_name":"mengyou658/spark-job-rest","owner":"mengyou658","description":"Spark-Job-REST","archived":false,"fork":false,"pushed_at":"2015-07-23T12:48:45.000Z","size":1180,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-01-13T14:33:33.145Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mengyou658.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-16T10:09:59.000Z","updated_at":"2016-01-15T22:05:38.000Z","dependencies_parsed_at":"2022-09-06T18:20:34.579Z","dependency_job_id":null,"html_url":"https://github.com/mengyou658/spark-job-rest","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mengyou658%2Fspark-job-rest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mengyou658%2Fspark-job-rest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mengyou658%2Fspark-job-rest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mengyou658%2Fspark-job-rest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mengyou658","download_url":"https://codeload.github.com/mengyou658/spark-job-rest/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241601124,"owners_count":19988868,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T00:36:43.999Z","updated_at":"2025-03-03T03:21:10.736Z","avatar_url":"https://github.com/mengyou658.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Atigeo/spark-job-rest?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge)\n\n## Features:\n\n**Supports multiple spark contexts created from the same server**\n\nThe main problem this project solves is the inability to run multiple Spark contexts from the same JVM. This is a bug in Spark core that was also present in Ooyala's Spark Job Server, from which this project is inspired. The project launches a new process for each Spark context/application, with its own driver memory setting and its own driver log. Each driver JVM is created with its own Spark UI port, sent back to the api caller. Inter-process communication is achieved with akka actors, and each process is shut down when a Spark context/application is deleted.\n\n## Version compatibility\n\nSJR Version   | Spark Version\n------------- | -------------\n0.3.0         |  1.1.0 \n0.3.1         |  1.3.1 \n0.3.3         |  1.4.0 \n\n## Building Spark-Job-Rest (SJR)\n\nThe project is build with Maven3 and Java7.\n```\nmake build\n```\nSJR can now be deployed from `spark-job-rest/spark-job-rest/target/scala-2.10/spark-job-rest.zip`\n\nIf running from IDE fails with:\n```\nException in thread \"main\" java.lang.NoClassDefFoundError: akka/actor/Props\n```\nThis happens because the spark dependency has the provided scope. In order to run from IDE you can remove the provided scope (check `project/BuildSugar.scala`) for the spark dependency or you can add the spark assembly jar to the running classpath.\n\n## Deploying Spark-Job-Rest\n\nYou can deploy Spark-Job-Rest locally to `deploy` directory inside the project by:\n```sh\nmake deploy\n```\nOptionally you can specifying install directory in `$SJR_DEPLOY_PATH` environment variable:\n```sh\nSJR_DEPLOY_PATH=/opt/spark-job-rest make deploy\n```\n\nBefore running JSR ensure that [working environment](#configure-spark-environment) is configured.\n\nIn order to have a proper installation you should set `$SPARK_HOME` to your Apache Spark distribution and `$SPARK_CONF_HOME` to directory which consists `spark-env.sh` (usually `$SPARK_HOME/conf` or `$SPARK_HOME/libexec/conf`).\nYou can do it in your bash profile (`~/.bash_profile` or `~/.bashrc`) by adding the following lines:\n```sh\nexport SPARK_HOME=\u003cPath to Apache Spark\u003e\nexport SPARK_CONF_HOME=$SPARK_HOME/libexec/conf  # or $SPARK_HOME/conf depending on your distribution\n```\nAfter that either run in the new terminal session or source your bash profile.\n\n### Deploying to remote host\n\nYou can deploy Spark-Job-REST to remote host via:\n```sh\nmake remote-deploy\n```\n\nFor remote deployment you should set following environment variables:\n```sh\n# Mandatory connection string\nexport SJR_DEPLOY_HOST=\u003cuser@hostname for remote machine\u003e\n# Optional parameters\nexport SJR_DEPLOY_KEY=\u003coptional path to your SSH key\u003e\nexport SJR_REMOTE_DEPLOY_PATH=\u003cwhere you want to install Spark-Job-REST on remote host\u003e\n```\nIf `SJR_REMOTE_DEPLOY_PATH` is not set then `SJR_DEPLOY_PATH` will be used during remote deploy.\n\n## Deploying from artifacts\n\nYou can deploy by putting `spark-job-rest-server.zip` (with optional `spark-job-rest-sql.zip`) and `deploy.sh` to one directory. Then perform:\n\n```sh\nsh deploy.sh\n```\n\nThat will extract everything to `\u003cfiles location/spark-job-rest\u003e` and set proper file permissions.\n \n## Starting Spark-Job-Rest\n\nTo start/stop SJR use\n```sh\ncd $SJR_DEPLOY_PATH\nbin/start_server.sh\nbin/stop_server.sh\n```\n\nor if it deployed to default destination just\n```sh\nmake start\nmake stop\n```\n\n### Run in non-detached mode\n\nBy default server runs in detached mode. To turn it off simply set `SJR_RUN_DETACHED` to `false`.\n\n## Configure Spark-job-rest\n\nSpark-Job-REST default configuration is stored in `resources/application.conf` (here and after under `spark-job-rest/src/main/`).\nTo add or override settings create `resources/deploy.conf` (ignored by VCS).\n\n### Spark context settings\nConfigure the default spark properties for context creation as they are normal Spark configuration options\n```\nspark.executor.memory=2g\nspark.master=\"local\"\nspark.path=\"/Users/user/spark-1.1.0\"\n........\n```\nTo set how much memory should be allocated for driver use `driver.xmxMemory` (default is `1g`).\n\n### Application settings\n\nConfigure settings like web server port and akka system ports\n```\nspark.job.rest.appConf{\n  web.services.port=8097\n  spark.ui.first.port = 16000\n  ........\n}\n```\n\n### Configure folders \u0026 class paths\n\nYou may configure folders by setting environment variables and by creating and editing `resources/deploy-settings.sh` (under `spark-job-rest/src/main/`):\n\n```sh\nexport SJR_LOG_DIR=\u003cpath to logs directory\u003e\nexport SJR_JAR_PATH=\u003cpath to jar files storage\u003e\nexport SJR_CONTEXTS_BASE_DIR=\u003cpath to the rood directory for contexts process directories\u003e\nexport JSR_EXTRA_CLASSPATH=\u003cadditional classes required for your application to run\u003e\nexport SJR_DATABASE_ROOT_DIR=\u003cdirectory where database storage will be created\u003e\n```\n\n### Java \u0026 GC options\n\nYou can extend or override Java and GC options in `resources/deploy-settings.sh`:\n\n```sh\nJAVA_OPTS=\"${JAVA_OPTS}\n           ${YOUR_EXTRA_JAVA_OPTIONS}\"\nGC_OPTS=\"${GC_OPTS}\n         ${YOUR_EXTRA_GC_OPTIONS}\"           \n```\n\n## Custom contexts\n\nSpark-Job-REST supports custom job context factories defined in `context.job-context-factory` property of config.\nBy default SJR uses `SparkContextFactory` which creates one Spark Context per JVM.\n\n### SQL contexts\n\nTo run jobs with provided SQL contexts include `spark-job-rest-sql` in your project, set context factory to one of SQLContext factories provided by this library and inherit your job from `api.SparkSqlJob`.\nCurrently supported contexts:\n\n1. `spark.job.rest.context.SparkSqlContextFactory` creates simple job SQLContext.\n2. `spark.job.rest.HiveContextFactory` creates Hive SQL context.\n\n## Configure Spark environment\n\nIn order to have a proper installation you should set `$SPARK_HOME` to your Apache Spark distribution and `$SPARK_CONF_HOME` to directory which consists `spark-env.sh` (usually `$SPARK_HOME/conf` or `$SPARK_HOME/libexec/conf`).\nYou can do it in your bash profile (`~/.bash_profile` or `~/.bashrc`) by adding the following lines:\n```sh\nexport SPARK_HOME=\u003cPath to Apache Spark\u003e\nexport SPARK_CONF_HOME=$SPARK_HOME/libexec/conf  # or $SPARK_HOME/conf depending on your distribution\n```\nAfter that either run in the new terminal session or source your bash profile.\n\nThe SJR can be run from outside the Spark cluster, but you need to at least copy the deployment folder from one of the slaves or master nodes.\n\n## Run Spark-job-rest\n\nAfter editing all the configuration files SJR can be run by executing the script `start-server.sh`\n\nThe UI can be accessed at `\u003cserver address\u003e:\u003cappConf.web.services.port\u003e`.\n\n## API\n\n**Contexts**\n\n- POST /contexts/{contextName}  -  Create Context\n\n * Body:  Raw entity with key-value pairs. \n * jars key is required and it should be in the form of a comma separated list of jar paths. These jars will be added at Spark context creation time to the class path of the newly created context's JVM process. There are 3 types of jar paths supported:\n    * Absolute path on the server side : /home/ubuntu/example.jar\n    * Name of the jar that was uploaded to the server : example.jar\n    * Hdfs path : hdfs://devbox.local:8020/user/test/example.jar\n  \n  ``` \n # Body example:\n jars=\"/home/ubuntu/example.jar,example.jar,hdfs://devbox.local:8020/user/test/example.jar”\n spark.executor.memory=2g\n driver.xmxMemory = 1g\n  ```\n\n- GET /contexts/{contextName}  -  returns Context JSON object | No such context.\n\n- DELETE /contexts/{contextName}  -  Delete Context\n\n**Jobs**\n\n- POST /jobs?runningClass={runningClass}\u0026context={contextName}  - Job Submission \n\n  * Body:  Raw entity with key-value pairs. Here you can set any configuration properties that will be passed to the config parameter of the validate and run methods of the provided jar (see the SparkJob definition below)\n\n- GET /jobs/{jobId}?contextName={contextName} - Gets the result or state of a specific job\n\n- GET /jobs - Gets the states/results of all jobs from all running contexts \n\n**Jars**\n\n- POST /jars/{jarName}  - Upload jar\n  * Body: Jar Bytes\n  \n- POST /jars  - Upload jar\n  * Body: MultiPart Form\n\n- GET /jars - Gets all the uploaded jars\n\n- DELETE /jars/{jarName} - Delete jar\n\n## HTTP Client\n\nAll the API methods can be called from Scala/Java with the help of an HTTP Client.\n\nMaven Spark-Job-Rest-Client dependency:\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.xpatterns\u003c/groupId\u003e\n    \u003cartifactId\u003espark-job-rest-client\u003c/artifactId\u003e\n    \u003cversion\u003e0.3.3\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## Create Spark Job Project\n\nAdd maven Spark-Job-Rest-Api dependency:\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.xpatterns\u003c/groupId\u003e\n    \u003cartifactId\u003espark-job-rest-api\u003c/artifactId\u003e\n    \u003cversion\u003e0.3.3\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nTo create a job that can be submitted through the server, the class must implement the SparkJob trait.\n\n```scala\nimport com.typesafe.config.Config\nimport org.apache.spark.SparkContext\nimport spark.job.rest.api.{SparkJobInvalid, SparkJobValid, SparkJobValidation, SparkJob}\n\nclass Example extends SparkJob {\n    override def runJob(sc:SparkContext, jobConfig: Config): Any = { ... }\n    override def validate(sc:SparkContext, config: Config): SparkJobValidation = { ... }\n}\n```\n\n- runJob method contains the implementation of the Job. SparkContext and Config objects are provided through parameters.\n- validate method allows for an initial validation. In order to run the job return SparkJobValid(), otherwise return SparkJobInvalid(message).\n\n## Example\n\nAn example for this project can be found here: ```spark-job-rest/examples/example-job```. In order to package it, run \n```sh\nmvn clean install\n```\n\n**Upload JAR**\n```sh\n# In the project root directory\ncurl --data-binary @spark-job-rest/examples/example-job/target/example-job.jar 'localhost:8097/jars/example-job.jar'\n\n{\n  \"contextName\": \"test-context\",\n  \"sparkUiPort\": \"16003\"\n}\n```\n\n**Create a context**\n```sh\ncurl -X POST -d \"jars=example-job.jar\" 'localhost:8097/contexts/test-context'\n\n{\n  \"contextName\": \"test-context\",\n  \"sparkUiPort\": \"16003\"\n}\n```\n\n**Check if context exists**\n\n```sh\ncurl 'localhost:8097/contexts/test-context'\n\n{\n  \"contextName\": \"test-context\",\n  \"sparkUiPort\": \"16003\"\n}\n```\n\n**Run job** - The example job creates an RDD from a Range(0,input) and applies count on it.\n\n```sh\ncurl -X POST -d \"input=10000\" 'localhost:8097/jobs?runningClass=com.job.SparkJobImplemented\u0026contextName=test-context'\n\n{\n  \"jobId\": \"2bd438a2-ac1e-401a-b767-5fa044b2bd69\",\n  \"contextName\": \"test-context\",\n  \"status\": \"Running\",\n  \"result\": \"\",\n  \"startTime\": 1430287260144\n}\n```\n\n```2bd438a2-ac1e-401a-b767-5fa044b2bd69``` represents the jobId. This id can be used to query for the job status/results.\n\n**Query for results**\n\n```sh\ncurl 'localhost:8097/jobs/2bd438a2-ac1e-401a-b767-5fa044b2bd69?contextName=test-context'\n\n{\n  \"jobId\": \"2bd438a2-ac1e-401a-b767-5fa044b2bd69\",\n  \"contextName\": \"test-context\",\n  \"status\": \"Finished\",\n  \"result\": \"10000\",\n  \"startTime\": 1430287261108\n}\n```\n\n**Delete context**\n\n```sh\ncurl -X DELETE 'localhost:8097/contexts/test-context'\n\n{\n  \"message\": \"Context deleted.\"\n}\n```\n\n**HTTP Client Example**\n\n```scala\nobject Example extends App {\n  implicit val system = ActorSystem()\n  val contextName = \"testContext\"\n\n  try {\n    val sjrc = new SparkJobRestClient(\"http://localhost:8097\")\n\n    val context = sjrc.createContext(contextName, Map(\"jars\" -\u003e \"/Users/raduchilom/projects/spark-job-rest/examples/example-job/target/example-job.jar\"))\n    println(context)\n\n    val job = sjrc.runJob(\"com.job.SparkJobImplemented\", contextName, Map(\"input\" -\u003e \"10\"))\n    println(job)\n\n    var jobFinal = sjrc.getJob(job.jobId, job.contextName)\n    while (jobFinal.status.equals(JobStates.RUNNING.toString())) {\n      Thread.sleep(1000)\n      jobFinal = sjrc.getJob(job.jobId, job.contextName)\n    }\n    println(jobFinal)\n\n    sjrc.deleteContext(contextName)\n  } catch {\n    case e:Exception =\u003e {\n      e.printStackTrace()\n    }\n  }\n\n  system.shutdown()\n}\n```\nRunning this would produce the output:\n\n```\nContext(testContext,16002)\nJob(ab63c19f-bbb4-461e-8c6f-f0a35f73a943,testContext,Running,,1430291077689)\nJob(ab63c19f-bbb4-461e-8c6f-f0a35f73a943,testContext,Finished,10,1430291078694)\n```\n\n\n## UI\n\nThe UI was added in a compiled and minified state. For sources and changes please refer to [spark-job-rest-ui](https://github.com/marianbanita82/spark-job-rest-ui) project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmengyou658%2Fspark-job-rest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmengyou658%2Fspark-job-rest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmengyou658%2Fspark-job-rest/lists"}