{"id":13488779,"url":"https://github.com/spark-jobserver/spark-jobserver","last_synced_at":"2025-05-13T20:10:48.580Z","repository":{"id":37736265,"uuid":"23205911","full_name":"spark-jobserver/spark-jobserver","owner":"spark-jobserver","description":"REST job server for Apache Spark","archived":false,"fork":false,"pushed_at":"2025-01-04T14:29:27.000Z","size":4962,"stargazers_count":2836,"open_issues_count":111,"forks_count":991,"subscribers_count":219,"default_branch":"master","last_synced_at":"2025-04-28T10:56:19.433Z","etag":null,"topics":["rest-api","scala","spark","spark-jobserver"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spark-jobserver.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-08-21T23:07:47.000Z","updated_at":"2025-03-14T15:32:46.000Z","dependencies_parsed_at":"2022-07-12T15:18:35.647Z","dependency_job_id":"34bca028-5be7-4f62-b661-e67f96bf779a","html_url":"https://github.com/spark-jobserver/spark-jobserver","commit_stats":{"total_commits":1504,"total_committers":163,"mean_commits":9.226993865030675,"dds":0.8045212765957447,"last_synced_commit":"f58280bbe4ecfef110bdf425f5aafccfbaa1bdba"},"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spark-jobserver%2Fspark-jobserver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spark-jobserver%2Fspark-jobserver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spark-jobserver%2Fspark-jobserver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spark-jobserver%2Fspark-jobserver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spark-jobserver","download_url":"https://codeload.github.com/spark-jobserver/spark-jobserver/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254020608,"owners_count":22000753,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["rest-api","scala","spark","spark-jobserver"],"created_at":"2024-07-31T18:01:21.660Z","updated_at":"2025-05-13T20:10:48.549Z","avatar_url":"https://github.com/spark-jobserver.png","language":"Scala","funding_links":[],"categories":["Scala","大数据","Packages"],"sub_categories":["Middleware"],"readme":"[![Build Status](https://travis-ci.org/spark-jobserver/spark-jobserver.svg?branch=master)](https://travis-ci.org/spark-jobserver/spark-jobserver) [![Coverage](https://img.shields.io/codecov/c/github/spark-jobserver/spark-jobserver/master.svg)](https://codecov.io/gh/spark-jobserver/spark-jobserver/branch/master)\n\n[![Join the chat at https://gitter.im/spark-jobserver/spark-jobserver](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/spark-jobserver/spark-jobserver?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\nspark-jobserver provides a RESTful interface for submitting and managing [Apache Spark](http://spark-project.org) jobs, jars, and job contexts.\nThis repo contains the complete Spark job server project, including unit tests and deploy scripts.\nIt was originally started at [Ooyala](http://www.ooyala.com), but this is now the main development repo.\n\nOther useful links: [Troubleshooting](doc/troubleshooting.md), [cluster](doc/cluster.md), [YARN client](doc/yarn.md), [YARN on EMR](doc/EMR.md), [Mesos](doc/mesos.md), [JMX tips](doc/jmx.md).\n\nAlso see [Chinese docs / 中文](doc/chinese/job-server.md).\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*\n\n- [Users](#users)\n- [Features](#features)\n- [Version Information](#version-information)\n- [Getting Started with Spark Job Server](#getting-started-with-spark-job-server)\n- [Development mode](#development-mode)\n  - [WordCountExample walk-through](#wordcountexample-walk-through)\n    - [Package Jar - Send to Server](#package-jar---send-to-server)\n    - [Ad-hoc Mode - Single, Unrelated Jobs (Transient Context)](#ad-hoc-mode---single-unrelated-jobs-transient-context)\n    - [Persistent Context Mode - Faster \u0026 Required for Related Jobs](#persistent-context-mode---faster--required-for-related-jobs)\n  - [Debug mode](#debug-mode)\n- [Create a Job Server Project](#create-a-job-server-project)\n  - [Creating a project from scratch using giter8 template](#creating-a-project-from-scratch-using-giter8-template)\n  - [Creating a project manually assuming that you already have sbt project structure](#creating-a-project-manually-assuming-that-you-already-have-sbt-project-structure)\n  - [NEW SparkJob API](#new-sparkjob-api)\n  - [NEW SparkJob API with Spark v2.1](#new-sparkjob-api-with-spark-v21)\n  - [Dependency jars](#dependency-jars)\n  - [Named Objects](#named-objects)\n    - [Using Named RDDs](#using-named-rdds)\n    - [Using Named Objects](#using-named-objects)\n  - [HTTPS / SSL Configuration](#https--ssl-configuration)\n    - [Server authentication](#server-authentication)\n    - [Client authentication](#client-authentication)\n  - [Access Control](#access-control)\n    - [Shiro Authentication](#shiro-authentication)\n    - [Keycloak Authentication](#keycloak-authentication)\n  - [User Authorization](#user-authorization)\n- [Deployment](#deployment)\n  - [Manual steps](#manual-steps)\n  - [Context per JVM](#context-per-jvm)\n  - [Configuring Spark Jobserver backend](#configuring-spark-jobserver-backend)\n  - [HA Deployment (beta)](#ha-deployment-beta)\n  - [Chef](#chef)\n- [Architecture](#architecture)\n- [API](#api)\n  - [Binaries](#binaries)\n  - [Contexts](#contexts)\n  - [Jobs](#jobs)\n  - [Data](#data)\n    - [Data API Example](#data-api-example)\n  - [Context configuration](#context-configuration)\n  - [Other configuration settings](#other-configuration-settings)\n  - [Job Result Serialization](#job-result-serialization)\n  - [HTTP Override](#http-override)\n- [Clients](#clients)\n- [Contribution and Development](#contribution-and-development)\n- [Contact](#contact)\n- [License](#license)\n- [TODO](#todo)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n## Users\n\n(Please add yourself to this list!)\n\nSpark Job Server is included in Datastax Enterprise!\n\n- [Ooyala](http://www.ooyala.com)\n- [Netflix](http://www.netflix.com)\n- [Avenida.com](http://www.avenida.com)\n- GumGum\n- Fuse Elements\n- Frontline Solvers\n- [Aruba Networks](http://www.arubanetworks.com/)\n- [Zed Worldwide](http://www.zed.com)\n- [KNIME](https://www.knime.org/)\n- [Azavea](http://azavea.com)\n- [Maana](http://maana.io/)\n- [Newsweaver](https://www.newsweaver.com)\n- [Instaclustr](http://www.instaclustr.com)\n- [SnappyData](http://www.snappydata.io)\n- [Linkfluence](http://www.linkfluence.com)\n- [Smartsct](http://www.smartsct.com)\n- [Datadog](https://www.datadoghq.com/)\n- [Planalytics](http://www.planalytics.com)\n- [Target](http://www.target.com/)\n- [Branch](http://branch.io)\n- [Informatica](https://www.informatica.com/)\n- [Lentra](https://lentra.ai/cadenz/)\n\n## Features\n\n- *\"Spark as a Service\"*: Simple REST interface (including HTTPS) for all aspects of job, context management\n- Support for Spark SQL, Hive, Streaming Contexts/jobs and custom job contexts!  See [Contexts](doc/contexts.md).\n- [Python](doc/python.md), Scala, and [Java](doc/javaapi.md) (see [TestJob.java](https://github.com/spark-jobserver/spark-jobserver/blob/master/job-server-api/src/main/java/spark/jobserver/api/TestJob.java)) support\n- LDAP Auth support via Apache Shiro integration\n- Separate JVM per SparkContext for isolation (EXPERIMENTAL)\n- Supports sub-second low-latency jobs via long-running job contexts\n- Start and stop job contexts for RDD sharing and low-latency jobs; change resources on restart\n- Kill running jobs via stop context and delete job\n- Separate jar uploading step for faster job startup\n- Asynchronous and synchronous job API.  Synchronous API is great for low latency jobs!\n- Works with Standalone Spark as well on [cluster](doc/cluster.md), [Mesos](doc/mesos.md), YARN [client](doc/yarn.md) and [on EMR](doc/EMR.md))\n- Job and jar info is persisted via a pluggable DAO interface\n- Named Objects (such as RDDs or DataFrames) to cache and retrieve RDDs or DataFrames by name, improving object sharing and reuse among jobs.\n- Supports Scala 2.11 and 2.12\n- Support for supervise mode of Spark (EXPERIMENTAL)\n- Possible to be deployed in an [HA setup](#ha-deployment-beta) of multiple jobservers (beta)\n\n## Version Information\n\n| Version     | Spark Version | Scala Version |\n|-------------|---------------|---------------|\n| 0.8.1       | 2.2.0         | 2.11          |\n| 0.10.2      | 2.4.4         | 2.11          |\n| 0.11.1      | 2.4.4         | 2.11, 2.12    |\n\nFor release notes, look in the `notes/` directory.\n\n[Due to the sunset of Bintray](https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/) all previous release binaries were deleted. Jobserver had to migrate to JFrog Platform and only recent releases are available there.\nTo use Spark Jobserver in your SBT project please include the following resolver in you `build.sbt` file:\n\n```scala\nresolvers += \"Artifactory\" at \"https://sparkjobserver.jfrog.io/artifactory/jobserver/\"\n```\n\nCheck [Creating a project manually assuming that you already have sbt project structure](#creating-a-project-manually-assuming-that-you-already-have-sbt-project-structure) for more information.\n\nIf you need non-released jars, please visit [Jitpack](https://jitpack.io) - they provide non-release jar builds for any Git repo.  :)\n\n## Getting Started with Spark Job Server\n\nThe easiest way to get started is to try the [Docker container](doc/docker.md) which prepackages a Spark distribution with the job server and lets you start and deploy it.\n\nAlternatives:\n\n* Build and run Job Server in local [development mode](#development-mode) within SBT.  NOTE:  This does NOT work for YARN, and in fact is only recommended with `spark.master` set to `local[*]`.  Please deploy if you want to try with YARN or other real cluster.\n* Deploy job server to a cluster.  There are two alternatives (see the [deployment section](#deployment)):\n  - `server_deploy.sh`  deploys job server to a directory on a remote host.\n  - `server_package.sh` deploys job server to a local directory, from which you can deploy the directory, or create a .tar.gz for Mesos or YARN deployment.\n* EC2 Deploy scripts - follow the instructions in [EC2](doc/EC2.md) to spin up a Spark cluster with job server and an example application.\n* EMR Deploy instruction - follow the instruction in [EMR](doc/EMR.md)\n\nNOTE: Spark Job Server can optionally run `SparkContext`s in their own, forked JVM process when the config option `spark.jobserver.context-per-jvm` is set to `true`.  This option does not currently work for SBT/local dev mode. See [Deployment](#deployment) section for more info.\n\n## Development mode\n\nThe example walk-through below shows you how to use the job server with an included example job, by running the job server in local development mode in SBT.  This is not an example of usage in production.\n\nYou need to have [SBT](http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html) installed.\n\nTo set the current version, do something like this:\n\n    export VER=`sbt version | tail -1 | cut -f2`\n\nFrom SBT shell, simply type \"reStart\".  This uses a default configuration file.  An optional argument is a\npath to an alternative config file.  You can also specify JVM parameters after \"---\".  Including all the\noptions looks like this:\n\n    job-server-extras/reStart /path/to/my.conf --- -Xmx8g\n\nNote that reStart (SBT Revolver) forks the job server in a separate process.  If you make a code change, simply\ntype reStart again at the SBT shell prompt, it will compile your changes and restart the jobserver.  It enables\nvery fast turnaround cycles.\n\n**NOTE2**: You cannot do `sbt reStart` from the OS shell.  SBT will start job server and immediately kill it.\n\nFor example jobs see the job-server-tests/ project / folder.\n\nWhen you use `reStart`, the log file goes to `job-server/job-server-local.log`.  There is also an environment variable\nEXTRA_JAR for adding a jar to the classpath.\n\n### WordCountExample walk-through\n\n#### Package Jar - Send to Server\nFirst, to package the test jar containing the WordCountExample: `sbt job-server-tests/package`.\nThen go ahead and start the job server using the instructions above.\n\nLet's upload the jar:\n\n    curl -X POST localhost:8090/binaries/test -H \"Content-Type: application/java-archive\" --data-binary @job-server-tests/target/scala-2.12/job-server-tests_2.12-$VER.jar\n    OK⏎\n\n#### Ad-hoc Mode - Single, Unrelated Jobs (Transient Context)\nThe above jar is uploaded as app `test`.  Next, let's start an ad-hoc word count job, meaning that the job\nserver will create its own SparkContext, and return a job ID for subsequent querying:\n\n    curl -d \"input.string = a b c a b see\" \"localhost:8090/jobs?appName=test\u0026classPath=spark.jobserver.WordCountExample\"\n    {\n      \"duration\": \"Job not done yet\",\n      \"classPath\": \"spark.jobserver.WordCountExample\",\n      \"startTime\": \"2016-06-19T16:27:12.196+05:30\",\n      \"context\": \"b7ea0eb5-spark.jobserver.WordCountExample\",\n      \"status\": \"STARTED\",\n      \"jobId\": \"5453779a-f004-45fc-a11d-a39dae0f9bf4\"\n    }⏎\n\nNOTE: If you want to feed in a text file config and POST using curl, you want the `--data-binary` option, otherwise\ncurl will munge your line separator chars.  Like:\n\n    curl --data-binary @my-job-config.json \"localhost:8090/jobs?appNam=...\"\n\nNOTE2: If you want to send in UTF-8 chars, make sure you pass in a proper header to CURL for the encoding, otherwise it may assume an encoding which is not what you expect.\n\nFrom this point, you could asynchronously query the status and results:\n\n    curl localhost:8090/jobs/5453779a-f004-45fc-a11d-a39dae0f9bf4\n    {\n      \"duration\": \"6.341 secs\",\n      \"classPath\": \"spark.jobserver.WordCountExample\",\n      \"startTime\": \"2015-10-16T03:17:03.127Z\",\n      \"context\": \"b7ea0eb5-spark.jobserver.WordCountExample\",\n      \"result\": {\n        \"a\": 2,\n        \"b\": 2,\n        \"c\": 1,\n        \"see\": 1\n      },\n      \"status\": \"FINISHED\",\n      \"jobId\": \"5453779a-f004-45fc-a11d-a39dae0f9bf4\"\n    }⏎\n\nNote that you could append `\u0026sync=true` when you POST to /jobs to get the results back in one request, but for\nreal clusters and most jobs this may be too slow.\n\nYou can also append `\u0026timeout=XX` to extend the request timeout for `sync=true` requests.\n\n#### Persistent Context Mode - Faster \u0026 Required for Related Jobs\nAnother way of running this job is in a pre-created context.  Start a new context:\n\n    curl -d \"\" \"localhost:8090/contexts/test-context?num-cpu-cores=4\u0026memory-per-node=512m\"\n    OK⏎\n\nYou can verify that the context has been created:\n\n    curl localhost:8090/contexts\n    [\"test-context\"]⏎\n\nNow let's run the job in the context and get the results back right away:\n\n    curl -d \"input.string = a b c a b see\" \"localhost:8090/jobs?appName=test\u0026classPath=spark.jobserver.WordCountExample\u0026context=test-context\u0026sync=true\"\n    {\n      \"result\": {\n        \"a\": 2,\n        \"b\": 2,\n        \"c\": 1,\n        \"see\": 1\n      }\n    }⏎\n\nNote the addition of `context=` and `sync=true`.\n\n### Debug mode\nSpark job server is started using SBT Revolver (which forks a new JVM), so debugging directly in an IDE is not feasible.\nTo enable debugging, the Spark job server should be started from the SBT shell with the following Java options :\n```bash\njob-server-extras/reStart /absolute/path/to/your/dev.conf --- -Xdebug -Xrunjdwp:transport=dt_socket,address=15000,server=y,suspend=y\n```\nThe above command starts a remote debugging server on port 15000. The Spark job server is not started until a debugging client\n(Intellij, Eclipse, telnet, ...) connects to the exposed port.\n\nIn your IDE you just have to start a Remote debugging debug job and use the above defined port. Once the client connects to the debugging server the Spark job server is started and you can start adding breakpoints and debugging requests.\n\nNote that you might need to adjust some server parameters to avoid short Spary/Akka/Spark timeouts, in your `dev.conf` add the following values :\n```bash\nspark {\n  jobserver {\n    # Dev debug timeouts\n    context-creation-timeout = 1000000 s\n    yarn-context-creation-timeout = 1000000 s\n    default-sync-timeout = 1000000 s\n  }\n\n  context-settings {\n    # Dev debug timeout\n    context-init-timeout = 1000000 s\n  }\n}\nakka.http.server {\n      # Debug timeouts\n      idle-timeout = infinite\n      request-timeout = infinite\n}\n```\n\nAdditionally, you might have to increase the Akka Timeouts by adding the following query parameter `timeout=1000000` in your HTTP requests :\n```bash\ncurl -d \"input.string = a b c a b see\" \"localhost:8090/jobs?appName=test\u0026classPath=spark.jobserver.WordCountExample\u0026sync=true\u0026timeout=100000\"\n```\n\n## Create a Job Server Project\n### Creating a project from scratch using giter8 template\n\nThere is a giter8 template available at https://github.com/spark-jobserver/spark-jobserver.g8\n\n    $ sbt new spark-jobserver/spark-jobserver.g8\n\nAnswer the questions to generate a project structure for you. This contains Word Count example spark job using both old API and new one.\n\n    $ cd /path/to/project/directory\n    $ sbt package\n\nNow you could remove example application and start adding your one.\n\n### Creating a project manually assuming that you already have sbt project structure\nIn your `build.sbt`, add this to use the job server jar:\n\n        resolvers += \"Artifactory\" at \"https://sparkjobserver.jfrog.io/artifactory/jobserver/\"\n\n        libraryDependencies += \"spark.jobserver\" %% \"job-server-api\" % \"0.11.1\" % \"provided\"\n\nIf a SQL or Hive job/context is desired, you also want to pull in `job-server-extras`:\n\n    libraryDependencies += \"spark.jobserver\" %% \"job-server-extras\" % \"0.11.1\" % \"provided\"\n\nFor most use cases it's better to have the dependencies be \"provided\" because you don't want SBT assembly to include the whole job server jar.\n\nTo create a job that can be submitted through the job server, the job must implement the `SparkJob` trait.\nYour job will look like:\n```scala\nobject SampleJob extends SparkJob {\n    override def runJob(sc: SparkContext, jobConfig: Config): Any = ???\n    override def validate(sc: SparkContext, config: Config): SparkJobValidation = ???\n}\n```\n\n- `runJob` contains the implementation of the Job. The SparkContext is managed by the JobServer and will be provided to the job through this method.\n  This relieves the developer from the boiler-plate configuration management that comes with the creation of a Spark job and allows the Job Server to\nmanage and re-use contexts.\n- `validate` allows for an initial validation of the context and any provided configuration. If the context and configuration are OK to run the job, returning `spark.jobserver.SparkJobValid` will let the job execute, otherwise returning `spark.jobserver.SparkJobInvalid(reason)` prevents the job from running and provides means to convey the reason of failure. In this case, the call immediately returns an `HTTP/1.1 400 Bad Request` status code.\n`validate` helps you preventing running jobs that will eventually fail due to missing or wrong configuration and save both time and resources.\n\n### NEW SparkJob API\n\nNote: As of version 0.7.0, a new SparkJob API that is significantly better than the old SparkJob API will take over.  Existing jobs should continue to compile against the old `spark.jobserver.SparkJob` API, but this will be deprecated in the future.  Note that jobs before 0.7.0 will need to be recompiled, older jobs may not work with the current SJS example.  The new API looks like this:\n\n```scala\nobject WordCountExampleNewApi extends NewSparkJob {\n  type JobData = Seq[String]\n  type JobOutput = collection.Map[String, Long]\n\n  def runJob(sc: SparkContext, runtime: JobEnvironment, data: JobData): JobOutput =\n    sc.parallelize(data).countByValue\n\n  def validate(sc: SparkContext, runtime: JobEnvironment, config: Config):\n    JobData Or Every[ValidationProblem] = {\n    Try(config.getString(\"input.string\").split(\" \").toSeq)\n      .map(words =\u003e Good(words))\n      .getOrElse(Bad(One(SingleProblem(\"No input.string param\"))))\n  }\n}\n```\n\nIt is much more type safe, separates context configuration, job ID, named objects, and other environment variables into a separate JobEnvironment input, and allows the validation method to return specific data for the runJob method.  See the [WordCountExample](job-server-tests/src/main/scala/spark/jobserver/WordCountExample.scala) and [LongPiJob](job-server-tests/src/main/scala/spark/jobserver/LongPiJob.scala) for examples.\n\nLet's try running our sample job with an invalid configuration:\n\n    curl -i -d \"bad.input=abc\" \"localhost:8090/jobs?appName=test\u0026classPath=spark.jobserver.WordCountExample\"\n    HTTP/1.1 400 Bad Request\n    Server: spray-can/1.3.4\n    Date: Thu, 14 Sep 2017 12:01:37 GMT\n    Access-Control-Allow-Origin: *\n    Content-Type: application/json; charset=UTF-8\n    Content-Length: 738\n\n    {\n      \"status\": \"VALIDATION FAILED\",\n      \"result\": {\n        \"message\": \"One(SparkJobInvalid(No input.string config param))\",\n        \"errorClass\": \"java.lang.Throwable\",\n        \"stack\": \"java.lang.Throwable: One(SparkJobInvalid(No input.string config param))\\n\\tat spark.jobserver.JobManagerActor$$anonfun$getJobFuture$4.apply(JobManagerActor.scala:327)\\n\\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)\\n\\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)\\n\\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\\n\\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\\n\\tat java.lang.Thread.run(Thread.java:748)\\n\"\n      }\n    }\n\n### NEW SparkJob API with Spark v2.1\n\nDeploying Spark JobServer with Spark v2.x cluster, you can create a SparkSession context which enables Spark-SQL and Hive support\n```scala\ncurl -i -d \"\" 'http://localhost:8090/contexts/sql-context-1?num-cpu-cores=2\u0026memory-per-node=512M\u0026context-factory=spark.jobserver.context.SessionContextFactory'\n```\nSpark JobServer application shall extend from the SparkSessionJob to use the spark.jobserver.context.SessionContextFactory, here is an example\n```scala\nimport com.typesafe.config.Config\nimport org.apache.spark.sql.SparkSession\nimport org.scalactic._\nimport spark.jobserver.SparkSessionJob\nimport spark.jobserver.api.{JobEnvironment, SingleProblem, ValidationProblem}\n\nimport scala.util.Try\n\nobject WordCountExampleSparkSession extends SparkSessionJob {\n  type JobData = Seq[String]\n  type JobOutput = collection.Map[String, Long]\n\n  override def runJob(sparkSession: SparkSession, runtime: JobEnvironment, data: JobData): JobOutput =\n    sparkSession.sparkContext.parallelize(data).countByValue\n\n  override def validate(sparkSession: SparkSession, runtime: JobEnvironment, config: Config):\n  JobData Or Every[ValidationProblem] = {\n    Try(config.getString(\"input.string\").split(\" \").toSeq)\n      .map(words =\u003e Good(words))\n      .getOrElse(Bad(One(SingleProblem(\"No input.string param\"))))\n  }\n}\n```\n\n### Dependency jars\n\nFor Java/Scala applications you have a couple options to package and upload dependency jars.\n\n* The easiest is to use something like [sbt-assembly](https://github.com/sbt/sbt-assembly) to produce a fat jar.  Be sure to mark the Spark and job-server dependencies as \"provided\" so it won't blow up the jar size.  This works well if the number of dependencies is not large.\n* When the dependencies are sizeable and/or you don't want to load them with every different job, you can package the dependencies separately and use one of several options:\n    - Use the `dependent-jar-uris` context configuration param. Then the jar gets loaded for every job.\n    - The `dependent-jar-uris` can also be used in job configuration param when submitting a job. On an ad-hoc context this has the same effect as `dependent-jar-uris` context configuration param. On a persistent context the jars will be loaded for the current job and then for every job that will be executed on the persistent context.\n        ````\n        curl -d \"\" \"localhost:8090/contexts/test-context?num-cpu-cores=4\u0026memory-per-node=512m\"\n        OK⏎\n        ````\n        ````\n        curl \"localhost:8090/jobs?appName=test\u0026classPath=spark.jobserver.WordCountExample\u0026context=test-context\u0026sync=true\" -d '{\n            dependent-jar-uris = [\"file:///myjars/deps01.jar\", \"file:///myjars/deps02.jar\"],\n            input.string = \"a b c a b see\"\n        }'\n        ````\n        The jars /myjars/deps01.jar \u0026 /myjars/deps02.jar (present only on the SJS node) will be loaded and made available for the Spark driver \u0026 executors.\n        Please note that only only `file`, `local`, `ftp`, `http` protocols will work (URIs will be added to standard java class loader).\n        Recent changes also allow to use names of the binaries, which were uploaded to Jobserver.\n    - Use the `--package` option with Maven coordinates with `server_start.sh`.\n    - Recent changes also allow you to use new parameters for the `POST /jobs` request:\n      ````\n      POST /jobs?cp=someURI,binName1,binName2\u0026mainClass=some.main.Class\n      ````\n      `cp` accepts list of binary names (under which you uploaded binary to Jobserver) and URIs,\n      `mainClass` is the main class of your application.\n      Main advantage of this approach in comparison to using `dependent-jar-uris` is that you don't need to\n      specify which jar is the main one and can just send all of needed jars in one list.\n\n### Named Objects\n#### Using Named RDDs\nInitially, the job server only supported Named RDDs. For backwards compatibility and convenience, the following is still supported even though it is now possible to use the more generic Named Object support described in the next section.\n\nNamed RDDs are a way to easily share RDDs among jobs. Using this facility, computed RDDs can be cached with a given name and later on retrieved.\nTo use this feature, the SparkJob needs to mixin `NamedRddSupport`:\n```scala\nobject SampleNamedRDDJob  extends SparkJob with NamedRddSupport {\n    override def runJob(sc:SparkContext, jobConfig: Config): Any = ???\n    override def validate(sc:SparkContext, config: Config): SparkJobValidation = ???\n}\n```\n\nThen in the implementation of the job, RDDs can be stored with a given name:\n```scala\nthis.namedRdds.update(\"french_dictionary\", frenchDictionaryRDD)\n```\nOther job running in the same context can retrieve and use this RDD later on:\n```scala\nval rdd = this.namedRdds.get[(String, String)](\"french_dictionary\").get\n```\n(note the explicit type provided to get. This will allow to cast the retrieved RDD that otherwise is of type RDD[_])\n\nFor jobs that depends on a named RDDs it's a good practice to check for the existence of the NamedRDD in the `validate` method as explained earlier:\n```scala\ndef validate(sc:SparkContext, config: Config): SparkJobValidation = {\n  ...\n  val rdd = this.namedRdds.get[(Long, scala.Seq[String])](\"dictionary\")\n  if (rdd.isDefined) SparkJobValid else SparkJobInvalid(s\"Missing named RDD [dictionary]\")\n}\n```\n#### Using Named Objects\nNamed Objects are a way to easily share RDDs, DataFrames or other objects among jobs. Using this facility, computed objects can be cached with a given name and later on retrieved.\nTo use this feature, the SparkJob needs to mixin `NamedObjectSupport`. It is also necessary to define implicit persisters for each desired type of named objects. For convencience, we have provided implementations for RDD persistence and for DataFrame persistence (defined in `job-server-extras`):\n```scala\nobject SampleNamedObjectJob  extends SparkJob with NamedObjectSupport {\n\n  implicit def rddPersister[T] : NamedObjectPersister[NamedRDD[T]] = new RDDPersister[T]\n  implicit val dataFramePersister = new DataFramePersister\n\n    override def runJob(sc:SparkContext, jobConfig: Config): Any = ???\n    override def validate(sc:SparkContext, config: Config): SparkJobValidation = ???\n}\n```\n\nThen in the implementation of the job, RDDs can be stored with a given name:\n```scala\nthis.namedObjects.update(\"rdd:french_dictionary\", NamedRDD(frenchDictionaryRDD, forceComputation = false, storageLevel = StorageLevel.NONE))\n```\nDataFrames can be stored like so:\n```scala\nthis.namedObjects.update(\"df:some df\", NamedDataFrame(frenchDictionaryDF, forceComputation = false, storageLevel = StorageLevel.NONE))\n```\nIt is advisable to use different name prefixes for different types of objects to avoid confusion.\n\nAnother job running in the same context can retrieve and use these objects later on:\n```scala\nval NamedRDD(frenchDictionaryRDD, _ ,_) = namedObjects.get[NamedRDD[(String, String)]](\"rdd:french_dictionary\").get\n\nval NamedDataFrame(frenchDictionaryDF, _, _) = namedObjects.get[NamedDataFrame](\"df:some df\").get\n\n```\n(Note the explicit type provided to get. This will allow to cast the retrieved RDD/DataFrame object to the proper result type.)\n\nFor jobs that depends on a named objects it's a good practice to check for the existence of the NamedObject in the `validate` method as explained earlier:\n```scala\ndef validate(sc:SparkContext, config: Config): SparkJobValidation = {\n  ...\n  val obj = this.namedObjects.get(\"dictionary\")\n  if (obj.isDefined) SparkJobValid else SparkJobInvalid(s\"Missing named object [dictionary]\")\n}\n```\n\n### HTTPS / SSL Configuration\n#### Server authentication\nTo activate server authentication and ssl communication, set these flags in your application.conf file (Section 'akka.http.server'):\n```\n  ssl-encryption = on\n  # absolute path to keystore file\n  keystore = \"/some/path/sjs.jks\"\n  keystorePW = \"changeit\"\n```\n\nYou will need a keystore that contains the server certificate. The bare minimum is achieved with this command which creates a self-signed certificate:\n```\n keytool -genkey -keyalg RSA -alias jobserver -keystore ~/sjs.jks -storepass changeit -validity 360 -keysize 2048\n```\nYou may place the keystore anywhere.\nHere is an example of a simple curl command that utilizes ssl:\n```\ncurl -k https://localhost:8090/contexts\n```\nThe ```-k``` flag tells curl to \"Allow connections to SSL sites without certs\". Export your server certificate and import it into the client's truststore to fully utilize ssl security.\n\n#### Client authentication\nClient authentication can be enabled by simply pointing Job Server to a valid Trust Store. \nAs for server authentication, this is done by setting appropriate values in the application.conf.\nThe minimum set of parameters to enable client authentication consists of:\n```\n  # truststore = \"/some/path/server-truststore.jks\"\n  # truststorePW = \"changeit\"\n```\nNote, client authentication implies server authentication, therefore client authentication will only be enabled once server authentication is activated.\n\n### Access Control\nBy default, access to the Job Server is not limited. Basic authentication (username and password) support is provided\nvia the [Apache Shiro](http://shiro.apache.org/index.html) framework or [Keycloak](https://www.keycloak.org/). Both\nauthentication frameworks have to be explicitly activated in the configuration file. \n\nAfter the configuration, you can provide credentials via basic auth. Here is an example of a simple curl command that\nauthenticates a user and uses ssl (you may want to use -H to hide the credentials, this is just a simple example to get\nyou started):\n```\ncurl -k --basic --user 'user:pw' https://localhost:8090/contexts\n```\n\n#### Shiro Authentication\nThe Shiro Authenticator can be activated in the configuration file by changing the authentication provider and\nproviding a shiro configuration file.\n```\naccess-control {\n  provider = spark.jobserver.auth.ShiroAccessControl\n\n  # absolute path to shiro config file, including file name\n  shiro.config.path = \"/some/path/shiro.ini\"\n}\n```\nShiro-specific configuration options should be placed into a file named 'shiro.ini' in the directory as specified by the config option 'config.path'.\nHere is an example that configures LDAP with user group verification:\n```\n# use this for basic ldap authorization, without group checking\n# activeDirectoryRealm = org.apache.shiro.realm.ldap.JndiLdapRealm\n# use this for checking group membership of users based on the 'member' attribute of the groups:\nactiveDirectoryRealm = spark.jobserver.auth.LdapGroupRealm\n# search base for ldap groups (only relevant for LdapGroupRealm):\nactiveDirectoryRealm.contextFactory.environment[ldap.searchBase] = dc=xxx,dc=org\n# allowed groups (only relevant for LdapGroupRealm):\nactiveDirectoryRealm.contextFactory.environment[ldap.allowedGroups] = \"cn=group1,ou=groups\", \"cn=group2,ou=groups\"\nactiveDirectoryRealm.contextFactory.environment[java.naming.security.credentials] = password\nactiveDirectoryRealm.contextFactory.url = ldap://localhost:389\nactiveDirectoryRealm.userDnTemplate = cn={0},ou=people,dc=xxx,dc=org\n\ncacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager\n\nsecurityManager.cacheManager = $cacheManager\n```\n\nMake sure to edit the url, credentials, userDnTemplate, ldap.allowedGroups and ldap.searchBase settings in accordance with your local setup.\n\nThe Shiro authenticator is able to perform fine-grained [user authorization](#user-authorization). Permissions are\nextracted from the provided user roles. Each role that matches a known permission is added to the authenticated user.\nUnknown roles are ignored. For a list of available permissions see [Permissions](doc/permissions.md).\n\n#### Keycloak Authentication\nThe Keycloak Authenticator can be activated in the configuration file by changing the authentication provider and\nproviding a keycloak configuration.\n```\naccess-control {\n  provider = spark.jobserver.auth.KeycloakAccessControl\n\n  keycloak {\n    authServerUrl = \"https://example.com\"\n    realmName = \"master\"\n    client = \"client\"\n    clientSecret = \"secret\"\n  }\n}\n```\n| Name          | Description                                | Mandatory |\n|---------------|--------------------------------------------|-----------|\n| authServerUrl | The URL to reach the keycloak instance.    | yes       |\n| realmName     | The realm to authenticate against.         | yes       |\n| client        | The client to authenticate against.        | yes       |\n| clientSecret  | An according client secret, if it exists.  | no        |\n\nFor better performance, authentication requests against Keycloak can be cached locally.\n\nThe Keycloak authenticator is able to perform fine-grained [user authorization](#user-authorization). Permissions are\nextracted from the provided client's roles. Each client role that matches a known permission is added to the\nauthenticated user. Unknown client roles are ignored. For a list of available permissions see\n[Permissions](doc/permissions.md).\n\n*Important:* If no client role matches a permission, the user is assigned the `ALLOW_ALL` role.\n\n### User Authorization\nSpark job server implements a basic authorization management system to control access to single resources. By default,\nusers always have access to all resources (`ALLOW_ALL`). Authorization is implemented by checking the *permissions* of a\nuser with the required permissions of an endpoint. For a detailed list of all available permissions see\n[Permissions](doc/permissions.md).\n\n## Deployment\n\nSee also running on [cluster](doc/cluster.md), [YARN client](doc/yarn.md), on [EMR](doc/EMR.md) and running on [Mesos](doc/mesos.md).\n\n### Manual steps\n\n1. Copy `config/local.sh.template` to `\u003cenvironment\u003e.sh` and edit as appropriate.  NOTE: be sure to set SPARK_VERSION if you need to compile against a different version.\n2. Copy `config/shiro.ini.template` to `shiro.ini` and edit as appropriate. NOTE: only required when `access-control.provider = spark.jobserver.auth.ShiroAccessControl`\n3. Copy `config/local.conf.template` to `\u003cenvironment\u003e.conf` and edit as appropriate.\n4. `bin/server_deploy.sh \u003cenvironment\u003e` -- this packages the job server along with config files and pushes\n   it to the remotes you have configured in `\u003cenvironment\u003e.sh`\n5. On the remote server, start it in the deployed directory with `server_start.sh` and stop it with `server_stop.sh`\n\nThe `server_start.sh` script uses `spark-submit` under the hood and may be passed any of the standard extra arguments from `spark-submit`.\n\nNOTE: Under the hood, the deploy scripts generate an assembly jar from the `job-server-extras` project.  Generating assemblies from other projects may not include all the necessary components for job execution.\n\n### Context per JVM\n\nEach context can be a separate process launched using SparkLauncher, if `context-per-jvm` is set to true.\nThis can be especially desirable when you want to run many contexts at once, or for certain types of contexts such as StreamingContexts which really need their own processes.\n\nAlso, the extra processes talk to the master HTTP process via random ports using the Akka Cluster gossip protocol.  If for some reason the separate processes causes issues, set `spark.jobserver.context-per-jvm` to `false`, which will cause the job server to use a single JVM for all contexts.\n\nAmong the known issues:\n- Launched contexts do not shut down by themselves.  You need to manually kill each separate process, or do `-X DELETE /contexts/\u003ccontext-name\u003e`\n\nLog files are separated out for each context (assuming `context-per-jvm` is `true`) in their own subdirs under the `LOG_DIR` configured in `settings.sh` in the deployed directory.\n\nNote: to test out the deploy to a local staging dir, or package the job server for Mesos,\nuse `bin/server_package.sh \u003cenvironment\u003e`.\n\n### Configuring Spark Jobserver backend\n\nPlease visit [setting up dao](doc/dao-setup.md) documentation page. Currently supported backend options:\n- H2\n- MySQL\n- PostgreSQL\n- HDFS for binaries with Zookeeper for metadata\n- HDFS for binaries with H2/MySQL/PostgreSQL for metadata\n\n### HA Deployment (beta)\n\nIt is possible to run multiple Spark Jobservers in a highly available setup. For a documentation of a Jobserver HA setup, refer to the [Jobserver HA documentation](doc/HA.md).\n\n### Chef\n\nThere is also a [Chef cookbook](https://github.com/spark-jobserver/chef-spark-jobserver) which can be used to deploy Spark Jobserver.\n\n## Architecture\n\nThe job server is intended to be run as one or more independent processes, separate from the Spark cluster\n(though it very well may be collocated with say the Master).\n\nAt first glance, it seems many of these functions (eg job management) could be integrated into the Spark standalone master.  While this is true, we believe there are many significant reasons to keep it separate:\n\n- We want the job server to work for Mesos and YARN as well\n- Spark and Mesos masters are organized around \"applications\" or contexts, but the job server supports running many discrete \"jobs\" inside a single context\n- We want it to support Shark functionality in the future\n- Loose coupling allows for flexible HA arrangements (multiple job servers targeting same standalone master, or possibly multiple Spark clusters per job server)\n\nFlow diagrams are checked in in the doc/ subdirectory.  .diagram files are for websequencediagrams.com... check them out, they really will help you understand the flow of messages between actors.\n\n## API\n\nA comprehensive (manually created) swagger specification of the spark jobserver WebApi can be found [here](doc/WebApi/).\n\n### Binaries\n\n    GET /binaries               - lists all current binaries\n    GET /binaries /\u003cappName\u003e    - gets info about the last binary uploaded under this name (app-name, binary-type, upload-time)\n    POST /binaries/\u003cappName\u003e    - upload a new binary file\n    DELETE /binaries/\u003cappName\u003e  - delete defined binary\n\nWhen POSTing new binaries, the content-type header must be set to one of the types supported by the subclasses of the `BinaryType` trait. e.g. \"application/java-archive\",  \"application/python-egg\" or \"application/python-wheel\". If you are using curl command, then you must pass for example \"-H 'Content-Type: application/python-wheel'\".\n\n### Contexts\n\n    GET /contexts               - lists all current contexts\n    GET /contexts/\u003cname\u003e        - gets info about a context, such as the spark UI url\n    POST /contexts/\u003cname\u003e       - creates a new context\n    DELETE /contexts/\u003cname\u003e     - stops a context and all jobs running in it. Additionally, you can pass ?force=true to stop a context forcefully. This is equivalent to killing the application from SparkUI (works for spark standalone only).\n    PUT /contexts?reset=reboot  - shuts down all contexts and re-loads only the contexts from config. Use ?sync=false to execute asynchronously.\n\nSpark context configuration params can follow `POST /contexts/\u003cname\u003e` as query params. See section below for more details.\n\n### Jobs\n\nJobs submitted to the job server must implement a `SparkJob` trait.  It has a main `runJob` method which is\npassed a SparkContext and a typesafe Config object.  Results returned by the method are made available through\nthe REST API.\n\n    GET /jobs                - Lists the last N jobs\n    POST /jobs               - Starts a new job, use ?sync=true to wait for results\n    GET /jobs/\u003cjobId\u003e        - Gets the result or status of a specific job\n    DELETE /jobs/\u003cjobId\u003e     - Kills the specified job\n    GET /jobs/\u003cjobId\u003e/config - Gets the job configuration\n\nFor additional information on `POST /jobs` check out [submitting jobs documentation](doc/submitting-jobs.md).\n\nFor details on the Typesafe config format used for input (JSON also works), see the [Typesafe Config docs](https://github.com/typesafehub/config).\n\n### Data\n\nIt is sometime necessary to programmatically upload files to the server. Use these paths to manage such files:\n\n    GET /data                - Lists previously uploaded files that were not yet deleted\n    POST /data/\u003cprefix\u003e      - Uploads a new file, the full path of the file on the server is returned, the\n                               prefix is the prefix of the actual filename used on the server (a timestamp is\n                               added to ensure uniqueness)\n    DELETE /data/\u003cfilename\u003e  - Deletes the specified file (only if under control of the JobServer)\n    PUT /data?reset=reboot   - Deletes all uploaded files. Use ?sync=false to execute asynchronously.\n\nThese files are uploaded to the server and are stored in a local temporary\ndirectory where the JobServer runs. The POST command returns the full\npathname and filename of the uploaded file so that later jobs can work with this\njust the same as with any other server-local file. A job could therefore add this file to HDFS or distribute\nit to worker nodes via the SparkContext.addFile command.\nFor files that are larger than a few hundred MB, it is recommended to manually upload these files to the server or\nto directly add them to your HDFS.\n\n#### Data API Example\n\n    $ curl -d \"Test data file api\" http://localhost:8090/data/test_data_file_upload.txt\n    {\n      \"result\": {\n        \"filename\": \"/tmp/spark-jobserver/upload/test_data_file_upload.txt-2016-07-04T09_09_57.928+05_30.dat\"\n      }\n    }\n\n    $ curl http://localhost:8090/data\n    [\"/tmp/spark-jobserver/upload/test_data_file_upload.txt-2016-07-04T09_09_57.928+05_30.dat\"]\n\n    $ curl -X DELETE http://localhost:8090/data/%2Ftmp%2Fspark-jobserver%2Fupload%2Ftest_data_file_upload.txt-2016-07-04T09_09_57.928%2B05_30.dat\n    OK\n\n    $ curl http://localhost:8090/data\n    []\n\nNote: Both POST and DELETE requests takes URI encoded file names.\n\n### Context configuration\n\nA number of context-specific settings can be controlled when creating a context (POST /contexts) or running an\nad-hoc job (which creates a context on the spot).  For example, add urls of dependent jars for a context.\n\n    POST '/contexts/my-new-context?dependent-jar-uris=file:///some/path/of/my-foo-lib.jar'\n\nNOTE: Only the latest `dependent-jar-uris` (btw it’s jar-uris, not jars-uri) takes effect.  You can specify multiple URIs by comma-separating them.  So like this:\n\n    \u0026dependent-jar-uris=file:///path/a.jar,file:///path/b.jar\n\nWhen creating a context via POST /contexts, the query params are used to override the default configuration in\nspark.context-settings.  For example,\n\n    POST /contexts/my-new-context?num-cpu-cores=10\n\nwould override the default spark.context-settings.num-cpu-cores setting.\n\nWhen starting a job, and the `context=` query param is not specified, then an ad-hoc context is created.  Any\nsettings specified in spark.context-settings will override the defaults in the job server config when it is\nstarted up.\n\nAny spark configuration param can be overridden either in POST /contexts query params, or through `spark\n.context-settings` job configuration.  In addition, `num-cpu-cores` maps to `spark.cores.max`, and `mem-per-\nnode` maps to `spark.executor.memory`.  Therefore the following are all equivalent:\n\n    POST /contexts/my-new-context?num-cpu-cores=10\n\n    POST /contexts/my-new-context?spark.cores.max=10\n\nor in the job config when using POST /jobs,\n\n    spark.context-settings {\n        spark.cores.max = 10\n    }\n\nUser impersonation for an already Kerberos authenticated user is supported via `spark.proxy.user` query param:\n\n  POST /contexts/my-new-context?spark.proxy.user=\u003cuser-to-impersonate\u003e\n\nHowever, whenever the flag `access-control.shiro.use-as-proxy-user` is set to `on` (and Shiro is used as provider) then this parameter\nis ignored and the name of the authenticated user is *always* used as the value of the `spark.proxy.user`\nparameter when creating contexts.\n\nTo pass settings directly to the sparkConf that do not use the \"spark.\" prefix \"as-is\", use the \"passthrough\" section.\n\n    spark.context-settings {\n        spark.cores.max = 10\n        passthrough {\n          some.custom.hadoop.config = \"192.168.1.1\"\n        }\n    }\n\nTo add to the underlying Hadoop configuration in a Spark context, add the hadoop section to the context settings\n\n    spark.context-settings {\n        hadoop {\n            mapreduce.framework.name = \"Foo\"\n        }\n    }\n\n`stop-context-on-job-error=true` can be passed to context if you want the context to stop immediately after first error is reported by a job. The default value is false but for StreamingContextFactory the default is true.\nYou can also change the default value globally in application.conf `context-settings` section.\n\nFor the exact context configuration parameters, see JobManagerActor docs as well as application.conf.\n\n### Other configuration settings\n\nFor all of the Spark Job Server configuration settings, see `job-server/src/main/resources/application.conf`.\n\n### Job Result Serialization\n\nThe result returned by the `SparkJob` `runJob` method is serialized by the job server into JSON for routes\nthat return the result (GET /jobs with sync=true, GET /jobs/\u003cjobId\u003e).  Currently the following types can be\nserialized properly:\n\n- String, Int, Long, Double, Float, Boolean\n- Scala Map's with string key values (non-string keys may be converted to strings)\n- Scala Seq's\n- Array's\n- Anything that implements Product (Option, case classes) -- they will be serialized as lists\n- Subclasses of java.util.List\n- Subclasses of java.util.Map with string key values (non-string keys may be converted to strings)\n- Maps, Seqs, Java Maps and Java Lists may contain nested values of any of the above\n- If a job result is of scala's Stream[Byte] type it will be serialised directly as a chunk encoded stream.\n  This is useful if your job result payload is large and may cause a timeout serialising as objects. Beware, this\n  will not currently work as desired with context-per-jvm=true configuration, since it would require serialising\n  Stream[\\_] blob between processes. For now use Stream[\\_] job results in context-per-jvm=false configuration, pending\n  potential future enhancements to support this in context-per-jvm=true mode.\n\nIf we encounter a data type that is not supported, then the entire result will be serialized to a string.\n\n### HTTP Override\n\nSpark Job Server offers HTTP override functionality.\nOften reverse proxies and firewall implement access limitations to, for example, DELETE and PUT requests.\nHTTP override allows overcoming these limitations by wrapping, for example, a DELETE request into a POST request.\n\nRequesting the destruction of a context can be accomplished through HTTP override using the following syntax:\n\n    $ curl -X POST \"localhost:8090/contexts/test_context?_method=DELETE\"\n\nHere, a DELETE request is passed to Spark Job Server \"through\" a POST request.\n\n\n## Clients\n\nSpark Jobserver project has a\n[python binding](https://github.com/spark-jobserver/python-sjsclient) package.\nThis can be used to quickly develop python applications that can interact with\nSpark Jobserver programmatically.\n\n## Contribution and Development\nContributions via Github Pull Request are welcome. Please start by taking a look at the [contribution guidelines](doc/contribution-guidelines.md) and check the TODO for some contribution ideas.\n\n- If you need to build with a specific scala version use ++x.xx.x followed by the regular command,\nfor instance: `sbt ++2.12.12 job-server/compile`\n- From the \"master\" project, please run \"test\" to ensure nothing is broken.\n   - You may need to set `SPARK_LOCAL_IP` to `localhost` to ensure Akka port can bind successfully\n   - Note for Windows users: very few tests fail on Windows. Thus, run `testOnly -- -l WindowsIgnore` from SBT shell to ignore them.\n- Logging for tests goes to \"job-server-test.log\". To see test logging in console also, add the following to your log4j.properties (`job-server/src/test/resources/log4j.properties`)\n```$xslt\nlog4j.rootLogger=INFO, LOGFILE, console\n\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\nlog4j.appender.console.target=System.err\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\nlog4j.appender.console.layout.ConversionPattern=[%d] %-5p %.26c [%X{testName}] [%X{akkaSource}] - %m%n\n```\n- Run `sbt clean coverage test` to check the code coverage and improve it. You can generate reports by running\n`sbt coverageReport` or `sbt coverageAggregate` for the full overview.\n  - Windows users: run `; coverage ; testOnly -- -l WindowsIgnore ; coverageReport` from SBT shell.\n- Please run scalastyle to ensure your code changes don't break the style guide.\n- Do \"reStart\" from SBT for quick restarts of the job server process\n- Please update the g8 template if you change the SparkJob API\n\nProfiling software generously provided by ![](https://www.yourkit.com/images/yklogo.png)\n\nYourKit supports open source projects with its full-featured [Java Profiler](https://www.yourkit.com/java/profiler/index.jsp).\n\n\n## Contact\n\nFor user/dev questions, we are using google group for discussions:\n\u003chttps://groups.google.com/forum/#!forum/spark-jobserver\u003e\n\nPlease report bugs/problems to:\n\u003chttps://github.com/spark-jobserver/spark-jobserver/issues\u003e\n\n## License\nApache 2.0, see LICENSE.md\n\n## TODO\n\n- More debugging for classpath issues\n- Add Swagger support.  See the spray-swagger project.\n- Implement an interactive SQL window.  See: [spark-admin](https://github.com/adatao/spark-admin)\n\n- Stream the current job progress via a Listener\n- Add routes to return stage info for a job.  Persist it via DAO so that we can always retrieve stage / performance info\n  even for historical jobs.  This would be pretty kickass.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspark-jobserver%2Fspark-jobserver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspark-jobserver%2Fspark-jobserver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspark-jobserver%2Fspark-jobserver/lists"}