{"id":24835781,"url":"https://github.com/tobilg/docker-spark-jobserver","last_synced_at":"2025-10-14T10:30:36.477Z","repository":{"id":86825749,"uuid":"41732213","full_name":"tobilg/docker-spark-jobserver","owner":"tobilg","description":"A Docker container for Spark Jobserver","archived":false,"fork":false,"pushed_at":"2015-11-09T11:04:52.000Z","size":168,"stargazers_count":3,"open_issues_count":0,"forks_count":8,"subscribers_count":3,"default_branch":"master","last_synced_at":"2023-02-26T19:16:35.192Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tobilg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-09-01T10:27:27.000Z","updated_at":"2023-03-08T16:17:15.875Z","dependencies_parsed_at":"2023-03-09T13:30:46.165Z","dependency_job_id":null,"html_url":"https://github.com/tobilg/docker-spark-jobserver","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fdocker-spark-jobserver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fdocker-spark-jobserver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fdocker-spark-jobserver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fdocker-spark-jobserver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tobilg","download_url":"https://codeload.github.com/tobilg/docker-spark-jobserver/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":236464817,"owners_count":19152979,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-31T04:51:45.494Z","updated_at":"2025-10-14T10:30:31.214Z","avatar_url":"https://github.com/tobilg.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark Jobserver\n\nSpark-Jobserver as a Docker container! This image has been created to support the most recent versions of Spark together with the most recent versions of Mesos. Have a look at the [\"Tags\" tab](https://hub.docker.com/r/tobilg/spark-jobserver/tags/) to see which combinations are supported.\nIt's loosely derived from `velvia/spark-jobserver`, and these docs have been adapted to match `tobilg/spark-jobserver`.\n\nTo get started:\n\n    docker run -d -p 8090:8090 tobilg/spark-jobserver:latest\n\nThis will start job server on port 8090 in a container, with H2 database and Mesos support, and expose that port to the host on which you run the container.\n\nIf you would like to debug job server using JMX / VisualVM etc., then also expose port 9999 via `-p 9999:9999`.\n\n## Configuration\n\nBy default, the container has an embedded Spark 1.4.1 distro and runs using Spark local mode (`local[4]`).\n\nTo change the spark master the container runs against, set SPARK_MASTER when you start the container:\n\n    docker run -d -p 8090:8090 -e SPARK_MASTER=mesos://zk://mesos.master:5050 tobilg/spark-jobserver:latest\n\nYou can easily change the amount of memory job server uses with `JOBSERVER_MEMORY`, or replace the entire config job server uses at startup with `JOBSERVER_CONFIG`.\n\nThe standard way to replace the config is to derive a custom Docker image from the job server one by overwriting the default config at `app/docker.conf`.  The Dockerfile would look like this:\n\n    from tobilg/spark-jobserver:latest\n    add /path/to/my/jobserver.conf /app/docker.conf\n\nSimilarly, to change the logging configuration, inherit from this container and overwrite `/app/log4j-server.properties`.\n\n## Jars / Passing Arguments to the Start Script\n\nAny `spark-submit` arguments can be passed to the tail of the `docker run` command.  A very common use of this is to add custom jars to your Spark job environment.  For example, to add the Datastax Spark-Cassandra Connector to your job:\n\n    docker run -d -p 8090:8090 tobilg/spark-jobserver:latest --packages com.datastax.spark:spark-cassandra-connector_2.10:1.3.0-M1\n\n## Database, Persistence, Logs\n\nDocker containers are usually stateless, but it wouldn't be very useful to have the jars and job config reset every time you had to kill and restart a container.\n\nThe job server docker image is configured to use H2 database by default and to write the database to a Docker volume at `/database`, which will be persisted between container restarts, and can even be shared amongst multiple job server containers on the same host. Note that in order to persist them to new containers, you need to create a local directory, something like this:\n\n    docker run -d -p 8090:8090 -v /opt/job-server-db:/database tobilg/spark-jobserver:latest\n\nSee the [Docker Volumes Guide](http://docs-stage.docker.com/userguide/dockervolumes/#volume) for more info.\n\nAnother option is to configure job server to persist metadata in PostGres, MySQL, or similar database.  To do that, create a new config, pass it into the docker container as above using `JOBSERVER_CONFIG` and the `/config` volume, and point to your shared database, perhaps using `--link` to a PostGres or MySQL container.\n\nLogging goes to stdout, as per standard Docker conventions.  Therefore:\n\n* Use `docker logs -f \u003ccontainerHash\u003e` to follow logs\n* Use `docker logs --tail=100 \u003ccontainerHash\u003e` to list the last 100 lines\n* Use Docker logging drivers to redirect logs to syslog, SumoLogic, etc.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobilg%2Fdocker-spark-jobserver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftobilg%2Fdocker-spark-jobserver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobilg%2Fdocker-spark-jobserver/lists"}