{"id":28613996,"url":"https://github.com/bluebreezecf/sparkjobserverclient","last_synced_at":"2025-06-12T01:10:44.502Z","repository":{"id":20517501,"uuid":"23796322","full_name":"bluebreezecf/SparkJobServerClient","owner":"bluebreezecf","description":"Java Client of the Spark Job Server implementing the arranged Rest APIs","archived":false,"fork":false,"pushed_at":"2021-06-04T01:00:01.000Z","size":2468,"stargazers_count":49,"open_issues_count":12,"forks_count":33,"subscribers_count":7,"default_branch":"master","last_synced_at":"2023-08-30T09:24:51.298Z","etag":null,"topics":["jar","java-client","pom","spark"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bluebreezecf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-09-08T15:28:05.000Z","updated_at":"2023-08-30T09:24:51.299Z","dependencies_parsed_at":"2022-09-09T13:20:56.121Z","dependency_job_id":null,"html_url":"https://github.com/bluebreezecf/SparkJobServerClient","commit_stats":null,"previous_names":[],"tags_count":2,"template":null,"template_full_name":null,"purl":"pkg:github/bluebreezecf/SparkJobServerClient","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluebreezecf%2FSparkJobServerClient","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluebreezecf%2FSparkJobServerClient/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluebreezecf%2FSparkJobServerClient/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluebreezecf%2FSparkJobServerClient/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bluebreezecf","download_url":"https://codeload.github.com/bluebreezecf/SparkJobServerClient/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluebreezecf%2FSparkJobServerClient/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259374947,"owners_count":22847873,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jar","java-client","pom","spark"],"created_at":"2025-06-12T01:10:42.690Z","updated_at":"2025-06-12T01:10:44.447Z","avatar_url":"https://github.com/bluebreezecf.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Spark-Job-Server-Client\n\n### Backgroud\nPeople always use curl or HUE to upload jar and run spark job in Spark Job Server.\nBut the Spark Job Server official only presents the rest apis to upload job jar and \nrun a job, doesn't give client lib with any language implementation.\n\nNow there is another option to communicate with spark job server in Java, that is Spark-Job-Server-Client, the Java Client of the Spark Job Server implementing the arranged Rest APIs.\n\nSpark-Job-Server-Client is a open-source program of **com.bluebreezecf** under Apache License v2. It aims to make the java applications use the spark more easily.\n\n### How to compile\nYou can execute the following commands to compile this client:\n```shell\ngit clone https://github.com/bluebreezecf/SparkJobServerClient.git\ncd SparkJobServerClient\nmvn clean package\n```\nThen you can find`spark-job-server-client-1.0.0.jar`in SparkJobServerClient/target, it is the main jar of spark-job-server-client. Besides, `spark-job-server-client-1.0.0-sources.jar`is the java source jar, and `spark-job-server-client-1.0.0-javadoc.jar` is the java doc api jar.\n\n### How to set dependency\nExcept the DIY way of getting the latest codes and compiling it, now the real and original **spark-job-server** has been published to [http://repo1.maven.org/maven2/](http://repo1.maven.org/maven2/com/bluebreezecf/spark-job-server-client/). So everyone, who has set the url as one of the remote maven repository, can download or get it via `mvn package` after setting the following dependency into your pom.xml.\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.bluebreezecf\u003c/groupId\u003e\n    \u003cartifactId\u003espark-job-server-client\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\nOne can find all the releases of `spark-job-server-client` via [search.maven.org](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.bluebreezecf%22%20AND%20a%3A%22spark-job-server-client%22`).\n\n### How to use\nThe following sample codes shows how to use spark-job-server-client:\n\n```java\nimport java.io.File;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.bluebreezecf.tools.sparkjobserver.api.ISparkJobServerClient;\nimport com.bluebreezecf.tools.sparkjobserver.api.ISparkJobServerClientConstants;\nimport com.bluebreezecf.tools.sparkjobserver.api.SparkJobConfig;\nimport com.bluebreezecf.tools.sparkjobserver.api.SparkJobInfo;\nimport com.bluebreezecf.tools.sparkjobserver.api.SparkJobJarInfo;\nimport com.bluebreezecf.tools.sparkjobserver.api.SparkJobResult;\nimport com.bluebreezecf.tools.sparkjobserver.api.SparkJobServerClientException;\nimport com.bluebreezecf.tools.sparkjobserver.api.SparkJobServerClientFactory;\n\n/**\n * A sample shows how to use spark-job-server-client.\n * \n * @author bluebreezecf\n * @since 2014-09-16\n *\n */\npublic class SparkJobServerClientTest {\n\t\n\tpublic static void main(String[] args) {\n\t\tISparkJobServerClient client = null;\n\t\ttry {\n\t\t\tclient = SparkJobServerClientFactory.getInstance().createSparkJobServerClient(\"http://localhost:8090/\");\n\t\t\t//GET /jars\n\t\t\tList\u003cSparkJobJarInfo\u003e jarInfos = client.getJars();\n\t\t\tfor (SparkJobJarInfo jarInfo: jarInfos) {\n\t\t\t\tSystem.out.println(jarInfo.toString());\n\t\t\t}\n\n\t\t\t//POST /jars/\u003cappName\u003e\n\t\t\tclient.uploadSparkJobJar(new File(\"d:\\\\spark-examples_2.10-1.0.2.jar\"), \"spark-test\");\n\t\t\t\n\t\t\t//GET /contexts\n\t\t\tList\u003cString\u003e contexts = client.getContexts();\n\t\t\tSystem.out.println(\"Current contexts:\");\n\t\t\tfor (String cxt: contexts) {\n\t\t\t\tSystem.out.println(cxt);\n\t\t\t}\n\t\t\t\n\t\t\t//POST /contexts/\u003cname\u003e--Create context with name ctxTest and null parameter\n\t\t\tclient.createContext(\"ctxTest\", null);\n\t\t\t//POST /contexts/\u003cname\u003e--Create context with parameters\n\t\t\tMap\u003cString, String\u003e params = new HashMap\u003cString, String\u003e();\n\t\t\tparams.put(ISparkJobServerClientConstants.PARAM_MEM_PER_NODE, \"512m\");\n\t\t\tparams.put(ISparkJobServerClientConstants.PARAM_NUM_CPU_CORES, \"10\");\n\t\t\tclient.createContext(\"cxtTest2\", params);\n\t\t\t\n\t\t\t//DELETE /contexts/\u003cname\u003e\n\t\t\tclient.deleteContext(\"ctxTest\");\n\t\t\t\n\t\t\t//GET /jobs\n\t\t\tList\u003cSparkJobInfo\u003e jobInfos = client.getJobs();\n\t\t\tSystem.out.println(\"Current jobs:\");\n\t\t\tfor (SparkJobInfo jobInfo: jobInfos) {\n\t\t\t\tSystem.out.println(jobInfo);\n\t\t\t}\n\t\t\t\n\t\t\t//Post /jobs---Create a new job\n\t\t\tparams.put(ISparkJobServerClientConstants.PARAM_APP_NAME, \"spark-test\");\n\t\t\tparams.put(ISparkJobServerClientConstants.PARAM_CLASS_PATH, \"spark.jobserver.WordCountExample\");\n\t\t\t//1.start a spark job asynchronously and just get the status information\n\t\t\tSparkJobResult result = client.startJob(\"input.string= fdsafd dfsf blullkfdsoflaw fsdfs\", params);\n\t\t\tSystem.out.println(result);\n\t\t\t\n\t\t\t//2.start a spark job synchronously and wait until the result\n\t\t\tparams.put(ISparkJobServerClientConstants.PARAM_CONTEXT, \"cxtTest2\");\n\t\t\tparams.put(ISparkJobServerClientConstants.PARAM_SYNC, \"true\");\n\t\t\tresult = client.startJob(\"input.string= fdsafd dfsf blullkfdsoflaw fsdffdsfsfs\", params);\n\t\t\tSystem.out.println(result);\n\t\t\t\n\t\t\t//GET /jobs/\u003cjobId\u003e---Gets the result or status of a specific job\n\t\t\tresult = client.getJobResult(\"fdsfsfdfwfef\");\n\t\t\tSystem.out.println(result);\n\t\t\t\n\t\t\t//GET /jobs/\u003cjobId\u003e/config - Gets the job configuration\n\t\t\tSparkJobConfig jobConfig = client.getConfig(\"fdsfsfdfwfef\");\n\t\t\tSystem.out.println(jobConfig);\n\t\t} catch (SparkJobServerClientException e1) {\n\t\t\te1.printStackTrace();\n\t\t} catch (Exception e) {\n\t\t\te.printStackTrace();\n\t\t}\n\t}\n}\n```\n### How to contribute\nAnyone interested in this program can do the following things:\n 1. `Fork` it to your own git repository.\n 2. Create a new branch for your feature via `git checkout -b your-new-feature`.\n 3. Add or modify new codes.\n 4. Commit the modifications through `git commit -am 'add your new feature'`.\n 5. Push the new branch by `git push origin your-new-feature`.\n 6. Create a new `pull request`.\n\nAny questions and discussions can be added in [SparkJobServerClient/issues] (https://github.com/bluebreezecf/SparkJobServerClient/issues)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluebreezecf%2Fsparkjobserverclient","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbluebreezecf%2Fsparkjobserverclient","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluebreezecf%2Fsparkjobserverclient/lists"}