{"id":17305793,"url":"https://github.com/s8sg/spark-py-submit","last_synced_at":"2026-05-01T01:31:40.616Z","repository":{"id":57469614,"uuid":"81062131","full_name":"s8sg/spark-py-submit","owner":"s8sg","description":"A python library to submit spark job in yarn cluster at different distributions (Currently CDH, HDP)","archived":false,"fork":false,"pushed_at":"2017-02-10T15:02:51.000Z","size":32,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-14T11:51:28.727Z","etag":null,"topics":["cdh","hdfs","hdp","python-library","spark","spark-clusters","spark-job"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/s8sg.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-06T07:55:38.000Z","updated_at":"2021-04-15T01:46:31.000Z","dependencies_parsed_at":"2022-09-19T09:50:30.929Z","dependency_job_id":null,"html_url":"https://github.com/s8sg/spark-py-submit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/s8sg/spark-py-submit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s8sg%2Fspark-py-submit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s8sg%2Fspark-py-submit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s8sg%2Fspark-py-submit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s8sg%2Fspark-py-submit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/s8sg","download_url":"https://codeload.github.com/s8sg/spark-py-submit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s8sg%2Fspark-py-submit/sbom","scorecard":{"id":793996,"data":{"date":"2025-08-11","repo":{"name":"github.com/s8sg/spark-py-submit","commit":"16cc184136c70118598a876c6a460c737702894d"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.6,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":0,"reason":"Found 0/14 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-23T08:24:07.130Z","repository_id":57469614,"created_at":"2025-08-23T08:24:07.131Z","updated_at":"2025-08-23T08:24:07.131Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32482460,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"ssl_error","status_checked_at":"2026-04-30T13:12:06.837Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdh","hdfs","hdp","python-library","spark","spark-clusters","spark-job"],"created_at":"2024-10-15T11:56:47.225Z","updated_at":"2026-05-01T01:31:40.598Z","avatar_url":"https://github.com/s8sg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Spark Py Submit\n===============\n\nA python library that can submit spark job to spark cluster and standalone using\nrest API\n\n| **Note: It Currently supports the CDH(5.6.1) and\n  HDP(2.3.2.0-2950,2.4.0.0-169) yarn cluster**\n| The Library is Inspired from:\n  ``github.com/bernhard-42/spark-yarn-rest-api``\n\nGetting Started:\n~~~~~~~~~~~~~~~~\n\nUse the library\n^^^^^^^^^^^^^^^\n\n.. code:: python\n\n    # Import the SparkJobHandler\n    from spark_job_handler import SparkJobHandler\n\n    ...\n\n    logger = logging.getLogger('TestLocalJobSubmit')\n    # Create a spark JOB\n    # jobName:           name of the Spark Job   \n    # jar:               location of the Jar (local/hdfs)  \n    # run_class:         entry class of the appliaction   \n    # hadoop_rm:         hadoop resource manager host ip  \n    # hadoop_web_hdfs:   hadoop web hdfs ip   \n    # hadoop_nn:         hadoop name node ip (Normally same as of web_hdfs)  \n    # env_type:          env type is CDH or HDP  \n    # local_jar:         flag to define if a jar is local (Local jar gets uploaded to hdfs)  \n    # spark_properties:  custom properties that need to be set \n    sparkJob = SparkJobHandler(logger=logger, job_name=\"test_local_job_submit\", \n                jar=\"./simple-project/target/scala-2.10/simple-project_2.10-1.0.jar\",\n                run_class=\"IrisApp\", hadoop_rm='rma', hadoop_web_hdfs='nn', hadoop_nn='nn',\n                env_type=\"CDH\", local_jar=True, spark_properties=None)\n    trackingUrl = sparkJob.run()\n    print \"Job Tracking URL: %s\" % trackingUrl\n\n| The above code starts an spark application using the local jar\n  (simple-project/target/scala-2.10/simple-project\\_2.10-1.0.jar)\n| For more example see the\n  `test\\_spark\\_job\\_handler.py \u003chttps://github.com/s8sg/spark-py-submit/blob/master/test_spark_job_handler.py\u003e`__\n\nBuild the simple-project\n^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code:: bash\n\n      $ cd simple-project\n      $ sbt package;cd ..\n\nThe above steps will create the target jar as:\n``./simple-project/target/scala-2.10/simple-project_2.10-1.0.jar``\n\nUpdate the nodes Ip in test:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n| Add the node IP for hadoop resource manager and Name node in the\n  test\\_cases:\n| \\* rm: Resource Manager \\* nn: Name Node\n\nload the data and make it available to HDFS:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code:: bash\n\n      $ wget https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data\n\nupload data to the HDFS:\n\n.. code:: bash\n\n      $ python upload_to_hdfs.py \u003cname_nodei_ip\u003e iris.data /tmp/iris.data\n\nRun the test cases:\n^^^^^^^^^^^^^^^^^^^\n\nMake the simple-project jar available in HDFS to test remote jar:\n\n.. code:: bash\n\n      $ python upload_to_hdfs.py \u003cname_nodei_ip\u003e simple-project/target/scala-2.10/simple-project_2.10-1.0.jar /tmp/test_data/simple-project_2.10-1.0.jar\n\nRun the test:\n\n.. code:: bash\n\n      $ python test_spark_job_handler.py \n\nUtility:\n~~~~~~~~\n\n-  upload\\_to\\_hdfs.py: upload local file to hdfs file system\n\nNotes:\n~~~~~~\n\n| The Library is still in early stage and need testing, bug-fixing and\n  documentation\n| Before running, follow the below steps:\n| \\* Update the ResourceManager,NameNode and WebHDFS Port if required in\n  settings.py\n| \\* Make the spark-jar available in hdfs as:\n  ``hdfs:/user/spark/share/lib/spark-assembly.jar``\n| For Contribution Please Create Issue corresponding PR at: github.com/s8sg/spark-py-submit\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs8sg%2Fspark-py-submit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fs8sg%2Fspark-py-submit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs8sg%2Fspark-py-submit/lists"}