{"id":13795303,"url":"https://github.com/Tubular/sparkly","last_synced_at":"2025-05-12T22:30:40.933Z","repository":{"id":13943759,"uuid":"70260962","full_name":"tubular/sparkly","owner":"tubular","description":"Helpers \u0026 syntactic sugar for PySpark. ","archived":false,"fork":false,"pushed_at":"2023-06-26T21:09:23.000Z","size":1614,"stargazers_count":62,"open_issues_count":5,"forks_count":9,"subscribers_count":41,"default_branch":"master","last_synced_at":"2025-05-01T13:53:30.326Z","etag":null,"topics":["pyspark","python","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tubular.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-10-07T15:50:02.000Z","updated_at":"2025-04-21T17:35:14.000Z","dependencies_parsed_at":"2024-01-23T02:03:44.872Z","dependency_job_id":"2b1fdc5a-224b-4d7c-9242-0c5202b28f1f","html_url":"https://github.com/tubular/sparkly","commit_stats":{"total_commits":107,"total_committers":14,"mean_commits":7.642857142857143,"dds":0.6542056074766356,"last_synced_commit":"2c5d50e2eabc5a891120585782abecd98dec9abe"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tubular%2Fsparkly","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tubular%2Fsparkly/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tubular%2Fsparkly/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tubular%2Fsparkly/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tubular","download_url":"https://codeload.github.com/tubular/sparkly/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253833077,"owners_count":21971357,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pyspark","python","spark"],"created_at":"2024-08-03T23:00:54.414Z","updated_at":"2025-05-12T22:30:40.354Z","avatar_url":"https://github.com/tubular.png","language":"Python","funding_links":[],"categories":["Packages"],"sub_categories":["Utilities"],"readme":"Sparkly\n=======\n\n|Sparkly PyPi Version| |Documentation Status|\n\nHelpers \u0026 syntax sugar for PySpark. There are several features to make your life easier:\n\n- Definition of spark packages, external jars, UDFs and spark options within your code;\n- Simplified reader/writer api for Cassandra, Elastic, MySQL, Kafka;\n- Testing framework for spark applications.\n\nMore details could be found in `the official\ndocumentation \u003chttps://sparkly.readthedocs.org\u003e`__.\n\nInstallation\n------------\n\nSparkly itself is easy to install::\n\n    pip install pyspark  # pick your version\n    pip install sparkly (compatible with spark \u003e= 2.4)\n\n\nGetting Started\n---------------\n\nHere is a small code snippet to show how to easily read Cassandra table\nand write its content to ElasticSearch index::\n\n    from sparkly import SparklySession\n\n\n    class MySession(SparklySession):\n        packages = [\n            'datastax:spark-cassandra-connector:2.0.0-M2-s_2.11',\n            'org.elasticsearch:elasticsearch-spark-20_2.11:6.5.4',\n        ]\n\n\n    if __name__ == '__main__':\n        spark = MySession()\n        df = spark.read_ext.cassandra('localhost', 'my_keyspace', 'my_table')\n        df.write_ext.elastic('localhost', 'my_index', 'my_type')\n\nSee `the online documentation \u003chttps://sparkly.readthedocs.org\u003e`__ for\nmore details.\n\nTesting\n-------\n\nTo run tests you have to have `docker \u003chttps://www.docker.com/\u003e`__ and\n`docker-compose \u003chttps://docs.docker.com/compose/\u003e`__ installed on your\nsystem. If you are working on MacOS we highly recommend you to use\n`docker-machine \u003chttps://docs.docker.com/machine/\u003e`__. As soon as the\ntools mentioned above have been installed, all you need is to run::\n\n    make test\n\nSupported Spark Versions\n------------------------\n\nAt the moment we support:\n\n+---------------------------------------------------------------------------+\n| sparkly \u003e= 2.7 | Spark 2.4.x                                              |\n+---------------------------------------------------------------------------+\n| sparkly 2.x    | Spark 2.0.x and Spark 2.1.x and Spark 2.2.x              |\n+---------------------------------------------------------------------------+\n| sparkly 1.x    | Spark 1.6.x                                              |\n+---------------------------------------------------------------------------+\n\n.. |Sparkly PyPi Version| image:: http://img.shields.io/pypi/v/sparkly.svg\n   :target: https://pypi.python.org/pypi/sparkly\n.. |Sparkly Build Status| image:: https://app.travis-ci.com/tubular/sparkly.svg?branch=master\n   :target: https://app.travis-ci.com/github/tubular/sparkly\n.. |Documentation Status| image:: https://readthedocs.org/projects/sparkly/badge/?version=latest\n   :target: http://sparkly.readthedocs.io/en/latest/?badge=latest\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTubular%2Fsparkly","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTubular%2Fsparkly","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTubular%2Fsparkly/lists"}