{"id":13905894,"url":"https://github.com/svenkreiss/pysparkling","last_synced_at":"2025-04-07T05:05:58.888Z","repository":{"id":31775591,"uuid":"35341934","full_name":"svenkreiss/pysparkling","owner":"svenkreiss","description":"A pure Python implementation of Apache Spark's RDD and DStream interfaces.","archived":false,"fork":false,"pushed_at":"2024-09-03T22:15:34.000Z","size":3620,"stargazers_count":268,"open_issues_count":11,"forks_count":44,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-31T04:03:28.597Z","etag":null,"topics":["apache-spark","data-processing","data-science","python"],"latest_commit_sha":null,"homepage":"https://pysparkling.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/svenkreiss.png","metadata":{"files":{"readme":"README.rst","changelog":"HISTORY.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-05-09T19:23:20.000Z","updated_at":"2025-01-15T15:04:11.000Z","dependencies_parsed_at":"2024-09-24T09:01:43.813Z","dependency_job_id":"4729cd54-b3ba-46fa-84a7-6ab0dc860bb3","html_url":"https://github.com/svenkreiss/pysparkling","commit_stats":{"total_commits":1454,"total_committers":10,"mean_commits":145.4,"dds":0.4092159559834938,"last_synced_commit":"431df12873bd9cf12af5f085cd7e283aabdcf097"},"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svenkreiss%2Fpysparkling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svenkreiss%2Fpysparkling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svenkreiss%2Fpysparkling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svenkreiss%2Fpysparkling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/svenkreiss","download_url":"https://codeload.github.com/svenkreiss/pysparkling/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247595331,"owners_count":20963943,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","data-processing","data-science","python"],"created_at":"2024-08-06T23:01:25.922Z","updated_at":"2025-04-07T05:05:58.853Z","avatar_url":"https://github.com/svenkreiss.png","language":"Python","funding_links":[],"categories":["Data Manipulation","数据管道和流处理"],"sub_categories":["Data Frames"],"readme":".. image:: https://raw.githubusercontent.com/svenkreiss/pysparkling/master/logo/logo-w100.png\n    :target: https://github.com/svenkreiss/pysparkling\n\npysparkling\n===========\n\n**Pysparkling** provides a faster, more responsive way to develop programs\nfor PySpark. It enables code intended for Spark applications to execute\nentirely in Python, without incurring the overhead of initializing and\npassing data through the JVM and Hadoop. The focus is on having a lightweight\nand fast implementation for small datasets at the expense of some data\nresilience features and some parallel processing features.\n\n**How does it work?** To switch execution of a script from PySpark to pysparkling,\nhave the code initialize a pysparkling Context instead of a SparkContext, and\nuse the pysparkling Context to set up your RDDs. The beauty is you don't have\nto change a single line of code after the Context initialization, because\npysparkling's API is (almost) exactly the same as PySpark's. Since it's so easy\nto switch between PySpark and pysparkling, you can choose the right tool for your\nuse case.\n\n**When would I use it?** Say you are writing a Spark application because you\nneed robust computation on huge datasets, but you also want the same application\nto provide fast answers on a small dataset. You're finding Spark is not responsive\nenough for your needs, but you don't want to rewrite an entire separate application\nfor the *small-answers-fast* problem. You'd rather reuse your Spark code but somehow\nget it to run fast. Pysparkling bypasses the stuff that causes Spark's long startup\ntimes and less responsive feel.\n\nHere are a few areas where pysparkling excels:\n\n* Small to medium-scale exploratory data analysis\n* Application prototyping\n* Low-latency web deployments\n* Unit tests\n\n\nInstall\n=======\n\n.. code-block:: bash\n\n    python3 -m pip install \"pysparkling[s3,hdfs,http,streaming]\"\n\n\n`Documentation \u003chttps://pysparkling.trivial.io\u003e`_:\n\n.. image:: https://raw.githubusercontent.com/svenkreiss/pysparkling/master/docs/readthedocs.png\n   :target: https://pysparkling.trivial.io\n\n\nOther links:\n`Github \u003chttps://github.com/svenkreiss/pysparkling\u003e`_,\n|pypi-badge|, |test-badge|, |docs-badge|\n\n.. |pypi-badge| image:: https://badge.fury.io/py/pysparkling.svg\n   :target: https://pypi.python.org/pypi/pysparkling/\n.. |test-badge| image:: https://github.com/svenkreiss/pysparkling/workflows/Tests/badge.svg\n   :target: https://github.com/svenkreiss/pysparkling/actions?query=workflow%3ATests\n.. |docs-badge| image:: https://readthedocs.org/projects/pysparkling/badge/?version=latest\n   :target: https://pysparkling.readthedocs.io/en/latest/?badge=latest\n   :alt: Documentation Status\n\n\nFeatures\n========\n\n* Supports URI schemes ``s3://``, ``hdfs://``, ``gs://``, ``http://`` and ``file://``\n  for Amazon S3, HDFS, Google Storage, web and local file access.\n  Specify multiple files separated by comma.\n  Resolves ``*`` and ``?`` wildcards.\n* Handles ``.gz``, ``.zip``, ``.lzma``, ``.xz``, ``.bz2``, ``.tar``,\n  ``.tar.gz`` and ``.tar.bz2`` compressed files.\n  Supports reading of ``.7z`` files.\n* Parallelization via ``multiprocessing.Pool``,\n  ``concurrent.futures.ThreadPoolExecutor`` or any other Pool-like\n  objects that have a ``map(func, iterable)`` method.\n* Plain pysparkling does not have any dependencies (use ``pip install pysparkling``).\n  Some file access methods have optional dependencies:\n  ``boto`` for AWS S3, ``requests`` for http, ``hdfs`` for hdfs\n\n\nExamples\n========\n\nSome demos are in the notebooks\n`docs/demo.ipynb \u003chttps://github.com/svenkreiss/pysparkling/blob/master/docs/demo.ipynb\u003e`_\nand\n`docs/iris.ipynb \u003chttps://github.com/svenkreiss/pysparkling/blob/master/docs/iris.ipynb\u003e`_\n.\n\n**Word Count**\n\n.. code-block:: python\n\n    from pysparkling import Context\n\n    counts = (\n        Context()\n        .textFile('README.rst')\n        .map(lambda line: ''.join(ch if ch.isalnum() else ' ' for ch in line))\n        .flatMap(lambda line: line.split(' '))\n        .map(lambda word: (word, 1))\n        .reduceByKey(lambda a, b: a + b)\n    )\n    print(counts.collect())\n\nwhich prints a long list of pairs of words and their counts.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsvenkreiss%2Fpysparkling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsvenkreiss%2Fpysparkling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsvenkreiss%2Fpysparkling/lists"}