{"id":17301684,"url":"https://github.com/wh1isper/sparglim","last_synced_at":"2026-03-11T01:31:07.918Z","repository":{"id":180375158,"uuid":"665047696","full_name":"Wh1isper/sparglim","owner":"Wh1isper","description":"Sparglim✨ makes PySpark App Configurable and Deploy Spark Connect Server Easier!","archived":false,"fork":false,"pushed_at":"2026-01-19T20:46:58.000Z","size":154,"stargazers_count":42,"open_issues_count":3,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-02-21T17:30:32.594Z","etag":null,"topics":["jupyter-magic","pyspark","spark","spark-connect","spark-connect-server","spark-on-kubernetes","spark-sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Wh1isper.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":"FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["wh1isper"]}},"created_at":"2023-07-11T10:15:08.000Z","updated_at":"2026-02-06T22:31:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"aa410a72-3c12-434f-814d-4f4387146245","html_url":"https://github.com/Wh1isper/sparglim","commit_stats":{"total_commits":138,"total_committers":3,"mean_commits":46.0,"dds":0.1376811594202898,"last_synced_commit":"4edba17564b86e25fd671a127532c68cb37bf59d"},"previous_names":["wh1isper/sparglim"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/Wh1isper/sparglim","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wh1isper%2Fsparglim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wh1isper%2Fsparglim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wh1isper%2Fsparglim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wh1isper%2Fsparglim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Wh1isper","download_url":"https://codeload.github.com/Wh1isper/sparglim/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wh1isper%2Fsparglim/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30366051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T21:41:54.280Z","status":"ssl_error","status_checked_at":"2026-03-10T21:40:59.357Z","response_time":106,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jupyter-magic","pyspark","spark","spark-connect","spark-connect-server","spark-on-kubernetes","spark-sql"],"created_at":"2024-10-15T11:45:13.801Z","updated_at":"2026-03-11T01:31:07.901Z","avatar_url":"https://github.com/Wh1isper.png","language":"Python","funding_links":["https://github.com/sponsors/wh1isper"],"categories":[],"sub_categories":[],"readme":"![](https://github.com/Wh1isper/sparglim/actions/workflows/python-package.yml/badge.svg)\n![](https://img.shields.io/pypi/dm/sparglim)\n![](https://img.shields.io/github/last-commit/wh1isper/sparglim)\n![](https://img.shields.io/pypi/pyversions/sparglim)\n![](https://img.shields.io/github/license/wh1isper/sparglim)\n![](https://img.shields.io/github/v/release/wh1isper/sparglim?logo=github)\n![](https://img.shields.io/github/v/release/wh1isper/sparglim?include_prereleases\u0026label=pre-release\u0026logo=github)\n\n# Sparglim ✨\n\nSparglim is aimed at providing a clean solution for PySpark applications in cloud-native scenarios (On K8S、Connect Server etc.).\n\n**This is a fledgling project, looking forward to any PRs, Feature Requests and Discussions!**\n\n🌟✨⭐ Start to support!\n\n## Quick Start\n\nRun Jupyterlab with `sparglim` docker image:\n\n```bash\ndocker run \\\n-it \\\n-p 8888:8888 \\\nwh1isper/jupyterlab-sparglim\n```\n\nAccess `http://localhost:8888` in browser to use jupyterlab with `sparglim`. Then you can try [SQL Magic](#sql-magic).\n\nRun and Daemon a Spark Connect Server:\n\n```bash\ndocker run \\\n-it \\\n-p 15002:15002 \\\n-p 4040:4040 \\\nwh1isper/sparglim-server\n```\n\nAccess `http://localhost:4040` for Spark-UI and `sc://localhost:15002` for Spark Connect Server. [Use sparglim to setup SparkSession to connect to Spark Connect Server](#connect-to-spark-connect-server).\n\n## Install: `pip install sparglim[all]`\n\n- Install only for config and daemon spark connect server `pip install sparglim`\n- Install for pyspark app `pip install sparglim[pyspark]`\n- Install for using magic within ipython/jupyter (will also install pyspark) `pip install sparglim[magic]`\n- **Install for all above** (such as using magic in jupyterlab on k8s) `pip install sparglim[all]`\n\n## Feature\n\n- [Config Spark via environment variables](./config.md)\n- `%SQL` and `%%SQL` magic for executing Spark SQL in IPython/Jupyter\n  - SQL statement can be written in multiple lines, support using `;` to separate statements\n  - Support config `connect client`, see [Spark Connect Overview](https://spark.apache.org/docs/latest/spark-connect-overview.html#spark-connect-overview)\n  - *TODO: Visualize the result of SQL statement(Spark Dataframe)*\n- `sparglim-server` for daemon Spark Connect Server\n\n## User cases\n\n### Basic\n\n```python\nfrom sparglim.config.builder import ConfigBuilder\nfrom datetime import datetime, date\nfrom pyspark.sql import Row\n\n# Create a local[*] spark session with s3\u0026kerberos config\nspark = ConfigBuilder().get_or_create()\n\ndf = spark.createDataFrame([\n    Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)),\n    Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)),\n    Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0))\n])\ndf.show()\n```\n\n### Building a PySpark App\n\nTo config Spark on k8s for Data explorations, see [examples/jupyter-sparglim-on-k8s](./examples/jupyter-sparglim-on-k8s)\n\nTo config Spark for ELT Application/Service, see project [pyspark-sampling](https://github.com/Wh1isper/pyspark-sampling/)\n\n### Deploy Spark Connect Server on K8S (And Connect to it)\n\nTo daemon Spark Connect Server on K8S, see [examples/sparglim-server](./examples/sparglim-server)\n\nTo daemon Spark Connect Server on K8S and Connect it in JupyterLab , see [examples/jupyter-sparglim-sc](./examples/jupyter-sparglim-sc)\n\n### Connect to Spark Connect Server\n\nOnly thing need to do is to set `SPARGLIM_REMOTE` env, format is `sc://host:port`\n\nExample Code:\n\n```python\nimport os\nos.environ[\"SPARGLIM_REMOTE\"] = \"sc://localhost:15002\" # or export SPARGLIM_REMOTE=sc://localhost:15002 before run python\n\nfrom sparglim.config.builder import ConfigBuilder\nfrom datetime import datetime, date\nfrom pyspark.sql import Row\n\n\nc = ConfigBuilder().config_connect_client()\nspark = c.get_or_create()\n\ndf = spark.createDataFrame([\n    Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)),\n    Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)),\n    Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0))\n])\ndf.show()\n\n```\n\n### SQL Magic\n\nInstall Sparglim with\n\n```bash\npip install sparglim[\"magic\"]\n```\n\nLoad magic in IPython/Jupyter\n\n```ipython\n%load_ext sparglim.sql\nspark # show SparkSession brief info\n```\n\nCreate a view:\n\n```python\nfrom datetime import datetime, date\nfrom pyspark.sql import Row\n\ndf = spark.createDataFrame([\n            Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)),\n            Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)),\n            Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0))\n        ])\ndf.createOrReplaceTempView(\"tb\")\n```\n\nQuery the view by `%SQL`:\n\n```ipython\n%sql SELECT * FROM tb\n```\n\n`%SQL` result dataframe can be assigned to a variable:\n\n```ipython\ndf = %sql SELECT * FROM tb\ndf\n```\n\nor `%%SQL` can be used to execute multiple statements:\n\n```ipython\n%%sql SELECT\n        *\n        FROM\n        tb;\n```\n\nYou can also using Spark SQL to load data from external data source, such as:\n\n```ipython\n%%sql CREATE TABLE tb_people\nUSING json\nOPTIONS (path \"/path/to/file.json\");\nShow tables;\n```\n\n## Develop\n\nInstall pre-commit before commit\n\n```\npip install pre-commit\npre-commit install\n```\n\nInstall package locally\n\n```\npip install -e .[test]\n```\n\nRun unit-test before PR, **ensure that new features are covered by unit tests**\n\n```\npytest -v\n```\n\n(Optional, python\u003c=3.10) Use [pytype](https://github.com/google/pytype) to check typed\n\n```\npytype ./sparglim\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwh1isper%2Fsparglim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwh1isper%2Fsparglim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwh1isper%2Fsparglim/lists"}