{"id":39336943,"url":"https://github.com/dan1elt0m/sqlfluff-pyspark","last_synced_at":"2026-01-18T02:13:08.748Z","repository":{"id":322097360,"uuid":"1088192678","full_name":"dan1elt0m/sqlfluff-pyspark","owner":"dan1elt0m","description":"Lints / fixes pyspark sql strings using sqlfluff. Comes with precommit hooks","archived":false,"fork":false,"pushed_at":"2025-11-02T15:09:45.000Z","size":70,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-02T16:09:49.958Z","etag":null,"topics":["precommit-hooks","pyspark","sqlfluff"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dan1elt0m.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-02T13:55:27.000Z","updated_at":"2025-11-02T15:09:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"5311cf56-2672-4bc3-8465-ac4d84d11aa2","html_url":"https://github.com/dan1elt0m/sqlfluff-pyspark","commit_stats":null,"previous_names":["dan1elt0m/sqlfluff-pyspark"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/dan1elt0m/sqlfluff-pyspark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dan1elt0m%2Fsqlfluff-pyspark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dan1elt0m%2Fsqlfluff-pyspark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dan1elt0m%2Fsqlfluff-pyspark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dan1elt0m%2Fsqlfluff-pyspark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dan1elt0m","download_url":"https://codeload.github.com/dan1elt0m/sqlfluff-pyspark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dan1elt0m%2Fsqlfluff-pyspark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28526569,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["precommit-hooks","pyspark","sqlfluff"],"created_at":"2026-01-18T02:13:08.195Z","updated_at":"2026-01-18T02:13:08.743Z","avatar_url":"https://github.com/dan1elt0m.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sqlfluff-pyspark\n\nLint and optionally fix SQL embedded in `spark.sql(...)` calls using [SQLFluff](https://github.com/sqlfluff/sqlfluff).\n\n## Installation\n\n```bash\npip install sqlfluff-pyspark\n```\n\n## Command Line Usage\n\n```bash\n# Lint spark.sql strings\nsqlfluff-pyspark path/to/file.py another_file.py\n\n# Apply fixes (writes changes back to the files)\nsqlfluff-pyspark --fix path/to/file.py\n```\n\nExit codes:\n- 0: Success / no actionable snippets / no violations\n- 1: Lint violations found (printed to stderr) or fixes applied and re-lint failed\n- \u003e1: Unexpected error from sqlfluff invocation\n\n## Pre-commit Hook Integration\n\nThis project provides pre-commit hooks so you can automatically lint (and optionally fix) `spark.sql` strings before committing.\n\nAdd the following to your `.pre-commit-config.yaml`:\n\n```yaml\nrepos:\n  - repo: https://github.com/dan1elt0m/sqlfluff-pyspark\n    rev: v0.1.1  # or the latest tag\n    hooks:\n      - id: sqlfluff-pyspark-lint\n      # Optional fix hook (will modify files). Normally run separately or in CI.\n      # - id: sqlfluff-pyspark-fix\n```\n\nThen install:\n\n```bash\npre-commit install\n```\n\n### Choosing lint vs fix hook\n\nUse the lint hook locally to keep commits clean. Run the fix hook manually:\n\n```bash\npre-commit run sqlfluff-pyspark-fix --all-files\n```\n\nor directly:\n\n```bash\nsqlfluff-pyspark --fix your_script.py\n```\n\n## How It Works\n\n1. Parses Python source with `ast` to find calls to `spark.sql(...)`.\n2. Extracts string literals (skips f-strings and non-constant expressions).\n3. Writes each snippet to a temp SQL file and calls `sqlfluff` on them.\n4. Translates violation line numbers back to original file positions.\n5. For fixes, rewrites the original string literals preserving style (indent, quoting) when possible.\n\nInline directive `sqlfluff:` anywhere in the literal will skip linting/fixing for that snippet.\n\n## Limitations / Notes\n\n- F-strings are skipped (dynamic content).\n- Concatenated string literals are supported (`\"SELECT\" + \" 1\"`).\n- Multi-line snippets are normalized with indentation preserved.\n- Only supports `.py` files passed explicitly; it does not auto-discover.\n\n## Development\n\n```bash\npip install -e .[dev]\npytest -vv\n```\n\n## Versioning\n\nThe hook `rev` should match a published tag. If using main, be aware of potential breaking changes.\n\n## License\n\nMIT\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdan1elt0m%2Fsqlfluff-pyspark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdan1elt0m%2Fsqlfluff-pyspark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdan1elt0m%2Fsqlfluff-pyspark/lists"}