{"id":18546449,"url":"https://github.com/adroll/plunger","last_synced_at":"2025-04-09T20:30:54.425Z","repository":{"id":68835546,"uuid":"102907712","full_name":"AdRoll/plunger","owner":"AdRoll","description":"A linter utility for Luigi jobs","archived":false,"fork":false,"pushed_at":"2017-09-14T21:29:24.000Z","size":12,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-03-24T11:12:15.028Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AdRoll.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-08T21:56:43.000Z","updated_at":"2021-07-28T03:15:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"bf2d1d35-a26c-440c-89ae-15ab4f68977d","html_url":"https://github.com/AdRoll/plunger","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Fplunger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Fplunger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Fplunger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Fplunger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AdRoll","download_url":"https://codeload.github.com/AdRoll/plunger/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248107183,"owners_count":21048874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T20:25:10.969Z","updated_at":"2025-04-09T20:30:54.410Z","avatar_url":"https://github.com/AdRoll.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# plunger\n\nPlunger is a python linter, with built-in [Luigi](https://github.com/spotify/luigi) support.\n\n## Rationale\n\nExisting Python linters and type checkers ([pylint](https://www.pylint.org/), [mypy](http://mypy-lang.org/)) are great, but have their limitations: you can only get so far when dealing with Python code that uses run-time metaprogramming magic. Unfortunately Luigi relies on this quite a bit, that makes generic linters less useful.\n\nFor example, they can't effectively detect errors related to Luigi task parameters when they are used as member variables or arguments to the constructor. `pylint` can detect undefined variables only when they are normal local variables, not class members -- most of the time it has no idea if `self.foo` exists or not.\n\n## Solution\n\nThese limitations are mostly due to highly dynamic nature of Python, you can't really get around this in a generic way. Plunger's answer is to just give up on that and treat Luigi tasks specially. Plunger only supports a certain well behaved subset of Python. Features like `import *` or inheritance (with the exception of inheriting from luigi.Task) are not supported, but you shouldnt use them anyway.\n\n## Installation\n\nGet [Stack](https://github.com/commercialhaskell/stack/releases)\n```\nstack build plunger\nstack install plunger\n```\n\nYou should get `plunger` executable under `~/.local/bin`. \n\n## Usage\n\n```\nplunger path/to/module.py [path/to/imported.py ...]\n```\n\nWhere first path is to the module to be checked, and the are paths to imported modules to be checked as well. \n\n`plunger` has only limited support for imports, so it will only process the imported modules that you specify on the command line, the rest will be ignored. Only `from X import Y` imports are supported at the moment.\n\nTo illustrate what this means in practice, consider two python modules:\n\n##### otherjobs.py\n```python\nimport luigi\n\nclass TestTask(luigi.Task):\n    foo = luigi.Parameter()\n    bar = luigi.Parameter()\n\n    def run(self):\n        pass\n```\n\n\n##### myjobs.py\n```python\nimport luigi\nfrom otherjobs import TestTask\n\nclass OtherTestTask(luigi.Task):\n\n    def requires(self):\n        return TestTask(foo=\"a\")  # missing argument for \"bar\" here \n\n    def run(self):\n        pass\n```\n\nYou'll get no warnings if you run `plunger` on **myjobs.py** only:\n```shell\n$ plunger myjobs.py\nNo warnings\n```\n\nSince `plunger` does not automatically process all imports, it has no idea what the signature of `TestTask` is, and cannot detect the error (missing parameter).\n\nBut if you specify path to the other module on the command line, it will parse it as well and will be able to verify the invocation of `TestTask` constructor:\n```shell\nplunger myjobs.py otherjobs.py\n=========== 1 WARNINGS ========\n\nMissing argument bar at row 8 column 16\n    Expected: foo:AnyType, bar:AnyType\n```\n\n# Examples of errors detected by Plunger\n\n#### Undefined class members\n\n```python\nfrom __future__ import print_function\nimport luigi\n\nclass TestTask(luigi.Task):\n    foo = luigi.Parameter()\n    bar = luigi.Parameter()\n\n    def run(self):\n        print(self.y)\n```\n\nResult:\n\n```\n=========== 1 WARNINGS ========\n\nUndefined y at row 10 column 20\n```\n\n#### Missing task constructor parameters\n\n```python\nfrom __future__ import print_function\nimport luigi\n\nclass TestTask(luigi.Task):\n    foo = luigi.Parameter()\n    bar = luigi.Parameter()\n\n    def run(self):\n        pass\n\nclass OtherTestTask(luigi.Task):\n    def requires(self):\n        return TestTask(foo=\"a\") \n\n    def run(self):\n        pass\n```\n\nResult:\n\n```\n=========== 1 WARNINGS ========\n\nMissing argument bar at row 16 column 16\n    Expected: foo:AnyType, bar:AnyType\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadroll%2Fplunger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadroll%2Fplunger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadroll%2Fplunger/lists"}