{"id":51762772,"url":"https://github.com/rafaelpierre/pyjaws","last_synced_at":"2026-07-19T16:01:06.551Z","repository":{"id":161539729,"uuid":"636210258","full_name":"rafaelpierre/pyjaws","owner":"rafaelpierre","description":"PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows","archived":false,"fork":false,"pushed_at":"2025-10-18T11:04:00.000Z","size":3633,"stargazers_count":43,"open_issues_count":9,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-10-19T06:56:22.060Z","etag":null,"topics":["airflow","apache-airflow","apache-spark","azure-databricks","cicd","data-engineering","databricks","mage","mlops","orchestration","pyspark","python","spark","workflow"],"latest_commit_sha":null,"homepage":"https://pyjaws.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rafaelpierre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-05-04T10:52:40.000Z","updated_at":"2025-07-10T11:14:56.000Z","dependencies_parsed_at":"2023-11-17T12:28:50.775Z","dependency_job_id":"9b9356d2-d09e-453a-8551-5b9e1fe7d182","html_url":"https://github.com/rafaelpierre/pyjaws","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rafaelpierre/pyjaws","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelpierre%2Fpyjaws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelpierre%2Fpyjaws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelpierre%2Fpyjaws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelpierre%2Fpyjaws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rafaelpierre","download_url":"https://codeload.github.com/rafaelpierre/pyjaws/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelpierre%2Fpyjaws/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35657587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-19T02:00:06.923Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","apache-airflow","apache-spark","azure-databricks","cicd","data-engineering","databricks","mage","mlops","orchestration","pyspark","python","spark","workflow"],"created_at":"2026-07-19T16:00:48.036Z","updated_at":"2026-07-19T16:01:06.413Z","avatar_url":"https://github.com/rafaelpierre.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PyJaws: A Pythonic Way to Define Databricks JaWs (Jobs and Workflows)\n\n\u003chr /\u003e\n\n![Dynamic XML Badge](https://img.shields.io/badge/dynamic/xml?url=https%3A%2F%2Fwww.pepy.tech%2Fprojects%2Fpyjaws\u0026query=%2F%2Fdiv%5B%40class%3D'MuiGrid-root%20MuiGrid-container%20MuiGrid-spacing-xs-1%20css-tuxzvu'%5D%2Fdiv%5B4%5D%2Ftext()\u0026style=for-the-badge\u0026label=downloads)\n ![black](https://img.shields.io/badge/code%20style-black-000000.svg?style=for-the-badge) ![ruff](https://img.shields.io/badge/lint-ruff-gold?style=for-the-badge) ![cov](https://raw.githubusercontent.com/rafaelpierre/pyjaws/main/img/coverage.svg) ![databricks](https://img.shields.io/badge/Databricks-FF3621.svg?style=for-the-badge\u0026logo=Databricks\u0026logoColor=white) ![Jinja](https://img.shields.io/badge/jinja-white.svg?style=for-the-badge\u0026logo=jinja\u0026logoColor=black)\n\n\u003cp align=\"center\"\u003e\n        \u003cimg src=\"https://raw.githubusercontent.com/rafaelpierre/pyjaws/main/img/pyjaws.png\" class=\"align-center\" /\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003chr /\u003e\n\n* **PyJaws** enables declaring [Databricks Jobs and Workflows](https://docs.databricks.com/workflows/index.html) as Python code, allowing for:\n  * Code Linting\n  * Formatting\n  * Parameter Validation\n  * Modularity and reusability\n* In addition to those, **PyJaws** also provides some nice features such as [cycle detection](https://networkx.org/documentation/stable/reference/algorithms/cycles.html) out of the box.\n\nFolks who have used Python-based orchestration tools such as [Apache Airflow](https://airflow.apache.org/), [Luigi](https://luigi.readthedocs.io/en/stable/) and [Mage](https://pypi.org/project/mage-ai/) will be familiar with the concepts and the API if **PyJaws**.\n\n## Getting Started\n\n* First step is installing `pyjaws`:\n\n```bash\npip install pyjaws\n```\n\n* Once it's installed, define your Databricks Workspace authentication variables:\n\n```bash\nexport DATABRICKS_HOST = ...\nexport DATABRICKS_TOKEN = ...\n```\n\n* Last, define your Workflow Tasks (see `examples`) and run:\n\n```bash\npyjaws create path/to/your/workflow_definitions\n```\n\n### Sample Job Definition\n\n```python\nfrom pyjaws.api.base import (\n    Cluster,\n    Runtime,\n    Workflow\n)\nfrom pyjaws.api.tasks import PythonWheelTask\n\ncluster = Cluster(\n    job_cluster_key = \"ai_cluster\",\n    spark_version = Runtime.DBR_13_ML,\n    num_workers = 2,\n    node_type_id = \"Standard_DS3_v2\",\n    cluster_log_conf = {\n        \"dbfs\": {\n            \"destination\": \"dbfs:/home/cluster_log\"\n        }\n    }\n)\n\n\n# Create a Task object.\n\ningest_task = PythonWheelTask(\n    key = \"ingest\",\n    cluster = cluster,\n    entrypoint = \"iot\",\n    task_name = \"ingest\",\n    parameters = [\n        f\"my_parameter_value\",\n        \"--output-table\", \"my_table\"\n    ]\n)\n\ntransform_task = PythonWheelTask(\n    key = \"transform\",\n    cluster = cluster,\n    entrypoint = \"iot\",\n    task_name = \"ingest\",\n    dependencies = [ingest_task],\n    parameters = [\n        f\"my_parameter_value2\",\n        \"--input-table\", \"my_table\"\n        \"--output-table\", \"output_table\"\n    ]\n)\n\n\n# Create a Workflow object to define dependencies\n# between previously defined tasks.\n\nworkflow = Workflow(\n    name = \"my_workflow\",\n    tasks = [ingest_task, transform_task]\n)\n```\n\n### Extra Features\n\n* **Context Manager** for **Cluster** declarations:\n\n```python\n# cluster created with context manager\nwith Cluster(\n    job_cluster_key=\"mycluster_2\",\n    spark_version=Runtime.DBR_13_ML,\n    node_type_id=\"Standard_E4ds_v4\",\n    num_workers=3,\n) as cluster_2:\n    task_2 = SparkPythonTask(\n        key=\"task_2\",\n        cluster=cluster_2,\n        python_file=\"/Workspace/Repos/bob@mail.com/utils/task_2.py\",\n        source=Source.WORKSPACE,\n    )\n```\n\n* **Workflow preview/visualization** on notebooks:\n\n```python\ndisplay(workflow)\n```\nResult:\n\n![preview](https://github.com/rafaelpierre/pyjaws/blob/main/img/preview.png?raw=true \"Preview\")\n\n## Deploying Jobs\n\nIf you have a folder containing multiple workflow definition files written in Python with **PyJaws**, it is quite simple to deploy all of them to your Databricks Workspace with a one liner:\n\n`pyjaws create examples/simple_workflow`\n\nThis would result in the following Workflow being deployed to your workspace:\n\n![workflow](https://github.com/rafaelpierre/pyjaws/blob/main/img/workflow.png?raw=true \"Workflow\")\n\nBy default, **pyjaws** also includes some useful tags into the workflows indicating which Git Repo hosts the Python definition, commit hash and when the workflow was last updated. For example:\n\n![workflow](https://github.com/rafaelpierre/pyjaws/blob/main/img/tags.png?raw=true \"Workflow\")\n\n## Documentation\n\n* [PyJaws Documentation (Read the Docs)](https://pyjaws.readthedocs.io)\n\n## Development \u0026 Testing\n\n* **PyJaws** can be tested locally for development purposes. To run unit tests, make sure `tox`, `pytest`, `pytest-cov`, and `coverage` are installed and from a bash terminal, simply run `tox`.\n\n## Disclaimer\n\n* **PyJaws** is not developed, endorsed not supported by Databricks. It is provided as-is; no warranty is derived from using this package. For more details, please refer to the [license](https://github.com/rafaelpierre/pyjaws/blob/main/LICENSE.md).\n\n## Reporting Bugs and Contributing\n\nFeel free to create an issue if you feel something is not right. Contribution guidelines can be found [here](https://githubcom/rafaelpierre/pyjaws/blob/main/CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafaelpierre%2Fpyjaws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frafaelpierre%2Fpyjaws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafaelpierre%2Fpyjaws/lists"}