{"id":13678266,"url":"https://github.com/databricks/ide-best-practices","last_synced_at":"2025-06-26T01:03:14.386Z","repository":{"id":37768639,"uuid":"499561071","full_name":"databricks/ide-best-practices","owner":"databricks","description":"Best practices for working with Databricks from an IDE","archived":false,"fork":false,"pushed_at":"2023-04-19T11:20:14.000Z","size":26,"stargazers_count":51,"open_issues_count":3,"forks_count":48,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-06-22T06:52:35.077Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-06-03T15:28:08.000Z","updated_at":"2025-05-11T17:46:06.000Z","dependencies_parsed_at":"2024-01-14T15:21:44.150Z","dependency_job_id":"a137a6e8-99a0-431b-a672-3568aac12eae","html_url":"https://github.com/databricks/ide-best-practices","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/databricks/ide-best-practices","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fide-best-practices","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fide-best-practices/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fide-best-practices/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fide-best-practices/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks","download_url":"https://codeload.github.com/databricks/ide-best-practices/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fide-best-practices/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261978909,"owners_count":23239417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T13:00:51.708Z","updated_at":"2025-06-26T01:03:14.341Z","avatar_url":"https://github.com/databricks.png","language":"Python","readme":"# Best practices for using an IDE with Databricks\n\nThis repository is a companion for the example article \"Use an IDE with Databricks\" for [AWS](https://docs.databricks.com/dev-tools/ide-how-to.html), [Azure](https://docs.microsoft.com/azure/databricks/dev-tools/ide-how-to), and [GCP](https://docs.gcp.databricks.com/dev-tools/ide-how-to.html). \n\nThis example features the use of Visual Studio Code, Python, `dbx` by Databricks Labs (for [AWS](https://docs.databricks.com/dev-tools/dbx.html), [Azure](https://docs.microsoft.com/azure/databricks/dev-tools/dbx), and [GCP](https://docs.gcp.databricks.com/dev-tools/dbx.html)), `pytest`, and GitHub Actions.\n\nYou can adapt this example for use with other Python-compatible IDEs such as PyCharm, IntelliJ IDEA with the Python plugin, and Eclipse with the PyDev plugin.\n\nGoing through the example, you will use your IDE to:\n\n* Get data from the [owid/covid-19-data](https://github.com/owid/covid-19-data) repo in GitHub.\n* Filter the data for a specific ISO country code.\n* Create a pivot table from the data.\n* Perform data cleansing on the data.\n* Modularize the code logic into reusable functions.\n* Unit test the functions.\n* Provide `dbx` project configurations and settings to enable the code to write the data to a Delta table in a remote Databricks workspace.\n\nThe only time you need to use the Databricks user interface for this example is to see the results of writing the data to your Databricks workspace. (And even then, you can use the Databricks Jobs REST API or the Databricks Jobs CLI for [AWS](https://docs.databricks.com/data-engineering/jobs/jobs.html), [Azure](https://docs.microsoft.com/azure/databricks/data-engineering/jobs/jobs), or [GCP](https://docs.gcp.databricks.com/data-engineering/jobs/jobs.html) to get some high-level information about those data writes programmatically.) Otherwise, you can complete all of these development tasks from within your favorite Python-compatible IDE.\n\nThe example is hands-on. We recommend working through the example article to learn how to apply these techniques to your own Databricks development tasks from within your favorite Python-compatible IDE.\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fide-best-practices","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks%2Fide-best-practices","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fide-best-practices/lists"}