{"id":20535719,"url":"https://github.com/e2fyi/databricks-utils","last_synced_at":"2026-05-09T07:35:21.290Z","repository":{"id":62566534,"uuid":"138819592","full_name":"e2fyi/databricks-utils","owner":"e2fyi","description":"`databricks-utils` is a python package that provide several utility classes/func that improve ease-of-use in databricks notebook.","archived":false,"fork":false,"pushed_at":"2018-07-03T07:26:15.000Z","size":5540,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-17T05:35:58.233Z","etag":null,"topics":["aws","databricks","jupyter-notebooks","notebook","pyspark","s3","spark","vega","vega-lite"],"latest_commit_sha":null,"homepage":"https://e2fyi.github.io/databricks-utils/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/e2fyi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-27T02:31:51.000Z","updated_at":"2021-01-15T20:06:30.000Z","dependencies_parsed_at":"2022-11-03T16:15:35.523Z","dependency_job_id":null,"html_url":"https://github.com/e2fyi/databricks-utils","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2fyi%2Fdatabricks-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2fyi%2Fdatabricks-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2fyi%2Fdatabricks-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2fyi%2Fdatabricks-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/e2fyi","download_url":"https://codeload.github.com/e2fyi/databricks-utils/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242140776,"owners_count":20078395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","databricks","jupyter-notebooks","notebook","pyspark","s3","spark","vega","vega-lite"],"created_at":"2024-11-16T00:33:39.850Z","updated_at":"2026-05-09T07:35:16.257Z","avatar_url":"https://github.com/e2fyi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# databricks-utils\n[![Python version](https://img.shields.io/badge/python-3.6-blue.svg)](https://shields.io/)\n[![Pyspark version](https://img.shields.io/badge/pyspark-2.3.1-blue.svg)](https://shields.io/)\n[![Build Status](https://travis-ci.org/e2fyi/databricks-utils.svg?branch=master)](https://travis-ci.org/e2fyi/databricks-utils)\n\n`databricks-utils` is a python package that provide several utility classes/func\nthat improve ease-of-use in databricks notebook.\n\n### Installation\n```bash\npip install databricks-utils\n```\n\n### Features\n- `S3Bucket` class to easily interact with a [S3 bucket](https://aws.amazon.com/s3/) via [`dbfs`](https://docs.databricks.com/user-guide/dbfs-databricks-file-system.html) and databricks spark.\n\n- `vega_embed` to render charts from [Vega](https://vega.github.io/vega/) and [Vega-Lite](https://vega.github.io/vega-lite/) specifications.\n\n### Documentation\nAPI documentation can be found at [https://e2fyi.github.io/databricks-utils/](https://e2fyi.github.io/databricks-utils/).\n\n\n### Quick start\n**S3Bucket**  \n```python\nimport json\nfrom databricks_utils.aws import S3Bucket\n\n# need to attach notebook's dbutils\n# before S3Bucket can be used\nS3Bucket.attach_dbutils(dbutils)\n\n# create an instance of the s3 bucket\nbucket = (S3Bucket(\"somebucketname\", \"SOMEACCESSKEY\", \"SOMESECRETKEY\")\n          .allow_spark(sc) # local spark context\n          .mount(\"somebucketname\")) # mount location name (resolves as `/mnt/somebucketname`)\n\n# show list of files/folders in the bucket \"resource\" folder\nbucket.ls(\"resource/\")\n\n# read in a json file from the bucket\ndata = json.load(open(bucket.local(\"resource/somefile.json\", \"r\")))\n\n# read from parquet via spark\ndataframe = spark.read.parquet(bucket.s3(\"resource/somedf.parquet\"))\n\n# umount\nbucket.umount()\n```\n\n**Vega**  \n[Vega](https://vega.github.io/vega/) and [Vega-Lite](https://vega.github.io/vega-lite/)\nare high-level grammars of interactive graphics. They provide concise JSON\nsyntax for rapidly generating visualizations to support analysis.\n\n```python\nfrom databricks_utils.vega import vega_embed\n\n# vega-lite spec for a bar chart\nspec = {\n  \"data\": {\n    \"values\": [\n      {\"a\": \"A\",\"b\": 28}, {\"a\": \"B\",\"b\": 55}, {\"a\": \"C\",\"b\": 43},\n      {\"a\": \"D\",\"b\": 91}, {\"a\": \"E\",\"b\": 81}, {\"a\": \"F\",\"b\": 53},\n      {\"a\": \"G\",\"b\": 19}, {\"a\": \"H\",\"b\": 87}, {\"a\": \"I\",\"b\": 52}\n    ]\n  },\n  \"mark\": \"bar\",\n  \"encoding\": {\n    \"x\": {\"field\": \"a\", \"type\": \"ordinal\"},\n    \"y\": {\"field\": \"b\", \"type\": \"quantitative\"}\n  }\n}\n\n# plot out the vega chart in databricks notebook\ndisplayHTML(vega_embed(spec=spec))\n```\n\n### Developer\n```bash\n# add a version to git tag and publish to pypi\n. add_tag.sh \u003cVERSION\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fe2fyi%2Fdatabricks-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fe2fyi%2Fdatabricks-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fe2fyi%2Fdatabricks-utils/lists"}