https://github.com/productiveanalytics/databricks_sandbox
Databricks (Azure and AWS)
https://github.com/productiveanalytics/databricks_sandbox
Last synced: 20 days ago
JSON representation
Databricks (Azure and AWS)
- Host: GitHub
- URL: https://github.com/productiveanalytics/databricks_sandbox
- Owner: productiveAnalytics
- Created: 2022-11-22T18:26:35.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2026-07-02T00:15:44.000Z (about 1 month ago)
- Last Synced: 2026-07-02T00:18:49.093Z (about 1 month ago)
- Language: Jupyter Notebook
- Size: 789 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# sandbox for Databricks on AWS and Azure
## Install Databricks CLI
- ```python3 -m pip install databricks-cli```
- Provide the token from Azure Databricks to the interactive CLI: ```databricks configure --token```
- Test the CLI by checking for the secret: ```databricks secrets list-scopes```
- Create secret scope: ```databricks secrets create-scope --scope myblob```
- Create secret: ```databricks secrets put --scope myblob --key my_access_key```
- Check ACL for secrets: ```databricks secrets list-acls --scope myblob```
## Create folder and upload files (Note: databricks fs = dbfs)
```
databricks fs mkdirs dbfs:/Users/productive_analytics@databricks.com/data
databricks fs cp ~/Dev/data/member_account_info.csv dbfs:/Users/productive_analytics@databricks.com/data/member_account_info.csv
databricks fs ls dbfs:/Users/productive_analytics@databricks.com/data/
```
### In notebook
```
%python
display(dbutils.fs.ls('dbfs:/Users/productive_analytics@databricks.com/data/'))
```
which is same as
```
%fs
dbfs ls /Users/productive_analytics@databricks.com/data/
```
### Then use the "dbutils" to read the secret
```
%python
val azure_blob_storage_dns = ".blob.core.windows.net"
val storageAccountName = "myazurestorageaccount"
val containerName = "myconatainer"
val relativeFilePath = "movies.csv"
val blobAccessKey = dbutils.secrets.get(scope="myblob", key="my_access_key")
spark.conf.set(
"fs.azure.account.key."+ storageAccountName + azure_blob_storage_dns,
blobAccessKey
)
val wasbs_path = "wasbs://"+ containerName + "@" + storageAccountName + azure_blob_storage_dns + "/" + relativeFilePath
val movies_df = spark.read
.option("hader", "true")
.csv(wasbs_path)
display(movies_df, 10)
```