{"id":24884581,"url":"https://github.com/stackql/stackql-databricks-openapi","last_synced_at":"2025-04-13T19:10:26.650Z","repository":{"id":269124117,"uuid":"900136770","full_name":"stackql/stackql-databricks-openapi","owner":"stackql","description":"generate openapi specs from Databricks REST API web documentation","archived":false,"fork":false,"pushed_at":"2024-12-23T22:15:00.000Z","size":10244,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-27T09:47:05.078Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stackql.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-08T00:36:00.000Z","updated_at":"2024-12-24T21:55:07.000Z","dependencies_parsed_at":"2025-03-27T09:53:39.317Z","dependency_job_id":null,"html_url":"https://github.com/stackql/stackql-databricks-openapi","commit_stats":null,"previous_names":["stackql/stackql-databricks-openapi"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackql%2Fstackql-databricks-openapi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackql%2Fstackql-databricks-openapi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackql%2Fstackql-databricks-openapi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackql%2Fstackql-databricks-openapi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stackql","download_url":"https://codeload.github.com/stackql/stackql-databricks-openapi/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248766736,"owners_count":21158301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-01T14:29:50.629Z","updated_at":"2025-04-13T19:10:26.629Z","avatar_url":"https://github.com/stackql.png","language":"Python","readme":"# stackql-databricks-openapi\n\nBuild `databricks_account` and `databricks_workspace` providers for `stackql` using the databricks web documentation:  \n\n- [api/workspace](https://docs.databricks.com/api/workspace/introduction)\n- [api/account](https://docs.databricks.com/api/account/introduction)\n\n## usage\n\nThe program requires `selenium` and the `chromedriver` for windows, use PowerShell to run the following code to extract web doc data into machine readable staging documents, the staging documents are then converted into tagged OpenAPI specification documents organized by service:\n\n```powershell\npython -m venv venv\n.\\venv\\Scripts\\Activate\npip install -r requirements.txt\npip freeze\n# scrape web docs\npython .\\process_web_docs.py account --clean --debug \npython .\\process_web_docs.py workspace --clean --debug\n# generate openapi specs\npython .\\generate_openapi_specs.py account --clean --debug\npython .\\generate_openapi_specs.py workspace --clean --debug\n\ndeactivate\nRemove-Item -Recurse -Force ./venv\n```\n\n## tests\n\nTo run tests locally, clone [stackql-provider-tests](https://github.com/stackql/stackql-provider-tests), and run locally:\n\n```bash\n# run from the directory you cloned into\ncd /mnt/c/LocalGitRepos/stackql/core/stackql-provider-tests\n# test account\nbash test-provider.sh \\\ndatabricks_account \\\nfalse \\\n/mnt/c/LocalGitRepos/stackql/openapi-conversion/stackql-databricks-openapi/openapi_providers \\\ntrue\n# test workspace\nbash test-provider.sh \\\ndatabricks_workspace \\\nfalse \\\n/mnt/c/LocalGitRepos/stackql/openapi-conversion/stackql-databricks-openapi/openapi_providers \\\ntrue\n# back to starting dir\ncd /mnt/c/LocalGitRepos/stackql/openapi-conversion/stackql-databricks-openapi\n```\n\n## inspect\n\n```bash\ncurl -L https://bit.ly/stackql-zip -O \u0026\u0026 unzip stackql-zip\n```\n\n```bash\nPROVIDER_REGISTRY_ROOT_DIR=\"$(pwd)/openapi_providers\"\nREG_STR='{\"url\": \"file://'${PROVIDER_REGISTRY_ROOT_DIR}'\", \"localDocRoot\": \"'${PROVIDER_REGISTRY_ROOT_DIR}'\", \"verifyConfig\": {\"nopVerify\": true}}'\n./stackql shell --registry=\"${REG_STR}\"\n```\n\nsome test queries...\n\n```sql\nSELECT \nu.id,\ndisplayName, \nuserName, \nactive \nFROM databricks_account.iam.users u, JSON_EACH(roles)\nWHERE account_id = 'ebfcc5a9-9d49-4c93-b651-b3ee6cf1c9ce'\nAND JSON_EXTRACT(json_each.value, '$.value') = 'account_admin';\n```\n\n```sql\nselect \ngr.id, \ndisplayName, \njson_extract(json_each.value, '$.value') as entitlement \nfrom databricks_workspace.iam.groups gr, JSON_EACH(entitlements) \nwhere deployment_name = 'dbc-ddbc0f51-c9cf';\n```\n\n```sql\nSELECT \nsp.id,\nactive,\napplicationId,\ndisplayName,\nexternalId\nFROM databricks_account.iam.service_principals sp, JSON_EACH(roles)\nWHERE account_id = 'ebfcc5a9-9d49-4c93-b651-b3ee6cf1c9ce'\nAND JSON_EXTRACT(json_each.value, '$.value') = 'account_admin';\n```\n\n```sql\nselect \nworkspace_id,\nworkspace_name,\ndeployment_name,\nworkspace_status,\npricing_tier, \naws_region, \ncredentials_id, \nstorage_configuration_id\nfrom\ndatabricks_account.provisioning.workspaces where account_id = 'ebfcc5a9-9d49-4c93-b651-b3ee6cf1c9ce';\n```\n\n```sql\nselect \ncluster_id,\naws_attributes,\nnode_type_id,\nstate\nfrom  \ndatabricks_workspace.compute.clusters \nwhere deployment_name = 'dbc-ddbc0f51-c9cf';\n```\n\n```sql\nselect\n*\nfrom databricks_account.provisioning.vw_workspaces \nwhere account_id = 'ebfcc5a9-9d49-4c93-b651-b3ee6cf1c9ce' \n```\n\n## check for new routes\n\n```powershell\npython .\\find_new_routes.py workspace\n# or\npython3 .\\find_new_routes.py account\n```\n\n## debugging with `curl`\n\n```bash\nDATABRICKS_TOKEN=$(curl --request POST \"https://accounts.cloud.databricks.com/oidc/accounts/${DATABRICKS_ACCOUNT_ID}/v1/token\" \\\n  --header \"Content-Type: application/x-www-form-urlencoded\" \\\n  --data-urlencode \"grant_type=client_credentials\" \\\n  --data-urlencode \"client_id=${DATABRICKS_CLIENT_ID}\" \\\n  --data-urlencode \"client_secret=${DATABRICKS_CLIENT_SECRET}\" \\\n  --data-urlencode \"scope=all-apis\" | jq -r .access_token)\n\ncurl --request GET \"https://accounts.cloud.databricks.com/api/2.0/accounts/${DATABRICKS_ACCOUNT_ID}/workspaces\" \\\n  --header \"Authorization: Bearer ${DATABRICKS_TOKEN}\" \\\n  -vvv \\\n  --header \"Accept: application/json\"\n  ```\n\n  ## generate user (web) docs\n\n```bash\nbash generate_user_docs.sh account\nbash generate_user_docs.sh workspace\n```","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackql%2Fstackql-databricks-openapi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstackql%2Fstackql-databricks-openapi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackql%2Fstackql-databricks-openapi/lists"}