{"id":34070670,"url":"https://github.com/lucasvieirasilva/databricks-aws-utils","last_synced_at":"2025-12-14T07:28:08.819Z","repository":{"id":43857789,"uuid":"430850436","full_name":"lucasvieirasilva/databricks-aws-utils","owner":"lucasvieirasilva","description":"Databricks AWS Utils is a library to abstract Databricks integration with AWS Services","archived":false,"fork":false,"pushed_at":"2024-04-12T09:29:17.000Z","size":68,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-27T04:44:27.115Z","etag":null,"topics":["aws","databricks","utils"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucasvieirasilva.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-11-22T20:10:20.000Z","updated_at":"2023-09-06T13:25:30.000Z","dependencies_parsed_at":"2024-04-12T10:44:30.164Z","dependency_job_id":"146e36d7-3123-49e5-9c2b-018eaecbab69","html_url":"https://github.com/lucasvieirasilva/databricks-aws-utils","commit_stats":{"total_commits":8,"total_committers":1,"mean_commits":8.0,"dds":0.0,"last_synced_commit":"8e03db0145d6401cbf4b0719332952dfe2abb564"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/lucasvieirasilva/databricks-aws-utils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasvieirasilva%2Fdatabricks-aws-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasvieirasilva%2Fdatabricks-aws-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasvieirasilva%2Fdatabricks-aws-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasvieirasilva%2Fdatabricks-aws-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucasvieirasilva","download_url":"https://codeload.github.com/lucasvieirasilva/databricks-aws-utils/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasvieirasilva%2Fdatabricks-aws-utils/sbom","scorecard":{"id":602245,"data":{"date":"2025-08-11","repo":{"name":"github.com/lucasvieirasilva/databricks-aws-utils","commit":"67d4686283e4ba7d685b6dc39d01cff6626b0670"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.8,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/lint.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/lucasvieirasilva/databricks-aws-utils/lint.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/lint.yml:23: update your workflow using https://app.stepsecurity.io/secureworkflow/lucasvieirasilva/databricks-aws-utils/lint.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/lint.yml:28: update your workflow using https://app.stepsecurity.io/secureworkflow/lucasvieirasilva/databricks-aws-utils/lint.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/lucasvieirasilva/databricks-aws-utils/release.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:23: update your workflow using https://app.stepsecurity.io/secureworkflow/lucasvieirasilva/databricks-aws-utils/release.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:28: update your workflow using https://app.stepsecurity.io/secureworkflow/lucasvieirasilva/databricks-aws-utils/release.yml/main?enable=pin","Info:   0 out of   3 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   3 third-party GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 0/26 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: jobLevel 'contents' permission set to 'write': .github/workflows/release.yml:14","Warn: no topLevel permission defined: .github/workflows/lint.yml:1","Warn: no topLevel permission defined: .github/workflows/release.yml:1"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 7 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"23 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2024-230 / GHSA-248v-346w-9cwc","Warn: Project is vulnerable to: PYSEC-2023-135 / GHSA-xqr8-7jwr-rhp7","Warn: Project is vulnerable to: PYSEC-2024-4 / GHSA-2mqj-m65w-jghx","Warn: Project is vulnerable to: PYSEC-2023-165 / GHSA-cwvm-v4w8-q58c","Warn: Project is vulnerable to: PYSEC-2024-60 / GHSA-jjg7-2v4v-x38h","Warn: Project is vulnerable to: GHSA-cpwx-vrp4-4pq7","Warn: Project is vulnerable to: GHSA-gmj6-6f8f-6699","Warn: Project is vulnerable to: GHSA-h5c8-rqwp-cp95","Warn: Project is vulnerable to: GHSA-h75v-3vvj-5mfj","Warn: Project is vulnerable to: GHSA-q2x7-8rv6-6q7h","Warn: Project is vulnerable to: GHSA-mr82-8j83-vxmv","Warn: Project is vulnerable to: PYSEC-2023-44","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q","Warn: Project is vulnerable to: PYSEC-2025-49 / GHSA-5rjg-fvgr-3xxf","Warn: Project is vulnerable to: GHSA-cx63-2mw6-8hw5","Warn: Project is vulnerable to: GHSA-34jh-p97f-mpxf","Warn: Project is vulnerable to: PYSEC-2023-212 / GHSA-g4mx-q9vg-27p4","Warn: Project is vulnerable to: GHSA-pq67-6m6q-mj2v","Warn: Project is vulnerable to: PYSEC-2023-192 / GHSA-v845-jxx5-vc9f","Warn: Project is vulnerable to: PYSEC-2024-187 / GHSA-rqc4-2hc7-8c8v","Warn: Project is vulnerable to: GHSA-jfmj-5v4g-7637"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-21T00:43:55.081Z","repository_id":43857789,"created_at":"2025-08-21T00:43:55.082Z","updated_at":"2025-08-21T00:43:55.082Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27621550,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-09T02:00:09.185Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","databricks","utils"],"created_at":"2025-12-14T07:28:08.169Z","updated_at":"2025-12-14T07:28:08.813Z","avatar_url":"https://github.com/lucasvieirasilva.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Databricks AWS Utils\n\nDatabricks AWS Utils is a library to abstract Databricks integration with AWS Services\n\n## Features\n\n- Convert Delta Table to be consumed by AWS Athena with Schema evolution\n- Run queries against AWS RDS using AWS Secrets Manager to retrieve the connection properties and returns as Spark DataFrame\n\n## Install\n\n`pip install databricks-aws-utils`\n\n## Delta Table to AWS Athena\n\n### Motivation\n\nCurrently, delta tables are only compatible with AWS Athena engine v3, however, even with the compatibility, there are some limitations regarding the schema evolution, where the schema is not fully or correctly synchronized with the AWS Glue catalog, causing problems when querying the table.\n\nTo solve this problem, we created this library to convert the delta table columns to be compatible with the AWS Glue catalog and update the table metadata, allowing the table to be queried correctly by AWS Athena.\n\n### Usage\n\n```python\nfrom databricks_aws_utils.delta_table import DeltaTableUtils\n\n...\n\nDeltaTableUtils(spark, 'my_schema.my_table_name').to_athena_v3()\n```\n\nThe `to_athena_v3` function uses the spark session to capture the current delta schema and update the glue table.\n\n**NOTE**: This feature is only compatible with AWS Athena engine v3, and the Databricks cluster must have access to the AWS Glue catalog.\n\n**NOTE**: This feature is not supported by Databricks Unity Catalog, since it does not allow queries from AWS Athena.\n\n#### Custom IAM Role\n\nIf you need to use a custom IAM Role to update the AWS Glue table, you can pass the role name as a parameter to the `DeltaTableUtils` class.\n\n```python\nfrom databricks_aws_utils.delta_table import DeltaTableUtils\n\n...\n\nDeltaTableUtils(\n    spark,\n    'my_schema.my_table_name',\n    iam_role='my_custom_iam_role'\n).to_athena_v3()\n```\n\n**NOTE**: The Databricks cluster must have permission to assume the custom IAM Role.\n\n### Athena Engine v2\n\nAWS Athena engine v2 doesn't support delta tables, so, to query a delta table using AWS Athena engine v2, it's necessary to generate Hive Symlink from the delta table and point to a different table.\n\n```python\nfrom databricks_aws_utils.delta_table import DeltaTableUtils\n\n...\n\nDeltaTableUtils(spark, 'my_schema.my_table_name').to_athena('my_schema', 'my_symlink_table_name')\n```\n\n**NOTE**: The schema name provided in the `to_athena` doesn't need to be the same as the delta table schema.\n\n## Contributing\n\n- See our [Contributing Guide](CONTRIBUTING.md)\n\n## Change Log\n\n- See our [Change Log](CHANGELOG.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucasvieirasilva%2Fdatabricks-aws-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucasvieirasilva%2Fdatabricks-aws-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucasvieirasilva%2Fdatabricks-aws-utils/lists"}