{"id":18554426,"url":"https://github.com/oracle/spark-oracle","last_synced_at":"2025-08-08T01:54:09.304Z","repository":{"id":66212460,"uuid":"444905368","full_name":"oracle/spark-oracle","owner":"oracle","description":"On the fly, translation of Spark programs to run natively on your Oracle DB. Your Spark programs require no changes.","archived":false,"fork":false,"pushed_at":"2025-03-10T12:26:51.000Z","size":3013,"stargazers_count":33,"open_issues_count":1,"forks_count":10,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-09T23:38:00.434Z","etag":null,"topics":["oracle","spark","sql"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oracle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-05T18:12:13.000Z","updated_at":"2025-04-04T04:32:33.000Z","dependencies_parsed_at":"2024-11-06T21:24:10.065Z","dependency_job_id":"1e556d50-b0a0-416d-8d4c-1c988a8cdf49","html_url":"https://github.com/oracle/spark-oracle","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oracle/spark-oracle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Fspark-oracle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Fspark-oracle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Fspark-oracle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Fspark-oracle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oracle","download_url":"https://codeload.github.com/oracle/spark-oracle/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Fspark-oracle/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269351901,"owners_count":24402677,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-07T02:00:09.698Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["oracle","spark","sql"],"created_at":"2024-11-06T21:21:53.783Z","updated_at":"2025-08-08T01:54:09.290Z","avatar_url":"https://github.com/oracle.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark_On_Oracle\r\n\r\n- Currently, data lakes comprising Oracle Data Warehouse and Apache Spark have these characteristics:\r\n  - They have **separate data catalogs,** even if they access the same data in an object store.\r\n  - Applications built entirely on Spark have to **compensate for gaps in data management.**\r\n  - Applications that federate across Spark and Oracle usually suffer from\r\n    **inefficient data movement.**\r\n  - Operating Spark clusters are expensive because they lack administration tooling\r\n    and they have gaps in data management. **Therefore, price-performance advantages of Spark are overstated.**\r\n\r\n![current deployments](https://github.com/oracle/spark-oracle/wiki/uploads/currentDeploymentDrawbacks.png)\r\n\r\nThis project fixes those issues:\r\n- It provides a single catalog: Oracle Data Dictionary.\r\n- Oracle is responsible for data management, including:\r\n  - Consistency\r\n  - Isolation\r\n  - Security\r\n  - Storage layout\r\n  - Data lifecycle\r\n  - Data in an object store managed by Oracle as external tables\r\n- It provides support for a full Spark programming model.\r\n- **Spark on Oracle** has these  characteristics:\r\n  - Full pushdown on SQL workloads: Query, DML on all tables, DDL for external tables.\r\n  - Push SQL operations of other workloads.\r\n  - Surface Oracle capabilities like machine learning and streaming in the Spark programming model.\r\n  - Co-processor on Oracle instances to run certain kinds of Scala code. Co-processors are isolated and limited and therefore are easy to manage.\r\n- Enable simpler, smaller Spark clusters.\r\n\r\n![spark on oracle](https://github.com/oracle/spark-oracle/wiki/uploads/spark-on-oracle.png)\r\n\r\n**Feature summary:**\r\n- Catalog integration. (See [this page](https://github.com/oracle/spark-oracle/wiki/Oracle-Catalog).)\r\n- Significant support for SQL pushdown, to the extent that more than 95 (of 99) [TPCDS queries](https://github.com/oracle/spark-oracle/wiki/TPCDS-Queries)\r\n  are completely pushed to Oracle instance. (See [Operator](https://github.com/oracle/spark-oracle/wiki/Operator-Translation) and [Expression](https://github.com/oracle/spark-oracle/wiki/Expression-Translation) translation pages.)\r\n- Deployable as a Spark extension jar for Spark 3 environments.\r\n- [Language integration beyond SQL](https://github.com/oracle/spark-oracle/wiki/Language-Integration)\r\n  and [DML](https://github.com/oracle/spark-oracle/wiki/Write-Path-Flow) support.\r\n\r\nSee [Project Wiki](https://github.com/oracle/spark-oracle/wiki/home) for complete documentation.\r\n\r\n\r\n## Installation\r\n\r\nSpark on Oracle can be deployed on any Spark 3.1 or above environment.\r\nSee the [Quick Start Guide](https://github.com/oracle/spark-oracle/wiki/Quick-Start-Guide).\r\n\r\n## Documentation\r\n\r\nSee the [wiki](https://github.com/oracle/spark-oracle/wiki/home).\r\n\r\n\r\n## Examples\r\n\r\nThe [demo script](https://github.com/oracle/spark-oracle/wiki/Demo) walks you\r\nthrough the features of the library.\r\n\r\n## Help\r\n\r\nPlease file Github issues.\r\n\r\n## Contributing\r\n\r\n\u003c!-- If your project has specific contribution requirements, update the\r\n    CONTRIBUTING.md file to ensure those requirements are clearly explained. --\u003e\r\n\r\nThis project welcomes contributions from the community. Before submitting a pull\r\nrequest, please [review our contribution guide](./CONTRIBUTING.md).\r\n\r\n## Security\r\n\r\nPlease consult the [security guide](./SECURITY.md) for our responsible security\r\nvulnerability disclosure process.\r\n\r\n## License\r\n\r\n\u003c!-- The correct copyright notice format for both documentation and software\r\n    is \"Copyright (c) [year,] year Oracle and/or its affiliates.\"\r\n    You must include the year the content was first released (on any platform) and\r\n    the most recent year in which it was revised. --\u003e\r\n\r\nCopyright (c) 2021, 2023 Oracle and/or its affiliates.\r\n\r\n\u003c!-- Replace this statement if your project is not licensed under the UPL --\u003e\r\n\r\nReleased under the Universal Permissive License v1.0 as shown at\r\n\u003chttps://oss.oracle.com/licenses/upl/\u003e.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foracle%2Fspark-oracle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foracle%2Fspark-oracle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foracle%2Fspark-oracle/lists"}