{"id":22050230,"url":"https://github.com/queukat/spark_oracle_hive_streaming","last_synced_at":"2026-04-12T23:44:05.309Z","repository":{"id":164600303,"uuid":"587742366","full_name":"queukat/spark_oracle_hive_streaming","owner":"queukat","description":null,"archived":false,"fork":false,"pushed_at":"2023-12-15T21:29:10.000Z","size":126,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-28T21:18:02.603Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/queukat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-11T13:34:50.000Z","updated_at":"2023-08-29T10:55:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"26081f37-b8ee-44e9-adf7-73cada570006","html_url":"https://github.com/queukat/spark_oracle_hive_streaming","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queukat%2Fspark_oracle_hive_streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queukat%2Fspark_oracle_hive_streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queukat%2Fspark_oracle_hive_streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queukat%2Fspark_oracle_hive_streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/queukat","download_url":"https://codeload.github.com/queukat/spark_oracle_hive_streaming/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245119592,"owners_count":20563763,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-30T14:18:23.286Z","updated_at":"2026-04-12T23:44:05.292Z","avatar_url":"https://github.com/queukat.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project: Spark Universal Migrator\n\nSpark Universal Migrator is a Scala/Spark library for full-load table-by-table migration from Oracle to Hive. It captures an Oracle snapshot SCN, reads source rows through Spark JDBC using `ROWID` range queries built from Oracle extent metadata, applies an explicit Oracle-to-Spark schema policy, and writes the result into Hive through a temporary table plus `INSERT OVERWRITE`.\n\n# Requirements\n\n- Apache Spark 3.5.7\n- Scala 2.12.x\n- Oracle as the source system\n- Hive metastore / Hive-enabled Spark as the target system\n- Oracle JDBC driver support on the runtime classpath\n\n# What It Supports\n\n- Full-load data migration\n- Table-by-table execution\n- Range-based reads using Oracle `ROWID`\n- Snapshot-based reads through captured SCN\n- Oracle type handling modes:\n  - `spark`: keep Spark JDBC inferred types for ambiguous Oracle `NUMBER`\n  - `oracle`: profile Oracle `NUMBER` columns without precision/scale metadata\n  - `skip`: avoid extra profiling and fall back to `StringType` for ambiguous `NUMBER`\n\n# What It Does Not Do\n\n- CDC / change capture\n- Incremental load orchestration\n- Schema evolution management across runs\n- Environment-variable based configuration\n- Standalone CLI entrypoint in this repository\n\n# Testing\n\nRun the test suite with:\n\n```bash\nsbt test\n```\n\nThe test suite covers SQL generation, schema conversion, Spark session creation, Spark-side JDBC load composition, and Hive overwrite behavior.\n\n# Using the Library\n\nUse the `NewSpark.migrate(...)` API from your own application entrypoint:\n\n```scala\nimport queukat.spark_universal.NewSpark\n\nobject ExampleMigration {\n  def main(args: Array[String]): Unit = {\n    NewSpark.migrate(\n      url = \"jdbc:oracle:thin:@//localhost:1521/ORCL\",\n      oracleUser = \"your_oracle_username\",\n      oraclePassword = \"your_oracle_password\",\n      tableName = \"employees\",\n      owner = \"HR\",\n      hivetable = \"employees_hive\",\n      numPartitions = 8,\n      fetchSize = 1000,\n      typeCheck = \"spark\"\n    )\n  }\n}\n```\n\n# Notes\n\n- The library expects Oracle users with access to the required metadata views used to compute extent ranges.\n- Unsupported Oracle types fail fast during schema conversion instead of being silently downgraded.\n- Temporary Hive table names are generated uniquely per migration run.\n- Logging stays on the existing `slf4j` facade so the host Spark application keeps control over the final backend.\n- The library now emits stage-oriented log messages such as `[MIGRATE]`, `[SCHEMA]`, `[LOAD]`, and `[HIVE]` for easier scanning.\n- ANSI color is opt-in/out through `-Dspark.universal.log.color=true|false`; by default the library only colors logs when it detects an interactive terminal.\n\n# Publishing\n\nThe repository includes GitHub Actions workflows for CI and Maven Central publishing. CI runs `sbt test`, and publishing uses `sbt +publishSigned` on release.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqueukat%2Fspark_oracle_hive_streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqueukat%2Fspark_oracle_hive_streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqueukat%2Fspark_oracle_hive_streaming/lists"}