Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/javrasya/crevasse
A data migration tool for data formats like Apache Iceberg
https://github.com/javrasya/crevasse
Last synced: 14 days ago
JSON representation
A data migration tool for data formats like Apache Iceberg
- Host: GitHub
- URL: https://github.com/javrasya/crevasse
- Owner: javrasya
- Created: 2024-04-30T09:50:39.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-05-03T21:38:04.000Z (8 months ago)
- Last Synced: 2024-12-20T13:50:24.959Z (24 days ago)
- Language: Java
- Size: 789 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
![crevasse-logo.png](asssets/crevasse-logo.png)
# Crevasse: Data Migration Tool for Iceberg
Crevasse is a powerful data migration tool designed specifically for Iceberg tables. It supports Groovy scripts with a Domain-Specific Language (DSL) to perform a variety of table operations, including:
* Adding columns
* Removing columns
* Updating primitive types
* Toggling nullability
* Adding table properties
* Removing table properties
* Adding partition columns
* Removing partition columnsCrevasse provides Gradle tasks to execute these migration scripts step by step, ensuring seamless migration of data structures while keeping track of the latest applied step number in the Iceberg table metadata. This enables efficient migration processes by running only the relevant new migration scripts on subsequent updates.
Additionally, Crevasse offers a Gradle task to automatically generate migration scripts based on changes made to Avro schemas, similar to the Django migration module. Instead of Django model classes, Crevasse operates on Avro schemas, making it easy to manage and update data structures as your project evolves.
With Crevasse, you can confidently manage and evolve your Iceberg tables with ease, ensuring data integrity and consistency throughout your data migration process.