https://github.com/kamu-data/kamu-engine-spark

Spark application for streaming transactional transformations between datasets
https://github.com/kamu-data/kamu-engine-spark

Last synced: 3 days ago
JSON representation

Spark application for streaming transactional transformations between datasets

• Host: GitHub
• URL: https://github.com/kamu-data/kamu-engine-spark
• Owner: kamu-data
• License: apache-2.0
• Created: 2019-05-19T19:45:33.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2024-08-02T23:10:31.000Z (3 months ago)
• Last Synced: 2024-08-03T01:10:56.638Z (3 months ago)
• Language: Scala
• Size: 394 KB
• Stars: 6
• Watchers: 8
• Forks: 0
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

        
# Apache Spark Engine

This the implementation of the `Engine` contract of [Open Data Fabric](http://opendatafabric.org/) using the [Apache Spark](https://spark.apache.org/) data processing framework. It is currently in use in [kamu-cli](https://github.com/kamu-data/kamu-cli) data management tool.

## Features

- Spark engine currently provides the most rich SQL dialect for map/filter style transformations

- Integrates [GeoSpark](http://geospark.datasyslab.org/) to provide geo-spatial SQL functions

- It is used by [kamu-cli](https://github.com/kamu-data/kamu-cli) for ingesting data into Parquet

- It is used by [kamu-cli](https://github.com/kamu-data/kamu-cli) along with [Apache Livy](https://livy.apache.org/) to provide SQL queries functionality in the Jupyter notebooks

## Known Issues

- Takes a long time to start up which is hurting the user experience

- Does not support temporal table joins

  - You might be better off using [Flink-based engine](https://github.com/kamu-data/kamu-engine-flink) for joining and aggregating event streams

- TODO

## Developing

See the [Developer Guide](DEVELOPER.md)