https://github.com/milenkovicm/ballista_delta
Datafusion Ballista support for Delta Table (showcase project)
https://github.com/milenkovicm/ballista_delta
ballista datafusion delta-lake deltatable distributed objectstore rust rustlang
Last synced: 2 months ago
JSON representation
Datafusion Ballista support for Delta Table (showcase project)
- Host: GitHub
- URL: https://github.com/milenkovicm/ballista_delta
- Owner: milenkovicm
- Created: 2025-01-21T17:34:30.000Z (4 months ago)
- Default Branch: master
- Last Pushed: 2025-03-05T09:31:30.000Z (3 months ago)
- Last Synced: 2025-03-05T10:31:32.674Z (3 months ago)
- Topics: ballista, datafusion, delta-lake, deltatable, distributed, objectstore, rust, rustlang
- Language: Rust
- Homepage:
- Size: 41 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Datafusion Ballista Support For Delta Table
Since version `43.0.0` [datafusion ballista](https://github.com/apache/datafusion-ballista) extending core components and functionalities.
This example demonstrate extending [datafusion ballista](https://github.com/apache/datafusion-ballista) to support [delta.rs](https://delta-io.github.io/delta-rs/) read operations.
>
> [!NOTE]
>
> This project has been part of Datafusion Ballista show case series
>
> - [Ballista (Datafusion) Python Support](https://github.com/milenkovicm/ballista_python)
> - [Datafusion Ballista Read Support For Delta Table](https://github.com/milenkovicm/ballista_delta)
>
>
> [!IMPORTANT]
>
> This is just a showcase project, it is not meant to be maintained.
>
Setting up [standalone ballista](examples/standalone.rs):
```rust
use ballista::prelude::{SessionConfigExt, SessionContextExt};
use ballista_delta::{BallistaDeltaLogicalCodec, BallistaDeltaPhysicalCodec};
use datafusion::{
common::Result,
execution::SessionStateBuilder,
prelude::{SessionConfig, SessionContext},
};
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<()> {
let config = SessionConfig::new_with_ballista()
.with_ballista_logical_extension_codec(Arc::new(BallistaDeltaLogicalCodec::default()))
.with_ballista_physical_extension_codec(Arc::new(BallistaDeltaPhysicalCodec::default()));
let state = SessionStateBuilder::new()
.with_config(config)
.with_default_features()
.build();
let table = deltalake::open_table("./data/people_countries_delta_dask")
.await
.unwrap();
let ctx = SessionContext::standalone_with_state(state).await?;
ctx.register_table("demo", Arc::new(table)).unwrap();
ctx.sql("select * from demo").await?.show().await?;
Ok(())
}
```
Other examples show extending [client](examples/cluster_client.rs), [scheduler](examples/cluster_scheduler.rs)
and [executor](examples/cluster_executor.rs) for cluster deployment.