https://github.com/stepfenshawn/predato-rs

A tiny but blazing fast distributed computation engine in rust. [wip]
https://github.com/stepfenshawn/predato-rs

Last synced: about 1 year ago
JSON representation

A tiny but blazing fast distributed computation engine in rust. [wip]

• Host: GitHub
• URL: https://github.com/stepfenshawn/predato-rs
• Owner: StepfenShawn
• Created: 2024-06-15T03:45:30.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2024-09-24T06:45:47.000Z (over 1 year ago)
• Last Synced: 2025-01-29T08:11:53.373Z (over 1 year ago)
• Language: Rust
• Homepage:
• Size: 25.4 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Predato-rs[WIP]

A tiny but blazing fast distributed computation engine in rust. 

# TODO Features

* distributed computation

* column-based

* vectorized query

* streaming batch

* llvm-jit query runtime

# Dataframe Expression API

```rust

let ctx = Context::new();

let mut lhs = ctx.read.format("csv").load("a.csv")?;

let mut rhs = ctx.read.format("csv").load("a.csv")?;

lhs = lhs.with_expr((

    case(col("*"))

    | (col("l1") > col("l2")) >>= (lit(1))

    | (col("l2") < col("l3")) >>= (lit(3))

    | otherwise >>= lit(2)

).as_col("res1"))

lhs = lhs.with_exprs([

    (col("l1") + col("l2")).as_col("sum_12"),

    col(*).sum().as_col("all_sum")

])

// Filter

lhs /= (col("l1") > 0)

// Or

lhs = lhs.filter(col("l1") > 0)

// Join

rhs = rhs.join(lhs, how: JoinType::Left)

         .left_on(&[col("l1"), col("l2")])

         .right_on(&[col("l1"), col("l2")]);

// Group By

rhs = rhs.group_by(keys: &[col("l1"), col("l2")])

         .fmap(|&chunk| {chunk})

pipe!{rhs, show . collect . select([col("l1"), col("l2"), col("l3")])}

// Or

rhs.select([col("l1"), col("l2"), col("l3")])

   .collect()

   .show();

```

# test

```

cargo test -r

```

# benchmarks

wip