https://github.com/juliaparallel/dtables.jl

Distributed table structures and data manipulation operations built on top of Dagger.jl
https://github.com/juliaparallel/dtables.jl

Last synced: about 1 year ago
JSON representation

Distributed table structures and data manipulation operations built on top of Dagger.jl

• Host: GitHub
• URL: https://github.com/juliaparallel/dtables.jl
• Owner: JuliaParallel
• License: mit
• Created: 2022-06-18T18:14:44.000Z (almost 4 years ago)
• Default Branch: main
• Last Pushed: 2024-04-08T12:57:36.000Z (about 2 years ago)
• Last Synced: 2025-04-21T18:05:52.934Z (about 1 year ago)
• Language: Julia
• Homepage:
• Size: 418 KB
• Stars: 76
• Watchers: 11
• Forks: 4
• Open Issues: 28
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# DTables

Distributed table structures and data manipulation operations built on top of [Dagger.jl](https://github.com/JuliaParallel/Dagger.jl)

[![Dev](https://img.shields.io/badge/docs-dev-blue.svg)](https://juliaparallel.github.io/DTables.jl/dev/)

[![Build Status](https://github.com/juliaparallel/DTables.jl/actions/workflows/CI.yml/badge.svg?branch=main)](https://github.com/juliaparallel/DTables.jl/actions/workflows/CI.yml?query=branch%3Amain)

[![Coverage](https://codecov.io/gh/juliaparallel/DTables.jl/branch/main/graph/badge.svg)](https://codecov.io/gh/juliaparallel/DTables.jl)

[![Code Style: Blue](https://img.shields.io/badge/code%20style-blue-4495d1.svg)](https://github.com/invenia/BlueStyle)

# Installation

The package registered in the general repository, so you can add it by typing:

```julia

julia> ]add DTables

```

# Usage

Below you can find a quick example on how to get started with DTables.

There's a lot more you can do though, so please refer to the documentation!

```julia

# launch a Julia session with threads/workers

julia> using DTables

julia> dt = DTable((a=rand(100), b=rand(100)), 10)

DTable with 10 partitions

Tabletype: NamedTuple

julia> m = map(r -> (x=sum(r), id=Threads.threadid(),), dt)

DTable with 10 partitions

Tabletype: NamedTuple

julia> xsum = reduce((x, y) -> x + y, m, init=0, cols=[:x])

EagerThunk (running)

julia> threads_used = reduce((acc, el) -> union(acc, el), m, init=Set(), cols=[:id])

EagerThunk (running)

julia> fetch(xsum)

(x = 95.71209812014976,)

julia> fetch(threads_used)

(id = Set(Any[5, 4, 6, 13, 2, 10, 9, 12, 8, 3]),)

```