https://github.com/danielbeach/RustForDataPipelines

Testing out if Rust can be used for a normal Data Engineering Pipeline.
https://github.com/danielbeach/RustForDataPipelines

Last synced: 3 months ago
JSON representation

Testing out if Rust can be used for a normal Data Engineering Pipeline.

• Host: GitHub
• URL: https://github.com/danielbeach/RustForDataPipelines
• Owner: danielbeach
• Created: 2023-01-14T01:14:06.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2023-03-03T01:07:34.000Z (over 1 year ago)
• Last Synced: 2024-06-30T13:41:20.580Z (5 months ago)
• Language: Rust
• Size: 3.91 KB
• Stars: 10
• Watchers: 3
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# RustForDataPipelines

Testing out if Rust can be used for a normal Data Engineering Pipeline.

Check out the full blog post here. 

https://www.confessionsofadataguy.com/using-rust-to-write-a-data-pipeline-thoughts-musings/

The idea was to try things like HTTP, File Systems, and Database calls (Postgres) out with Rust.

These are normal data engineering functions, is it too verbose to write data pipelines in Rust?

The data flow is as follows

- download `.zip` file.

- unpack the `.zip` file into `.csv`, mess with files.

- connect Rust to `postgres`

- Process CSV file into `postgres`

- Run some `sql` and push some `rows`