Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jac18281828/rustdata
Example data ingestion and processing in rust
https://github.com/jac18281828/rustdata
Last synced: about 2 months ago
JSON representation
Example data ingestion and processing in rust
- Host: GitHub
- URL: https://github.com/jac18281828/rustdata
- Owner: jac18281828
- Created: 2024-08-23T19:03:07.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2024-08-27T18:33:45.000Z (5 months ago)
- Last Synced: 2024-10-15T19:13:58.132Z (3 months ago)
- Language: Rust
- Size: 66.4 KB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# rustdata
## Disclaimer
This is an unscientific comparision of Parquet and Postgress for a simple stat calculation.
The dataset is 2500 rows of Ethereum Event data. Eigenlayer IRewardsCoordinator.RewardsClaimed. The data is stored to the database and then serialized back with the purpose of computing some common statistics, mean, min, max, etc.
## Postgres
#### 1. Start Postgres database
```bash
docker run --name db-postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=password -e POSTGRES_DB=edb -p 5432:5432 -d postgres
```
#### 2. Upload Postgres data
```bash
cargo run --release --bin pg
```
## Parquet
#### 1. run the client
```bash
cargo run --release --bin pq
```
## Comparison Chart
| Storage | Write Time (us) | Read Time (us) | Storage Size (kB) |
| -------------------------------------- | --------------- | -------------- | ----------------- |
| [Parquet](https://parquet.apache.org/) | 2687 | 339 | 583 |
| [Postgres](https://www.postgresql.org) | 1037531 | 2539 | 1048 |
| [DuckDB](https://duckdb.org/) | 277615 | 798 | 1300 |
| SQLite | 1040026 | 320 | 792 |