https://github.com/oeo/traqq
High-performance event processing system for data indexing
https://github.com/oeo/traqq
json redis rust
Last synced: about 1 month ago
JSON representation
High-performance event processing system for data indexing
- Host: GitHub
- URL: https://github.com/oeo/traqq
- Owner: oeo
- License: mit
- Created: 2024-11-03T02:19:40.000Z (7 months ago)
- Default Branch: master
- Last Pushed: 2024-11-05T17:43:24.000Z (7 months ago)
- Last Synced: 2025-03-27T22:23:32.231Z (about 2 months ago)
- Topics: json, redis, rust
- Language: Rust
- Homepage: https://crates.io/crates/traqq
- Size: 97.7 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
- License: LICENSE
Awesome Lists containing this project
README
# traqq
A high-performance event processing system that transforms JSON events into optimized Redis commands for real-time analytics, enabling complex queries without post-processing.
Traqq is designed to be a high-performance, low-memory event processing system that can handle large volumes of events and scale to meet the demands of even the largest organizations. It will be used to process events from a variety of sources, including web servers, mobile apps, and IoT devices.
Eventually, Traqq will be able to support RocksDB, Redis, and other storage solutions.
### @done
- [x] Event parsing and validation
- [x] Property sanitization
- [x] Compound key generation
- [x] Metric generation
- [x] Concurrent processing
- [x] Performance benchmarking
### @todo
- [ ] Convert Redis commands to RocksDB commands
- [ ] Network interface wrapper for RocksDB
- Examples in ./src/idea/*
- [ ] Command line interface exposed in main.rs
## Performance Highlights
PC: Apple M2 Max 2023 64GB
- **Processing Speed**: 40,000+ events/second on a single thread
- **Memory Efficient**: ~6.7MB for 5000 test events
- **Concurrent Support**: Multi-threaded event processing
- **Scaling Performance**:
- Level 1: ~19,685 events/sec
- Level 5: ~11,214 events/sec
- Level 10: ~7,543 events/sec
- Level 20: ~4,343 events/sec
## Quick Start
See [main.rs](src/main.rs) for a basic usage demonstration.
```rust
use traqq::prelude::*;
let config = TraqqConfig::default();
let event = IncomingEvent::from_json(serde_json::json!({
"event": "purchase",
"amount": 99.99,
"ip": "127.0.0.1",
"utm_source": "google",
"utm_medium": "cpc"
})).unwrap();
match ProcessedEvent::from_incoming(event, &config) {
Ok(processed) => processed.pretty_print(),
Err(e) => println!("Error: {}", e),
}
```
### Installation
```toml
[dependencies]
traqq = "0.1.3"
```
```bash
# run unit tests
cargo test
# run tests w benchmarking output
cargo test -- --nocapture
# run example
cargo run
```
## Core Features
### 1. Configuration System
```rust
TraqqConfig {
time: TimeConfig {
store_hourly: true,
timezone: "America/New_York".to_string(),
},
mapping: MappingConfig {
bitmap: vec!["ip".to_string()],
add: vec!["event".to_string()],
add_value: vec![/* value metrics */],
},
limits: LimitsConfig {
max_field_length: 128,
max_value_length: 512,
max_combinations: 1000,
max_metrics_per_event: 1000,
},
}
```
### 2. Event Processing Pipeline
1. Event Ingestion
2. Sanitization
3. Property Extraction
4. Metric Generation
5. Redis Command Generation
### 3. Redis Integration
#### Key Structure
```
::::
```
#### Example Commands
```redis
PFADD bmp:d:1696118400:ip 127.0.0.1
INCR add:d:1696118400:event:purchase
INCRBY adv:d:1696118400:amount:event:purchase 99.99
```
## Components
- Core Types: `TraqqConfig`, `IncomingEvent`, `ProcessedEvent`, `RedisCommand`, `RedisCommandType`
- Utility Modules: `constants.rs`, `utils.rs`
## License
MIT