{"id":18395404,"url":"https://github.com/oeo/traqq","last_synced_at":"2025-08-30T12:34:37.697Z","repository":{"id":260850261,"uuid":"882525106","full_name":"oeo/traqq","owner":"oeo","description":"High-performance event processing system for data indexing","archived":false,"fork":false,"pushed_at":"2024-11-05T17:43:24.000Z","size":100,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-27T22:23:32.231Z","etag":null,"topics":["json","redis","rust"],"latest_commit_sha":null,"homepage":"https://crates.io/crates/traqq","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oeo.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-03T02:19:40.000Z","updated_at":"2024-11-30T01:33:09.000Z","dependencies_parsed_at":"2024-11-03T03:19:11.327Z","dependency_job_id":"a5e1d370-160a-427e-9697-6e85c649681e","html_url":"https://github.com/oeo/traqq","commit_stats":null,"previous_names":["oeo/traqq"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oeo%2Ftraqq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oeo%2Ftraqq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oeo%2Ftraqq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oeo%2Ftraqq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oeo","download_url":"https://codeload.github.com/oeo/traqq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248852109,"owners_count":21171839,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","redis","rust"],"created_at":"2024-11-06T02:10:52.377Z","updated_at":"2025-08-30T12:34:37.691Z","avatar_url":"https://github.com/oeo.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# traqq\n\nA high-performance event processing system that transforms JSON events into optimized Redis commands for real-time analytics, enabling complex queries without post-processing.\n\nTraqq is designed to be a high-performance, low-memory event processing system that can handle large volumes of events and scale to meet the demands of even the largest organizations. It will be used to process events from a variety of sources, including web servers, mobile apps, and IoT devices.\n\nEventually, Traqq will be able to support RocksDB, Redis, and other storage solutions.\n\n### @done\n- [x] Event parsing and validation\n- [x] Property sanitization\n- [x] Compound key generation\n- [x] Metric generation\n- [x] Concurrent processing\n- [x] Performance benchmarking\n\n### @todo\n- [ ] Convert Redis commands to RocksDB commands\n- [ ] Network interface wrapper for RocksDB\n    - Examples in ./src/idea/*\n- [ ] Command line interface exposed in main.rs\n\n## Performance Highlights\nPC: Apple M2 Max 2023 64GB\n\n- **Processing Speed**: 40,000+ events/second on a single thread\n- **Memory Efficient**: ~6.7MB for 5000 test events\n- **Concurrent Support**: Multi-threaded event processing\n- **Scaling Performance**:\n  - Level 1: ~19,685 events/sec\n  - Level 5: ~11,214 events/sec\n  - Level 10: ~7,543 events/sec\n  - Level 20: ~4,343 events/sec\n\n## Quick Start\n\nSee [main.rs](src/main.rs) for a basic usage demonstration.\n\n```rust\nuse traqq::prelude::*;\n\nlet config = TraqqConfig::default();\nlet event = IncomingEvent::from_json(serde_json::json!({\n    \"event\": \"purchase\",\n    \"amount\": 99.99,\n    \"ip\": \"127.0.0.1\",\n    \"utm_source\": \"google\",\n    \"utm_medium\": \"cpc\"\n})).unwrap();\n\nmatch ProcessedEvent::from_incoming(event, \u0026config) {\n    Ok(processed) =\u003e processed.pretty_print(),\n    Err(e) =\u003e println!(\"Error: {}\", e),\n}\n```\n\n### Installation\n\n```toml\n[dependencies]\ntraqq = \"0.1.3\"\n```\n\n```bash\n# run unit tests\ncargo test\n\n# run tests w benchmarking output\ncargo test -- --nocapture\n\n# run example\ncargo run\n```\n\n## Core Features\n\n### 1. Configuration System\n\n```rust\nTraqqConfig {\n    time: TimeConfig {\n        store_hourly: true,\n        timezone: \"America/New_York\".to_string(),\n    },\n    mapping: MappingConfig {\n        bitmap: vec![\"ip\".to_string()],\n        add: vec![\"event\".to_string()],\n        add_value: vec![/* value metrics */],\n    },\n    limits: LimitsConfig {\n        max_field_length: 128,\n        max_value_length: 512,\n        max_combinations: 1000,\n        max_metrics_per_event: 1000,\n    },\n}\n```\n\n### 2. Event Processing Pipeline\n\n1. Event Ingestion\n2. Sanitization\n3. Property Extraction\n4. Metric Generation\n5. Redis Command Generation\n\n### 3. Redis Integration\n\n#### Key Structure\n```\n\u003cmetric_type\u003e:\u003cbucket_type\u003e:\u003ctimestamp\u003e:\u003cpattern\u003e:\u003cvalues\u003e\n```\n\n#### Example Commands\n\n```redis\nPFADD bmp:d:1696118400:ip 127.0.0.1\nINCR add:d:1696118400:event:purchase\nINCRBY adv:d:1696118400:amount:event:purchase 99.99\n```\n\n## Components\n\n- Core Types: `TraqqConfig`, `IncomingEvent`, `ProcessedEvent`, `RedisCommand`, `RedisCommandType`\n- Utility Modules: `constants.rs`, `utils.rs`\n\n## License\n\nMIT\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foeo%2Ftraqq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foeo%2Ftraqq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foeo%2Ftraqq/lists"}