{"id":51665978,"url":"https://github.com/geeknik/streamtrace","last_synced_at":"2026-07-14T20:30:49.680Z","repository":{"id":350332965,"uuid":"1206269502","full_name":"geeknik/streamtrace","owner":"geeknik","description":"An open forensic runtime for reconstructing what actually happened.","archived":false,"fork":false,"pushed_at":"2026-04-24T17:02:17.000Z","size":325,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-03T04:33:57.332Z","etag":null,"topics":["forensic","open","runtime"],"latest_commit_sha":null,"homepage":"https://deepforkcyber.com/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geeknik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-09T18:42:15.000Z","updated_at":"2026-05-04T09:56:32.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/geeknik/streamtrace","commit_stats":null,"previous_names":["geeknik/streamtrace"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/geeknik/streamtrace","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geeknik%2Fstreamtrace","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geeknik%2Fstreamtrace/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geeknik%2Fstreamtrace/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geeknik%2Fstreamtrace/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geeknik","download_url":"https://codeload.github.com/geeknik/streamtrace/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geeknik%2Fstreamtrace/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35478630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-14T02:00:06.603Z","response_time":114,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["forensic","open","runtime"],"created_at":"2026-07-14T20:30:45.630Z","updated_at":"2026-07-14T20:30:49.670Z","avatar_url":"https://github.com/geeknik.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# StreamTrace\n\n**An open forensic runtime for reconstructing what actually happened.**\n\nYour systems already know the truth.\nThey just say it in different languages.\n\nStreamTrace ingests raw machine exhaust -- logs, events, traces, webhooks -- and turns it into a unified, defensible timeline across systems.\n\nNo vendor lock-in.\nNo black box.\nNo guessing.\n\n---\n\n## What This Is\n\nStreamTrace is not:\n\n* a SIEM\n* an observability dashboard\n* another \"AI analytics platform\"\n\nStreamTrace is:\n\n\u003e **A system for reconstructing reality from fragmented data.**\n\nIt answers:\n\n* What actually happened?\n* In what order?\n* Across which systems?\n* With what evidence?\n* Can we prove it?\n\n---\n\n## Tech Stack\n\n| Component | Technology | Why |\n|---|---|---|\n| Backend | Rust | Memory safety, performance, zero-cost abstractions |\n| Database | PostgreSQL + TimescaleDB | Time-series hypertables, chunk pruning, full-text search |\n| Frontend | SolidJS + TypeScript | Fine-grained reactivity, small bundle, type safety |\n| Deployment | Docker Compose | Single-command setup, self-hosted, no vendor lock-in |\n| Hashing | BLAKE3 | 3x faster than SHA-256, cryptographically secure |\n| Auth | Argon2 + Bearer tokens | Timing-safe API key validation |\n\nSee [ARCHITECTURE.md](ARCHITECTURE.md) for detailed design decisions.\n\n---\n\n## Why This Exists\n\nMost tools optimize for:\n\n* uptime\n* alerts\n* dashboards\n* summaries\n\nThey do not optimize for truth.\n\nWhen something breaks -- or gets exploited -- teams still:\n\n* stitch logs manually\n* jump between tools\n* lose context\n* argue about timelines\n* fail to prove causality\n\nThat's unacceptable.\n\nStreamTrace exists to make reconstruction:\n\n* fast\n* unified\n* evidence-backed\n* reproducible\n\n---\n\n## Core Capabilities\n\n### 1. Ingest Anything\n\n* JSON logs\n* CSV exports\n* API/webhook events\n* cloud audit logs\n* auth events\n* application telemetry\n* arbitrary data streams\n\nIf it emits events, it works.\n\n---\n\n### 2. Preserve Raw Evidence\n\nEvery event is stored:\n\n* immutably\n* with hashes\n* with provenance\n\nNothing is rewritten.\nNothing is hidden.\n\n---\n\n### 3. Normalize Without Losing Meaning\n\nEvents are mapped into a shared forensic model:\n\n* timestamps (occurred / observed / received)\n* actors, subjects, objects\n* network + device context\n* source attribution\n\nOriginal structure is always preserved.\n\n---\n\n### 4. Correlate Across Systems\n\nLink events by:\n\n* identity (users, accounts)\n* sessions and tokens\n* IPs and devices\n* hosts and workloads\n* custom keys\n\nBuild chains like:\n\n```\nlogin -\u003e token -\u003e repo access -\u003e data export\n```\n\n---\n\n### 5. Investigate via Timeline\n\nThe primary interface is a **global timeline**:\n\n* zoom from hours to milliseconds\n* filter by entity, source, type\n* stack related events\n* pivot instantly\n\nTime is the spine. Everything else hangs off it.\n\n---\n\n### 6. Build Cases\n\n* pin evidence\n* annotate events\n* track hypotheses\n* export reports\n\nNo more screenshots. No more guesswork.\n\n---\n\n### 7. Export Evidence\n\nGenerate:\n\n* structured timelines\n* raw event bundles\n* integrity manifests\n* investigation summaries\n\nBuilt for:\n\n* incident response\n* postmortems\n* compliance\n* legal review\n\n---\n\n## Quick Start\n\n### 1. Clone\n\n```bash\ngit clone https://github.com/geeknik/streamtrace\ncd streamtrace\n```\n\n### 2. Run\n\n```bash\ncp .env.example .env\ndocker compose up\n```\n\n### 3. Open UI\n\n```\nhttp://localhost:3000\n```\n\nAPI is available at `http://localhost:8080`.\n\n### 4. Send Data\n\n```bash\ncurl -X POST http://localhost:8080/v1/ingest/events \\\n  -H \"Authorization: Bearer dev-token\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"event_type\": \"auth.login\",\n    \"occurred_at\": \"2026-04-09T12:00:00Z\",\n    \"actor\": {\"id\": \"alice\"},\n    \"network\": {\"src_ip\": \"203.0.113.10\"}\n  }'\n```\n\nOpen the timeline. You'll see it immediately.\n\n### Development Setup\n\nPrerequisites: Rust 1.75+, Node.js 22+, Docker\n\n```bash\n# Start database\ndocker compose up timescaledb -d\n\n# Copy environment\ncp .env.example .env\n\n# Run migrations and start API\ncargo run --bin streamtrace\n\n# Start frontend (separate terminal)\ncd frontend \u0026\u0026 npm install \u0026\u0026 npm run dev\n```\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for the full development workflow.\n\n---\n\n## Architecture\n\n```\nIngestion -\u003e Raw Store -\u003e Normalize -\u003e Index -\u003e Correlate -\u003e Investigate\n```\n\n### Crates\n\n| Crate | Responsibility |\n|---|---|\n| `st-common` | Shared types, forensic event model, errors, config |\n| `st-crypto` | BLAKE3/SHA-256 hashing, integrity verification |\n| `st-store` | PostgreSQL/TimescaleDB access layer |\n| `st-parser` | Pluggable parser framework (JSON, CSV, syslog) |\n| `st-ingest` | Ingestion pipeline orchestration |\n| `st-index` | Timeline queries, full-text search |\n| `st-correlate` | Cross-system event correlation |\n| `st-cases` | Case management and evidence export |\n| `st-api` | REST API (axum), auth, rate limiting |\n| `st-server` | Binary entry point, config, graceful shutdown |\n\nSee [ARCHITECTURE.md](ARCHITECTURE.md) for the full dependency graph, database schema, and design decisions.\n\n---\n\n## Example Use Cases\n\n### Incident Response\n\nReconstruct a breach across:\n\n* identity provider\n* cloud logs\n* application events\n* endpoints\n\n### SRE / Outage Analysis\n\nCorrelate:\n\n* deploys\n* errors\n* traffic shifts\n* queue failures\n\n### Fraud Investigation\n\nTrack:\n\n* account clusters\n* device reuse\n* payment anomalies\n* session patterns\n\n### Internal Investigations\n\nBuild a defensible timeline with:\n\n* raw evidence\n* annotations\n* exportable reports\n\n---\n\n## Design Principles\n\n* **Truth \u003e convenience**\n* **Raw data is sacred**\n* **Time is the source of truth**\n* **Correlation beats aggregation**\n* **Every view must lead to evidence**\n* **No black boxes**\n\n---\n\n## What This Is Not\n\n* Not a compliance checkbox\n* Not a dashboard farm\n* Not \"AI that guesses what happened\"\n* Not a replacement for every tool you have\n\nIt is the layer that tells you what actually happened across them.\n\n---\n\n## Roadmap\n\n### Phase 1 -- Delivered\n\n* Ingestion pipeline (JSON, CSV, syslog)\n* Timeline queries with cursor-based pagination\n* Evidence viewer with raw/normalized split\n* Basic correlation (identity, session, IP, device, host)\n* Case management and evidence export\n\n### Phase 2 -- Delivered\n\n* Entity graph (resolution, relationships, entity timeline)\n* Sequence detection (built-in and custom patterns)\n* Replay mode (SSE-based chronological event streaming)\n\n### Phase 3 -- Delivered\n\n* Signed evidence bundles (Ed25519)\n* Legal hold management\n* Advanced correlation search\n* Audit logging for all security-relevant operations\n\n### Phase 4 -- Production Hardening (planned)\n\n* Horizontal scaling (stateless API nodes behind a load balancer)\n* Signing key rotation without invalidating existing bundles\n* Configurable retention policies with legal hold enforcement\n* Terraform modules for managed PostgreSQL/TimescaleDB deployments\n* Helm chart for Kubernetes deployments\n* Backup and disaster recovery runbooks\n\n---\n\n## Contributing\n\nWe want:\n\n* new parsers\n* better correlation logic\n* real-world datasets\n* investigation workflows\n* performance improvements\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for setup, standards, and workflow.\nSee [Parser Guide](crates/st-parser/PARSERS.md) for step-by-step instructions on adding support for your log format.\n\n---\n\n## Philosophy\n\nMost systems optimize for visibility.\n\nStreamTrace optimizes for **reality reconstruction**.\n\nThose are not the same thing.\n\n---\n\n## License\n\n[Apache 2.0](LICENSE)\n\n---\n\n## Final Note\n\nYour logs are not noise.\n\nThey are fragmented truth.\n\nStreamTrace turns them into something you can actually trust.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeeknik%2Fstreamtrace","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeeknik%2Fstreamtrace","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeeknik%2Fstreamtrace/lists"}