{"id":49567986,"url":"https://github.com/jordandelbar/detr-mmap","last_synced_at":"2026-05-03T12:12:57.713Z","repository":{"id":335535948,"uuid":"1119986903","full_name":"jordandelbar/detr-mmap","owner":"jordandelbar","description":"A RF-DETR implementation with Rust ort/cxx, FlatBuffers over mmap and k3d","archived":false,"fork":false,"pushed_at":"2026-01-30T18:52:58.000Z","size":1492,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-31T11:37:27.661Z","etag":null,"topics":["computer-vision","flatbuffers","machine-learning","ort","rf-detr","rust","video-processing"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jordandelbar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-20T08:43:10.000Z","updated_at":"2026-01-30T18:53:02.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jordandelbar/detr-mmap","commit_stats":null,"previous_names":["jordandelbar/detr-mmap"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jordandelbar/detr-mmap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandelbar%2Fdetr-mmap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandelbar%2Fdetr-mmap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandelbar%2Fdetr-mmap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandelbar%2Fdetr-mmap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jordandelbar","download_url":"https://codeload.github.com/jordandelbar/detr-mmap/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandelbar%2Fdetr-mmap/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32568235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T06:36:36.687Z","status":"ssl_error","status_checked_at":"2026-05-03T06:36:09.306Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","flatbuffers","machine-learning","ort","rf-detr","rust","video-processing"],"created_at":"2026-05-03T12:12:56.282Z","updated_at":"2026-05-03T12:12:57.690Z","avatar_url":"https://github.com/jordandelbar.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RF-DETR on Edge Devices with Zero-Serialization Memory-Mapped IPC\n[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)\n[![Build](https://github.com/jordandelbar/detr-mmap/actions/workflows/ci.yaml/badge.svg)](https://github.com/jordandelbar/detr-mmap/actions/workflows/ci.yaml)\n[![codecov](https://codecov.io/gh/jordandelbar/detr-mmap/branch/main/graph/badge.svg?token=GFI0VJOZ9G)](https://codecov.io/gh/jordandelbar/detr-mmap)\n\nAn RF-DETR implementation with Rust, CXX | ORT, FlatBuffers and k3d\n\n## Overview\n\nThis project implements a complete edge AI pipeline with RF-DETR object detection, designed for minimal latency and maximum throughput through zero-serialization shared memory communication.\nA sentry mode state machine with hysteresis reduces computation by switching to standby when no humans are detected.\nThe state machine requires multiple consecutive detections before transitioning to alarmed mode (30 FPS), and multiple consecutive non-detections before returning to standby (3 FPS), preventing rapid flickering between modes.\n\n![screencast](https://github.com/user-attachments/assets/814a93cf-61cf-4937-8a9b-a6c5bb66af69)\n\n\u003csub\u003eVideo by [Martina Tomšič](https://www.pexels.com/video/dog-waiting-in-front-of-a-door-6477490/) on Pexels\u003c/sub\u003e\n\n## Tech Stack\n\n  - Capture: Camera frame acquisition using [v4l]\n  - Preprocessing: CPU (default) or CUDA (automatic with TensorRT backend)\n  - Inference: [RF-DETR] model inference:\n    - Rust + [ONNX] [ORT] Runtime (CPU / CUDA)\n    - TensorRT via C++ (integrated into Rust using [CXX])\n  - Controller: State machine managing sentry mode (Standby/Alarmed) based on human detection, publishes events to [MQTT]\n  - Gateway: WebSocket [Axum] server streaming frames + detections to clients\n  - IPC: Memory-mapped files with [FlatBuffers] for zero-serialization, synchronized with POSIX mqueue semaphores.\n  - Broker: [mosquitto] MQTT broker for centralized event collection\n  - Observability: [OpenTelemetry] + [Jaeger] for tracing and metrics\n\n## Architecture\n\n![Alt text](./docs/schemas/detr-mmap-context.svg)\n\n![Alt text](./docs/schemas/detr-mmap-app.svg)\n\n## Technical decisions\n\nEdge devices have limited CPU and memory. Network protocol overhead (TCP, HTTP/2, serialization) adds latency and CPU usage.\n\nMemory-mapped files (`mmap`) provide true zero-serialization IPC. There is no runtime serialization or copying; data is written once and read zero-copy via memory mapping.\nThe trade-off is that it only works for local IPC, this is not secured for cloud deployment with shared machine but clearly fitting for edge deployments.\n\nI used k3s even though it adds some memory footprint for the ease of use when it comes to edge deployment.\n\nThe pipeline is intentionally asynchronous. Frames are displayed immediately while detections correspond to the previous frame,\nintroducing \u003c1-frame temporal skew (multiple frame skew if using the CPU). This design maximizes throughput and minimizes perceived latency.\n\nGPU preprocessing is enabled automatically with the TensorRT backend. On edge devices where the CPU is typically the bottleneck, offloading preprocessing to the GPU frees CPU resources for other tasks.\n\n## Installation\n\nFor quick demo:\n - **Docker** with buildx\n\nFor full setup:\n - **Docker** with buildx\n - **Rust** 1.92+ (`rustup default 1.92.0`)\n - **k3d** (lightweight k3s in docker) - [install guide](https://k3d.io/stable/#installation)\n - **uv** (Python package manager) - [install guide](https://docs.astral.sh/uv/getting-started/installation/)\n - **ONNX Runtime** (installed automatically in containers)\n - **CUDA** (to run GPU inference)\n\n### Quick Start\n\n```bash\n# Clone repository\ngit clone https://github.com/jordandelbar/detr-mmap.git\ncd detr-mmap\n```\n\n#### Quick Demo (pre-built images)\n\nRun with pre-built images from GHCR (model baked in, no build required):\n\n```bash\n# Start all services\njust demo-up\n\n# Open webpage\njust open-webpage\n\n# Stop services\njust demo-down\n```\n\n#### Local Development\n\nBuild and run locally from source with CPU inference:\n\n```bash\n# Download ONNX model from HuggingFace\njust download-model\n\n# Build and start all services\njust dev-up\n\n# Open webpage\njust open-webpage\n\n# Stop services\njust dev-down\n```\n\n#### k3d Deployment (GPU)\n\nFor GPU inference with TensorRT, you need to build the INT8 engine for your specific GPU:\n\n```bash\n# Build TensorRT INT8 engine (requires NVIDIA GPU, CUDA, TensorRT)\njust build-engine\n\n# Create k3d cluster + deploy services\njust up\n\n# Check deployment\nkubectl get pods -n detr-mmap\n\n# View logs\nkubectl logs -n detr-mmap -l component=inference --follow\n```\nBy default, this runs the TensorRT version of inference, so ensure you have configured your Docker\ndaemon to run with CUDA. See the next section for setup instructions.\n\n## Running with CUDA\n\nFollow [this guide](https://github.com/jordandelbar/yolo-tonic/blob/a146a7820c173545c47c5c1bac7cdf0417773150/docs/setup/nvidia_docker.md) to set up CUDA correctly.\n\n## Benchmarks \u0026 Performance\n\nBenchmarks run on NVIDIA RTX 2060 Super and AMD Ryzen 7 9800x3D with 1920x1080 RGB input frames.\n\n\u003e [!NOTE]\n\u003e These benchmarks were run on high-end desktop hardware. Edge device performance will vary.\n\u003e Benchmarks on edge device (Raspberry Pi, Jetson) are welcome.\n\n### Benchmarks\n\nAll benchmarks measured with 1920x1080 RGB input frames.\n\n#### Inference\n\n| Backend    | Latency   | Throughput |\n|------------|-----------|------------|\n| ORT (CPU)  | 66.8 ms   | ~15 FPS    |\n| ORT (CUDA) | 15.1 ms   | ~66 FPS    |\n| TensorRT   |  3.7 ms   | ~270 FPS   |\n\n#### Preprocessing\n\n| Backend                   | Latency  | Speedup |\n|---------------------------|----------|---------|\n| CPU (AMD Ryzen 9800x3D)   | 444 µs   | 1x      |\n| GPU (NVIDIA RTX 2060S)    | 21 µs    | 21x     |\n\n\u003e [!NOTE]\n\u003e GPU preprocessing benchmarks measure kernel execution only, excluding host-to-device transfer.\n\u003e In production with TensorRT, frames stay on GPU (decoder → preprocess → inference), achieving these speeds.\n\n#### Other Components\n\n| Component      | Latency  |\n|----------------|----------|\n| Postprocessing | 23 µs    |\n| Frame write    | 309 µs   |\n| Frame read     | 46 ns    |\n\n#### IPC Performance (FlatBuffers + mmap)\n\n| Scenario           | Write    | Read     | Roundtrip |\n|--------------------|----------|----------|-----------|\n| No detections      | 22 ns    | 29 ns    | 53 ns     |\n| Single detection   | 51 ns    | 51 ns    | 101 ns    |\n| Few detections (5) | 113 ns   | 120 ns   | 228 ns    |\n| Many detections    | 352 ns   | 350 ns   | 703 ns    |\n| Crowded scene      | 1.6 µs   | 1.6 µs   | 3.2 µs    |\n\nRun benchmarks yourself:\n```bash\njust bench\n# HTML reports output to benchmark-reports/\n```\n\n### Performance\n\nHere is an example of a typical trace span:\n\n![Alt text](./docs/images/traces.png)\n\n| Stage                    | Component | Latency     |\n|--------------------------|-----------|-------------|\n| MJPEG decoding           | Capture   | ~2.5 ms     |\n| Frame write (mmap)       | Capture   | ~300 µs     |\n| Host-to-device transfer  | Inference | ~600 µs     |\n| Preprocessing (kernel)   | Inference | ~150 µs     |\n| Model inference          | Inference | ~4 ms       |\n| Postprocessing           | Inference | ~30 µs      |\n| Detection write (mmap)   | Inference | ~50 µs      |\n| **Total**                |           | **~7.5 ms** |\n| JPEG encoding (async)    | Gateway   | ~2.5 ms     |\n\n\u003e [!NOTE]\n\u003e Gateway runs asynchronously, encoding the previous frame while inference processes the current frame.\n\n## Testing without a camera\n\n```bash\nsudo modprobe v4l2loopback video_nr=0\nffmpeg -re -stream_loop -1 -i video.mp4 -vf \"scale=1920:1080\" -c:v mjpeg -f v4l2 /dev/video0\n```\n\n## Ideas about what to do with this repo\n\n - DevOps: Deploy with KubeEdge instead of K3s (KinD + KubeEdge)\n - SWE: Replace WebSocket streaming with proper H.264 setup\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md)\n\n## Citation\n\nThis project uses RF-DETR by Roboflow. If you use this software, please cite:\n\n```bibtex\n@software{robinson2025rf-detr,\n  author = {Robinson, Isaac and Robicheaux, Peter and Popov, Matvei},\n  title = {RF-DETR},\n  year = {2025},\n  publisher = {GitHub},\n  url = {https://github.com/roboflow/rf-detr}\n}\n```\n\n## License\n\nLicensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for details.\n\n\u003e [!NOTE]\n\u003e This project is intended for educational and research purposes.\n\n\u003c!--references--\u003e\n[RF-DETR]: https://github.com/roboflow/rf-detr\n[v4l]: https://crates.io/crates/v4l\n[ONNX]: https://onnx.ai/\n[CXX]: https://cxx.rs/\n[MQTT]: https://mqtt.org/\n[Axum]: https://docs.rs/axum/latest/axum/\n[mosquitto]: https://mosquitto.org/\n[FlatBuffers]: https://flatbuffers.dev/\n[ORT]: https://ort.pyke.io/\n[OpenTelemetry]: https://opentelemetry.io/\n[Jaeger]: https://www.jaegertracing.io/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjordandelbar%2Fdetr-mmap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjordandelbar%2Fdetr-mmap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjordandelbar%2Fdetr-mmap/lists"}