https://github.com/anuragxxd/redrush
Event-driven, low-latency filesystem-to-S3 sync daemon. Apache-2.0.
https://github.com/anuragxxd/redrush
devops filesystem-watcher golang kubernetes minio s3 sidecar sync
Last synced: 1 day ago
JSON representation
Event-driven, low-latency filesystem-to-S3 sync daemon. Apache-2.0.
- Host: GitHub
- URL: https://github.com/anuragxxd/redrush
- Owner: anuragxxd
- License: apache-2.0
- Created: 2026-04-26T08:58:37.000Z (about 2 months ago)
- Default Branch: master
- Last Pushed: 2026-04-26T09:41:07.000Z (about 2 months ago)
- Last Synced: 2026-06-14T18:30:04.945Z (9 days ago)
- Topics: devops, filesystem-watcher, golang, kubernetes, minio, s3, sidecar, sync
- Language: Go
- Size: 30.3 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
RedRush
Event-driven, low-latency filesystem-to-S3 sync daemon.
Watches a local directory for changes via kernel events and uploads files to any S3-compatible object storage.
Files remain browsable in the bucket exactly as they appear on disk.
One binary. Zero config files.
Quick Start ·
Features ·
Configuration ·
Architecture ·
Kubernetes Sidecar
---
## Why
No production-grade, permissively-licensed, event-driven local-to-S3 sync daemon exists:
| Tool | Problem |
|------|---------|
| `mc mirror --watch` | AGPL, silent failures, no crash recovery |
| `s3sync-service` | GPL-3.0, polling-based, latency = poll interval |
| `rclone` | MIT but no daemon/watch mode |
| `lsyncd` | GPL, no native S3 |
RedRush fills this gap. Apache-2.0. Event-driven. Crash-recoverable.
## Quick Start
```bash
export REDRUSH_WATCH_DIR=/workspace
export REDRUSH_S3_ENDPOINT=http://minio:9000
export REDRUSH_S3_BUCKET=my-bucket
redrush
```
Or with Docker:
```bash
docker run \
-e REDRUSH_WATCH_DIR=/workspace \
-e REDRUSH_S3_ENDPOINT=http://minio:9000 \
-e REDRUSH_S3_BUCKET=my-bucket \
-v /workspace:/workspace:ro \
redrush:latest
```
## Features
- **Event-driven uploads** — file changes detected via kernel events (fsnotify), not polling
- **Per-path debouncing** — coalesces rapid writes with a configurable quiet window and max-wait ceiling
- **Periodic reconciliation** — full-scan safety net on startup, on timer, and on event overflow
- **Delete sync** — optionally mirrors local deletions to S3
- **Worker pool** — bounded concurrent uploads with configurable parallelism
- **Multipart uploads** — automatic multipart for large files via AWS SDK manager
- **Crash recovery** — reconciler detects and uploads anything missed while offline, including same-size content changes via mtime metadata
- **Graceful shutdown** — SIGTERM/SIGINT triggers watcher stop, debounce flush, upload drain with configurable timeout
- **One-shot mode** — run a single reconciliation pass and exit
- **Distroless Docker image** — ~10MB, runs as nonroot
- **Zero config files** — everything via environment variables
## Configuration
All configuration is via environment variables. No config files.
| Variable | Default | Description |
|----------|---------|-------------|
| `REDRUSH_WATCH_DIR` | — | **(required)** Directory to watch |
| `REDRUSH_S3_ENDPOINT` | — | **(required)** S3/MinIO endpoint URL |
| `REDRUSH_S3_BUCKET` | — | **(required)** Target bucket |
| `REDRUSH_S3_PREFIX` | `""` | Key prefix for all uploads |
| `REDRUSH_S3_REGION` | `us-east-1` | S3 region |
| `REDRUSH_DEBOUNCE_MS` | `500` | Debounce quiet window (ms) |
| `REDRUSH_MAX_WAIT_MS` | `2000` | Max debounce ceiling (ms) |
| `REDRUSH_UPLOAD_WORKERS` | `8` | Concurrent upload goroutines |
| `REDRUSH_QUEUE_SIZE` | `1000` | Upload queue depth |
| `REDRUSH_RECONCILE_INTERVAL` | `60s` | Periodic full-scan interval |
| `REDRUSH_DELETE_MODE` | `ignore` | `ignore` or `sync` |
| `REDRUSH_FOLLOW_SYMLINKS` | `false` | Resolve and upload symlink targets |
| `REDRUSH_MULTIPART_THRESHOLD` | `16MB` | File size threshold for multipart |
| `REDRUSH_DRAIN_TIMEOUT` | `30s` | Shutdown drain timeout |
| `REDRUSH_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error` |
| `REDRUSH_ONE_SHOT` | `false` | Run one reconciliation and exit |
## Architecture
```
┌──────────────────────────────────────────┐
│ RedRush │
│ │
│ Watcher ──▶ Debouncer ──▶ Upload Queue │
│ (fsnotify) (per-path) (bounded chan) │
│ │ │
│ Reconciler ──────────────▶ S3 Uploader │
│ (periodic) (worker pool) │
│ │
│ Lifecycle (signal handling, drain) │
└──────────────────────────────────────────┘
│ │
/watch-dir S3-compatible
(read-only) object storage
```
**Watcher** detects filesystem events and feeds them through a per-path **Debouncer** into a bounded upload queue. A pool of **S3 Uploader** workers consume from the queue. The **Reconciler** runs on startup, periodically, and on event overflow to catch anything the watcher missed. The **Syncer** orchestrates all components and manages graceful shutdown.
## Expected Latency
| Phase | Time |
|-------|------|
| Kernel event delivery | < 1ms |
| Debounce window | 500ms (configurable) |
| S3 PUT (small file, same network) | 10–50ms |
| **Total: small file visible in S3** | **~1–2s** |
## Kubernetes Sidecar
RedRush is designed for use as a K8s native sidecar (`restartPolicy: Always` init container):
```yaml
initContainers:
- name: redrush
restartPolicy: Always
image: redrush:latest
env:
- name: REDRUSH_WATCH_DIR
value: /workspace
- name: REDRUSH_S3_ENDPOINT
value: http://minio.svc:9000
- name: REDRUSH_S3_BUCKET
value: workflows
- name: REDRUSH_S3_PREFIX
value: v1/runs/$(RUN_ID)/workspace/
volumeMounts:
- name: workspace
mountPath: /workspace
readOnly: true
```
## Development
```bash
# Build
make build
# Unit tests
make test
# Integration tests (Docker + MinIO)
make integration
# Lint
make lint
# Docker image
make docker
```
## What RedRush Does NOT Do
- Bidirectional sync
- Conflict resolution
- File locking or coordination
- FUSE mounting
- Metrics endpoint (planned)
## Dependencies
| Package | License |
|---------|---------|
| [fsnotify](https://github.com/fsnotify/fsnotify) v1.9.0 | BSD-3-Clause |
| [aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | Apache-2.0 |
| Go stdlib (`log/slog`, `os`, `path/filepath`, `sync`, `time`) | BSD-3-Clause |
## License
Apache-2.0 — see [LICENSE](LICENSE).