{"id":30772092,"url":"https://github.com/solodynamo/redis-exactly-once-processing","last_synced_at":"2026-06-27T23:31:49.084Z","repository":{"id":312654268,"uuid":"1048218029","full_name":"solodynamo/redis-exactly-once-processing","owner":"solodynamo","description":"SQS's exactly once processing w/ Logic Control and Data structures of Redis","archived":false,"fork":false,"pushed_at":"2025-09-01T05:59:34.000Z","size":33,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-01T07:43:25.083Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/solodynamo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-01T05:44:56.000Z","updated_at":"2025-09-01T05:59:37.000Z","dependencies_parsed_at":"2025-09-01T07:43:30.003Z","dependency_job_id":"023a1e5f-de56-4cce-a4a5-493bf415c825","html_url":"https://github.com/solodynamo/redis-exactly-once-processing","commit_stats":null,"previous_names":["solodynamo/redis-exactly-once-processing"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/solodynamo/redis-exactly-once-processing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solodynamo%2Fredis-exactly-once-processing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solodynamo%2Fredis-exactly-once-processing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solodynamo%2Fredis-exactly-once-processing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solodynamo%2Fredis-exactly-once-processing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/solodynamo","download_url":"https://codeload.github.com/solodynamo/redis-exactly-once-processing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solodynamo%2Fredis-exactly-once-processing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34872279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-27T02:00:06.362Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-05T00:50:17.580Z","updated_at":"2026-06-27T23:31:49.079Z","avatar_url":"https://github.com/solodynamo.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Redis Timeout Tracking POC\n\nA scalable, Redis-based timeout tracking system for the Care customer support platform that tracks customer response times and sends progressive notifications.\n\n## Overview\n\nThis POC implements a two-phase approach:\n- **Phase 1**: Single leader architecture (handles up to 10K conversations/sec)\n- **Phase 2**: Multi-consumer with leader election (scales to 100K+ conversations/sec)\n\n## Architecture\n\n### Phase 1: Single Leader\n```\n┌─────────────────────────────────────────────────────────────┐\n│                   Kubernetes Cluster                         │\n├─────────────────────────────────────────────────────────────┤\n│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │\n│  │   Pod 1      │  │   Pod 2      │  │   Pod N      │     │\n│  │  (Leader)    │  │  (Standby)   │  │  (Standby)   │     │\n│  └──────┬───────┘  └──────────────┘  └──────────────┘     │\n│         ▼                                                    │\n│  ┌────────────────────────────────────────────────────┐    │\n│  │                    Redis                            │    │\n│  │ • waiting_conversations (Sorted Set)               │    │\n│  │ • notification_states (Hash)                       │    │\n│  │ • timeout:leader (String with TTL)                 │    │\n│  └────────────────────────────────────────────────────┘    │\n└─────────────────────────────────────────────────────────────┘\n```\n\n### Phase 2: Multi-Consumer\n```\n┌──────────────────────────────────────────────────────────────┐\n│  ┌──────────────┐  Leader - Timeout Detector                │\n│  │   Pod 1      │  Checks for timeouts                      │\n│  │  (Leader)    │  Pushes to Redis Stream                   │\n│  └──────┬───────┘                                           │\n│         ▼                                                     │\n│  ┌────────────────────────────────────────────────────┐     │\n│  │              Redis Stream: timeout_events           │     │\n│  └──────┬───────────┬───────────┬────────────────────┘     │\n│         │           │           │                            │\n│  ┌──────▼─────┐ ┌──▼─────┐ ┌──▼─────┐                     │\n│  │   Pod 1    │ │  Pod 2  │ │ Pod N   │  All pods consume  │\n│  │ (Consumer) │ │(Consumer)│ │(Consumer)│  different msgs   │\n│  └────────────┘ └─────────┘ └──────────┘                   │\n└──────────────────────────────────────────────────────────────┘\n```\n\n## Quick Start\n\n### 🚀 Setup and Testing\n```bash\n# 1. Setup development environment\n./scripts/setup-dev.sh\n\n# 2. Test multi-instance leader election\n./scripts/test-multi-leader.sh\n```\n\n### 🏃 Manual Local Development\n```bash\n# Build applications\nmake local-build\n\n# Run Phase 1 locally\nmake local-run-phase1\n\n# Run Phase 2 locally\nmake local-run-phase2\n```\n\n### 🐳 Docker Cluster (Production-like)\n```bash\nmake docker-up    # Start full cluster\nmake docker-logs  # View logs\nmake docker-down  # Stop cluster\n```\n\n### Prerequisites\n- Go 1.21+\n- Docker and Docker Compose\n- `jq` (for demo scripts): `brew install jq` (macOS) or `apt-get install jq` (Ubuntu)\n\n### Running Phase 2\nmake run-phase2\n```\n\n## Configuration\n\n### Environment Variables\n- `REDIS_URL`: Redis connection string (default: redis://localhost:6379)\n- `TIMEOUT_INTERVAL_MS`: Base timeout interval in milliseconds (default: 30000)\n- `LEADER_ELECTION_TTL`: Leader lock TTL in seconds (default: 10)\n- `CHECK_INTERVAL_MS`: How often to check for timeouts in ms (default: 1000)\n- `POD_ID`: Unique identifier for this pod (default: auto-generated)\n- `PORT`: HTTP server port (default: 8080)\n\n## API Endpoints\n\n### POST /conversations/:id/agent-message\nTrack when an agent sends a message to start timeout monitoring.\n\n**Request Body:**\n```json\n{\n  \"agent_id\": \"agent_123\",\n  \"message_id\": \"msg_456\",\n  \"timestamp\": \"2024-01-01T12:00:00Z\"\n}\n```\n\n### POST /conversations/:id/customer-response\nClear timeout tracking when customer responds.\n\n**Request Body:**\n```json\n{\n  \"customer_id\": \"customer_123\",\n  \"message_id\": \"msg_789\",\n  \"timestamp\": \"2024-01-01T12:05:00Z\"\n}\n```\n\n### GET /health\nHealth check endpoint.\n\n### GET /metrics\nPrometheus metrics endpoint.\n\n## Redis Data Structures\n\n| Key | Type | Purpose | Example |\n|-----|------|---------|---------|\n| `waiting_conversations` | Sorted Set | Tracks waiting conversations | Score: timestamp, Member: conv_id |\n| `notification_states` | Hash | Prevents duplicate notifications | Field: conv_id, Value: level (1,2,3) |\n| `timeout:leader` | String | Leader election lock | Value: pod_id, TTL: 10s |\n| `metrics:timeouts` | Hash | Monitoring metrics | Fields: total, level1, level2, level3 |\n| `timeout_events` | Stream | Phase 2 event queue | Messages with conversation timeouts |\n\n## Testing\n\n```bash\n# Run all tests\nmake test\n\n# Run with coverage\nmake test-coverage\n\n# Load testing\nmake load-test\n```\n\n## Monitoring\n\nKey metrics tracked:\n- `waiting_conversations_count`: Current conversations being tracked\n- `timeout_notifications_sent`: Notifications sent by level\n- `timeout_leader_changes`: Number of leader changes\n- `timeout_check_duration`: Performance of timeout checks\n\n## Production Considerations\n\n- Monitor Redis memory usage\n- Set up Redis persistence (RDB + AOF)\n- Configure Redis cluster for high availability\n- Implement circuit breakers for external services\n- Set up proper logging and alerting\n- Use Redis connection pooling\n- Implement graceful shutdown handling\n\n## Development\n\n```bash\n# Format code\nmake fmt\n\n# Lint code\nmake lint\n\n# Build binaries\nmake build\n\n# Clean up\nmake clean\n``` ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolodynamo%2Fredis-exactly-once-processing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsolodynamo%2Fredis-exactly-once-processing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolodynamo%2Fredis-exactly-once-processing/lists"}