{"id":24625787,"url":"https://github.com/retail-ai-inc/beanq","last_synced_at":"2026-04-10T08:07:06.360Z","repository":{"id":65598209,"uuid":"577268601","full_name":"retail-ai-inc/beanq","owner":"retail-ai-inc","description":"A simple production grade golang Job queue system with a slick dashboard","archived":false,"fork":false,"pushed_at":"2025-05-06T07:19:34.000Z","size":19202,"stargazers_count":9,"open_issues_count":10,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-05-06T08:31:51.101Z","etag":null,"topics":["dashboard","golang","job-queue","production"],"latest_commit_sha":null,"homepage":"https://retail-ai.jp","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/retail-ai-inc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-12-12T11:10:34.000Z","updated_at":"2025-04-18T23:07:02.000Z","dependencies_parsed_at":"2024-01-02T05:33:25.861Z","dependency_job_id":"6d89b762-89a6-4d9a-9a56-f6bff374b4c1","html_url":"https://github.com/retail-ai-inc/beanq","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/retail-ai-inc%2Fbeanq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/retail-ai-inc%2Fbeanq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/retail-ai-inc%2Fbeanq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/retail-ai-inc%2Fbeanq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/retail-ai-inc","download_url":"https://codeload.github.com/retail-ai-inc/beanq/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931555,"owners_count":21827112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dashboard","golang","job-queue","production"],"created_at":"2025-01-25T04:39:54.131Z","updated_at":"2026-04-10T08:07:06.348Z","avatar_url":"https://github.com/retail-ai-inc.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Beanq v4\n\n\u003cdiv align=\"center\"\u003e\n\n[![Go Version](https://img.shields.io/badge/go-1.24.0-blue.svg)](https://golang.org/)\n[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n[![Redis](https://img.shields.io/badge/redis-5.0.12+-red.svg)](https://redis.io/)\n[![MongoDB](https://img.shields.io/badge/mongodb-8.0+-green.svg)](https://www.mongodb.com/)\n\n**A powerful message queue system built on Redis Stream**\n\n[Features](#-features) • [Quick Start](#-quick-start) • [Documentation](#-documentation) • [Examples](#-examples) • [Configuration](#-configuration)\n\n\u003c/div\u003e\n\n---\n\n## 📖 Overview\n\nBeanq is a high-performance message queue system developed based on **Redis Stream**, providing three types of queues:\n- ✅ **Normal Queues** - Immediate message processing\n- ⏱️ **Delay Queues** - Scheduled message delivery with priority support\n- 🔒 **Sequence Queues** - Ordered message processing with locking mechanisms\n\n### Core Architecture\n\n```mermaid\ngraph TB\n    Publisher[Message Publisher] --\u003e Redis[Redis Stream]\n    Redis --\u003e Consumer[Message Consumer]\n    Redis --\u003e DLQ[Dead Letter Queue]\n    Redis --\u003e History[MongoDB History Storage]\n    Consumer --\u003e UI[Monitoring Dashboard]\n```\n\n---\n\n## ✨ Features\n\n### 🚀 High Performance\n- Built on Redis Stream for fast message processing\n- Concurrent consumer pools with configurable sizing\n- Efficient dead-letter message handling\n\n### ⏰ Advanced Scheduling\n- Delay queue with timestamp-based scheduling\n- Priority support (max level 999) for time-sensitive messages\n- Format: `timestamp.priority` (e.g., `1734399237.999`)\n\n### 🔐 Reliable Processing\n- Sequence queues ensure ordered message processing\n- Redis hash-based status synchronization\n- Automatic retry mechanisms (configurable max retries)\n- Dead-letter queue for failed messages\n\n### 📊 Comprehensive Monitoring\n- Web-based UI dashboard (port 9090)\n- Real-time queue statistics\n- Message history tracking in MongoDB\n- Health check endpoints\n\n### 🛠️ Enterprise Ready\n- JWT authentication for UI access\n- Google OAuth integration\n- Email notifications via SendGrid\n- Slack integration for error reporting\n- Multi-instance deployment support\n\n---\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Docker \u0026 Docker Compose\n- Go 1.24.0 or higher\n- Redis 5.0.12+\n- MongoDB 8.0+\n\n### 1. Clone and Setup\n\n```bash\ngit clone https://github.com/retail-ai-inc/beanq.git\ncd beanq\n```\n\n### 2. Start Dependencies\n\n```bash\n# Start Redis and MongoDB\ndocker-compose up -d --build\n\n# Verify containers are running\ndocker-compose ps\n```\n\n### 3. Run Examples\n\n```bash\n# Enter the example container\ndocker exec -it beanq-example bash\n\n# Run normal queue example\nmake normal\n\n# Run delay queue example\nmake delay\n\n# Run sequence queue example\nmake sequential\n```\n\n### 4. Launch UI Dashboard\n\n```bash\n# Start the monitoring UI\nmake ui\n```\n\nAccess at: `http://localhost:9090`\n- Default username: `rai`\n- Default password: `mysecretpass`\n\n---\n\n## 📚 Documentation\n\n### Queue Types\n\n#### 1. Normal Queue\n\nMessages are consumed immediately upon publishing.\n\n```mermaid\ngraph LR\n    Publisher[Publisher] --\u003e RedisStream[(Redis Stream)]\n    RedisStream --\u003e ConsumerPool[Consumer Pool]\n    \n    subgraph Normal Queue Flow\n        Publisher --\u003e|Publish| RedisStream\n        RedisStream --\u003e|Immediate| ConsumerPool\n        ConsumerPool --\u003e|Success| Success[✓ Success]\n        ConsumerPool --\u003e|Fail| Retry{Retry?}\n        Retry --\u003e|Yes| ConsumerPool\n        Retry --\u003e|No| DLQ[Dead Letter Queue]\n        DLQ --\u003e LogQueue[Log Queue]\n        LogQueue --\u003e Analysis[Analysis]\n    end\n    \n    style Publisher fill:#4CAF50,stroke:#388E3C,color:#fff\n    style RedisStream fill:#2196F3,stroke:#1976D2,color:#fff\n    style ConsumerPool fill:#FF9800,stroke:#F57C00,color:#fff\n    style Success fill:#4CAF50,stroke:#388E3C,color:#fff\n    style DLQ fill:#f44336,stroke:#D32F2F,color:#fff\n    style LogQueue fill:#FFC107,stroke:#FFA000,color:#000\n    style Analysis fill:#9C27B0,stroke:#7B1FA2,color:#fff\n```\n\n**Characteristics:**\n- Immediate processing\n- Dead-letter detection and logging\n- Automatic retry on failure\n- Concurrent consumption\n\n**Example:**\n```go\n// Publisher\npub := beanq.New(config)\nerr := pub.BQ().WithContext(ctx).Publish(\"channel\", \"topic\", messageBytes)\n\n// Consumer\nconsumer.Subscribe(channel, topic, beanq.DefaultHandle{\n    DoHandle: func(ctx context.Context, message *beanq.Message) error {\n        // Process message\n        return nil\n    },\n})\n```\n\n#### 2. Delay Queue\n\nMessages are processed at a scheduled time with optional priority.\n\n```mermaid\ngraph LR\n    Publisher[Publisher] --\u003e DelayQueue[Delay Queue]\n    \n    subgraph Storage Format\n        DelayQueue --\u003e TS[timestamp.priority]\n        TS --\u003e TimePart[1734399237]\n        TS --\u003e PriorityPart[.999 max]\n    end\n    \n    DelayQueue --\u003e Scheduler[Scheduler]\n    Scheduler --\u003e TimeCheck{Time Reached?}\n    TimeCheck --\u003e|No| Wait[Wait]\n    TimeCheck --\u003e|Yes| PrioritySort[Priority Sort]\n    PrioritySort --\u003e|High First| ConsumerPool[Consumer Pool]\n    ConsumerPool --\u003e Processing[Process]\n    Processing --\u003e Success[✓ Success]\n    \n    style Publisher fill:#4CAF50,stroke:#388E3C,color:#fff\n    style DelayQueue fill:#FF9800,stroke:#F57C00,color:#fff\n    style Scheduler fill:#2196F3,stroke:#1976D2,color:#fff\n    style PrioritySort fill:#9C27B0,stroke:#7B1FA2,color:#fff\n    style ConsumerPool fill:#FF5722,stroke:#E64A19,color:#fff\n    style Success fill:#4CAF50,stroke:#388E3C,color:#fff\n```\n\n**Characteristics:**\n- Time-based scheduling using Unix timestamp\n- Priority levels: 0-999 (higher = earlier execution)\n- Format: `timestamp.priority`\n- Same-time messages sorted by priority\n\n**Example:**\n```go\n// Publish with delay and priority\ndelayTime := time.Now().Add(10 * time.Second)\nerr := pub.BQ().WithContext(ctx).\n    Priority(8).\n    PublishAtTime(\"delay-channel\", \"topic\", messageBytes, delayTime)\n```\n\n#### 3. Sequence Queue\n\nEnsures ordered processing for messages with the same key.\n\n```mermaid\ngraph LR\n    Publisher[Publisher] --\u003e LockMechanism[Lock Mechanism]\n    \n    subgraph Publish Flow\n        Publisher --\u003e|OrderKey| LockKey[orderKey: aa]\n        LockKey --\u003e SetTTL[Set TTL 10s]\n        SetTTL --\u003e RedisHash[Redis Hash]\n        RedisHash --\u003e QueueMsg[Queue Message]\n    end\n    \n    QueueMsg --\u003e CheckLock{Check Lock}\n    CheckLock --\u003e|Free| AcquireLock[Acquire Lock]\n    CheckLock --\u003e|Locked| BlockWait[Block Wait]\n    BlockWait --\u003e|TTL Expire| DeleteKey[Delete Key]\n    DeleteKey --\u003e AcquireLock\n    AcquireLock --\u003e ProcessMsg[Process Message]\n    ProcessMsg --\u003e UpdateStatus[Update Status]\n    UpdateStatus --\u003e Ack[Acknowledge]\n    Ack --\u003e ReleaseLock[Release Lock]\n    \n    style Publisher fill:#4CAF50,stroke:#388E3C,color:#fff\n    style LockMechanism fill:#FF9800,stroke:#F57C00,color:#fff\n    style RedisHash fill:#2196F3,stroke:#1976D2,color:#fff\n    style AcquireLock fill:#9C27B0,stroke:#7B1FA2,color:#fff\n    style ProcessMsg fill:#FF5722,stroke:#E64A19,color:#fff\n    style ReleaseLock fill:#4CAF50,stroke:#388E3C,color:#fff\n```\n\n**Characteristics:**\n- Lock-based ordering by order key\n- Redis hash status synchronization\n- Configurable lock TTL\n- Acknowledgment support\n\n**Example:**\n```go\n// Publish with sequence lock\nresult, err := pub.BQ().WithContext(ctx).\n    SetId(messageId).\n    SetLockOrderKeyTTL(10 * time.Second).\n    PublishInSequenceByLock(\"channel\", \"topic\", \"orderKey\", messageBytes).\n    WaitingAck()\n```\n\n---\n\n## 🔧 Configuration\n\n### Environment Configuration (`env.json`)\n\n```json\n{\n  \"redis\": {\n    \"host\": \"localhost\",\n    \"port\": \"6379\",\n    \"password\": \"secret\",\n    \"database\": 0,\n    \"prefix\": \"beanq_\",\n    \"poolSize\": 30,\n    \"minIdleConnections\": 10\n  },\n  \"mongo\": {\n    \"host\": \"localhost\",\n    \"port\": \"27017\",\n    \"username\": \"beanq\",\n    \"password\": \"secret\",\n    \"database\": \"beanq_logs\",\n    \"connectTimeout\": \"10s\",\n    \"maxConnectionPoolSize\": 200,\n    \"maxConnectionLifeTime\": \"600s\",\n    \"collections\": {\n      \"config\": {\n        \"name\": \"config\",\n        \"shard\": false\n      },\n      \"event\":{\n        \"name\": \"event_logs\",\n        \"shard\": true\n      },\n      \"opt\":{\n        \"name\": \"opt_logs\",\n        \"shard\": true\n      },\n      \"workflow\": {\n        \"name\": \"workflow_records\",\n        \"shard\": true\n      },\n      \"tenant\": {\n        \"name\": \"tenants\",\n        \"shard\": true\n      },\n      \"manager\": {\n        \"name\": \"managers\",\n        \"shard\": true\n      },\n      \"role\": {\n        \"name\": \"roles\",\n        \"shard\": true\n      }\n    }\n  },\n  \"broker\": \"redis\",\n  \"consumerPoolSize\": 100,\n  \"deadLetterIdle\": \"60s\",\n  \"jobMaxRetries\": 1,\n  \"keepFailedJobsInHistory\": \"3600s\",\n  \"keepSuccessJobsInHistory\": \"3600s\",\n  \"minConsumers\": 10,\n  \"publishTimeOut\": \"10s\",\n  \"consumeTimeOut\": \"10s\",\n  \"ui\": {\n    \"on\": true,\n    \"issuer\": \"rai\",\n    \"subject\": \"beanq monitor ui\",\n    \"port\": \"9090\",\n    \"jwtKey\": \"your-secret-key\",\n    \"expiresAt\": \"3600s\",\n    \"root\": {\n      \"username\": \"admin\",\n      \"password\": \"your-password\"\n    }\n  },\n  \"history\": {\n    \"on\": true,\n    \"storage\": \"mongo\"\n  },\n  \"workflow\": {\n    \"on\": true,\n    \"retry\": 3,\n    \"async\": true,\n    \"storage\": \"mongo\"\n  }\n}\n```\n\n### Key Parameters\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `consumerPoolSize` | 10 | Number of concurrent consumers |\n| `jobMaxRetries` | 3 | Maximum retry attempts for failed jobs |\n| `deadLetterIdle` | 60s | Idle time before moving to DLQ |\n| `publishTimeOut` | 10s | Publishing timeout |\n| `consumeTimeOut` | 10s | Consumption timeout |\n| `minConsumers` | 100 | Minimum consumer count |\n\n---\n\n## 💡 Examples\n\n### Basic Publisher-Consumer\n\n```bash\n# Terminal 1: Start consumer\ncd examples/normal/consumer\ngo run main.go\n\n# Terminal 2: Publish messages\ncd examples/normal/publisher\ngo run main.go\n```\n\n### Workflow Example\n\nWorkflow allows defining multi-step tasks with rollback support:\n\n```go\nconsumer.SubscribeToSequence(\"channel\", \"topic\", beanq.WorkflowHandler(func(ctx context.Context, wf *beanq.Workflow) error {\n    // Task 1\n    wf.NewTask().OnExecute(func(task beanq.Task) error {\n        log.Println(\"Executing task 1\")\n        return nil\n    }).OnRollback(func(task beanq.Task) error {\n        log.Println(\"Rolling back task 1\")\n        return nil\n    })\n    \n    // Task 2\n    wf.NewTask().OnExecute(func(task beanq.Task) error {\n        log.Println(\"Executing task 2\")\n        return nil\n    })\n    \n    return wf.Run()\n}))\n```\n\n### Scaling Consumers\n\n```bash\n# Scale to 3 consumer instances\ndocker-compose up --build -d --scale example-normal-consumer=3\n```\n\n---\n\n## 🏗️ Architecture\n\n### Components\n\n```\nbeanq/\n├── internal/          # Core implementation\n│   ├── driver/       # Redis/MongoDB drivers\n│   ├── routers/      # HTTP handlers\n│   └── boptions/     # Configuration options\n├── helper/           # Utility packages\n│   ├── logger/       # Logging\n│   ├── json/         # JSON handling\n│   ├── email/        # Email notifications\n│   └── slack/        # Slack integration\n├── examples/         # Usage examples\n└── ui/              # Web dashboard\n```\n\n### Data Flow\n\n1. **Publish**: Message → Redis Stream → Status Log\n2. **Consume**: Redis Stream → Consumer Pool → Processing\n3. **History**: Success/Failure → MongoDB Collections\n4. **Monitoring**: UI Dashboard ← Redis Stats + MongoDB\n\n---\n\n## 🔍 Monitoring \u0026 Observability\n\n### Health Check\n\n```bash\ncurl http://localhost:7777/health\n```\n\n### UI Dashboard Features\n\n- 📊 Real-time queue metrics\n- 📝 Message history viewer\n- 🔍 Dead-letter queue inspection\n- 👥 User management\n- 🔐 Role-based access control\n- 📈 Performance analytics\n\n---\n\n## ⚠️ Important Notes\n\n### Redis Persistence\n\n**CRITICAL**: To ensure data safety, enable AOF persistence:\n\n```conf\n# redis.conf\nappendonly yes\nappendfsync everysec\n```\n\nReference: [Redis Persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/)\n\n### Production Recommendations\n\n1. Enable Redis AOF persistence\n2. Configure appropriate pool sizes\n3. Set up monitoring alerts\n4. Use strong passwords for UI and databases\n5. Enable SSL/TLS for production deployments\n6. Regular backup of MongoDB data\n\n---\n\n## 🧪 Testing\n\n```bash\n# Run all tests with coverage\nmake test\n\n# Run specific test suite\ngo test -v ./... -run TestNormalQueue\n\n# View coverage report\ngo tool cover -html=coverage.txt\n```\n\n---\n\n## 🛠️ Development Tools\n\n```bash\n# Run linters\nmake lint\n\n# Fix field alignment issues\nmake vet-fix\n\n# Clean Docker resources\nmake clean-docker-compose\n```\n\n---\n\n## 📦 Dependencies\n\n### Core\n- [Redis](https://redis.io/) - Message broker\n- [MongoDB](https://www.mongodb.com/) - History storage\n- [Go](https://golang.org/) - Programming language\n\n### Libraries\n- `go-redis/redis/v8` - Redis client\n- `mongodb/mongo-driver` - MongoDB driver\n- `labstack/gommon` - HTTP framework\n- `spf13/viper` - Configuration management\n- `sendgrid/sendgrid-go` - Email service\n- `slack-go/slack` - Slack notifications\n\n---\n\n## 🤝 Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.\n\n### Development Workflow\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Run tests: `make test`\n5. Run linters: `make lint`\n6. Submit a pull request\n\n---\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\n## 🙏 Acknowledgments\n\n- Redis team for the amazing data store\n- MongoDB team for the flexible document database\n- All contributors and supporters of this project\n\n---\n\n## 📞 Support\n\n- **Issues**: [GitHub Issues](https://github.com/retail-ai-inc/beanq/issues)\n- **Discussions**: [GitHub Discussions](https://github.com/retail-ai-inc/beanq/discussions)\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**Built with ❤️ by Retail AI Inc.**\n\n[Star this repo](https://github.com/retail-ai-inc/beanq/stargazers) if you find it helpful!\n\n\u003c/div\u003e\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fretail-ai-inc%2Fbeanq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fretail-ai-inc%2Fbeanq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fretail-ai-inc%2Fbeanq/lists"}