{"id":43961710,"url":"https://github.com/vnykmshr/ledgerq","last_synced_at":"2026-02-07T05:33:39.427Z","repository":{"id":319280968,"uuid":"1078126731","full_name":"vnykmshr/ledgerq","owner":"vnykmshr","description":"Production-ready, disk-backed message queue for single-node Go applications. Zero dependencies, crash-safe, with priorities, DLQ, compression \u0026 deduplication.","archived":false,"fork":false,"pushed_at":"2025-11-19T05:45:50.000Z","size":2170,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-19T07:17:50.658Z","etag":null,"topics":["append-only-log","crash-recovery","dead-letter-queue","embedded-database","golang","local-first","message-queue","priority-queue","task-queue","zero-dependencies"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vnykmshr.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"lfx_crowdfunding":null,"polar":null,"buy_me_a_coffee":null,"thanks_dev":"u/gh/vnykmshr","custom":null}},"created_at":"2025-10-17T08:47:55.000Z","updated_at":"2025-11-19T05:45:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"f847a2c3-db63-417a-b2ea-9ca2aaefe729","html_url":"https://github.com/vnykmshr/ledgerq","commit_stats":null,"previous_names":["vnykmshr/ledgerq"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/vnykmshr/ledgerq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vnykmshr%2Fledgerq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vnykmshr%2Fledgerq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vnykmshr%2Fledgerq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vnykmshr%2Fledgerq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vnykmshr","download_url":"https://codeload.github.com/vnykmshr/ledgerq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vnykmshr%2Fledgerq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29187224,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T05:07:31.176Z","status":"ssl_error","status_checked_at":"2026-02-07T05:06:15.227Z","response_time":63,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["append-only-log","crash-recovery","dead-letter-queue","embedded-database","golang","local-first","message-queue","priority-queue","task-queue","zero-dependencies"],"created_at":"2026-02-07T05:33:38.412Z","updated_at":"2026-02-07T05:33:39.420Z","avatar_url":"https://github.com/vnykmshr.png","language":"Go","funding_links":["https://thanks.dev/u/gh/vnykmshr"],"categories":[],"sub_categories":[],"readme":"# LedgerQ\n\n[![CI](https://github.com/vnykmshr/ledgerq/actions/workflows/ci.yml/badge.svg)](https://github.com/vnykmshr/ledgerq/actions/workflows/ci.yml)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n[![Go Reference](https://pkg.go.dev/badge/github.com/vnykmshr/ledgerq.svg)](https://pkg.go.dev/github.com/vnykmshr/ledgerq/pkg/ledgerq)\n[![Feature Complete](https://img.shields.io/badge/status-feature--complete-success.svg)](https://github.com/vnykmshr/ledgerq/releases/tag/v1.4.0)\n[![Go Version](https://img.shields.io/badge/go-%3E%3D1.23-blue.svg)](https://golang.org/dl/)\n\nA production-ready, disk-backed message queue for single-node applications - written in pure Go with zero dependencies.\n\n**Feature-complete as of v1.4.0** - LedgerQ focuses on being excellent at local message queuing rather than trying to replace distributed systems like Kafka.\n\n## Features\n\n- Crash-safe durability with append-only log design\n- Zero dependencies beyond the Go standard library\n- High throughput with batch operations\n- **Priority queue support (v1.1.0+)** with starvation prevention\n- **Dead Letter Queue (DLQ) support (v1.2.0+)** for failed message handling\n- **Payload compression (v1.3.0+)** with GZIP to reduce disk usage\n- **Message deduplication (v1.4.0+)** for exactly-once semantics\n- Replay from message ID or timestamp\n- Message TTL and headers\n- Metrics and pluggable logging\n- Automatic compaction with retention policies\n- Thread-safe concurrent access\n- CLI tool for management\n\n## Why LedgerQ?\n\n**Single-node message queue with disk durability.** No network, no clustering, no external dependencies. If you're building CLI tools, desktop apps, or edge devices that need crash-safe task queues, LedgerQ stores messages on disk using an append-only log.\n\n### When to Use LedgerQ\n\n**Perfect for:**\n- **Offline-first applications** - Mobile sync, desktop apps that work without network\n- **Local task processing** - CI/CD pipelines, backup jobs, batch processing\n- **Edge/IoT devices** - Limited connectivity, single-node operation\n- **Event sourcing** - Local audit trails and event logs\n- **Development/testing** - Simpler than running Kafka/RabbitMQ locally\n- **Embedded systems** - Zero external dependencies, small footprint\n\n### When NOT to Use LedgerQ\n\n**Use these instead:**\n- **Need multiple consumers on different machines?** → Use **Kafka** (distributed streaming)\n- **Need pub/sub fan-out patterns?** → Use **NATS** or **RabbitMQ** (message broker)\n- **Need distributed consensus?** → Use **Kafka** or **Pulsar** (distributed log)\n- **Need cross-language support via HTTP?** → Use **RabbitMQ** or **ActiveMQ** (AMQP/STOMP)\n- **Already using a database?** → Use **PostgreSQL LISTEN/NOTIFY** or **Redis Streams**\n\n### Comparison\n\n| Feature | LedgerQ | Kafka | NATS | RabbitMQ |\n|---------|---------|-------|------|----------|\n| **Deployment** | Single binary | Cluster (ZooKeeper/KRaft) | Single/Cluster | Single/Cluster |\n| **Dependencies** | None | Java, ZooKeeper | None | Erlang |\n| **Consumer Model** | Single reader | Consumer groups | Subjects/Queues | Queues/Exchanges |\n| **Use Case** | Local queuing | Distributed streaming | Lightweight messaging | Enterprise messaging |\n| **Setup Time** | Instant | Hours | Minutes | Minutes |\n\n**LedgerQ's sweet spot**: You need reliability without the operational overhead of distributed systems.\n\n### Trade-offs\n\n- ✅ Zero config, just a directory path\n- ✅ Survives crashes and reboots\n- ✅ Fast sequential I/O (append-only)\n- ✅ No network protocols or ports\n- ✅ Embeddable in any Go application\n- ❌ Single process only (file locking)\n- ❌ Not distributed (single node)\n- ❌ Single consumer (FIFO order)\n\n## Quick Start\n\n### Installation\n\n```bash\ngo get github.com/vnykmshr/ledgerq/pkg/ledgerq@latest\n```\n\n### Basic Usage\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"log\"\n\n    \"github.com/vnykmshr/ledgerq/pkg/ledgerq\"\n)\n\nfunc main() {\n    q, err := ledgerq.Open(\"/tmp/myqueue\", nil)\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer q.Close()\n\n    offset, _ := q.Enqueue([]byte(\"Hello, World!\"))\n    fmt.Printf(\"Enqueued at offset: %d\\n\", offset)\n\n    msg, _ := q.Dequeue()\n    fmt.Printf(\"Dequeued [ID:%d]: %s\\n\", msg.ID, msg.Payload)\n}\n```\n\n### Configuration\n\n```go\nopts := ledgerq.DefaultOptions(\"/tmp/myqueue\")\nopts.AutoSync = true\nopts.MaxSegmentSize = 100 * 1024 * 1024\nopts.CompactionInterval = 5 * time.Minute\n\nq, _ := ledgerq.Open(\"/tmp/myqueue\", opts)\n```\n\n## Core Operations\n\n**Batch operations** (10-100x faster):\n\n```go\npayloads := [][]byte{[]byte(\"msg1\"), []byte(\"msg2\"), []byte(\"msg3\")}\noffsets, _ := q.EnqueueBatch(payloads)\n\nmessages, _ := q.DequeueBatch(10)\n```\n\n**Message TTL:**\n\n```go\nq.EnqueueWithTTL([]byte(\"temporary\"), 5*time.Second)\n```\n\n**Message headers:**\n\n```go\nheaders := map[string]string{\"content-type\": \"application/json\"}\nq.EnqueueWithHeaders(payload, headers)\n```\n\n**Payload compression (v1.3.0+):**\n\n```go\n// Enable compression by default\nopts := ledgerq.DefaultOptions(\"/tmp/myqueue\")\nopts.DefaultCompression = ledgerq.CompressionGzip\nopts.CompressionLevel = 6  // 1 (fastest) to 9 (best compression)\nopts.MinCompressionSize = 1024  // Only compress \u003e= 1KB\nq, _ := ledgerq.Open(\"/tmp/myqueue\", opts)\n\n// Messages are automatically compressed/decompressed\nq.Enqueue(largePayload)  // Compressed if \u003e= 1KB\n\n// Or control compression per-message\nq.EnqueueWithCompression(payload, ledgerq.CompressionGzip)\n```\n\n**Message deduplication (v1.4.0+):**\n\n```go\n// Enable deduplication with a 5-minute window\nopts := ledgerq.DefaultOptions(\"/tmp/myqueue\")\nopts.DefaultDeduplicationWindow = 5 * time.Minute\nopts.MaxDeduplicationEntries = 100000  // Max 100K unique messages tracked\nq, _ := ledgerq.Open(\"/tmp/myqueue\", opts)\n\n// Enqueue with deduplication\noffset, isDup, _ := q.EnqueueWithDedup(payload, \"order-12345\", 0)\nif isDup {\n    fmt.Printf(\"Duplicate detected! Original at offset %d\\n\", offset)\n} else {\n    fmt.Printf(\"New message enqueued at offset %d\\n\", offset)\n}\n\n// Custom per-message window\nq.EnqueueWithDedup(payload, \"request-789\", 10*time.Minute)\n\n// View deduplication statistics\nstats := q.Stats()\nfmt.Printf(\"Tracked entries: %d\\n\", stats.DedupTrackedEntries)\n```\n\n**Priority queue (v1.1.0+):**\n\n```go\n// Enable priority mode (set at queue creation, cannot be changed later)\nopts := ledgerq.DefaultOptions(\"/tmp/myqueue\")\nopts.EnablePriorities = true\nq, _ := ledgerq.Open(\"/tmp/myqueue\", opts)\n\n// Enqueue with priority\nq.EnqueueWithPriority([]byte(\"urgent\"), ledgerq.PriorityHigh)\nq.EnqueueWithPriority([]byte(\"normal\"), ledgerq.PriorityMedium)\nq.EnqueueWithPriority([]byte(\"background\"), ledgerq.PriorityLow)\n\n// Dequeue in priority order (High → Medium → Low)\nmsg, _ := q.Dequeue() // Returns \"urgent\" first\n```\n\n**Batch operations with priorities (v1.1.0+):**\n\n```go\n// Batch enqueue with per-message options\nmessages := []ledgerq.BatchEnqueueOptions{\n    {Payload: []byte(\"urgent task\"), Priority: ledgerq.PriorityHigh},\n    {Payload: []byte(\"normal task\"), Priority: ledgerq.PriorityMedium},\n    {Payload: []byte(\"background task\"), Priority: ledgerq.PriorityLow, TTL: time.Hour},\n}\noffsets, _ := q.EnqueueBatchWithOptions(messages)\n```\n\n**Dead Letter Queue (v1.2.0+):**\n\n```go\n// Enable DLQ with max retries\nopts := ledgerq.DefaultOptions(\"/tmp/myqueue\")\nopts.DLQPath = \"/tmp/myqueue/dlq\"\nopts.MaxRetries = 3\nq, _ := ledgerq.Open(\"/tmp/myqueue\", opts)\n\n// Process messages with Ack/Nack\nmsg, _ := q.Dequeue()\n\n// Check retry info for custom backoff logic\nif info := q.GetRetryInfo(msg.ID); info != nil {\n    backoff := time.Duration(1\u003c\u003cuint(info.RetryCount)) * time.Second\n    time.Sleep(backoff) // Exponential backoff\n}\n\nif processSuccessfully(msg.Payload) {\n    q.Ack(msg.ID) // Mark as successfully processed\n} else {\n    q.Nack(msg.ID, \"processing failed\") // Record failure (moves to DLQ after MaxRetries)\n}\n\n// Inspect DLQ messages\ndlq := q.GetDLQ()\ndlqMsg, _ := dlq.Dequeue()\nfmt.Printf(\"Failed: %s, Reason: %s\\n\",\n    string(dlqMsg.Payload),\n    dlqMsg.Headers[\"dlq.failure_reason\"])\n\n// Requeue from DLQ after fixing issues\nq.RequeueFromDLQ(dlqMsg.ID)\n```\n\n**Note**: `Nack()` tracks failures but does not auto-retry messages. After `MaxRetries` failures, messages move to DLQ. For retry patterns with backoff, see [USAGE.md](docs/USAGE.md#dead-letter-queue-dlq---v120).\n\n**Streaming:**\n\n```go\nctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)\ndefer cancel()\n\nq.Stream(ctx, func(msg *ledgerq.Message) error {\n    fmt.Printf(\"Received: %s\\n\", msg.Payload)\n    return nil\n})\n```\n\n**Replay:**\n\n```go\nq.SeekToMessageID(100)\nq.SeekToTimestamp(time.Now().Add(-1 * time.Hour).UnixNano())\n```\n\n## Performance\n\nBenchmarks (Go 1.21, macOS, Intel i5-8257U @ 2.3GHz):\n\n| Operation | Throughput | Latency |\n|-----------|------------|---------|\n| Enqueue (buffered) | 3M ops/sec | 300-400 ns |\n| Enqueue (fsync) | 50 ops/sec | 19 ms |\n| EnqueueBatch (100 msgs) | 50M msgs/sec | 200 ns/msg |\n| Dequeue | 1.4K ops/sec | 700 μs |\n\nUse batch operations for high throughput.\n\n## Examples\n\nSee [examples/](examples/) for runnable code with comprehensive READMEs:\n\n**Getting Started:**\n- [simple](examples/simple) - Basic operations ⭐ **Start here!**\n- [producer-consumer](examples/producer-consumer) - Concurrent access with multiple goroutines\n\n**Core Features:**\n- [ttl](examples/ttl) - Message expiration (time-to-live)\n- [headers](examples/headers) - Message metadata and routing\n- [replay](examples/replay) - Seeking and time-travel through message history\n- [streaming](examples/streaming) - Real-time event streaming\n- [metrics](examples/metrics) - Monitoring and observability\n\n**Advanced Features (v1.1.0+):**\n- **[priority](examples/priority)** - Priority ordering (v1.1.0+)\n- **[dlq](examples/dlq)** - Dead Letter Queue for failed messages (v1.2.0+)\n- **[compression](examples/compression)** - GZIP payload compression (v1.3.0+)\n- **[deduplication](examples/deduplication)** - Exactly-once semantics (v1.4.0+)\n\n## CLI Tool\n\n```bash\ngo install github.com/vnykmshr/ledgerq/cmd/ledgerq@latest\n```\n\n**View queue statistics:**\n```bash\n$ ledgerq stats /path/to/queue\n\nQueue Statistics\n================\nDirectory:         /path/to/queue\nTotal Messages:    1500\nPending Messages:  342\nNext Message ID:   1501\nRead Message ID:   1159\nSegment Count:     3\n```\n\n**Inspect queue metadata:**\n```bash\n$ ledgerq inspect /path/to/queue\n```\n\n**Compact queue (remove processed messages):**\n```bash\n$ ledgerq compact /path/to/queue\n```\n\n**Peek at messages (without dequeuing):**\n```bash\n$ ledgerq peek /path/to/queue 5\n```\n\n## Documentation\n\n- [Usage Guide](docs/USAGE.md) - Complete reference\n- [Architecture](docs/ARCHITECTURE.md) - Internal design\n- [API Reference](https://pkg.go.dev/github.com/vnykmshr/ledgerq/pkg/ledgerq) - GoDoc\n- [Contributing](CONTRIBUTING.md) - Development guide\n\n## Limitations\n\n### Batch Operations and Priority Queues\n\n**Important**: `DequeueBatch()` always returns messages in FIFO order (by message ID), even when `EnablePriorities=true`. This is a performance trade-off: batch dequeue optimizes for sequential I/O rather than priority ordering.\n\nFor priority-aware consumption, use `Dequeue()` in a loop or the `Stream()` API.\n\n```go\n// FIFO batch dequeue (fast, sequential I/O)\nmessages, _ := q.DequeueBatch(100)\n\n// Priority-aware dequeue (respects priority order)\nfor i := 0; i \u003c 100; i++ {\n    msg, err := q.Dequeue()\n    // ...\n}\n```\n\n### Priority Queue Memory Usage\n\nWhen `EnablePriorities=true`, the queue maintains an in-memory index of all unprocessed message IDs, organized by priority level. Memory usage scales linearly with queue depth (measured with runtime.MemStats):\n\n- 1M pending messages: ~24 MB\n- 10M pending messages: ~240 MB\n- 100M pending messages: ~2.4 GB\n\nFor very deep queues (\u003e10M messages), consider:\n- Using FIFO mode (`EnablePriorities=false`)\n- Implementing application-level batching to keep queue depth low\n- Running compaction regularly to remove processed messages\n\n## Testing\n\n```bash\ngo test ./...\ngo test -race ./...\ngo test -bench=. ./internal/queue\ngo test -fuzz=FuzzEnqueueDequeue -fuzztime=30s ./internal/queue\n```\n\n## License\n\nApache License 2.0 - see [LICENSE](LICENSE)\n\n## Credits\n\nInspired by Apache Kafka, NATS Streaming, and Python's persist-queue.\n\nBuilt as a learning project for message queue internals and Go systems programming.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvnykmshr%2Fledgerq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvnykmshr%2Fledgerq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvnykmshr%2Fledgerq/lists"}