{"id":29019189,"url":"https://github.com/raiyanyahya/zapq","last_synced_at":"2025-07-05T22:39:18.658Z","repository":{"id":301153332,"uuid":"1008328383","full_name":"raiyanyahya/zapq","owner":"raiyanyahya","description":"⚡ A single‑binary Go microservice that exposes a First‑In First‑Out message queue completely in RAM. ","archived":false,"fork":false,"pushed_at":"2025-06-25T11:30:38.000Z","size":14,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-06-25T12:29:11.402Z","etag":null,"topics":["concurency","fifo","golang","microservice","queue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raiyanyahya.png","metadata":{"files":{"readme":"ReadMe.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-25T11:27:03.000Z","updated_at":"2025-06-25T11:38:22.000Z","dependencies_parsed_at":"2025-06-25T12:42:08.256Z","dependency_job_id":null,"html_url":"https://github.com/raiyanyahya/zapq","commit_stats":null,"previous_names":["raiyanyahya/zapq"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/raiyanyahya/zapq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raiyanyahya%2Fzapq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raiyanyahya%2Fzapq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raiyanyahya%2Fzapq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raiyanyahya%2Fzapq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raiyanyahya","download_url":"https://codeload.github.com/raiyanyahya/zapq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raiyanyahya%2Fzapq/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261973746,"owners_count":23238588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurency","fifo","golang","microservice","queue"],"created_at":"2025-06-26T00:08:23.360Z","updated_at":"2025-06-26T00:08:29.751Z","avatar_url":"https://github.com/raiyanyahya.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ⚡ ZapQ In‑Memory FIFO Queue Server\n\nA single‑binary Go microservice that exposes a **First‑In / First‑Out message queue** completely in RAM. This README explains **what every endpoint does, why it works internally, and the key computer‑science principles** behind each subsystem.\n\n\u003e **TL;DR**\n\u003e * Ultra‑low‑latency queue for transient messages (≤ 128 KB each).\n\u003e * Hard caps (`maxBytes`, `maxMsgs`) guard RAM.\n\u003e * Concurrency‑safe via a mutex + atomic counters.\n\u003e * JSON metrics, TLS option, graceful shutdown.\n\n---\n\n## 1. Architecture Overview\n\n```\n ┌────────────┐   HTTP/HTTPS    ┌────────────────────────┐   Goroutines   ┌─────────────┐\n │  Producers │ ───────────────▶│  queue ([]byte slices) │◀──────────────▶│  Consumers  │\n └────────────┘                 └────────────────────────┘                └─────────────┘\n```\n\n* **queue.data** – a slice of byte‑slices (`[][]byte`). Index **0** is always the next item to pop, satisfying FIFO.\n* **queue.mu** – a single `sync.Mutex` serialises structural modifications while permitting **parallel HTTP clients**.\n* **queue.bytes** – running total for O(1) memory‑cap checks.\n* **Atomic counters** record events without taking the mutex (cache‑friendly, lock‑free).\n\n### Time \u0026 Space Complexity\n| Operation  | Time | Space Comments |\n|-----------|------|----------------|\n| `enqueue` | **O(1)** append; amortised as Go grows slice capacity exponentially.\n| `dequeue` | **O(1)** slice header moves, old entry set to `nil` for GC.\n| `clear`   | **O(n)** nils everything so GC can reclaim.\n\nMutex ensures **linearizability** (each op appears instantaneous). Because we hold the lock only long enough to modify the slice, throughput scales linearly until contention on `enqueue`/`dequeue` dominates.\n\n---\n\n## 2. Endpoint Walk‑Through \u0026 CS Background\n\n| Path / Verb | Semantics | Response | Internals \u0026 Rationale |\n|-------------|-----------|----------|----------------------|\n| **`POST /enqueue`** | Push a message (≤ 128 KB). | `202 Accepted` or `429 Queue Full` or `413 Payload Too Large`. | *Reads the body into a byte‑slice*. Calls `enqueue`. Checks both **count‑cap** (`maxMsgs`) and **byte‑cap** (`maxBytes`) in **O(1)**. On success increments `enqueueCnt` atomically. |\n| **`GET /dequeue`** | Pop next message. | `200` with raw bytes or `204 No Content` if empty. | `dequeue` grabs mutex, returns first item, shifts slice head. Failure path increments `dequeueMiss` without lock. |\n| **`POST /clear`** | Purge all data. | `200 OK`. | Nils every slice entry so Go’s tri‑colour GC can reclaim memory quickly; resets counters. |\n| **`POST /persist?file=path.json`** | Snapshot queue to disk (blocking). | `200` or `500`. | **Snapshot copy** removes lock early to minimise stall, then JSON‑encodes for human‑readability. O(n) IO. |\n| **`POST /load?file=path.json`** | Replace in‑memory queue with file contents. | `200` or `500`. | Validates file, recomputes `bytes` in one pass. |\n| **`GET /metrics`** | Human‑friendly JSON telemetry. | `{…}` | Shows uptime, counters, queue size, Go goroutines, and FD usage (reads `/proc/self/fd`). Ideal for dashboards or Prometheus “textfile” collector. |\n| **`GET /health`** | Liveness probe. | `ok` | Constant‑time, lock‑free. |\n\n### Why JSON not Protobuf?\n* Zero client codegen, trivial `curl` debugging, small payload (few hundred bytes).\n\n---\n\n## 3. Compile \u0026 Run\n\n### Prerequisites\n* Go ≥ 1.22\n* (Linux) raise FD limit for heavy load tests: `ulimit -n 65535`\n\n### Quick start (HTTP)\n```bash\n# Build \u0026 run with 2 GiB RAM cap and 100 k messages cap\nQUEUE_MAX_BYTES=2G go run fifo_queue_server.go \\\n   --addr :8080 --max-msgs 100000\n```\n\n### Enable HTTPS\n```bash\nopenssl req -x509 -newkey rsa:2048 -days 365 -nodes \\\n        -keyout key.pem -out cert.pem -subj \"/CN=localhost\"\n\ngo run fifo_queue_server.go --addr :8443 \\\n        --tls-cert cert.pem --tls-key key.pem\n```\nNow `curl -k https://localhost:8443/health` ➜ `ok`.\n\n### Flags \u0026 Env\n| Flag / Env          | Default    | Description |\n|---------------------|-----------|-------------|\n| `--addr`            | `:8080`   | Where to listen (`host:port`). |\n| `--tls-cert/key`    | *empty*   | Enable TLS. |\n| `--max-bytes` / `QUEUE_MAX_BYTES` | `256M` | Memory cap (bytes, M, G). |\n| `--max-msgs` / `QUEUE_MAX_MSGS`   | `50000` | Count cap. |\n\n*Flags beat env vars.*\n\n---\n\n## 4. Graceful Shutdown Explained\n\nThe server goroutine listens for **`SIGINT`/`SIGTERM`**:\n```go\nsigCh := make(chan os.Signal,1)\nsignal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)\n\u003c-sigCh // blocking\n```\nThen runs `srv.Shutdown(ctx)` which:\n1. Stops `Listen` accepting new sockets.  \n2. Waits for in‑flight handlers to return (10 s timeout).  \n3. Active goroutines finish `enqueue/ dequeue`, so no message corruption.\n\nThis follows Go’s **graceful‑shutdown contract**, ensuring linearizability even during redeploys.\n\n---\n\n## 5. Computer‑Science Nuggets\n\n* **Critical Section** – guarded by a single mutex. Simpler than read‑write locks because writes dominate.\n* **Copy‑on‑enqueue** – `cp := append([]byte(nil), b...)` avoids aliasing if caller reuses slice; preserves queue immutability.\n* **Slice‑shifting** – Cheap header move; memory of popped item reclaimed after next GC cycle.\n* **Atomic counters** – Use `sync/atomic` so metrics don’t need mutex (lock‑free, wait‑free).\n* **Memory Cap Enforcement** – Prevents unbounded growth, a common cause of **OOM‑killer** crashes.\n* **File‑descriptor telemetry** – Reading `/proc/self/fd` counts live FDs, helping detect `EMFILE` risk.\n* **Back‑pressure via 429** – Follows HTTP semantics; clients must retry or throttle. This protects server latency (queue spill‑over ≈ head‑of‑line blocking).\n* **Payload limit 128 KB** – Small enough to avoid pathological GC pauses; tweak `maxBody` constant if needed.\n\n---\n\n## 6. Load‑Testing Examples\n\n```bash\n# 100 parallel enqueues, 50k messages total\nhey -m POST -d @\u003c(head -c 16 /dev/urandom | base64) \\\n    -c 100 -n 50000 http://localhost:8080/enqueue\n\n# In another shell\nwatch -n1 'curl -s http://localhost:8080/metrics | jq \"{len:.queue_length, bytes:.queue_bytes}\"'\n```\nObserve `queue_length` grow, then drain as consumers dequeue.\n\n---\n\n## 7. Production Checklist (Why It Matters)\n\n| Item | Reason |\n|------|--------|\n| **`ulimit -n` ≥ 10 k** | Prevent `EMFILE` under heavy concurrency; FD count exposed via `/metrics`. |\n| **Caps tuned** | Protect node‑RAM; shrink caps in staging. |\n| **Process supervisor** | Auto‑restart on crash; passes SIGTERM for graceful shutdown. |\n| **TLS or upstream termination** | Protect messages on the wire (privacy, integrity). |\n| **Alerting** (`queue_length`, `uptime_sec`) | Spot backlog or unexpected restarts. |\n| **Regular `/persist` cron** (optional) | Hot‑backup if message loss is costly. |\n| **Client retry w/ jitter** | Avoid thundering herd when server returns 429. |\n\n---\n\n## 8. Extending the Server (Ideas)\n\n* Replace global mutex with **segment locks** or a **lock‑free ring buffer** for higher parallelism.\n* Add **priority queues** using a heap instead of FIFO slice.\n* Swap JSON snapshot with **binary protobuf** for faster I/O.\n* Integrate with **systemd‑sdnotify** for readiness signals.\n\nPull requests welcome! 🎉\n\n---\n\n## License\n\nMIT – Hack, learn, enjoy.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraiyanyahya%2Fzapq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraiyanyahya%2Fzapq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraiyanyahya%2Fzapq/lists"}