{"id":51369809,"url":"https://github.com/fandangolas/ratatoskr","last_synced_at":"2026-07-03T05:07:25.914Z","repository":{"id":311059743,"uuid":"1042304709","full_name":"fandangolas/ratatoskr","owner":"fandangolas","description":"🐿️ High-performance message broker built with Elixir/OTP. High throughput, fault-tolerant supervision trees, and comprehensive testing.","archived":false,"fork":false,"pushed_at":"2025-11-08T02:52:58.000Z","size":8067,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-08T04:06:55.567Z","etag":null,"topics":["elixir","functional-programming","kafka","message-broker","pubsub"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fandangolas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-21T19:57:44.000Z","updated_at":"2025-11-08T02:53:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"325f2a8f-8afd-401b-89a5-50ff0ac18a31","html_url":"https://github.com/fandangolas/ratatoskr","commit_stats":null,"previous_names":["fandangolas/ratatoskr"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/fandangolas/ratatoskr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fandangolas%2Fratatoskr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fandangolas%2Fratatoskr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fandangolas%2Fratatoskr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fandangolas%2Fratatoskr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fandangolas","download_url":"https://codeload.github.com/fandangolas/ratatoskr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fandangolas%2Fratatoskr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35072872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","functional-programming","kafka","message-broker","pubsub"],"created_at":"2026-07-03T05:07:25.254Z","updated_at":"2026-07-03T05:07:25.854Z","avatar_url":"https://github.com/fandangolas.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🐿️ Ratatoskr\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./ratatoskr.webp\" width=\"200\" align=\"right\" alt=\"Ratatoskr - The messenger squirrel\" /\u003e\n  \n  **A lightweight, high-performance message broker built with Elixir/OTP**\n  \n  Named after the Norse mythology squirrel who carries messages between the eagle at the top of Yggdrasil and the dragon at its roots, Ratatoskr delivers messages reliably across your distributed system.\n\u003c/div\u003e\n\n## 🎯 What is Ratatoskr?\n\nRatatoskr is a lightweight, educational message broker inspired by Apache Kafka, built entirely in Elixir/OTP. It demonstrates how to build a distributed messaging system using Elixir's powerful concurrency primitives.\n\n**Key Features:**\n- 🚀 High throughput (10,000+ msg/s target)\n- ⚡ Low latency (P99 \u003c 10ms target)\n- 🛡️ Fault-tolerant with OTP supervision\n- 📊 Topic-based publish/subscribe\n- 🔀 Partition-based parallelism\n- 💾 In-memory storage with ETS (Phase 1)\n\n**Purpose:**\n- 📚 Educational: Showcase Elixir/OTP best practices\n- 🎓 Demonstrative: Illustrate GenServers, supervision trees, and the Actor model\n- 🛠️ Practical: Provide a lightweight messaging solution for small-to-medium applications\n\n## 🚀 Quick Start\n\n```bash\n# Clone the repository\ngit clone https://github.com/fandangolas/ratatoskr.git\ncd ratatoskr\n\n# Install dependencies\nmix deps.get\n\n# Run tests\nmix test\n\n# Start the broker (interactive shell)\niex -S mix\n```\n\n### Basic Usage\n\n```elixir\n# Create a topic with 3 partitions\n{:ok, _topic} = Ratatoskr.create_topic(\"orders\", partition_count: 3)\n\n# Publish a message\n{:ok, offset} = Ratatoskr.publish(\"orders\", %{id: 123, amount: 99.99})\n\n# Publish with a key (routes to specific partition)\n{:ok, offset} = Ratatoskr.publish(\"orders\", \"user-123\", %{id: 456, amount: 49.99})\n\n# Subscribe to messages\n{:ok, subscription_id} = Ratatoskr.subscribe(\"orders\", 0, fn message -\u003e\n  IO.inspect(message, label: \"Received\")\n  :ack\nend)\n\n# List all topics\n{:ok, topics} = Ratatoskr.list_topics()\n\n# Delete a topic\n:ok = Ratatoskr.delete_topic(\"orders\")\n```\n\n## 🏗️ Architecture\n\nRatatoskr leverages OTP patterns for maximum reliability and performance:\n\n```\nApplication Supervisor (one_for_one)\n├── Registry (process discovery)\n├── Broker (topic lifecycle coordinator)\n└── Topic.Supervisor (dynamic)\n    └── Topic.Server (per topic)\n        └── Partition.Server (per partition)\n            ├── ETS storage\n            └── :queue (FIFO ordering)\n```\n\n### Core Components\n\n**Broker** (`Ratatoskr.Broker`)\n- Manages topic lifecycle (create/delete/list)\n- Maintains topic metadata\n- Coordinates with Topic.Supervisor\n\n**Topic Server** (`Ratatoskr.Topic.Server`)\n- Routes messages to partitions\n- Manages partition processes\n- Handles key-based routing: `hash(key) % partition_count`\n\n**Partition Server** (`Ratatoskr.Partition.Server`)\n- Stores messages in FIFO order\n- Manages subscribers (push model)\n- Assigns auto-incrementing offsets\n- Uses ETS for fast offset-based lookups\n\n**Registry** (Built-in)\n- Fast process discovery\n- Topic and partition name resolution\n- No external dependencies\n\n### Design Principles\n\n1. **Fault Isolation**: One GenServer per partition - failures are isolated\n2. **Let It Crash**: OTP supervisors handle recovery\n3. **Immutability**: Messages are append-only\n4. **Process Isolation**: Each partition is independent\n5. **Simple First**: Start minimal, add complexity as needed\n\n## 📊 Performance Targets (Phase 1)\n\n| Metric | Target |\n|--------|--------|\n| Throughput | 10,000+ msg/s per node |\n| Latency (P99) | \u003c 10ms |\n| Concurrent Consumers | 100+ per partition |\n| Message Size | Up to 1MB |\n| Memory per Partition | ~10MB baseline |\n\n## 🗺️ Roadmap\n\n### Phase 1: Core Engine (MVP) - **Current Phase**\n- [x] Project setup with supervision tree\n- [ ] Broker GenServer (topic management)\n- [ ] Topic.Server (routing logic)\n- [ ] Partition.Server (message storage + delivery)\n- [ ] Producer API\n- [ ] Consumer API (callback mode)\n- [ ] Basic tests (\u003e80% coverage)\n- [ ] Performance benchmarks\n\n### Phase 2: Persistence\n- [ ] Append-only log files per partition\n- [ ] Write-ahead logging (WAL)\n- [ ] Configurable retention policies\n- [ ] Recovery on crash\n\n### Phase 3: Advanced Features\n- [ ] Consumer groups (load balancing)\n- [ ] Offset management (commit/reset)\n- [ ] Dead letter queues\n- [ ] Message TTL\n- [ ] Compression\n\n### Phase 4: Observability\n- [ ] Telemetry integration\n- [ ] Metrics dashboard (Phoenix LiveView)\n- [ ] Distributed tracing\n- [ ] Health checks\n\n### Phase 5: Clustering (Stretch Goal)\n- [ ] Multi-node deployment\n- [ ] Partition replication\n- [ ] Leader election\n\n## 🧪 Testing\n\n```bash\n# Run all tests\nmix test\n\n# Run with coverage\nmix test --cover\n\n# Run performance benchmarks (future)\nmix run benchmark/throughput.exs\n```\n\n### Testing Strategy\n\n- **Unit Tests**: Each GenServer module tested independently\n- **Integration Tests**: Full publish-subscribe flows\n- **Property Tests**: Message ordering guarantees\n- **Performance Tests**: Throughput and latency benchmarks\n- **Target**: \u003e90% code coverage\n\n## 🔧 Development\n\n### Requirements\n\n- Elixir 1.17+\n- Erlang/OTP 27+\n\n### Code Quality\n\n```bash\n# Format code\nmix format\n\n# Run static analysis (future)\nmix credo\n\n# Run dialyzer (future)\nmix dialyzer\n```\n\n## 📚 Documentation\n\nFor detailed architecture and design decisions, see [ARCHITECTURE.md](ARCHITECTURE.md).\n\n## 📝 License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## 🙏 Acknowledgments\n\n- Inspired by [Apache Kafka](https://kafka.apache.org/)\n- Built with [Elixir](https://elixir-lang.org/) and [OTP](https://www.erlang.org/)\n\n---\n\n\u003cdiv align=\"center\"\u003e\n  \u003ci\u003eBuilt with ❤️ and Elixir\u003c/i\u003e\n\u003c/div\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffandangolas%2Fratatoskr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffandangolas%2Fratatoskr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffandangolas%2Fratatoskr/lists"}