{"id":34674662,"url":"https://github.com/rbilleci/boxy","last_synced_at":"2025-12-24T20:02:24.901Z","repository":{"id":324033118,"uuid":"1021760121","full_name":"rbilleci/boxy","owner":"rbilleci","description":"Boxy","archived":false,"fork":false,"pushed_at":"2025-11-23T20:54:26.000Z","size":2246,"stargazers_count":2,"open_issues_count":32,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-23T21:20:54.616Z","etag":null,"topics":["ai","apache-pulsar","java","kafka","mysql","pgsql","postgres","postgresql","pulsar","streaming","streams","transactional-outbox","transactional-outbox-pattern","transactional-outbox-patterns"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rbilleci.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"buy_me_a_coffee":"rbilleci","github":"rbilleci","thanks_dev":"gh/rbilleci","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"lfx_crowdfunding":null,"polar":null,"custom":null}},"created_at":"2025-07-17T22:57:16.000Z","updated_at":"2025-11-19T11:21:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rbilleci/boxy","commit_stats":null,"previous_names":["rbilleci/boxy"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rbilleci/boxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbilleci%2Fboxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbilleci%2Fboxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbilleci%2Fboxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbilleci%2Fboxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rbilleci","download_url":"https://codeload.github.com/rbilleci/boxy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbilleci%2Fboxy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28007455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-24T02:00:07.193Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","apache-pulsar","java","kafka","mysql","pgsql","postgres","postgresql","pulsar","streaming","streams","transactional-outbox","transactional-outbox-pattern","transactional-outbox-patterns"],"created_at":"2025-12-24T20:01:08.572Z","updated_at":"2025-12-24T20:02:24.895Z","avatar_url":"https://github.com/rbilleci.png","language":"Java","funding_links":["https://buymeacoffee.com/rbilleci","https://github.com/sponsors/rbilleci","https://thanks.dev/gh/rbilleci"],"categories":[],"sub_categories":[],"readme":"\n# Boxy\n\u003cimg src=\"docs/images/boxy-logo.png\" alt=\"Boxy Logo\" style=\"width:50%\" align=\"right\"/\u003e\n\nBoxy is a multi-tenant event streaming library modeled that exposes Apache Pulsar-like semantics \ndirectly over a database’s transactional outbox. It targets monolithic applications that need event streaming without \ntaking on the operational cost of running a Pulsar/Kafka deployment. \nBoxy turns your transactional outbox into an event stream and allows you to build asynchronous consumers in your \nfavorite programming language. \nTenant isolation is provided through a hierarchical namespace system.\n\nBoxy is released under the **Apache License 2.0** and is under active development.\n\n## Table of Contents\n\n- [Design Highlights](#design-highlights)\n- [Architecture Decisions](#architecture-decisions)\n- [Limits](#limits)\n- [Roadmap](#roadmap)\n- [Boxy Core and Boxy DB Overview](#boxy-core-and-boxy-db-overview)\n  - [Key Tables and Views](#key-tables-and-views)\n  - [Subscription Statistics](#subscription-statistics)\n- [Domain Classes](#domain-classes)\n- [Consumer State](#consumer-state)\n- [Lease State](#lease-state)\n- [State Change Example](#state-change-example)\n- [Heartbeats](#heartbeats)\n- [Work-Stealing Lease Protocol](#work-stealing-lease-protocol)\n- [Building the Project](#building-the-project)\n- [Getting Started](#getting-started)\n- [FAQ](#faq)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Design Highlights\n\n- **Decentralized, Randomized Work-Stealing** for lease distribution\n- **Fair-share Load Balancing** across consumers, proportional to capacity weights\n- **Low Consumer Lag**: p99 ~5ms for active partitions, and ~100ms for cold partitions\n- **Scalable Polling**: consumers stagger lease grabs to minimize database queries (e.g. ≈10 checks/s instead of hundreds)\n\n## Architecture Decisions\n- APIs for consumers and producers are kept simple, easy to integrate, and easy to use.\n- Publishing an event should be possible when only knowing the path and topic name.\n- Third-party libraries are minimized to those that are necessary.\n- For safety: boxy never deletes events. Event deletion is left to be orchestrated by you.\n- For easy portability across programming languages and runtimes, all mutations are strictly performed by stored procedures.\n- Namespace and Topic Names are case-sensitive.\n\n## Limits\n- Each topic has a practical limit of 1024 partitions, and a technical limit of 65536 partitions\n- Each subscription has a practical limit of 1024 consumers.\n- The fully qualified namespace path and topic name has a limit of 4000 characters\n- Each namespace name a limit of 500 characters.\n- Each topic name has a limit of 500 characters.\n\n## Roadmap\n\n### V1 (August 2025)\n\n1. Simple Consumer API for Java\n2. Simple Producer API for Java\n\n### V2 (September 2025)\n\n1. Multi-language Consumer APIs (Java, Go, Rust, Python, CLI) \n2. Multi-language Producer APIs\n3. Postgres support\n\n---\n\n## Boxy Core and Boxy DB Overview\n\nLiquibase migrations for the schema are under `boxy-db/src/main/resources/db/changelog`. The schema models:\n\n```mermaid\n    erDiagram\n        namespaces ||--o{ topics : owns\n        topics ||--o{ partitions : has\n        partitions ||--o{ unprocessed_events : queues\n        partitions ||--o{ sequences : sequences\n        events ||--|| unprocessed_events : references\n        events ||--|| sequences : references\n        subscriptions ||--o{ subscription_topics : links\n        subscriptions ||--o{ cursors : positions\n        subscription_topics ||--o{ cursors : positions\n        cursors ||--o{ leases : locks\n        consumers ||--o{ leases : holds\n```\n\n### Key Tables and Views\n\n- **namespaces**: hierarchical containers for topics.\n- **topics**: belong to namespaces and declare a partition count.\n- **partitions**: per-topic shards that track a `high_watermark`.\n  - **events**: raw event payloads; partition and sequence metadata are tracked separately.\n  - **unprocessed_events**: queue linking newly published events to partitions until sequenced.\n  - **sequences**: per-partition sequence numbers referencing events.\n- **subscription_topics**: links subscriptions to the topics they consume and stores precomputed statistics.\n- **cursors**: tracks the position per subscription and partition and stores the `subscription_id`,\n  `subscription_topic_id`, and `topic_id` for join-free lookups. Each row is assigned a persistent\n  `random_key` used for evenly distributing the start position when acquiring leases.\n- **consumers**: registers each consumer’s `subscription_id`, `weight`, and `heartbeat_detected_at`.\n- **leases**: one row per `cursor` when a consumer holds a lease, tracking `subscription_id`, `topic_id`, and `partition_id` alongside `state` to support join-free lookups.\n- **subscriptions**: defines logical groups of consumers.\n- **heartbeat_policies**, **lease_policies**, **metrics_policies**: singleton tables providing cluster-wide configuration.\n- **topics_cache**: in-memory table for quick topic lookups.\n\n### Subscription Statistics\n\nThe `subscription_topics` table stores precomputed statistics that are updated with each consumer check-in:\n\n- **heartbeat_interval**: Adaptive interval used for consumer heartbeats.\n- **active_partitions**: Count of partitions with new events (high_watermark \u003e position).\n- **active_consumers**: Count of consumers with valid heartbeats.\n- **active_consumers_weight**: Sum of weights of all active consumers in the subscription.\n- **last_modified_at**: Timestamp of the last statistics update.\n\nThese statistics are used for:\n\n1. **Fair Share Calculation**: The ideal share of leases for each consumer is calculated as `(consumer_weight / active_consumers_weight) * active_partitions`.\n2. **Adaptive Heartbeat Intervals**: The heartbeat interval is adjusted based on the number of active consumers to maintain a target QPS (queries per second) for the cluster.\n3. **Garbage Collection**: The cluster's lease policy (`lease_release_period`) determines how long a lease remains in the 'RELEASING' state before being deleted.\n\nPrecomputing these statistics reduces the need for expensive queries during consumer check-ins and ensures consistent fair share calculations across all consumers.\n\n### Cluster Policies\n\nThe following tables define cluster-wide defaults and each contains exactly one row:\n\n- `heartbeat_policies`: `heartbeat_deadline_multiplier`, `heartbeat_interval_baseline`, `heartbeat_interval_limit`, `heartbeat_target_qps`\n- `lease_policies`: `active_consumers_limit`, `lease_release_period`\n- `metrics_policies`: `metrics_refresh_interval`\n\nOperators can adjust these records to tune cluster behavior.\n\n## Domain Classes\n\nThe Boxy Core module uses Java records to model the schema. Relevant classes:\n\n```mermaid\nclassDiagram\n    class Consumer {\n        +String id\n        +long subscriptionId\n        +double weight\n        +Instant heartbeatDetectedAt\n        +double heartbeatInterval\n        +Instant heartbeatDeadline\n    }\n    class Cursor {\n        +long id\n        +long subscriptionId\n        +long partitionId\n        +int randomKey\n        +long position\n    }\n    class Lease {\n        +long cursorId\n        +String consumerId\n        +long version\n        +Instant acquiredAt\n        +Instant releasedAt\n        +Instant releaseDeadline\n        +LeaseState state\n    }\n    class LeaseState {\n        \u003c\u003cenumeration\u003e\u003e\n        ACTIVE\n        RELEASING\n    }\n    Consumer --\u003e \"*\" Lease\n    Cursor --\u003e \"0..1\" Lease\n    Lease --\u003e \"1\" LeaseState\n```\n\n\n\n## Consumer State\n\nEach consumer runs in one of three high-level states with respect to a given partition:\n \n      [Idle] ──(grab lease)──\u003e [Processing] ──(drain completed)──\u003e [Releasing] ──(confirm release)──\u003e [Idle]\n\n**Idle**\n- No lease held on this partition; not processing.\n- Periodically (per the work-steal loop) it may grab new leases if under-loaded.\n\n**Processing**\n- Lease acquired with state='ACTIVE' and an event batch is in-flight.\n- Consumer reads events, calls handlers, and updates the position as it goes.\n\n**Releasing**\n- Release can occur in two scenarios: 1) all in-flight work is done and the cursor position is update, 2) the consumer is determined to have an unfair share of leases.\n- The lease is marked as state='RELEASING' with a timestamp in released_at.\n- The consumer must finish its current batch and commit positions before the lease is fully released.\n- After a configurable deadline (default 10 seconds), the lease is deleted by the garbage collection process.\n- This controlled release is intended to reduce the occurrence of duplicate message processing.\n- Once the lease is deleted, the consumer transitions back to Idle.\n\nThis design minimizes leases on cold partitions, to reduce the total number of queries to the database, and ensures that consumers can finish processing in-flight events before leases are reassigned.\n\n## Lease State\n\nEach lease row goes through the following states:\n\n      [Available] ──(INSERT/UPSERT)──\u003e [ACTIVE] ──(UPDATE)──\u003e [RELEASING] ──(DELETE)──\u003e [Available]\n\n**Available (unleased_cursors_view)**\n\n- Partition is active (high_watermark \u003e position) but unleased.\n- Any under-loaded consumer can pick it up via a randomized grab.\n\n**ACTIVE (leases row exists with state='ACTIVE')**\n- The consumer is actively processing events from this partition.\n- The lease can be marked for release if the consumer has more leases than its fair share.\n\n**RELEASING (leases row exists with state='RELEASING')**\n- The consumer is finishing processing any in-flight events before the lease is fully released.\n- The `released_at` timestamp tracks when the release process started.\n- After a configurable deadline (default 10 seconds), the lease is deleted by the garbage collection process.\n- Leases in the RELEASING state are not available for acquisition by other consumers.\n\n```mermaid\nstateDiagram-v2\n[*] --\u003e Idle\nIdle --\u003e Processing      : grab lease\nProcessing --\u003e Releasing : commit final positions\nReleasing --\u003e Idle       : delete lease\n\n    state Processing {\n      [*] --\u003e InFlight\n      InFlight --\u003e InFlight : continue processing events\n      InFlight --\u003e Draining  : no more in-flight\n      Draining  --\u003e [*]      : delete lease (back to Idle)\n    }\n```\n\n## State Change Example\n\n      t=0s    Consumer A grabs lease on P42 → state Idle→Processing, lease created\n      t=0–2s  A processes events 101–105 → in-flight\n      t=3s    A commits position=105, no more in-flight → transition to Releasing\n      t=3s    A issues DELETE FROM leases WHERE X → lease row gone\n      t=4s    New event arrives in P42 → shows up in unleased_cursors_view\n      t=5s    Consumer B grabs lease on P42 → begins Processing\n\n\n## Heartbeats\n\nConsumer-level heartbeat (in the consumers table) remains independent of per-partition state.\n\nRather than each consumer writing every X seconds, we define:\n\n- T\u003csub\u003ecycle\u003c/sub\u003e: a fixed “heartbeat cycle” (e.g. 1 s)\n\n- QPS\u003csub\u003etarget\u003c/sub\u003e: the desired total heartbeats/sec for the whole cluster (e.g. 10 qps)\n\nOn each cycle, each consumer flips a weighted coin with probability `p = min(1, target_QPS / N_active)`, \nand only writes a heartbeat if it “wins” that flip.\n\nProperties\n- When N_active ≤ Q_target, then p = 1 → everyone writes → we get N_active QPS (fine for small clusters).\n- When N_active \u003e Q_target, then p = Q_target / N_active → expected cluster rate ≈ Q_target, irrespective of N.\n- We detect failures in a bounded time: expected per-consumer heartbeat interval = 1 s / p = N_active / Q_target seconds; \n  we pick the dead‐timeout to be a small multiple of that (e.g. 3×).\n\n\n  \n## Work-Stealing Lease Protocol\n\nThe work-stealing algorithm is implemented in the `sp_consumers__check_in` stored procedure and its sub-procedures. The algorithm works as follows:\n\n1. **Subscription Statistics Update**:\n   - The procedure updates precomputed statistics in the `subscription_topics` table:\n     - `heartbeat_interval`: Adaptive interval for consumer heartbeats\n     - `active_partitions`: Count of partitions with new events (high_watermark \u003e position)\n     - `active_consumers`: Count of consumers with valid heartbeats\n     - `active_consumers_weight`: Sum of weights of all active consumers\n   - These statistics are used for fair share calculation and adaptive heartbeat intervals.\n\n2. **Fair-Share Calculation**:\n    - Let `Wᵢ` = consumer weight, `T` = total active weight, `P` = active partitions.\n    - Ideal share `Sᵢ = (Wᵢ / T) * P` with slack Δ (default 10%) to prevent oscillation.\n    - Min leases = ⌊Sᵢ * (1 - Δ)⌋, Max leases = ⌈Sᵢ * (1 + Δ)⌉\n\n3. **Release Excess**: \n   - If held leases \u003e Max leases, mark the least-backlogged leases as 'RELEASING'.\n   - Leases are prioritized for release based on the smallest backlog (high_watermark - position).\n   - Released leases are not immediately deleted but enter a 'RELEASING' state with a timestamp.\n\n4. **Grab More**: \n   - If held leases \u003c Min leases, acquire more leases from the `unleased_cursors_view`.\n   - The algorithm uses a randomized pivot point to minimize contention.\n   - It performs two passes if necessary: first from the pivot to the end, then from the beginning to the pivot.\n   - Leases are acquired with state='ACTIVE' and no released_at timestamp.\n\n5. **Process**: \n   - For each held lease in the 'ACTIVE' state, read events \u003e `position`, process in-order, then update `position`.\n   - Leases in the 'RELEASING' state are allowed to finish processing before being deleted.\n\n6. **Garbage Collection**:\n   - Leases in the 'RELEASING' state are deleted after a configurable deadline (default 10 seconds).\n   - Expired consumers and their leases are deleted if they miss their heartbeat deadline.\n\nThis algorithm ensures:\n\n- **Proportional fairness** by weight\n- **Low churn** via slack Δ and batched grabs/releases\n- **Minimal DB load** through randomized, staggered scans\n- **In-order processing** (one lease-holder per partition)\n- **Duplicate-safety** on failover\n- **Graceful handover** of leases through the 'RELEASING' state\n\n\n## Building the Project\n\nBoxy uses Maven and requires Java 17+. From the root:\n\n```bash\nmvn clean package\n```\n\nIntegration tests use Testcontainers with MySQL (default) or Postgres. Configure via env vars:\n\n| Variable      | Default       | Description           |\n| ------------- | ------------- | --------------------- |\n| `DB_TYPE`     | `mysql`       | `mysql` or `postgres` |\n| `DB_HOST`     | `localhost`   | Database host         |\n| `DB_PORT`     | `3306`/`5432` | Database port         |\n| `DB_NAME`     | `events_db`   | Schema name           |\n| `DB_USER`     | `user`        | Database user         |\n| `DB_PASSWORD` | `password`    | Database password     |\n\n## Getting Started\n\n### Prerequisites\n\n1. Java 17 or higher\n2. MySQL 8.0+ or PostgreSQL 12+\n3. Maven 3.6+\n\n### Database Setup\n\n1. Create a database for Boxy:\n\n```sql\nCREATE DATABASE events_db;\nCREATE USER 'user'@'localhost' IDENTIFIED BY 'password';\nGRANT ALL PRIVILEGES ON events_db.* TO 'user'@'localhost';\n```\n\n2. Run the Liquibase migrations to set up the schema:\n\n```bash\nmvn liquibase:update -Dliquibase.url=jdbc:mysql://localhost:3306/events_db -Dliquibase.username=user -Dliquibase.password=password\n```\n\n\n### Advanced Configuration\n\nYou can configure various aspects of Boxy:\n\n```java\nBoxySubscription subscription = BoxySubscription.builder()\n    .dataSource(dataSource)\n    .name(\"order-processor\")\n    .build();\n\nBoxyConsumer consumer = subscription.createConsumer(BoxyConsumer.builder()\n    .id(\"consumer-1\")\n    .weight(2)                     // Higher weight gets proportionally more partitions\n    .build());\n```\n\nCluster-wide heartbeat, lease, and metrics settings can be adjusted by updating the `heartbeat_policies`, `lease_policies`, and `metrics_policies` tables.\n\n## FAQ\n\n#### How is the schema managed?\nLiquibase is used for schema management, but we do not use the database agnostic schema definitions. When this was attempted it was found that 1) the resulting YAML files were overly complex and required too many exceptions, and 2) the generated schemas would not perform as well as hand-crafted schemas without additional exceptions. Since we aim to support a wide range of databases, a decision was made to maintain complete control over the schema.\n\n## Contributing\n\nContributions to Boxy are welcome! Here's how you can contribute:\n\n1. **Fork the Repository**: Start by forking the repository on GitHub.\n\n2. **Create a Branch**: Create a branch for your feature or bugfix.\n   ```bash\n   git checkout -b feature/your-feature-name\n   ```\n\n3. **Make Changes**: Implement your changes, following the existing code style.\n\n4. **Write Tests**: Add tests for your changes to ensure they work correctly.\n\n5. **Run Tests**: Make sure all tests pass before submitting your changes.\n   ```bash\n   mvn test\n   ```\n\n6. **Submit a Pull Request**: Push your changes to your fork and submit a pull request to the main repository.\n\n### Development Guidelines\n\n- Follow the existing code style and conventions.\n- Keep changes focused on a single issue or feature.\n- Document new code with Javadoc comments.\n- Update the README.md if your changes affect the public API or usage instructions.\n- Add appropriate tests for your changes.\n\n## License\n\nBoxy is released under the Apache License 2.0. See the [LICENSE](LICENSE) file for details.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frbilleci%2Fboxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frbilleci%2Fboxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frbilleci%2Fboxy/lists"}