{"id":21436748,"url":"https://github.com/modfin/delta","last_synced_at":"2025-03-16T23:23:49.167Z","repository":{"id":263583602,"uuid":"880513994","full_name":"modfin/delta","owner":"modfin","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-01T15:42:43.000Z","size":44,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-23T09:33:59.846Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/modfin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-29T21:31:41.000Z","updated_at":"2024-11-18T15:52:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"6c0f5983-2aea-4688-99c6-cde8c73976cd","html_url":"https://github.com/modfin/delta","commit_stats":null,"previous_names":["modfin/delta"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fdelta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fdelta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fdelta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fdelta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/modfin","download_url":"https://codeload.github.com/modfin/delta/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243945786,"owners_count":20372931,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T00:14:39.612Z","updated_at":"2025-03-16T23:23:49.162Z","avatar_url":"https://github.com/modfin.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Delta\n\n[![goreportcard.com](https://goreportcard.com/badge/github.com/modfin/delta)](https://goreportcard.com/report/github.com/modfin/delta)\n[![PkgGoDev](https://pkg.go.dev/badge/github.com/modfin/delta)](https://pkg.go.dev/github.com/modfin/delta)\n\nDelta is a message queue backed by SQLite for persistence. It provides a simple and efficient way to handle message queuing with the durability and reliability of SQLite.\n\nIt is somewhat inspired by nats.io, but is single instance and persistence from the start.\n\n## Features\n\n- **Persistence**: Messages are stored in an SQLite database, ensuring durability.\n- **Pub/Sub**: Supports publish/subscribe messaging pattern.\n- **Queue**: Supports message queuing with load balancing.\n- **Request/Reply**: Supports request/reply messaging pattern.\n- **Multiple Streams**: Allows creating multiple streams for different namespaces.\n- **Globs**: Supports subscribing on a Glob for pattern-based subscriptions.\n\n## Installation\n\nTo install Delta, use `go get`:\n\n```sh\ngo get github.com/modfin/delta\n```\n\n## Usage\n\n### Creating a Message Queue Instance\n\n```go\npackage main\n\nimport (\n\t\"github.com/modfin/delta\"\n\t\"log/slog\"\n)\n\nfunc main() {\n\tmq, err := delta.New(\"file:delta.db\", delta.WithLogger(slog.Default()))\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tdefer mq.Close()\n}\n```\n\n\n### Pub/Sub\n\nThe publish/subscribe messaging pattern allows multiple subscribers to receive messages from a topic, \"One to many\" or broadcasting\n\n[![](https://mermaid.ink/img/pako:eNptjrEKgzAQhl8l3FRBB9vNoVC0nTqU2jFLjKcGEiMxoZTouzdSCxV6t9x9_8dxHriuETJopH7yjhlLrnfak1An72-ukmLs0MzzykiSHMmkxjadSL7bPfQgeBR9wvw3LLwvXTVyIyo0JP0e2DjnjbP_61w2zmFxliYEYlBoFBN1-N4vhILtUCGFLIw1NsxJS4H2c1CZs7p89RwyaxzGYLRrO8gaJsewuaFmFgvBWsPUSuc3JgZclg?type=png)](https://mermaid.live/edit#pako:eNptjrEKgzAQhl8l3FRBB9vNoVC0nTqU2jFLjKcGEiMxoZTouzdSCxV6t9x9_8dxHriuETJopH7yjhlLrnfak1An72-ukmLs0MzzykiSHMmkxjadSL7bPfQgeBR9wvw3LLwvXTVyIyo0JP0e2DjnjbP_61w2zmFxliYEYlBoFBN1-N4vhILtUCGFLIw1NsxJS4H2c1CZs7p89RwyaxzGYLRrO8gaJsewuaFmFgvBWsPUSuc3JgZclg)\n\n\n\n### Publishing Messages\n\nPublish a message to a specific topic using the `Publish` method. No need to create the topic on beforehand or declare it.\n\n```go\npub, err := mq.Publish(\"a.b.c.d\", []byte(\"hello world\"))\nif err != nil {\n    panic(err)\n}\n```\n\n### Subscribing to Messages\n\nSubscribe to a specific topic using the `Subscribe` method. The subscription can use a glob pattern to match multiple topics.\nExample might be `location.*` will match `location.us`, `location.eu` etc. but not `location.us.new-york`. For this we \ncan use `**` which basically translates to prefix match, eg `location.**` will match `location.us`, `location.us.new-york`, `location.us.new-york.new-york-city` and so on.\n\n\nThis can be done using a chan or by using the `Next` method which will block until a message is received or subscription is closed .\n\n```go\nsub, err := mq.Subscribe(\"a.b.*.d\")\nif err != nil {\n    panic(err)\n}\n\ngo func() {\n    for msg := range sub.Chan() { // or 'msg, ok := sub.Next()' can be used\n        fmt.Printf(\"[Chan] Received message: %s \u003c- %s\\n\", msg.Topic, string(msg.Payload))\n    }\n}()\n\n\n```\n\n\n### Using Queues\n\nA queue is a group of subscribers that receive messages from a topic. Each message is delivered to _only one_ subscriber.\nIt does not put any constraint how things are being published to the queue, it is only the consumption that is load balanced.\n\n[![](https://mermaid.ink/img/pako:eNplzz0LgzAQBuC_ctxkwQ7azaFQtJ06lLZjlhhPDRhTYkIp0f_eSL-Q5uC4e3k4iEehK8IM607fRcuNheOZ9RDezvuTKzs5tGSm6Z3Ber2FUQ1NEoeWzm0zQh5FV32TYrV6sfzL0hEK7y-uHISRJRlIPqd-JhzYL0z6b5IRDguzmc1cABijIqO4rMI__JwwtC0pYpiFsaKau84yZP0UKHdWXx69wMwaRzEa7ZoWs5p3Q9jcreKWCskbw9U7nZ7ap1_k?type=png)](https://mermaid.live/edit#pako:eNplzz0LgzAQBuC_ctxkwQ7azaFQtJ06lLZjlhhPDRhTYkIp0f_eSL-Q5uC4e3k4iEehK8IM607fRcuNheOZ9RDezvuTKzs5tGSm6Z3Ber2FUQ1NEoeWzm0zQh5FV32TYrV6sfzL0hEK7y-uHISRJRlIPqd-JhzYL0z6b5IRDguzmc1cABijIqO4rMI__JwwtC0pYpiFsaKau84yZP0UKHdWXx69wMwaRzEa7ZoWs5p3Q9jcreKWCskbw9U7nZ7ap1_k)\n\n```go\n\nfor i := 0; i \u003c 3; i++ {\n\ti:= i\n    sub, err := mq.Queue(\"a.b.*.d\", \"queue1\") // queue1 is the name of the queue which is used to identify it\n    if err != nil {\n        panic(err)\n    }\n\n    go func() {\n        for msg := range sub.Chan() {\n            fmt.Printf(\"Received message from queue by worker %d: %s\\n\", \n\t\t\t\ti, \n\t\t\t\tstring(msg.Payload),\n\t\t\t)\n        }\n    }()\n}\n\n```\n\n### Request/Reply\n\nThe request/reply messaging pattern allows a client to send a request to a service and receive a response.\n\n[![](https://mermaid.ink/img/pako:eNpVj08LwjAMxb9KyGkDFdTbDoL45-RBpsdeui7bCu06uhYZ3b67VSdicsl7v5dAAgpTEmZYKfMQDbcOLjlrIdY-hKsvlOwbstM0e7Bc7mDUfT3CIUnuppMiTT_sAKvIjiHcfNELKwuysP4ufuDpD25-8Hf1_BfZfiPndyRPkpw6NaTpW-5Z-2oAXKAmq7ks4yfh5TB0DWlimMWxpIp75RiydopR7p25Da3AzFlPC7TG1w1mFVd9VL4ruaOj5LXlenanJxw1XZc?type=png)](https://mermaid.live/edit#pako:eNpVj08LwjAMxb9KyGkDFdTbDoL45-RBpsdeui7bCu06uhYZ3b67VSdicsl7v5dAAgpTEmZYKfMQDbcOLjlrIdY-hKsvlOwbstM0e7Bc7mDUfT3CIUnuppMiTT_sAKvIjiHcfNELKwuysP4ufuDpD25-8Hf1_BfZfiPndyRPkpw6NaTpW-5Z-2oAXKAmq7ks4yfh5TB0DWlimMWxpIp75RiydopR7p25Da3AzFlPC7TG1w1mFVd9VL4ruaOj5LXlenanJxw1XZc)\n\n\n```go\n\n// test is the name of the queue which is used to identify it\n// it is important to use a queue group to ensure that the request is \n//not hadled by multiple workers\nfor i := 0; i \u003c 3; i++ {\n    i:= i\n    sub, err := mq.Queue(\"greet.*\", \"test\") \n    if err != nil {\n        panic(err)\n    }\n    go func() {\n        for msg := range sub.Chan() {\n            _, name, _ := strings.Cut(msg.Topic, \".\")\n            _, err := msg.Reply([]byte(fmt.Sprintf(\"from worker %d \u003e hello %s, \", i, name))\n            if err != nil {\n                panic(err)\n            }\n        }\n    }()\n}\n\n\nresp, err := mq.Request(context.Background(), \"greet.alice\", nil)\nif err != nil {\n    panic(err)\n}\nmsg, ok := resp.Next()\nif ok {\n    fmt.Printf(\"Received reply: %s\\n\", string(msg.Payload))\n}\n```\n\n\n\n### Subscription From\n\nThe `SubscribeFrom` method allows you to subscribe to messages from a specific topic starting from a given historical time. This is useful when you want to process messages that were published after a certain point in time or you want to re-process messages.\n\n#### Example Usage\n\n```go\n\n// Publish some messages\nfor i := 0; i \u003c 10; i++ {\n    payload := []byte(\"message \" + strconv.Itoa(i))\n    _, err := mq.Publish(\"example.topic\", payload)\n    if err != nil {\n        panic(err)\n    }\n}\n\n// Subscribe from a specific time\nfrom := time.Now()\n\n// Publish more messages\nfor i := 10; i \u003c 20; i++ {\n    payload := []byte(\"message \" + strconv.Itoa(i))\n    _, err := mq.Publish(\"example.topic\", payload)\n    if err != nil {\n        panic(err)\n    }\n}\n\nsub, err := mq.SubscribeFrom(\"example.topic\", from)\nif err != nil {\n    panic(err)\n}\n// Read messages from the subscription\nfor i := 10; i \u003c 20; i++ {\n    msg, ok := sub.Next()\n    if !ok {\n        panic(\"failed to read message\")\n    }\n    fmt.Println(\"Received historic message:\", string(msg.Payload))\n}\n\n```\n\n## Benchmarks\n\nSome benchmarks, but remember, generic benchmarks are shit :) Anyway, it seems it performs decently.\nIt fans out with very little overhead and seems and writes per second seems to be pretty stable (since its singe threaded.).  \n\n```text\n\n// Performed with waking up readers, which takes a performance hit on writers.\n\nBenchmarkMultipleSubscribers/1-22                     8_242 read-msg/s    9_196 write-msg/s\nBenchmarkMultipleSubscribers/4-22                    31_810 read-msg/s    8_037 write-msg/s\nBenchmarkMultipleSubscribers/num-cpu_(22)-22        212_433 read-msg/s    9_774 write-msg/s\nBenchmarkMultipleSubscribers/2x_num-cpu_(44)-22     411_696 read-msg/s    9_631 write-msg/s\n\n\nBenchmarkMultipleSubscribersSize/_0.1mb-22   1_313 read-MB/s     13_130 read-msg/s     59.7 write-MB/s     597.3 write-msg/s\nBenchmarkMultipleSubscribersSize/_1.0mb-22   3_517 read-MB/s      3_517 read-msg/s    160.6 write-MB/s     160.6 write-msg/s\nBenchmarkMultipleSubscribersSize/10.0mb-22   3_856 read-MB/s        386 read-msg/s    184.5 write-MB/s      18.5 write-msg/s\n\n```\n\n\n## License\n\nThis project is licensed under the MIT License. See the `LICENSE` file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodfin%2Fdelta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodfin%2Fdelta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodfin%2Fdelta/lists"}