{"id":17543586,"url":"https://github.com/lopcode/daisy","last_synced_at":"2025-04-24T00:02:11.474Z","repository":{"id":49130131,"uuid":"342974433","full_name":"lopcode/Daisy","owner":"lopcode","description":"A coroutine based message processing library, for Kotlin backend services 🌼","archived":false,"fork":false,"pushed_at":"2021-06-27T17:17:22.000Z","size":155,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-22T15:15:26.983Z","etag":null,"topics":["kotlin","server","sns","sqs"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lopcode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-27T22:46:03.000Z","updated_at":"2023-02-06T21:23:50.000Z","dependencies_parsed_at":"2022-08-26T00:13:59.655Z","dependency_job_id":null,"html_url":"https://github.com/lopcode/Daisy","commit_stats":null,"previous_names":["lopcode/daisy"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lopcode%2FDaisy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lopcode%2FDaisy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lopcode%2FDaisy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lopcode%2FDaisy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lopcode","download_url":"https://codeload.github.com/lopcode/Daisy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250535030,"owners_count":21446506,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kotlin","server","sns","sqs"],"created_at":"2024-10-21T00:24:54.923Z","updated_at":"2025-04-24T00:02:11.001Z","avatar_url":"https://github.com/lopcode.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Daisy 🌼\n\nDaisy is a coroutine based message processing library, for Kotlin backend services. It removes a lot of messaging-related\nboilerplate, and can scale up to meet your throughput needs.\n\nIt supports the basics, like message routing based on attributes, and message deletion after processing. More advanced features\nare also made easy - such as exponential backoff, and message delaying, to save you money during slow periods.\n\nMost parts of Daisy are configurable, but the defaults are good enough for production use, and will help you build\na fast and reliable message processor; without repeating yourself across services.\n\nThe library is used and proved in production at [Adopt Animals](https://www.adopt.app), but it is developed independently\nof anything else, including Kale Charity and my employer.\n\n## Project goals\n\n* ✅ Limit scope to SQS and SNS support to begin with\n* ✅ Limit scope to workers deployed as independently scalable processes, paired/deployed with a partner microservice (write vs read separation)\n* ✅ Support the consumption of messages using coroutines: https://github.com/Kotlin/kotlinx.coroutines\n* ✅ Support message routing and serialization helpers: https://github.com/Kotlin/kotlinx.serialization\n* ✅ Support changing message visibility to delay retries\n* ✅ Gracefully scale to large amounts of messages (don't be the bottleneck)\n* ✅ Support backoff strategies for scaling down processing of messages during slow periods\n* 🟨 Include documentation, and example projects\n* Support composition of processors - e.g. applying or removing GZIP compression\n* Support permanently failing the processing of a message, by delivering to a DLQ\n\n## Core\n\nThe library includes a `core` implementation. For now, the [demo](https://github.com/CarrotCodes/Daisy/tree/main/demo/src/main/kotlin/dev/skye/daisy)\nprojects described below are the best place to find examples of usage.\n\nBefore the API stabilises, and the library goes 1.0, this section will be expanded.\n\nThe first prerelease versions are published to Maven Central: `dev.skye.daisy:daisy-core:0.0.4`\n\nYou can use it with Gradle, for example:\n\n```groovy\nplugins {\n    id 'org.jetbrains.kotlin.jvm' version '1.5.10'\n}\n\nrepositories {\n    mavenCentral()\n}\n\ndependencies {\n    implementation \"dev.skye.daisy:daisy-core:0.0.4\"\n\n    // Kotlin\n    implementation platform(\"org.jetbrains.kotlin:kotlin-bom\")\n    implementation platform(\"org.jetbrains.kotlinx:kotlinx-coroutines-bom:1.5.0\")\n    implementation \"org.jetbrains.kotlin:kotlin-stdlib-jdk8\"\n    implementation \"org.jetbrains.kotlinx:kotlinx-coroutines-core\"\n    implementation \"org.jetbrains.kotlinx:kotlinx-coroutines-jdk8\"\n\n    // AWS\n    implementation platform(\"software.amazon.awssdk:bom:2.16.8\")\n    implementation \"software.amazon.awssdk:sqs\"\n\n    // Metrics\n    implementation \"io.micrometer:micrometer-core:1.6.5\"\n}\n```\n\n## Ktor integration\n\nDaisy can be integrated with Ktor, to start and stop message processing as the Ktor application starts and stops.\n\nThe first prerelease versions are published to Maven Central: `dev.skye.daisy:daisy-ktor:0.0.4`\n\n```kotlin\nimport dev.skye.daisy.DaisyFeature\n\n// ...\n\ninstall(DaisyFeature) {\n    configuration = DaisyConfiguration(\n        // ...\n    )\n}\n```\n\nSee the demo projects section below for instructions on how to run the sample Ktor project.\n\n## Demo projects\n\n### Throughput demo\n\nThe throughput demo uses an in-memory SQS implementation to generate messages when requested, and will run until\nterminated, or until zero messages are processed.\n\nThe demo shows a theoretical maximum for your machine, to prove that this library probably isn't the bottleneck in your\nmessage processing pipeline. The example output is from my laptop (a 16\" MacBook Pro), but in this demo, Daisy scales to\nmillions of messages on more powerful computers.\n\nIn real-world applications, where HTTP requests are used to perform actions on queues, you'll likely see much lower\nthroughput.\n\nTo run the demo project:\n* Open the project in IntelliJ IDEA\n* Run `main` in `ThroughputDemo.kt`, in the `demo` subproject\n\nExample output:\n```\nmessages.deleted{queue=https://test.local/0000/queue-1} throughput=130959/s\nmessages.deleted{queue=https://test.local/0000/queue-1-dlq} throughput=130760/s\nmessages.polled{queue=https://test.local/0000/queue-1-dlq} throughput=130810/s\nmessages.polled{queue=https://test.local/0000/queue-1} throughput=131020/s\nmessages.processed{queue=https://test.local/0000/queue-1} throughput=130990/s\nmessages.processed{queue=https://test.local/0000/queue-1-dlq} throughput=130780/s\nmessages.processed.total{} throughput=261771/s\n```\n\n### Ktor demo\n\nSee `KtorDemo.kt` in the `demo` subproject for an example of how to integrate. The demo includes a server that starts on\nport 8080, and includes Prometheus metrics. The supplied dummy SQS implementation produces a low number of messages,\nto generate some metrics.\n\nExample application output:\n```\n[DefaultDispatcher-worker-1] INFO ktor.application - Autoreload is disabled because the development mode is off.\n[DefaultDispatcher-worker-1] INFO ktor.application - Responding at http://0.0.0.0:8080\n[DefaultDispatcher-worker-1] INFO ktor.application - Daisy starting...\n[DefaultDispatcher-worker-9] INFO dev.skye.daisy.DaisyFeature - received message 8b3f1fba-3a32-4987-bbfb-8c3bed4a5d1f: MessageBody(message=hello, world!)\n[DefaultDispatcher-worker-5] INFO dev.skye.daisy.DaisyFeature - received message c5b1f675-527d-4649-bd45-e90a05ed0e89: MessageBody(message=hello, world!)\n```\n\nExample metrics output:\n\n```\n➜ http localhost:8080/metrics | grep messages\n# HELP messages_polled_total  \n# TYPE messages_polled_total counter\nmessages_polled_total{queue=\"https://test.local/0000/queue-1\",} 640.0\nmessages_polled_total{queue=\"https://test.local/0000/queue-1-dlq\",} 630.0\n# HELP messages_deleted_total  \n# TYPE messages_deleted_total counter\nmessages_deleted_total{queue=\"https://test.local/0000/queue-1\",} 640.0\nmessages_deleted_total{queue=\"https://test.local/0000/queue-1-dlq\",} 630.0\n# HELP messages_processed_total  \n# TYPE messages_processed_total counter\nmessages_processed_total{queue=\"https://test.local/0000/queue-1\",} 640.0\nmessages_processed_total{queue=\"https://test.local/0000/queue-1-dlq\",} 630.0\n```\n\n## Copyright\n\nThis project is licensed under the Apache License: [LICENSE.txt](LICENSE.txt)\n\n```\nCopyright 2021 Skye Welch\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flopcode%2Fdaisy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flopcode%2Fdaisy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flopcode%2Fdaisy/lists"}