{"id":16119438,"url":"https://github.com/codingchili/jet-pluginifer","last_synced_at":"2025-04-06T10:19:21.052Z","repository":{"id":87434379,"uuid":"217731038","full_name":"codingchili/jet-pluginifer","owner":"codingchili","description":"The pluginifier is a tiny wrapper that makes it possible to run Hazelcast Jet processors like plugins.","archived":false,"fork":false,"pushed_at":"2019-10-27T10:09:56.000Z","size":173,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-12T15:51:45.777Z","etag":null,"topics":["clustering","hazelcast","hazelcast-jet","jet","kotlin"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codingchili.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-26T15:43:22.000Z","updated_at":"2019-10-27T10:09:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"6dfc30c0-01ae-49dc-8546-9910e4467724","html_url":"https://github.com/codingchili/jet-pluginifer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingchili%2Fjet-pluginifer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingchili%2Fjet-pluginifer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingchili%2Fjet-pluginifer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingchili%2Fjet-pluginifer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codingchili","download_url":"https://codeload.github.com/codingchili/jet-pluginifer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247464387,"owners_count":20943001,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clustering","hazelcast","hazelcast-jet","jet","kotlin"],"created_at":"2024-10-09T20:54:09.609Z","updated_at":"2025-04-06T10:19:21.045Z","avatar_url":"https://github.com/codingchili.png","language":"Kotlin","readme":"# jet-pluginifier  \n\nHazelcast jet is a distributed stream processing engine. It can be used for low-latency streaming of events.\nIn this lab I'm evaluating if it can be used for processing cases/incoming messages in a case management system.\n\nThe idea is that Jet instances are very easy to deploy and allows the application to scale out painlessly. The\nserver can still be responsive for the clients even when processing large amounts of data/events. It could also\nserve as a replacement to the process engine completely, as it's very easy to use and even supports visualizations with the\nDOT format. \n\nIt would be cool to use distributed tracing along with the js-viz library for realtime surveillance as the events\nget passed through the system. By supporting the DOT format process designs can be shared bidirectionally from \ndevelopers to system designers.\n    \nPerformance wise Hazelcast Jet would potentially be a great improvement, the API's used could also be improved. By\nkeeping the whole case object in memory for example, it would be possible and very EASY to modify cases even\nwith deep hierarchies. When the process is started, the whole case tree is simply loaded into memory and transferred\nbetween nodes. The size of a case is usually very small, as it's only textual data - documents doesn't need to be\nincluded (but could of course).\n\n    \n##### Features\n\nI spent a day on the lab and this is what was done.\n\n- Using a custom datasource, the Hazelcast Queue to feed the DAG.\n- Wrapping the low-level DAG core API's in a `ProcessBuilder`\n- Creating a \"pluginified\" wrapper on top of the API's.\n- Added distributed tracing using Hazelcast topics.\n\nFind the real API's [here](https://docs.hazelcast.org/docs/jet/3.2/manual/)\n\n\n##### Stuff to think about\n\n- Check out a complete case tree including it's hierarchy. (only ONCE per process)\n- How should multiple processes per case be handled.\n- How does fork/join work? any way to wait for another item to arrive?\n- Hazelcast clustering really helps with messaging etc.\n- Any way to provide callbacks when processes completes? (for stream sources)\n- 1MB is around 75 _PAGES_ of paper, additionally text compresses very well.\n- case objects may be persisted at any stage, controlled by the plugins.\n- jobs can spawn other jobs, can jobs wait for external/other jobs?\n- support for building charts from existing xml configuration?\n- Pipeline is easy to use, DAG is harder - \"pluginified\" is easiest.\n- jobs can be paused unpaused, with snapshot support.\n- DAG's can be checked for errors without starting Jet.\n- local vs distributed edges, local edges are probably always better in this scenario.\n- are there business cases that REALLY requires cycles in the graph?\n- calling blocking code requires the processor to yield. (see context/nonCooperative)\n\nSee, \"edge forwarding patterns\", \"blocking processors\", \"custom data sources\" etc in the manual.\n\n    \n### Result\n\nPlugins are implemented like this, in Kotlin\n\n```kotlin\nclass TestPluginTransform : ProcessPlugin\u003cContextImpl, TestObject\u003e {\n    override fun process(context: ContextImpl, item: TestObject): TestObject {\n        // the context in this case contains a method for transforming the given item. cool.\n        return context.transform(item)\n    }\n}\n```\n\nUsing the `ProcessBuilder` API's to build a distributed process\n\n```kotlin\nfun process() {\n    // create a builder to create the process\n    val process = ProcessFactory.create\u003cContextImpl, TestObject\u003e(\n        ContextImpl::class.java\n    )\n    process.setName(\"MessageSource\")\n    \n    process.vertex(TestPluginStartNode::class.java)\n        .edge(TestPluginTransform::class.java)\n        .edge(TestPluginValidate::class.java)\n\n    // one of the fork paths\n    process.vertex(TestPluginTransform::class.java)\n        .edge(TestPluginJoin::class.java)\n\n    // the second fork path\n    process.vertex(TestPluginValidate::class.java)\n        .edge(TestPluginJoin::class.java)\n\n    // join node, forward to end node \"save\".\n    process.vertex(TestPluginJoin::class.java)\n        .edge(TestPluginSave::class.java)\n}\n```\n\nProduces the following DOT output (`DAG#toDOTString()`)\n\n```json\ndigraph DAG {\n\t\"MessageSource\" [tooltip=\"local-parallelism=1\"];\n\t\"TestPluginStartNode\" [tooltip=\"local-parallelism=4\"];\n\t\"TestPluginValidate\" [tooltip=\"local-parallelism=4\"];\n\t\"TestPluginTransform\" [tooltip=\"local-parallelism=4\"];\n\t\"TestPluginJoin\" [tooltip=\"local-parallelism=4\"];\n\t\"TestPluginSave\" [tooltip=\"local-parallelism=4\"];\n\t\"MessageSource\" -\u003e \"TestPluginStartNode\";\n\t\"TestPluginStartNode\" -\u003e \"TestPluginTransform\" [label=\"distributed\", taillabel=0];\n\t\"TestPluginStartNode\" -\u003e \"TestPluginValidate\" [label=\"distributed\", taillabel=1];\n\t\"TestPluginValidate\" -\u003e \"TestPluginJoin\" [label=\"distributed\", headlabel=1];\n\t\"TestPluginTransform\" -\u003e \"TestPluginJoin\" [label=\"distributed\", headlabel=0];\n\t\"TestPluginJoin\" -\u003e \"TestPluginSave\" [label=\"distributed\"];\n}\n```\n\nWhich can be visualized on [viz-js.com](http://viz-js.com/) and looks like this\n\n![result](screenshot/graphviz.png)\n\n#### Conclusion\n\nHazelcast Jet is very cool, I started out with the `Pipeline` API but wanted to go deeper and using the `DAG` API directly. \nThe hardest part was figuring out how to use custom data sources with this low-level API. It was much easier\nusing the `SourceBuilder` with the Pipeline API than creating one from `SourceProcessors`.\n\nThe result is pretty cool and easy, it pluginifies Hazelcast Jet which is nice for bigger operations I guess. :smiley_cat:\n\n### Building\n\nBuild the project with\n\n```console\ngradlew jar\n```\n\nRun with \n\n```console\n# create two distributed processing instances\njava -jar \u003cjet-sample\u003e.jar --instance\njava -jar \u003cjet-sample\u003e.jar --instance\n\n# create the instance that will start processing items.\njava -jar \u003cjet-sample\u003e.jar --process\n\n```\n\nThe following output should appear\n\n```$xslt\nNOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED\n--add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/sun.net.dns=ALL-UNNAMED --add-opens=java.base/java.lan\ng.reflect=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-open\ns=java.base/java.net=ALL-UNNAMED\nokt. 26, 2019 7:21:40 EM com.hazelcast.config.AbstractConfigLocator\nINFO: Loading 'hazelcast-jet-default.xml' from the classpath.\nokt. 26, 2019 7:21:40 EM com.hazelcast.jet.config.JetConfig\nINFO: jet.home is C:\\Users\\chili\\Documents\\jet-sample\\build\\libs\nokt. 26, 2019 7:21:40 EM com.hazelcast.config.AbstractConfigLocator\nINFO: Loading 'hazelcast-jet-member-default.xml' from the classpath.\nokt. 26, 2019 7:21:40 EM com.hazelcast.instance.HazelcastInstanceFactory\nWARNING: Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Ja\nva packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is\nused to get the best performance results. Arguments to be used:\n --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED\n--add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.\nmanagement=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED\nokt. 26, 2019 7:21:40 EM com.hazelcast.instance.AddressPicker\nINFO: [LOCAL] [jet] [3.2] Prefer IPv4 stack is true, prefer IPv6 addresses is false\nokt. 26, 2019 7:21:40 EM com.hazelcast.instance.AddressPicker\nINFO: [LOCAL] [jet] [3.2] Picked [10.0.75.1]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind\n any local is true\nokt. 26, 2019 7:21:40 EM com.hazelcast.system\nINFO: [10.0.75.1]:5701 [jet] [3.2] Hazelcast Jet 3.2 (20191018 - 61a9fa1) starting at [10.0.75.1]:5701\nokt. 26, 2019 7:21:40 EM com.hazelcast.system\nINFO: [10.0.75.1]:5701 [jet] [3.2]\n        o   o   o   o---o o---o o     o---o   o   o---o o-o-o        o o---o o-o-o\n        |   |  / \\     /  |     |     |      / \\  |       |          | |       |\n        o---o o---o   o   o-o   |     o     o---o o---o   |          | o-o     |\n        |   | |   |  /    |     |     |     |   |     |   |      \\   | |       |\n        o   o o   o o---o o---o o---o o---o o   o o---o   o       o--o o---o   o\nokt. 26, 2019 7:21:40 EM com.hazelcast.system\nINFO: [10.0.75.1]:5701 [jet] [3.2] Copyright (c) 2008-2019, Hazelcast, Inc. All Rights Reserved.\nokt. 26, 2019 7:21:41 EM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator\nINFO: [10.0.75.1]:5701 [jet] [3.2] Backpressure is disabled\nokt. 26, 2019 7:21:41 EM com.hazelcast.instance.Node\nINFO: [10.0.75.1]:5701 [jet] [3.2] Creating MulticastJoiner\nokt. 26, 2019 7:21:42 EM com.hazelcast.jet.impl.execution.TaskletExecutionService\nINFO: [10.0.75.1]:5701 [jet] [3.2] Creating idler with jet.idle.cooperative.min.microseconds=25Ás,jet.idle.cooperative.max.microseconds=500Ás\nokt. 26, 2019 7:21:42 EM com.hazelcast.jet.impl.execution.TaskletExecutionService\nINFO: [10.0.75.1]:5701 [jet] [3.2] Creating idler with jet.idle.noncooperative.min.microseconds=25Ás,jet.idle.noncooperative.max.microseconds=5000Ás\nokt. 26, 2019 7:21:42 EM com.hazelcast.jet.impl.metrics.JetMetricsService\nINFO: [10.0.75.1]:5701 [jet] [3.2] Configuring metrics collection, collection interval=5 seconds, retention=5 seconds, publishers=[Management Center Publisher, Job Metrics Publisher, JMX Publish\ner]\nokt. 26, 2019 7:21:42 EM com.hazelcast.jet.impl.JetService\nINFO: [10.0.75.1]:5701 [jet] [3.2] Setting number of cooperative threads and default parallelism to 4\nokt. 26, 2019 7:21:42 EM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl\nINFO: [10.0.75.1]:5701 [jet] [3.2] Starting 4 partition threads and 3 generic threads (1 dedicated for priority tasks)\nokt. 26, 2019 7:21:42 EM com.hazelcast.internal.diagnostics.Diagnostics\nINFO: [10.0.75.1]:5701 [jet] [3.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.\nokt. 26, 2019 7:21:42 EM com.hazelcast.core.LifecycleService\nINFO: [10.0.75.1]:5701 [jet] [3.2] [10.0.75.1]:5701 is STARTING\nokt. 26, 2019 7:21:44 EM com.hazelcast.internal.cluster.ClusterService\nINFO: [10.0.75.1]:5701 [jet] [3.2]\n\nMembers {size:1, ver:1} [\n        Member [10.0.75.1]:5701 - 44439093-250f-4477-8df6-558ee6af7c87 this\n]\n\nokt. 26, 2019 7:21:44 EM com.hazelcast.core.LifecycleService\nINFO: [10.0.75.1]:5701 [jet] [3.2] [10.0.75.1]:5701 is STARTED\nokt. 26, 2019 7:21:44 EM com.hazelcast.internal.partition.impl.PartitionStateManager\nINFO: [10.0.75.1]:5701 [jet] [3.2] Initializing cluster partition table arrangement...\nokt. 26, 2019 7:21:44 EM com.hazelcast.jet.impl.JobCoordinationService\nINFO: [10.0.75.1]:5701 [jet] [3.2] Starting job 0356-8480-eac0-0001 based on submit request\nokt. 26, 2019 7:21:44 EM com.hazelcast.jet.impl.MasterJobContext\nINFO: [10.0.75.1]:5701 [jet] [3.2] Didn't find any snapshot to restore for job '0356-8480-eac0-0001', execution 0356-8480-eac1-0001\nokt. 26, 2019 7:21:44 EM com.hazelcast.jet.impl.MasterJobContext\nINFO: [10.0.75.1]:5701 [jet] [3.2] Start executing job '0356-8480-eac0-0001', execution 0356-8480-eac1-0001, execution graph in DOT format:\ndigraph DAG {\n        \"MessageSource\" [tooltip=\"local-parallelism=1\"];\n        \"TestPluginStartNode\" [tooltip=\"local-parallelism=4\"];\n        \"TestPluginValidate\" [tooltip=\"local-parallelism=4\"];\n        \"TestPluginTransform\" [tooltip=\"local-parallelism=4\"];\n        \"TestPluginJoin\" [tooltip=\"local-parallelism=4\"];\n        \"TestPluginSave\" [tooltip=\"local-parallelism=4\"];\n        \"MessageSource\" -\u003e \"TestPluginStartNode\";\n        \"TestPluginStartNode\" -\u003e \"TestPluginTransform\" [label=\"distributed\", taillabel=0];\n        \"TestPluginStartNode\" -\u003e \"TestPluginValidate\" [label=\"distributed\", taillabel=1];\n        \"TestPluginValidate\" -\u003e \"TestPluginJoin\" [label=\"distributed\", headlabel=1];\n        \"TestPluginTransform\" -\u003e \"TestPluginJoin\" [label=\"distributed\", headlabel=0];\n        \"TestPluginJoin\" -\u003e \"TestPluginSave\" [label=\"distributed\"];\n}\nHINT: You can use graphviz or http://viz-js.com to visualize the printed graph.\nokt. 26, 2019 7:21:45 EM com.hazelcast.jet.impl.JobExecutionService\nINFO: [10.0.75.1]:5701 [jet] [3.2] Execution plan for jobId=0356-8480-eac0-0001, jobName='0356-8480-eac0-0001', executionId=0356-8480-eac1-0001 initialized\nokt. 26, 2019 7:21:45 EM com.hazelcast.jet.impl.JobExecutionService\nINFO: [10.0.75.1]:5701 [jet] [3.2] Start execution of job '0356-8480-eac0-0001', execution 0356-8480-eac1-0001 from coordinator [10.0.75.1]:5701\nokt. 27, 2019 10:57:10 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginStartNode on machine DESKTOP-TDNIONG in jet_instance_52ebc00\nokt. 27, 2019 10:57:10 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginValidate on machine DESKTOP-TDNIONG in jet_instance_d3e0ce6\nokt. 27, 2019 10:57:10 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginTransform on machine DESKTOP-TDNIONG in jet_instance_d3e0ce6\nokt. 27, 2019 10:57:10 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginJoin on machine DESKTOP-TDNIONG in jet_instance_52ebc00\nokt. 27, 2019 10:57:10 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginJoin on machine DESKTOP-TDNIONG in jet_instance_52ebc00\nokt. 27, 2019 10:57:10 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginSave on machine DESKTOP-TDNIONG in jet_instance_d3e0ce6\nokt. 27, 2019 10:57:10 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginSave on machine DESKTOP-TDNIONG in jet_instance_6b0ce21\nokt. 27, 2019 10:57:11 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginStartNode on machine DESKTOP-TDNIONG in jet_instance_6b0ce21\nokt. 27, 2019 10:57:11 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginTransform on machine DESKTOP-TDNIONG in jet_instance_d3e0ce6\nokt. 27, 2019 10:57:11 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginJoin on machine DESKTOP-TDNIONG in jet_instance_d3e0ce6\nokt. 27, 2019 10:57:11 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginValidate on machine DESKTOP-TDNIONG in jet_instance_d3e0ce6\nokt. 27, 2019 10:57:11 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginSave on machine DESKTOP-TDNIONG in jet_instance_52ebc00\nokt. 27, 2019 10:57:11 FM logging\nINFO: [10.0.75.1]:5703 [dev] [3.2] running plugin com.github.codingchili.plugins.TestPluginJoin on machine DESKTOP-TDNIONG in jet_instance_d3e0ce6\nokt. 27, 2019 10:57:11 FM logging\n...\n```\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodingchili%2Fjet-pluginifer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodingchili%2Fjet-pluginifer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodingchili%2Fjet-pluginifer/lists"}