{"id":20694799,"url":"https://github.com/davidgeorgewilliams/workflows","last_synced_at":"2025-03-11T02:28:16.246Z","repository":{"id":223719768,"uuid":"761331777","full_name":"davidgeorgewilliams/Workflows","owner":"davidgeorgewilliams","description":"Workflows is an advanced Java library that simplifies the creation and management of parallel workflows, offering robust concurrency controls and high-performance execution for complex applications.","archived":false,"fork":false,"pushed_at":"2024-03-04T10:17:52.000Z","size":2561,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-17T18:02:08.364Z","etag":null,"topics":["concurrency","concurrent-programming","domain-specific-language","executor","java","multithreading","threading","threads","threadsafe"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidgeorgewilliams.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-21T17:10:09.000Z","updated_at":"2024-02-25T18:06:30.000Z","dependencies_parsed_at":"2024-02-21T18:45:55.279Z","dependency_job_id":"329becb6-9fb3-41d3-a9c1-927085fafa51","html_url":"https://github.com/davidgeorgewilliams/Workflows","commit_stats":null,"previous_names":["davidgeorgewilliams/workflow"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidgeorgewilliams%2FWorkflows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidgeorgewilliams%2FWorkflows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidgeorgewilliams%2FWorkflows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidgeorgewilliams%2FWorkflows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidgeorgewilliams","download_url":"https://codeload.github.com/davidgeorgewilliams/Workflows/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242958499,"owners_count":20212812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","concurrent-programming","domain-specific-language","executor","java","multithreading","threading","threads","threadsafe"],"created_at":"2024-11-17T00:06:44.111Z","updated_at":"2025-03-11T02:28:16.216Z","avatar_url":"https://github.com/davidgeorgewilliams.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Workflows\n\nWorkflows is an advanced Java library that furnishes a high-level abstraction layer for parallel workflow management and\nexecution engine construction. It empowers developers to articulate and execute complex, dependent tasks in a concurrent\nenvironment with ease and precision. Built on the foundation of modern concurrent programming paradigms, Workflows\nstreamlines the process of defining and managing task relationships and execution order through a domain-specific\nlanguage crafted for simplicity and power.\n\n![Workflows Visual Representation](docs/Workflow.png)\n\n## Introduction\n\nAt the heart of Workflows lies a suite of elegantly designed constructs: **Workflow** orchestrates sequences of\n**WorkerPool** objects, enabling cascading task execution; **WorkerPool** serves as an innovative abstraction over\nthread\npools, utilizing 'after' semantics for defining task dependencies and ensuring logical execution order; **Worker**\nencapsulates tasks with execution metadata, leveraging the `Supplier` interface for clear task definitions and detailed\nexecution tracking; and **ThreadPool** streamlines asynchronous task management, providing a simplified interface for\nJava's `ExecutorService` and `CompletableFuture` APIs.\n\nWorkflows is designed with performance, scalability, and ease of use in mind, providing a robust toolkit for developers\nbuilding complex applications requiring sophisticated concurrency controls. Whether you're developing high-throughput\ndata processing systems, real-time computation engines, or complex application logic, Workflows offers the constructs\nand control you need to build reliable and efficient concurrent applications.\n\nThis library is the culmination of dedicated research and innovation in the field of concurrent programming. It stands\nas a testament to the collaborative spirit of the development community and is dedicated to visionaries who push the\nboundaries of technology.\n\n## Getting Started Guide\n\nThis guide will help you get started with the `Workflows` library, a sophisticated domain-specific language for\ncreating, managing, and executing parallel workflows in Java applications.\n\n### Maven Coordinate\n\nTo integrate `Workflows` into your Maven project, you'll first need to clone the repository and build the artifact\nlocally. Execute the following commands in your terminal:\n\n```bash\ngit clone https://github.com/davidgeorgewilliams/Workflows.git\ncd Workflows\nmvn clean install\n```\n\nAfter successfully building the project, incorporate `Workflows` into your application by adding the following\ndependency block to your project's `pom.xml` file:\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.davidgeorgewilliams\u003c/groupId\u003e\n    \u003cartifactId\u003eWorkflows\u003c/artifactId\u003e\n    \u003cversion\u003e1.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nThis ensures that the locally built version of `Workflows` is recognized and used by your Maven project.\n\n### Recommended JDK\n\nFor optimal performance and compatibility, it is recommended to\nuse [Azul Zulu 17](https://www.azul.com/downloads/?version=java-17-lts\u0026package=jdk#zulu), provided\nby [Azul Systems Inc.](https://www.azul.com/)\n\n### Self-Contained Example\n\nThe following example demonstrates how to use Workflows to process a set of tasks in parallel, leveraging the power of\nmulti-threading to enhance performance.\n\nImagine you have a list of 100 mathematical operations that you need to perform - specifically, calculating the natural\nlogarithm of numbers 1 through 100. Using Workflows, you can efficiently distribute these calculations across multiple\nthreads, corresponding to the number of available processors on your system.\n\nHere's a simple yet powerful example:\n\n```java\n@Slf4j\npublic class WorkflowExample implements Callable\u003cInteger\u003e {\n    \n    @Override\n    public Integer call() {\n        final Set\u003cWorker\u003c?\u003e\u003e workers = new HashSet\u003c\u003e();\n        for (int i = 0; i \u003c 100; i++) {\n            final double value = 1.0 * (i + 1);\n            final Worker\u003cDouble\u003e worker = Worker.of(() -\u003e Math.log(value));\n            workers.add(worker);\n        }\n        final int processors = Runtime.getRuntime().availableProcessors();\n        final ThreadPool threadPool = ThreadPool.of(processors);\n        final WorkerPool workerPool = WorkerPool.of(workers, threadPool);\n        Workflow.of(workerPool).process();\n        for (final Worker\u003c?\u003e worker : workers) {\n            final double logValue = (double) worker.result();\n            final double value = Math.exp(logValue);\n            final ThreadLocalTime completed = worker.completed();\n            log.info(String.format(\"completed=%s value=%s logValue=%s\", completed, value, logValue));\n        }\n        return 0;\n    }\n}\n```\n\nIn this code snippet:\n\nWe create a Worker for each operation, which contains the task to be executed - in this case, calculating the logarithm.\nThese `Worker` instances are added to a `WorkerPool`, signifying a group of tasks to be run together.\nA `ThreadPool` manages the execution of these tasks across multiple threads.\nOnce the `WorkerPool` is instructed to process the tasks, each Worker executes its operation in parallel, utilizing the\nfull capabilities of the system's CPU.\nAfter processing, we log the results of each task, including the completion time and the values calculated.\n\nThis example not only showcases the simplicity and elegance with which Workflows handles concurrent operations but also\nhighlights the ease with which computationally intensive tasks can be parallelized, resulting in significant performance\ngains. It's an ideal framework for businesses and engineers looking to optimize their processing capabilities, reduce\nlatency, and achieve superior throughput in their applications.\n\n### After Semantics\n\nThe **Workflows** library significantly streamlines the management of logical execution sequences with an elegantly simple concept: specifying that a worker should run `after` another. This approach is exemplified in the scenario where `workerB` and `workerC` are configured to execute following `workerA`, and `workerD` is designated to run subsequent to both `workerB` and `workerC`.\n\nConsider the following code snippet for a clearer understanding:\n\n```java\n@Test\npublic void testAfterSemantics() {\n    final String messageTemplate = \"Hello from worker%s\";\n    final Worker\u003cString\u003e workerA = Worker.of(() -\u003e String.format(messageTemplate, \"A\"));\n    final Worker\u003cString\u003e workerB = Worker.of(() -\u003e String.format(messageTemplate, \"B\"));\n    final Worker\u003cString\u003e workerC = Worker.of(() -\u003e String.format(messageTemplate, \"C\"));\n    final Worker\u003cString\u003e workerD = Worker.of(() -\u003e String.format(messageTemplate, \"D\"));\n    workerB.after(workerA);\n    workerC.after(workerA);\n    workerD.after(Set.of(workerB, workerC));\n    final WorkerPool workerPool = WorkerPool.of(Set.of(workerA, workerB, workerC, workerD));\n    final Workflow workflow = Workflow.of(workerPool);\n    workflow.process();\n    assertEquals(workerB.completed().compareTo(workerA.completed()), 1);\n    assertEquals(workerC.completed().compareTo(workerA.completed()), 1);\n    assertEquals(workerD.completed().compareTo(workerB.completed()), 1);\n    assertEquals(workerD.completed().compareTo(workerC.completed()), 1);\n}\n```\n\nThis setup illustrates a \"diamond\" pattern of execution over time: `workerA` initiates the process, followed by a concurrent execution of `workerB` and `workerC`, and culminating with `workerD` merging the paths forged by `workerB` and `workerC`. The assertions included in the code validate that each worker completes its execution in the intended order, ensuring a coherent and predictable workflow.\n\n## Class Overview and Usage\n\nThe `Workflows` library exposes a high-level API for constructing and executing workflows composed of dependent tasks,\nenabling efficient parallel processing. The API is centered around four main\nconstructs: `Workflow`, `WorkerPool`, `Worker`, and `ThreadPool`.\n\n### Workflow\n\nThe `Workflow` class in Workflows library offers a streamlined approach to orchestrating complex task sequences\nwithin `WorkerPool` groups, ensuring orderly and dependent execution with dynamic, runtime-adjustable workflows. By\nfacilitating sequential and parallel processing through a unique identifier for each workflow, it simplifies the\nconstruction and management of data processing pipelines and other sophisticated task sequences. This makes `Workflow`\nan invaluable tool for developers requiring a high degree of control and efficiency in concurrent programming, perfectly\nsuited for applications demanding structured task execution and scalability.\n\n### WorkerPool\n\nThe `WorkerPool` class encapsulates a sophisticated thread pool mechanism for executing `Worker` tasks in parallel,\nenriched with 'after' semantics for defining task interdependencies and execution order. By allowing specification of\nprerequisites for each task, it ensures a logical and efficient progression of work. Constructed dynamically with a\nunique identifier and an optional linkage to subsequent `WorkerPool`s via the `next` supplier, `WorkerPool` facilitates\ncomplex workflows with ease. Its processing logic intelligently manages task readiness and execution mode, leveraging a\ncombination of immediate execution and `CompletableFuture` for asynchronous tasks, thereby achieving optimal utilization\nof computing resources in concurrent applications.\n\n### Worker\n\nThe `Worker` class is a key component of the Workflows library, designed to encapsulate individual units of work\nalongside their execution metadata, such as start, completion times, and outcomes. It leverages Java's `Supplier`\ninterface to define tasks that can be executed concurrently, capturing the results and handling any exceptions that\narise. Through attributes like `parallel` and dependency management via the `after` set, `Worker` allows for\nfine-grained control over task execution order and concurrency. Each `Worker` is uniquely identified and can track its\nlifecycle events, from submission to completion, facilitating detailed monitoring and analysis of concurrent operations.\nThe class provides methods for task readiness checks and processing, ensuring that dependencies are resolved before\nexecution and exceptions are appropriately managed, making `Worker` an essential tool for implementing complex,\nreliable, and efficient concurrent workflows.\n\n### ThreadPool\n\nThe `ThreadPool` class serves as an intuitive interface to Java's concurrent execution frameworks,\nspecifically `ExecutorService` and `CompletableFuture`. It abstracts the complexity of managing threads and executing\nasynchronous tasks, allowing developers to focus on the logic of their applications rather than the intricacies of\nthread management. With the capability to create a pool of threads specified by the number of threads, `ThreadPool`\nfacilitates efficient task execution by distributing tasks across the available threads. This class supports the\nsubmission of both `Runnable` and `Supplier\u003cT\u003e` tasks, returning a `Future\u003c?\u003e` or `CompletableFuture\u003cT\u003e` respectively,\nthus enabling flexible execution strategies including synchronous waits or non-blocking callbacks.\nEssentially, `ThreadPool` enhances the scalability and responsiveness of applications by optimizing resource utilization\nand parallelizing task execution.\n\n## About the Author\n\n**Name**: David George Williams  \n**Location**: United Kingdom  \n**Email**: david@davidgeorgewilliams.com  \n**Website**: [davidgeorgewilliams.com](https://davidgeorgewilliams.com)  \n**LinkedIn**: [linkedin.com/in/david-george-williams-1025804b](https://linkedin.com/in/david-george-williams-1025804b)  \n**Upwork**: [upwork.com/freelancers/~01f6c903d62beb9032](https://upwork.com/freelancers/~01f6c903d62beb9032)  \n**Fiverr**: [fiverr.com/davidgw_uk/advance-ethical-ai](https://fiverr.com/davidgw_uk/advance-ethical-ai)\n\nI bring a unique blend of analytical prowess and human-centered insight to the field of AI and computer science. My\nacademic background includes graduating from the MIT Center for Transportation \u0026 Logistics, where I specialized in the\nintricate dynamics of international vaccine supply chains, influencing strategies adopted by organizations such as the\nUS CDC for vaccine distribution. With a solid foundation in pure mathematics from the University of California, Santa\nCruz, I exhibit a commitment to rigorous analytical thinking.\n\nMy professional journey extends beyond academia, with impactful contributions made at renowned companies like Apple and\nPayPal, where I have demonstrated both my ability to collaborate effectively within teams and my capacity for\ninnovation. Beyond the realm of algorithms and numbers, I am driven by a core ethos of utilizing technology to effect\nmeaningful and enduring change in the world.\n\n## Special Thanks\n\nI extend my sincere gratitude to Oliver Saleh, Deepak Kapur, and the Applied Machine Learning team at Apple Inc. for\ntheir invaluable support and encouragement throughout the research and development of this groundbreaking paradigm in\nconcurrent programming.\n\nOliver and Deepak's dedication, expertise, and creative contributions were instrumental in transforming the core concept\nof a Workflow into a tangible reality. Their passion, innovation, and tireless efforts have laid the groundwork for the\nadvancement of high-performance, reliable, and efficient systems.\n\n### Oliver Saleh\n\nOliver Saleh, an Expert Associate Partner in Data Science and Machine Learning, has a remarkable track record of\ndelivering innovative solutions at internet scale. With over 15 years of experience in the industry, Oliver has held\npivotal roles at leading companies in the AI domain, including Microsoft and Apple. His expertise spans various domains\nsuch as generative models, fraud detection, natural language processing, computer vision, and distributed systems. At\nApple, Oliver's focus on Applied Machine Learning \u0026 Strategic Health Initiatives significantly contributed to the\ncompany's advancements. [Oliver's LinkedIn Profile](https://www.linkedin.com/in/oliver-s-95103610/)\n\n### Deepak Kapur\n\nDeepak Kapur, the Co-founder and CEO of Qualification AI, brings extensive experience from his tenure at Google and\nApple. Deepak's innovative work at Qualification AI has revolutionized sales processes by providing AI-driven sales\ncoaching and personalized roleplays. During his time at Apple, Deepak led a team of talented engineers responsible for\ndeveloping Apple’s 24/7 hardware diagnostics platform, which plays a crucial role in preventing warranty fraud. His\ntechnical expertise and leadership have been pivotal in driving platform changes and ensuring the efficacy of diagnostic\nsystems across Apple's global retail network. [Deepak's LinkedIn Profile](https://www.linkedin.com/in/deepakkapur/)\n\nTheir contributions have not only enhanced the development of this project but have also inspired a new wave of\ninnovation in the field of concurrent programming.\n\n## Dedication\n\nThis library is dedicated to Steve Jobs and is officially published on his birthday as a tribute to his enduring legacy\nin technology and innovation.\n\nSteve Jobs, the co-founder of Apple Computer, Inc. (now Apple Inc.), was a charismatic pioneer of the personal computer\nera. Raised by adoptive parents in Cupertino, California, located in what is now known as Silicon Valley, Jobs' journey\nencompassed various interests from engineering to spirituality. He dropped out of Reed College, worked at Atari\nCorporation as a video game designer, and embarked on a pilgrimage to India before returning to Silicon Valley in 1974.\nThere, he reconnected with Stephen Wozniak, and together they founded Apple Computer in 1976, introducing the Apple I\nand later the highly successful Apple II.\n\nJobs's visionary leadership led Apple to become synonymous with innovation, evident in groundbreaking products like the\nMacintosh, which featured a graphical user interface, and later the iMac, iPod, iPhone, and iPad. Despite setbacks,\nincluding his temporary departure from Apple and subsequent founding of NeXT Inc. and Pixar, Jobs' return to Apple in\n1997 marked a renaissance for the company.\n\nHis relentless pursuit of excellence and belief in the power of innovation continues to inspire entrepreneurs and\nbusiness leaders worldwide. In his honor, here are 10 quotes from Steve Jobs that resonate with the entrepreneurial\nspirit:\n\n1. **\"Innovation distinguishes between a leader and a follower.\"**\n2. **\"Your time is limited, so don’t waste it living someone else’s life.\"**\n3. **\"Don’t let the noise of others’ opinions drown out your own inner voice.\"**\n4. **\"You can’t connect the dots looking forward; you can only connect them looking backwards. So you have to trust that\n   the dots will somehow connect in your future.\"**\n5. **\"Be a yardstick of quality. Some people aren’t used to an environment where excellence is expected.\"**\n6. **\"Stay hungry. Stay foolish.\"**\n7. **\"I'm convinced that about half of what separates the successful entrepreneurs from the nonsuccessful ones is pure\n   perseverance.\"**\n8. **\"You can't just ask customers what they want and then try to give that to them. By the time you get it built,\n   they'll want something new.\"**\n9. **\"People think focus means saying yes to the thing you've got to focus on. It means saying no to the hundred other\n   good ideas that there are. You have to pick carefully.\"**\n10. **\"We’re here to put a dent in the universe. Otherwise why else even be here?\"**\n\nSteve Jobs' indelible mark on the world serves as a reminder of the transformative power of vision, perseverance, and\ninnovation.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidgeorgewilliams%2Fworkflows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidgeorgewilliams%2Fworkflows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidgeorgewilliams%2Fworkflows/lists"}