{"id":17992017,"url":"https://github.com/coin8086/cloudworker","last_synced_at":"2026-01-02T16:52:17.999Z","repository":{"id":226159334,"uuid":"751156863","full_name":"coin8086/cloudworker","owner":"coin8086","description":"A queue-based system on cloud for high-throughput and parallel workload","archived":false,"fork":false,"pushed_at":"2024-05-25T14:18:53.000Z","size":382,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-29T20:24:29.217Z","etag":null,"topics":["azure","cloud-computing","dotnet","htc","message-queue","soa"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coin8086.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-01T03:18:49.000Z","updated_at":"2024-06-08T01:41:56.000Z","dependencies_parsed_at":"2024-03-16T10:41:35.988Z","dependency_job_id":"38f9abe0-027b-4953-9840-a0c90459a87b","html_url":"https://github.com/coin8086/cloudworker","commit_stats":null,"previous_names":["coin8086/cloud-native-soa","coin8086/cloudwork","coin8086/cloudworker"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coin8086%2Fcloudworker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coin8086%2Fcloudworker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coin8086%2Fcloudworker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coin8086%2Fcloudworker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coin8086","download_url":"https://codeload.github.com/coin8086/cloudworker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243806031,"owners_count":20350773,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","cloud-computing","dotnet","htc","message-queue","soa"],"created_at":"2024-10-29T19:26:41.977Z","updated_at":"2026-01-02T16:52:17.959Z","avatar_url":"https://github.com/coin8086.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CloudWorker\n\n## Overview\n\nCloudWorker is for a queue based system on cloud for [embarrassingly parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel) workload. CloudWorker provides runtime components for such a system and also tools for building it on cloud. CloudWorker itself is in .NET and the system can run on Linux or Windows.\n\n```mermaid\nflowchart\n    subgraph Cloud\n        cluster((workers))\n        inq(request queue)\n        outq(response queue)\n        storage[(file shares)]\n        cluster --\u003e |get| inq\n        cluster --\u003e |put| outq\n        cluster --\u003e storage\n    end\n\n    client((clients))\n    client --\u003e |put| inq\n    client --\u003e |get| outq\n    client --\u003e storage\n```\n\nThe system is composed of a cluster of workers, a pair of messaging queues and file shares. Clients put request messages into the request queue and get response messages from the response queue, while the workers get requests from the request queue and put responses into the response queue. File shares are optional. They can be used for files of user service, which the workers depend on.\n\nThough for now only Azure is supported, CloudWorker is also designed for on-premises and thus on other cloud. It makes this by abstractions in interface.\n\n### Worker\n\nA worker is a process that consumes requests and produces responses. Its input comes from the request queue and its output goes into the response queue. Each worker works independently of others.\n\nA worker is composed of a service host and a User Defined Service (UDS). The UDS is the real message processor.\n\n```mermaid\nflowchart\n    subgraph worker\n        svchost(service host) --\u003e |load| svc(UDS)\n        svchost --\u003e |call| svc\n    end\n```\n\n### User Defined Service (UDS)\n\nA UDS is a C# class that implements the following interface:\n\n```cs\ninterface IUserService : IAsyncDisposable\n{\n    Task InitializeAsync(CancellationToken cancel);\n\n    Task\u003cstring\u003e InvokeAsync(string input, CancellationToken token)\n}\n```\n\nThe `input` is the content of a message from the request queue and the returned string will be the content of a message into the response queue.\n\nThough a UDS is in C#, it can be implemented to support other programming languages, like Java, Python, etc.\n\nFor example,\n\n```mermaid\nflowchart LR\n    uds(UDS) --\u003e |start| grpc(gRPC server)\n    uds --\u003e |talk to| grpc\n```\n\nHere a UDS starts a gRPC server and then talks to it (such a UDS is an adapter between the service host and the real user service). The gRPC server can be implemented in whatever programming language.\n\n## Build\n\nCloudWorker is in .NET 8. Build it with standard .NET tools.\n\nWith [.NET CLI](https://learn.microsoft.com/en-us/dotnet/core/tools/), build it like\n\n```bash\ndotnet build\n```\n\nfrom the `src` directory.\n\n## Usage\n\nCloudWorker provides runtime components for a queue based system, and also tools for building the whole system on Azure.\n\n### Runtime Components\n\nThe following runtime components are provided\n\n* [`ServiceHost`](src/ServiceHost/)\n* [`ServiceInferface`](src/ServiceInterface/)\n* Official services [`Echo`](src/Services/Echo/), [`CGI`](src/Services/CGI/) and service adapter [`GRpc`](src/Services/GRpc/) (more to be added, like adapter for WCF)\n* [Queue abstractions `IMessageQueue`, `IQueueMessage` and implementations for Azure Service Bus queue and Azure Storage queue](src/MessageQueue/)\n\nWith these runtime components, users can build their own systems on Azure, on premises or in a mixed way. For example, users can make use of Azure Service Bus queues, while running service hosts on premises.\n\nSee the following examples about using the runtime components\n\n* [`SendAndReceive`](src/Samples/QueueClient/SendAndReceive/) A basic example on sending requests to and receiving responses from queues.\n* [`PerfTest`](src/Samples/QueueClient/PerfTest/) An example of sending requests and receiving responses in a high performance way.\n\nBesides, please note that you don't have to use C# for your client and service code\n\n* On client side, you can use whatever programming languages that can read and write the queues. For example, you can use Python SDK for Azure Service Bus queues. Or you can use Azure Service Bus REST API directly, without the SDK.\n* On service side, you can choose  `GRpc` or `CGI` as UDS and then use your favorite programming languages for the real service code.\n\n### Building on Azure\n\nCloudWorker provides tools for building the whole system on Azure, including\n\n* C# [SDK](src/Client/SDK/)\n* Local API Server for multiple programming languages (todo)\n\nExamples\n\n* [`Cluster`](src/Samples/SDK/Cluster/) Create, update or destroy a system on Azure, using `Cluster` class in C# SDK.\n* [`Session`](src/Samples/SDK/Session/) Create, update or destroy a system on Azure, and send and receive messages for Echo/GRpc services, using `Session` class in C# SDK.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoin8086%2Fcloudworker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoin8086%2Fcloudworker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoin8086%2Fcloudworker/lists"}