{"id":22538378,"url":"https://github.com/shuttle/shuttle.core.pipelines","last_synced_at":"2025-10-05T15:45:14.092Z","repository":{"id":54349673,"uuid":"116103011","full_name":"Shuttle/Shuttle.Core.Pipelines","owner":"Shuttle","description":"Observable event-based pipelines based broadly on pipes and filters.","archived":false,"fork":false,"pushed_at":"2025-02-02T11:55:34.000Z","size":188,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-25T13:59:34.588Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Shuttle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-03T06:54:50.000Z","updated_at":"2025-02-02T11:53:27.000Z","dependencies_parsed_at":"2024-01-13T12:40:55.696Z","dependency_job_id":"7b1384f4-43c2-41c7-b076-91ef73dcd73d","html_url":"https://github.com/Shuttle/Shuttle.Core.Pipelines","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/Shuttle/Shuttle.Core.Pipelines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shuttle%2FShuttle.Core.Pipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shuttle%2FShuttle.Core.Pipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shuttle%2FShuttle.Core.Pipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shuttle%2FShuttle.Core.Pipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Shuttle","download_url":"https://codeload.github.com/Shuttle/Shuttle.Core.Pipelines/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shuttle%2FShuttle.Core.Pipelines/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278477817,"owners_count":25993540,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-05T02:00:06.059Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-07T11:11:47.430Z","updated_at":"2025-10-05T15:45:14.064Z","avatar_url":"https://github.com/Shuttle.png","language":"C#","readme":"# Shuttle.Core.Pipelines\n\n```\nPM\u003e Install-Package Shuttle.Core.Pipelines\n```\n\nObservable event-based pipelines based broadly on pipes and filters.\n\n## Configuration\n\nIn order to more easily make use of pipelines an implementation of the `IPipelineFactory` should be used.  The following will register the `PipelineFactory` implementation:\n\n```c#\nservices.AddPipelineProcessing(builder =\u003e {\n    builder.AddAssembly(assembly);\n});\n```\n\nThis will register the `IPipelineFactory` and as `Singleton` and, using the builder, add all `IPipeline` imeplemtation as `Transient` and all `IPipelineObserver` implementations as `Singleton`.  The pipeline instances are re-used as they are kept in a pool.\n\nSince pipelines are quite frequently extended by adding observers, the recommended pattern is to make use of an `IHostedService` implementation that accepts the `IPipelineFactory` dependency:\n\n```c#\npublic class CustomHostedService : IHostedService\n{\n    private readonly Type _pipelineType = typeof(InterestingPipeline);\n\n    public CustomHostedService(IPipelineFactory pipelineFactory)\n    {\n        Guard.AgainstNull(pipelineFactory);\n\n        pipelineFactory.PipelineCreated += PipelineCreated;\n    }\n\n    private void PipelineCreated(object sender, PipelineEventArgs e)\n    {\n        if (e.Pipeline.GetType() != _pipelineType\n        {\n            return;\n        }\n\n        e.Pipeline.AddObserver(new SomeObserver());\n    }\n\n    public async Task StartAsync(CancellationToken cancellationToken)\n    {\n        await Task.CompletedTask;\n    }\n\n    public async Task StopAsync(CancellationToken cancellationToken)\n    {\n        _pipelineFactory.PipelineCreated -= OnPipelineCreated;\n\n        await Task.CompletedTask;\n    }\n}\n```\n\nTypically you would also have a way to register the above `CustomHostedService` with the `IServiceCollection`:\n\n```c#\npublic static class ServiceCollectionExtensions\n{\n    public static IServiceCollection AddCustomPipelineObserver(this IServiceCollection services)\n    {\n        services.AddHostedService\u003cCustomHostedService\u003e();\n\n        return services;\n    }\n}\n```\n\nThe above is a rather naive example but it should give you an idea of how to extend pipelines using the `IPipelineFactory` and `IHostedService` implementations.\n\n## Overview\n\nA `Pipeline` is a variation of the pipes and filters pattern and consists of 1 or more stages that each contain one or more events.  When the pipeline is executed each event in each stage is raised in the order that they were registered.  One or more observers should be registered to handle the relevant event(s).\n\nEach `Pipeline` always has its own state that is simply a name/value pair with some convenience methods to get and set/replace values.  The `State` class will use the full type name of the object as a key should none be specified:\n\n``` c#\nvar state = new State();\nvar list = new List\u003cstring\u003e {\"item-1\"};\n\nstate.Add(list); // key = System.Collections.Generic.List`1[[System.String...]]\nstate.Add(\"my-key\", \"my-key-value\");\n\nConsole.WriteLine(state.Get\u003cList\u003cstring\u003e\u003e()[0]);\nConsole.Write(state.Get\u003cstring\u003e(\"my-key\"));\n```\n\nThe `Pipeline` class has a `AddStage` method that will return a `PipelineStage` instance.  The `PipelineStage` instance has a `WithEvent` method that will return a `PipelineStageEvent` instance.  This allows for a fluent interface to register events for a pipeline:\n\n### IPipelineObserver\n\nThe `IPipelineObserver` interface is used to define the observer that will handle the events:\n\n``` c#\npublic interface IPipelineObserver\u003cin TPipelineEvent\u003e : IPipelineObserver where TPipelineEvent : IPipelineEvent\n{\n    void Execute(TPipelineEvent pipelineEvent);\n    Task ExecuteAsync(TPipelineEvent pipelineEvent);\n}\n```\n\nThe interface has two methods that can be implemented.  The `Execute` method is used for synchronous processing whereas the `ExecuteAsync` method is used for asynchronous processing.\n\n## Example\n\nEvents should derive from `PipelineEvent`.\n\nWe will use the following events:\n\n``` c#\npublic class OnAddCharacterA : PipelineEvent\n{\n}\n\npublic class OnAddCharacter : PipelineEvent\n{\n\tpublic char Character { get; private set; }\n\n\tpublic OnAddCharacter(char character)\n\t{\n\t\tCharacter = character;\n\t}\n}\n```\n\nThe `OnAddCharacterA` event represents a very explicit event whereas with the `OnAddCharacter` event one can specify some data.  In this case the character to add.\n\nIn order for the pipeline to process the events we will have to define one or more observers to handle the events.  We will define only one for this sample but we could very easily add another that will handle one or more of the same, or other, events:\n\n``` c#\n    public class CharacterPipelineObserver : \n        IPipelineObserver\u003cOnAddCharacterA\u003e,\n        IPipelineObserver\u003cOnAddCharacter\u003e\n    {\n        public void Execute(OnAddCharacterA pipelineEvent)\n        {\n            var state = pipelineEvent.Pipeline.State;\n            var value = state.Get\u003cstring\u003e(\"value\");\n\n            value = string.Format(\"{0}-A\", value);\n\n            state.Replace(\"value\", value);\n        }\n\n        public async Task ExecuteAsync(OnAddCharacterA pipelineEvent)\n        {\n\t\t\tExecute(pipelineEvent);\n\n            await Task.CompletedTask;\n        }\n\n        public void Execute(OnAddCharacter pipelineEvent)\n        {\n            var state = pipelineEvent.Pipeline.State;\n            var value = state.Get\u003cstring\u003e(\"value\");\n\n            value = string.Format(\"{0}-{1}\", value, pipelineEvent.Character);\n\n            state.Replace(\"value\", value);\n        }\n\n        public async Task ExecuteAsync(OnAddCharacter pipelineEvent)\n        {\n            Execute(pipelineEvent);\n\n\t\t\tawait Task.CompletedTask;\n        }\n    }\n```\n\nNext we will define the pipeline itself:\n\n``` c#\nvar pipeline = new Pipeline();\n\npipeline.AddStage(\"process\")\n\t.WithEvent\u003cOnAddCharacterA\u003e()\n\t.WithEvent(new OnAddCharacter('Z'));\n\npipeline.AddObserver(new CharacterPipelineObserver());\n\npipeline.State.Add(\"value\", \"start\");\n\nif (sync)\n{\n    pipeline.Execute();\n}\nelse\n{\n\tawait pipeline.ExecuteAsync();\n}\n\nConsole.WriteLine(pipeline.State.Get\u003cstring\u003e(\"value\")); // outputs start-A-Z\n```\n\nWe can now execute this pipeline with predictable results.\n\nPipelines afford us the ability to better decouple functionality.  This means that we could re-use the same observer in multiple pipelines enabling us to compose functionality at a more granular level.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshuttle%2Fshuttle.core.pipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshuttle%2Fshuttle.core.pipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshuttle%2Fshuttle.core.pipelines/lists"}