https://github.com/thomasbleijendaal/tplpipeline

Dataflow based batched pipeline
https://github.com/thomasbleijendaal/tplpipeline

Last synced: about 6 hours ago
JSON representation

Dataflow based batched pipeline

• Host: GitHub
• URL: https://github.com/thomasbleijendaal/tplpipeline
• Owner: ThomasBleijendaal
• License: mit
• Created: 2017-03-13T20:14:39.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2017-05-23T20:05:08.000Z (over 7 years ago)
• Last Synced: 2024-09-14T10:52:09.766Z (21 days ago)
• Language: C#
• Size: 50.8 KB
• Stars: 1
• Watchers: 2
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
# TPL Pipeline

A TPL Dataflow based pipeline with job based tracking.

## Example

### Pipeline layout

```

[Start]

  |

  |

[Download]

  |_____________________ 

  |                     |

  | Condition           | Condition

  |                     |

[Merge files]         [Save as image]

  |                     |

  |                     |

[Save file]             |

  |_____________________|

  |

[Finish]

```

### Pipeline class

```

public class Pipeline : BasePipeline

{

  TransformManyBlock> PipelineBegin;

  TransformBlock, IPipelineJobElement> DownloadBlock;

  TransformBlock>, IPipelineJobElement> MergeBlock;

  TransformBlock, IPipelineJobElement> DiskWriteBlock;

  TransformBlock, IPipelineJobElement> ImageBlock;

  ActionBlock>> FinishBlock;

  

  public Pipeline()

  {

  public Pipeline()

  {

    PipelineBegin = StartBlock();

    DownloadBlock = TransformBlock(

      async (job, request) =>

      {

        [..]

      }, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 25 });

    MergeBlock = MergeTransformBlock(

      (job, byteArrays) =>

      {

        [..]

      }, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 5 });

    DiskWriteBlock = TransformBlock(

      async (job, data) =>

      {

        [..]

      }, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 1 });

    ImageBlock = TransformBlock(

      async (job, data) =>

      {

        [..]

      }, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 1 });

    FinishBlock = ActionBlock>(

      (job, flags) =>

      {

        [..]

      }, true);

    PipelineBegin.LinkTo(DownloadBlock);

    DownloadBlock.LinkTo(MergeBlock, e => e.Properties["Type"] == "Website");

    DownloadBlock.LinkTo(ImageBlock, e => e.Properties["Type"] == "Thumbnail");

    MergeBlock.LinkTo(DiskWriteBlock);

      

    FinishBlock.LinkFrom(DiskWriteBlock, ImageBlock);

    

  }

}

```

### Job class

```

public class Job : BaseJob

{

  public Job() : base()

  {

  }

  public override void OnJobStart()

  {

  Console.WriteLine("Job started");

  }

  public override void OnJobComplete()

  {

  Console.WriteLine("Job completed");

  }

}

```