{"id":15533345,"url":"https://github.com/lobocv/simpleflow","last_synced_at":"2025-04-23T13:44:21.870Z","repository":{"id":39622067,"uuid":"450848543","full_name":"lobocv/simpleflow","owner":"lobocv","description":"Generic simple workflows and concurrency patterns","archived":false,"fork":false,"pushed_at":"2023-05-03T16:22:45.000Z","size":77,"stargazers_count":45,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-18T00:11:34.774Z","etag":null,"topics":["batching","concurrency","counter","deduplication","generics","go","golang","timeseries","worflows","workerpool"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/lobocv/simpleflow","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lobocv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-22T14:58:40.000Z","updated_at":"2025-02-10T09:19:36.000Z","dependencies_parsed_at":"2024-06-19T01:39:40.035Z","dependency_job_id":"59f5b0ee-86c2-4f72-b1fb-687f339af886","html_url":"https://github.com/lobocv/simpleflow","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lobocv%2Fsimpleflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lobocv%2Fsimpleflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lobocv%2Fsimpleflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lobocv%2Fsimpleflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lobocv","download_url":"https://codeload.github.com/lobocv/simpleflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250441705,"owners_count":21431204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batching","concurrency","counter","deduplication","generics","go","golang","timeseries","worflows","workerpool"],"created_at":"2024-10-02T11:35:58.846Z","updated_at":"2025-04-23T13:44:21.853Z","avatar_url":"https://github.com/lobocv.png","language":"Go","readme":"[![Go Reference](https://pkg.go.dev/badge/github.com/lobocv/simplerr.svg)](https://pkg.go.dev/github.com/lobocv/simpleflow)\n[![Github tag](https://badgen.net/github/tag/lobocv/simpleflow)](https://github.com/lobocv/simpleflow/tags)\n![Go version](https://img.shields.io/github/go-mod/go-version/lobocv/simpleflow)\n![Build Status](https://github.com/lobocv/simpleflow/actions/workflows/build.yaml/badge.svg)\n[![GoReportCard](https://goreportcard.com/badge/github.com/lobocv/simpleflow)](https://goreportcard.com/report/github.com/lobocv/simpleflow)\n\u003ca href='https://github.com/jpoles1/gopherbadger' target='_blank'\u003e![gopherbadger-tag-do-not-edit](https://img.shields.io/badge/Go%20Coverage-100%25-brightgreen.svg?longCache=true\u0026style=flat)\u003c/a\u003e\n[\u003cimg src=\"https://img.shields.io/github/license/lobocv/simpleflow\"\u003e](https://img.shields.io/github/license/lobocv/simpleflow)\n\n\n# SimpleFlow\n\nSimpleFlow is a a collection of generic functions and patterns that help building common workflows.\nPlease see the tests for examples on how to use these functions.\n\n## Why should I use Simpleflow?\n\n- A single library for common workflows so you do not have to reinvent the wheel, maintain your own library or\n  copy-paste code.\n- Simple and easy to use API.\n- Detailed documentation and examples. \n- Worker pools are simple, worker pools with error handling are not.\n- 100% test coverage\n\n# Installation\n```\ngo get -u github.com/lobocv/simpleflow\n```\n\n## Table of Contents\n\n1. [Channels](https://github.com/lobocv/simpleflow#channels)\n2. [Work Pools](https://github.com/lobocv/simpleflow#worker-pools)\n   1. [Example](https://github.com/lobocv/simpleflow#workerpoolfromslice-example)\n   2. [Canceling a running worker pool](https://github.com/lobocv/simpleflow#canceling-a-running-worker-pool)\n3. [Fan-Out and Fan-In](https://github.com/lobocv/simpleflow#fan-out-and-fan-in)\n4. [Round Robin](https://github.com/lobocv/simpleflow#round-robin)\n5. [Batching](https://github.com/lobocv/simpleflow#batching)\n6. [Incremental Batching](https://github.com/lobocv/simpleflow#incremental-batching)\n7. [Transforming](https://github.com/lobocv/simpleflow#transforming)\n8. [Filtering](https://github.com/lobocv/simpleflow#filtering)\n9. [Extracting](https://github.com/lobocv/simpleflow#extracting)\n10. [Segmenting](https://github.com/lobocv/simpleflow#segmenting)\n11. [Deduplication](https://github.com/lobocv/simpleflow#deduplication)\n12. [Counter](https://github.com/lobocv/simpleflow#counter)\n13. [Time](https://github.com/lobocv/simpleflow#time)\n14. [Time Series](https://github.com/lobocv/simpleflow#timeseries)\n\n## Channels\n\nSome common but tedious operations on channels are done by the channel functions:\n\nExample:\n\n```go\nitems := make(chan int, 3)\n// push 1, 2, 3 onto the channel\nLoadChannel(items, 1, 2, 3)\n// Close the channel so ChannelToSlice doesn't block.\nclose(items) \nout := ChannelToSlice(items)\n// out == []int{1, 2, 3}\n```\n\n## Worker Pools\n\nWorker pools provide a way to spin up a finite set of go routines to process items in a collection.\n\n- `WorkerPoolFromSlice` - Starts a fixed pool of workers that process elements in the `slice`\n- `WorkerPoolFromMap` - Starts a fixed pool of workers that process key-value pairs in the `map`\n- `WorkerPoolFromChan` - Starts a fixed pool of workers that process values read from a `channel`\n\nThese functions block until all workers finish processing.\n\n### WorkerPoolFromSlice example\n\n```go\nctx := context.Background()\nitems := []int{0, 1, 2, 3, 4, 5}\nout := NewSyncMap(map[int]int{})\nnWorkers := 2\nf := func(_ context.Context, v int) error {\n    out.Set(v, v*v)\n    return nil\n}\nerrors := WorkerPoolFromSlice(ctx, items, nWorkers, f)\n// errors == []error{}\n// out == map[int]int{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}\n```\n\n### Canceling a running worker pool \n\n```go\n// Create a cancel-able context\nctx, cancel := context.WithCancel(context.Background())\n\nitems := []int{0, 1, 2, 3, 4, 5}\nout := NewSyncMap(map[int]int{}) // threadsafe map used in tests\nnWorkers := 2\n\nf := func(_ context.Context, v int) error {\n    // Cancel as soon as we hit v \u003e 2\n    if v \u003e 2 {\n        cancel()\n        return nil\n    }\n    out.Set(v, v*v)\n    return nil\n}\nWorkerPoolFromSlice(ctx, items, nWorkers, f)\n// errors == []error{}\n// out == map[int]int{0: 0, 1: 1, 2: 4}\n```\n\n## Fan-Out and Fan-In\n\n`FanOut` and `FanIn` provide means of fanning-in and fanning-out channel to other channels. \n\nExample:\n\n```go\n// Generate some data on a channel (source for fan out)\nN := 3\nsource := make(chan int, N)\ndata := []int{1, 2, 3}\nfor _, v := range data {\n    source \u003c- v\n}\nclose(source)\n\n// Fan out to two channels. Each will get a copy of the data\nfanoutSink1 := make(chan int, N)\nfanoutSink2 := make(chan int, N)\nFanOutAndClose(source, fanoutSink1, fanoutSink2)\n\n// Fan them back in to a single channel. We should get the original source data with two copies of each item\nfanInSink := make(chan int, 2*N)\nFanInAndClose(fanInSink, fanoutSink1, fanoutSink2)\nfanInResults := ChannelToSlice(fanInSink)\n// fanInResults == []int{1, 2, 3, 1, 2, 3}\n```\n\n## Round Robin\n\n`RoundRobin` distributes values from a channel over other channels in a round-robin fashion\n\nExample:\n\n```go\n// Generate some data on a channel\nN := 5\nsource := make(chan int, N)\ndata := []int{1, 2, 3, 4, 5}\nfor _, v := range data {\n    source \u003c- v\n}\nclose(source)\n\n// Round robin the data into two channels, each should have half the data\nsink1 := make(chan int, N)\nsink2 := make(chan int, N)\nRoundRobin(source, fanoutSink1, sink2)\nCloseManyWriters(fanoutSink1, sink2)\n\nsink1Data := ChannelToSlice(sink1)\n// sink1Data == []int{1, 3, 5}\nsink2Data := ChannelToSlice(sink2)\n// sink2Data == []int{2, 4}\n```\n\n## Batching\n\n`BatchMap`, `BatchSlice` and `BatchChan` provide ways to break `maps`, `slices` and `channels` into smaller\ncomponents of at most `N` size.\n\nExample:\n\n```go\nitems := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}\nbatches := BatchSlice(items, 2)\n// batches == [][]int{{0, 1}, {2, 3}, {4, 5}, {6, 7}, {8, 9}}\n```\n\n```go\nitems := map[int]int{0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5}\nbatches := BatchMap(items, 3)\n// batches == []map[int]int{ {0: 0, 3: 3, 4: 4}, {1: 1, 2: 2, 5: 5} }\n\n```\n\n## Incremental Batching\n\nBatching can also be done incrementally by using `IncrementalBatchSlice` and `IncrementalBatchMap` functions.\nThese functions are meant to be called repeatedly, adding elements until a full batch can be processed, at which time,\nthe batch is returned.\n\nExample:\n\n```go\nbatchSize := 3\nvar items, batch []int\nitems, batch = IncrementalBatchSlice(items, batchSize, 1)\n// items == []int{1}, batch == nil\n\nitems, batch = IncrementalBatchSlice(items, batchSize, 2)\n// items == []int{1, 2}, batch == nil\n\nitems, batch = IncrementalBatchSlice(items, batchSize, 3)\n// Batch size reached\n// items == []int{}, batch == []int{1, 2, 3}\n\nitems, batch = IncrementalBatchSlice(items, batchSize, 4)\n// items == []int{4}, batch == nil\n```\n\n## Transforming\n\nTransformation operations (often named `map()` in other languages) allow you to transform each element of a slice to\nanother value using a given transformation function.  \n\n```go\nout := Transform([]int{1, 2, 3}, func(t int) string {\n    return strconv.Itoa(t)\n})\n// out == []string{\"1\", \"2\", \"3\"}\n\nout = TransformAndFilter([]int{1, 2, 3, 4, 5}, func(t int) (int, bool) {\n    return 2 * t,  t%2 == 0\n})\n// out == []string{4, 8}\n```\n\n## Filtering\n\nFiltering operations allows you to remove elements from slices or maps. Filtering can done either in-place with\n`FilterSliceInplace`, `FilterMapInPlace` or by creating a copy `FilterSlice`, `FilterMap`. The filtering function\nargument must return `true` if you want to keep the element.\n\nThe following example filters the positive numbers from the slice of integers:\n\n```go\ngetPositive := func(t int) bool {\n    return t \u003e 0\n}\n\nout := FilterSlice([]int{5, -2, 3, 1, 0, -3, -5, -6}, getPositive)\n// out == []int{5, 3, 1}\n```\n\n## Extracting\n\nExtraction operations allow you to extract one or more elements from a slice. Extraction functions accept a function which takes in each element of the slice and returns the\nelement to extract (potentially different than the input type) and a boolean for whether the element should be extracted.\n\nThe following example extracts the `Name` field from a slice of `Object`.\n\n```go\ntype Object struct {\n    Name string\n}\n\nin := []Object{\n    {Name: \"John\"},\n    {Name: \"Paul\"},\n    {Name: \"George\"},\n    {Name: \"Ringo\"},\n    {Name: \"Bob\"},\n}\nvar names []string\n\nfn := func(t Object) (string, bool) {\n    if t.Name == \"Bob\" {\n        return \"\", false\n    }\n    return t.Name, true\n}\n\nnames = ExtractToSlice(in, fn, names)\n// names == []string{\"John\", \"Paul\", \"George\", \"Ringo\"}\n```\n\n`ExtractFirst()` can be used to extract the first element in the slice. The following example extracts the first value\nlarger than 4.\n\n```go\nvalues := []int{4, 1, 5, 7}\nfn := func(v int) bool {\n    return v \u003e 4\n}\n\nv, found := ExtractFirst(values, fn)\n// v == 5, found == true\n```\n\n## Segmenting\n\n`SegmentSlice`, `SegmentMap` and `SegmentChan` allow you to split a `slice` or `map` into sub-slices or maps based on the provided\nsegmentation function:\n\n### Segmenting a slice into even and odd values\n```go\nitems := []int{0, 1, 2, 3, 4, 5}\n\nsegments := SegmentSlice(items, func(v int) int {\n    if v % 2 == 0 {\n        return \"even\"\n\t}\n        return \"odd\"\n})\n// segments == map[string][]int{\"even\": {0, 2, 4}, \"odd\": {1, 3, 5}}\n```\n\n## Deduplication\nA series of values can be deduplicated using the `Deduplicator{}`. It can either accept the entire slice:\n\n```go\ndeduped := Deduplicate([]int{1, 1, 2, 2, 3, 3})\n// deduped == []int{1, 2, 3}\n```\nor iteratively deduplicate for situations where you want fine control with a `for` loop.\n```go\ndd := NewDeduplicator[int]()\nvalues := []int{1, 1, 2, 3, 3}\ndeduped := []int{}\n\nfor _, v := range values {\n    seen := dd.Seen(v) \n    // seen == true for index 1 and 4\n    isNew := dd.Add(v) \n    // isNew == true for index 0, 2 and 3\n    if isNew {\n        deduped = append(deduped, v)\t\n    }\n}\n```\n\nComplex objects can also be deduplicated using the `ObjectDeduplicator{}`, which requires providing a function that\ncreates unique IDs for the provided objects being deduplicated. This is useful for situations where the values being \ndeduplicated are not comparable (ie, have a slice field) or if you want more fine control over just what constitutes a \nduplicate.\n\n```go\n// Object is a complex structure that cannot be used with a regular Deduplicator as it contains \n// a slice field, and thus is not `comparable`.\ntype Object struct {\n    slice   []int\n    pointer *int\n    value   string\n}\n\n// Create a deduplicator that deduplicates Object's by their \"value\" field.\ndd := NewObjectDeduplicator[Object](func(v Object) string {\n        return v.value\n    })\n```\n\n\n## Counter\nThe `Counter{}` and `ObjectCounter{}` can be used to count the number of occurrences\nof values. Much like the `Deduplicator{}`, the `Counter{}` works well for simple types.\n\n```go\ncounter := NewCounter[int]()\nvalues := []int{1, 1, 2, 3, 3, 3, 3}\n\n// Add the values to the counter, values can also be added individually with counter.Add()\ncurrentCount := counter.AddMany(values) \n\nnumberOfOnes := counter.Count(1) // returns 2\nnumberOfTwos := counter.Count(2) // returns 1\nnumberOfThrees := counter.Count(3) // returns 4\n```\n\nComplex objects can also be counted using the `ObjectCounter{}`, which requires providing a function that\ncreates buckets for the provided objects being deduplicated. This is useful for situations where the values being\ncounted are not comparable (ie, have a slice field) or if you want more fine control over the bucketing logic (ie\nbucket objects by a certain field value).\n\n```go\n// Object is a complex structure that cannot be used with a regular Counter as it contains \n// a slice field, and thus is not `comparable`.\ntype Object struct {\n    slice   []int\n    pointer *int\n    value   string\n}\n\n// Create a counter that counts Object's bucketed by their \"value\" field.\ncounter := NewObjectCounter[Object](func(v Object) string {\n        return v.value\n    })\n```\n\n\n## Time\n\nThe `simeplflow/time` package provides functions that assist with working with the standard library `time` package\nand `time.Time` objects. The package contains functions to define, compare and iterate time ranges.\n\n## Timeseries\n\nThe `simpleflow/timeseries` packages contains a generic `TimeSeries` object that allows you\nto manipulate timestamped data. `TimeSeries` store unordered time series data in an underlying \n`map[time.Time]V`. Each `TimeSeries` is configured with a `TimeTransformation` which applies to each\n`time.Time` key when accessed. This makes storing time series data with a particular time granularity\neasy. For example, with a `TimeTransformation` that truncates to the day, any \n`time.Time` object in the given day will access the same key.\n\nExample:\n\n```go\n// TF is a TimeTransformation that truncates the time to the start of the day\nfunc TF(t time.Time) time.Time {\n    return t.UTC().Truncate(24 * time.Hour)\n}\n\n// Day is a function to create a Time object on a given day offset from Jan 1st 2022 by the `i`th day\nfunc Day(i int) time.Time {\n    return time.Date(2022, 01, i, 0, 0, 0, 0, time.UTC)\n}\n\nfunc main() {\n    data := map[time.Time]int{\n            Day(0): 0, // Jan 1st 2022\n            Day(1): 1, // Jan 2nd 2022\n            Day(2): 2, // Jan 3rd 2022\n        }\n    ts := timeseries.NewTimeSeries(data, TF)\n\t\n\t// Get the value on Jan 2th at 4am and at 5 am\n\t// The values for `a` and `b` are both == 1 because the hour is irrelevant\n\t// when accessing data using the TF() time transform\n\ta := ts.Get(time.Date(2022, 01, 2, 4, 0, 0, 0, time.UTC))\n\tb := ts.Get(time.Date(2022, 01, 2, 5, 0, 0, 0, time.UTC))\n\t// a == b == 1 \n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flobocv%2Fsimpleflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flobocv%2Fsimpleflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flobocv%2Fsimpleflow/lists"}