{"id":13814002,"url":"https://github.com/mailgun/holster","last_synced_at":"2025-05-15T15:07:02.439Z","repository":{"id":37492931,"uuid":"82601233","full_name":"mailgun/holster","owner":"mailgun","description":"A place to keep useful golang functions and small libraries","archived":false,"fork":false,"pushed_at":"2025-05-12T12:06:03.000Z","size":25742,"stargazers_count":293,"open_issues_count":11,"forks_count":34,"subscribers_count":57,"default_branch":"master","last_synced_at":"2025-05-12T13:24:51.669Z","etag":null,"topics":["cache","cassandra","election","encryptor","fanout","golang","library","lru-cache","queue","utilities","waitgroup"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mailgun.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-02-20T20:49:44.000Z","updated_at":"2025-04-30T09:35:25.000Z","dependencies_parsed_at":"2024-05-02T13:34:49.606Z","dependency_job_id":"e40b7ad1-ac7a-4938-bf7e-6a10116a3b74","html_url":"https://github.com/mailgun/holster","commit_stats":{"total_commits":268,"total_committers":27,"mean_commits":9.925925925925926,"dds":0.6156716417910448,"last_synced_commit":"4c3ed06a0558df3961aed821c446c37f407fdc7c"},"previous_names":[],"tags_count":133,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mailgun%2Fholster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mailgun%2Fholster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mailgun%2Fholster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mailgun%2Fholster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mailgun","download_url":"https://codeload.github.com/mailgun/holster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254364270,"owners_count":22058878,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","cassandra","election","encryptor","fanout","golang","library","lru-cache","queue","utilities","waitgroup"],"created_at":"2024-08-04T04:01:39.554Z","updated_at":"2025-05-15T15:07:02.405Z","avatar_url":"https://github.com/mailgun.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Holster\nA place to holster mailgun's golang libraries and tools\n\n## Installation\n\nTo use, run the following command: \n```bash\ngo get github.com/mailgun/holster/v4\n```\n\n## Clock\nA drop in (almost) replacement for the system `time` package to make scheduled\nevents deterministic in tests. See the [clock readme](https://github.com/mailgun/holster/blob/master/clock/README.md) for details\n\n## HttpSign\nHttpSign is a library for signing and authenticating HTTP requests between web services.\nSee the [httpsign readme](https://github.com/mailgun/holster/blob/master/httpsign/README.md) for details\n\n## Distributed Election\nA distributed election implementation using etcd to coordinate elections\nSee the [etcd v3 readme](https://github.com/mailgun/holster/blob/master/etcdutil/README.md) for details\n\n## Errors\nErrors is a fork of [https://github.com/pkg/errors](https://github.com/pkg/errors) with additional\n functions for improving the relationship between structured logging and error handling in go\nSee the [errors readme](https://github.com/mailgun/holster/blob/master/errors/README.md) for details\n\n## WaitGroup\nWaitgroup is a simplification of `sync.Waitgroup` with item and error collection included.\n\nRunning many short term routines over a collection with `.Run()`\n```go\nimport \"github.com/mailgun/holster/v4/syncutil\"\nvar wg syncutil.WaitGroup\nfor _, item := range items {\n    wg.Run(func(item interface{}) error {\n        // Do some long running thing with the item\n        fmt.Printf(\"Item: %+v\\n\", item.(MyItem))\n        return nil\n    }, item)\n}\nerrs := wg.Wait()\nif errs != nil {\n    fmt.Printf(\"Errs: %+v\\n\", errs)\n}\n```\n\nClean up long running routines easily with `.Loop()`\n```go\nimport \"github.com/mailgun/holster/v4/syncutil\"\npipe := make(chan int32, 0)\nvar wg syncutil.WaitGroup\nvar count int32\n\nwg.Loop(func() bool {\n    select {\n    case inc, ok := \u003c-pipe:\n        // If the pipe was closed, return false\n        if !ok {\n            return false\n        }\n        atomic.AddInt32(\u0026count, inc)\n    }\n    return true\n})\n\n// Feed the loop some numbers and close the pipe\npipe \u003c- 1\npipe \u003c- 5\npipe \u003c- 10\nclose(pipe)\n\n// Wait for the loop to exit\nwg.Wait()\n```\n\nLoop `.Until()` `.Stop()` is called\n```go\nimport \"github.com/mailgun/holster/v4/syncutil\"\nvar wg syncutil.WaitGroup\n\nwg.Until(func(done chan struct{}) bool {\n    select {\n    case \u003c- time.Tick(time.Second):\n        // Do some periodic thing\n    case \u003c- done:\n        return false\n    }\n    return true\n})\n\n// Close the done channel and wait for the routine to exit\nwg.Stop()\n```\n\n## FanOut\nFanOut spawns a new go-routine each time `.Run()` is called until `size` is reached,\nsubsequent calls to `.Run()` will block until previously `.Run()` routines have completed.\nAllowing the user to control how many routines will run simultaneously. `.Wait()` then\ncollects any errors from the routines once they have all completed. FanOut allows you\nto control how many goroutines spawn at a time while WaitGroup will not.\n\n```go\nimport \"github.com/mailgun/holster/v4/syncutil\"\n// Insert records into the database 10 at a time\nfanOut := syncutil.NewFanOut(10)\nfor _, item := range items {\n    fanOut.Run(func(cast interface{}) error {\n        item := cast.(Item)\n        return db.ExecuteQuery(\"insert into tbl (id, field) values (?, ?)\",\n            item.Id, item.Field)\n    }, item)\n}\n\n// Collect errors\nerrs := fanOut.Wait()\nif errs != nil {\n    // do something with errs\n}\n```\n\n## LRUCache\nImplements a Least Recently Used Cache with optional TTL and stats collection\n\nThis is a LRU cache based off [github.com/golang/groupcache/lru](http://github.com/golang/groupcache/lru) expanded\nwith the following\n\n* `Peek()` - Get the value without updating the expiration or last used or stats\n* `Keys()` - Get a list of keys at this point in time\n* `Stats()` - Returns stats about the current state of the cache\n* `AddWithTTL()` - Adds a value to the cache with a expiration time\n* `Each()` - Concurrent non blocking access to each item in the cache\n* `Map()` - Efficient blocking modification to each item in the cache\n\nTTL is evaluated during calls to `.Get()` if the entry is past the requested TTL `.Get()`\nremoves the entry from the cache counts a miss and returns not `ok`\n\n```go\nimport \"github.com/mailgun/holster/v4/collections\"\ncache := collections.NewLRUCache(5000)\ngo func() {\n    for {\n        select {\n        // Send cache stats every 5 seconds\n        case \u003c-time.Tick(time.Second * 5):\n            stats := cache.GetStats()\n            metrics.Gauge(metrics.Metric(\"demo\", \"cache\", \"size\"), int64(stats.Size), 1)\n            metrics.Gauge(metrics.Metric(\"demo\", \"cache\", \"hit\"), stats.Hit, 1)\n            metrics.Gauge(metrics.Metric(\"demo\", \"cache\", \"miss\"), stats.Miss, 1)\n        }\n    }\n}()\n\ncache.Add(\"key\", \"value\")\nvalue, ok := cache.Get(\"key\")\n\nfor _, key := range cache.Keys() {\n    value, ok := cache.Get(key)\n    if ok {\n        fmt.Printf(\"Key: %+v Value %+v\\n\", key, value)\n    }\n}\n```\n\n## ExpireCache\nExpireCache is a cache which expires entries only after 2 conditions are met\n\n1. The Specified TTL has expired\n2. The item has been processed with ExpireCache.Each()\n\nThis is an unbounded cache which guaranties each item in the cache\nhas been processed before removal. This cache is useful if you need an\nunbounded queue, that can also act like an LRU cache.\n\nEvery time an item is touched by `.Get()` or `.Set()` the duration is\nupdated which ensures items in frequent use stay in the cache. Processing\nthe cache with `.Each()` can modify the item in the cache without\nupdating the expiration time by using the `.Update()` method.\n\nThe cache can also return statistics which can be used to graph cache usage\nand size.\n\n*NOTE: Because this is an unbounded cache, the user MUST process the cache\nwith `.Each()` regularly! Else the cache items will never expire and the cache\nwill eventually eat all the memory on the system*\n\n```go\nimport \"github.com/mailgun/holster/v4/collections\"\n// How often the cache is processed\nsyncInterval := time.Second * 10\n\n// In this example the cache TTL is slightly less than the sync interval\n// such that before the first sync; items that where only accessed once\n// between sync intervals should expire. This technique is useful if you\n// have a long syncInterval and are only interested in keeping items\n// that where accessed during the sync cycle\ncache := collections.NewExpireCache((syncInterval / 5) * 4)\n\ngo func() {\n    for {\n        select {\n        // Sync the cache with the database every 10 seconds\n        // Items in the cache will not be expired until this completes without error\n        case \u003c-time.Tick(syncInterval):\n            // Each() uses FanOut() to run several of these concurrently, in this\n            // example we are capped at running 10 concurrently, Use 0 or 1 if you\n            // don't need concurrent FanOut\n            cache.Each(10, func(key interface{}, value interface{}) error {\n                item := value.(Item)\n                return db.ExecuteQuery(\"insert into tbl (id, field) values (?, ?)\",\n                    item.Id, item.Field)\n            })\n        // Periodically send stats about the cache\n        case \u003c-time.Tick(time.Second * 5):\n            stats := cache.GetStats()\n            metrics.Gauge(metrics.Metric(\"demo\", \"cache\", \"size\"), int64(stats.Size), 1)\n            metrics.Gauge(metrics.Metric(\"demo\", \"cache\", \"hit\"), stats.Hit, 1)\n            metrics.Gauge(metrics.Metric(\"demo\", \"cache\", \"miss\"), stats.Miss, 1)\n        }\n    }\n}()\n\ncache.Add(\"domain-id\", Item{Id: 1, Field: \"value\"},\nitem, ok := cache.Get(\"domain-id\")\nif ok {\n    fmt.Printf(\"%+v\\n\", item.(Item))\n}\n```\n\n## TTLMap\nProvides a threadsafe time to live map useful for holding a bounded set of key'd values\n that can expire before being accessed. The expiration of values is calculated\n when the value is accessed or the map capacity has been reached.\n```go\nimport \"github.com/mailgun/holster/v4/collections\"\nttlMap := collections.NewTTLMap(10)\nclock.Freeze(time.Now())\n\n// Set a value that expires in 5 seconds\nttlMap.Set(\"one\", \"one\", 5)\n\n// Set a value that expires in 10 seconds\nttlMap.Set(\"two\", \"twp\", 10)\n\n// Simulate sleeping for 6 seconds\nclock.Sleep(time.Second * 6)\n\n// Retrieve the expired value and un-expired value\n_, ok1 := ttlMap.Get(\"one\")\n_, ok2 := ttlMap.Get(\"two\")\n\nfmt.Printf(\"value one exists: %t\\n\", ok1)\nfmt.Printf(\"value two exists: %t\\n\", ok2)\n\n// Output: value one exists: false\n// value two exists: true\n```\n\n## Default values\nThese functions assist in determining if values are the golang default\n and if so, set a value\n```go\nimport \"github.com/mailgun/holster/v4/setter\"\nvar value string\n\n// Returns true if 'value' is zero (the default golang value)\nsetter.IsZero(value)\n\n// Returns true if 'value' is zero (the default golang value)\nsetter.IsZeroValue(reflect.ValueOf(value))\n\n// If 'dest' is empty or of zero value, assign the default value.\n// This panics if the value is not a pointer or if value and\n// default value are not of the same type.\nvar config struct {\n    Foo string\n    Bar int\n}\nsetter.SetDefault(\u0026config.Foo, \"default\")\nsetter.SetDefault(\u0026config.Bar, 200)\n\n// Supply additional default values and SetDefault will\n// choose the first default that is not of zero value\nsetter.SetDefault(\u0026config.Foo, os.Getenv(\"FOO\"), \"default\")\n\n// Use 'SetOverride() to assign the first value that is not empty or of zero\n// value.  The following will override the config file if 'foo' is provided via\n// the cli or defined in the environment.\n\nloadFromFile(\u0026config)\nargFoo = flag.String(\"foo\", \"\", \"foo via cli arg\")\n\nsetter.SetOverride(\u0026config.Foo, *argFoo, os.Env(\"FOO\"))\n```\n\n## Check for Nil interface\n```go\nfunc NewImplementation() MyInterface {\n    // Type and Value are not nil\n    var p *MyImplementation = nil\n    return p\n}\n\nthing := NewImplementation()\nassert.False(t, thing == nil)\nassert.True(t, setter.IsNil(thing))\nassert.False(t, setter.IsNil(\u0026MyImplementation{}))\n```\n\n## GetEnv\nGet a value from an environment variable or return the provided default\n```go\nimport \"github.com/mailgun/holster/v4/config\"\n\nvar conf = sandra.CassandraConfig{\n   Nodes:    []string{config.GetEnv(\"CASSANDRA_ENDPOINT\", \"127.0.0.1:9042\")},\n   Keyspace: \"test\",\n}\n```\n\n## Random Things\nA set of functions to generate random domain names and strings useful for testing\n\n```go\n// Return a random string 10 characters long made up of runes passed\nutil.RandomRunes(\"prefix-\", 10, util.AlphaRunes, holster.NumericRunes)\n\n// Return a random string 10 characters long made up of Alpha Characters A-Z, a-z\nutil.RandomAlpha(\"prefix-\", 10)\n\n// Return a random string 10 characters long made up of Alpha and Numeric Characters A-Z, a-z, 0-9\nutil.RandomString(\"prefix-\", 10)\n\n// Return a random item from the list given\nutil.RandomItem(\"foo\", \"bar\", \"fee\", \"bee\")\n\n// Return a random domain name in the form \"random-numbers.[gov, net, com, ..]\"\nutil.RandomDomainName()\n```\n\n## GoRoutine ID\nGet the go routine id (useful for logging)\n```go\nimport \"github.com/mailgun/holster/v4/callstack\"\nlogrus.Infof(\"[%d] Info about this go routine\", stack.GoRoutineID())\n```\n\n## ContainsString\nChecks if a given slice of strings contains the provided string.\nIf a modifier func is provided, it is called with the slice item before the comparation.\n```go\nimport \"github.com/mailgun/holster/v4/slice\"\n\nhaystack := []string{\"one\", \"Two\", \"Three\"}\nslice.ContainsString(\"two\", haystack, strings.ToLower) // true\nslice.ContainsString(\"two\", haystack, nil) // false\n```\n\n## Priority Queue\nProvides a Priority Queue implementation as described [here](https://en.wikipedia.org/wiki/Priority_queue)\n\n```go\nimport \"github.com/mailgun/holster/v4/collections\"\nqueue := collections.NewPriorityQueue()\n\nqueue.Push(\u0026collections.PQItem{\n    Value: \"thing3\",\n    Priority: 3,\n})\n\nqueue.Push(\u0026collections.PQItem{\n    Value: \"thing1\",\n    Priority: 1,\n})\n\nqueue.Push(\u0026collections.PQItem{\n    Value: \"thing2\",\n    Priority: 2,\n})\n\n// Pops item off the queue according to the priority instead of the Push() order\nitem := queue.Pop()\n\nfmt.Printf(\"Item: %s\", item.Value.(string))\n\n// Output: Item: thing1\n```\n\n## Broadcaster\nAllow the user to notify multiple goroutines of an event. This implementation guarantees every goroutine will wake\nfor every broadcast sent. In the event the goroutine falls behind and more broadcasts() are sent than the goroutine\nhas handled the broadcasts are buffered up to 10,000 broadcasts. Once the broadcast buffer limit is reached calls\n to broadcast() will block until goroutines consuming the broadcasts can catch up.\n \n```go\nimport \"github.com/mailgun/holster/v4/syncutil\"\n    broadcaster := syncutil.NewBroadcaster()\n    done := make(chan struct{})\n    var mutex sync.Mutex\n    var chat []string\n\n    // Start some simple chat clients that are responsible for\n    // sending the contents of the []chat slice to their clients\n    for i := 0; i \u003c 2; i++ {\n        go func(idx int) {\n            var clientIndex int\n            for {\n                mutex.Lock()\n                if clientIndex != len(chat) {\n                    // Pretend we are sending a message to our client via a socket\n                    fmt.Printf(\"Client [%d] Chat: %s\\n\", idx, chat[clientIndex])\n                    clientIndex++\n                    mutex.Unlock()\n                    continue\n                }\n                mutex.Unlock()\n\n                // Wait for more chats to be added to chat[]\n                select {\n                case \u003c-broadcaster.WaitChan(string(idx)):\n                case \u003c-done:\n                    return\n                }\n            }\n        }(i)\n    }\n\n    // Add some chat lines to the []chat slice\n    for i := 0; i \u003c 5; i++ {\n        mutex.Lock()\n        chat = append(chat, fmt.Sprintf(\"Message '%d'\", i))\n        mutex.Unlock()\n\n        // Notify any clients there are new chats to read\n        broadcaster.Broadcast()\n    }\n\n    // Tell the clients to quit\n    close(done)\n```\n\n## UntilPass\nFunctional test helper which will run a suite of tests until the entire suite\npasses, or all attempts have been exhausted.\n\n```go\nimport (\n    \"github.com/mailgun/holster/v4/testutil\"\n    \"github.com/stretchr/testify/require\"\n    \"github.com/stretchr/testify/assert\"\n)\n\nfunc TestUntilPass(t *testing.T) {\n    rand.Seed(time.Now().UnixNano())\n    var value string\n\n    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n        if r.Method == http.MethodPost {\n            // Sleep some rand amount to time to simulate some\n            // async process happening on the server\n            time.Sleep(time.Duration(rand.Intn(10))*time.Millisecond)\n            // Set the value\n            value = r.FormValue(\"value\")\n        } else {\n            fmt.Fprintln(w, value)\n        }\n    }))\n    defer ts.Close()\n\n    // Start the async process that produces a value on the server\n    http.PostForm(ts.URL, url.Values{\"value\": []string{\"batch job completed\"}})\n\n    // Keep running this until the batch job completes or attempts are exhausted\n    testutil.UntilPass(t, 10, time.Millisecond*100, func(t testutil.TestingT) {\n        r, err := http.Get(ts.URL)\n\n        // use of `require` will abort the current test here and tell UntilPass() to\n        // run again after 100 milliseconds\n        require.NoError(t, err)\n\n        // Or you can check if the assert returned true or not\n        if !assert.Equal(t, 200, r.StatusCode) {\n            return\n        }\n\n        b, err := ioutil.ReadAll(r.Body)\n        require.NoError(t, err)\n\n        assert.Equal(t, \"batch job completed\\n\", string(b))\n    })\n}\n```\n\n## UntilConnect\nWaits until the test can connect to the TCP/HTTP server before continuing the test\n```go\nimport (\n    \"github.com/mailgun/holster/v4/testutil\"\n    \"golang.org/x/net/nettest\"\n    \"github.com/stretchr/testify/require\"\n)\n\nfunc TestUntilConnect(t *testing.T) {\n    ln, err := nettest.NewLocalListener(\"tcp\")\n    require.NoError(t, err)\n\n    go func() {\n        cn, err := ln.Accept()\n        require.NoError(t, err)\n        cn.Close()\n    }()\n    // Wait until we can connect, then continue with the test\n    testutil.UntilConnect(t, 10, time.Millisecond*100, ln.Addr().String())\n}\n```\n\n### Retry Until\nRetries a function until the function returns error = nil or until the context is deadline is exceeded\n```go\nctx, cancel := context.WithTimeout(context.Background(), time.Second*20)\ndefer cancel()\nerr := retry.Until(ctx, retry.Interval(time.Millisecond*10), func(ctx context.Context, att int) error {\n    res, err := http.Get(\"http://example.com/get\")\n    if err != nil {\n        return err\n    }\n    if res.StatusCode != http.StatusOK {\n        return errors.New(\"expected status 200\")\n    }\n    // Do something with the body\n    return nil\n})\nif err != nil {\n    panic(err)\n}\n```\n\nBackoff functions provided\n\n* `retry.Attempts(10, time.Millisecond*10)` retries up to `10` attempts\n* `retry.Interval(time.Millisecond*10)` retries at an interval indefinitely or until context is cancelled\n* `retry.ExponentialBackoff{ Min: time.Millisecond, Max: time.Millisecond * 100, Factor: 2}` retries\n at an exponential backoff interval. Can accept an optional `Attempts` which will limit the number of attempts\n\n\n### Retry Async\nRuns a function asynchronously and retries it until it succeeds, or the context is \ncancelled or `Stop()` is called. This is useful in distributed programming where\nyou know a remote thing will eventually succeed, but you need to keep trying until\nthe remote thing succeeds, or we are told to shutdown.\n\n```go\nctx := context.Background()\nasync := retry.NewRetryAsync()\n\nbackOff := \u0026retry.ExponentialBackoff{\n    Min:      time.Millisecond,\n    Max:      time.Millisecond * 100,\n    Factor:   2,\n    Attempts: 10,\n}\n\nid := createNewEC2(\"my-new-server\")\n\nasync.Async(id, ctx, backOff, func(ctx context.Context, i int) error {\n    // Waits for a new EC2 instance to be created then updates the config and exits\n    if err := updateInstance(id, mySettings); err != nil {\n        return err\n    }\n    return nil\n})\n// Wait for all the asyncs to complete\nasync.Wait()\n```\n\n\n## OpenTelemetry\nTracing tools using OpenTelemetry client SDK and Jaeger Tracing server.\n\nSee [tracing\nreadme](https://github.com/mailgun/holster/blob/master/tracing/README.md) for\ndetails.\n\n\n## Context Utilities\nSee package directory `ctxutil`.\n\nUse functions `ctxutil.WithDeadline()`/`WithTimeout()` instead of the `context`\nequivalents to log details of the deadline and source file:line where it was\nset.  Must enable debug logging.\n\n## Contributing code\nPlease read the [Contribution Guidelines](./CONTRIBUTING.md) before sending patches.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmailgun%2Fholster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmailgun%2Fholster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmailgun%2Fholster/lists"}