{"id":37151771,"url":"https://github.com/thisisdevelopment/mightymap","last_synced_at":"2026-01-14T17:55:21.297Z","repository":{"id":259887350,"uuid":"879726546","full_name":"thisisdevelopment/mightymap","owner":"thisisdevelopment","description":"Mightymap: Conveys both robustness and high capability, fitting for a powerful concurrent map. ","archived":false,"fork":false,"pushed_at":"2025-12-01T20:19:30.000Z","size":104,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-12-04T10:32:58.471Z","etag":null,"topics":["badgerdb","concurrent-programming","high-performance","maps","persistence","swiss","thread-safe"],"latest_commit_sha":null,"homepage":"https://this.nl","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thisisdevelopment.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-28T12:51:50.000Z","updated_at":"2025-12-01T20:16:24.000Z","dependencies_parsed_at":"2024-10-28T16:43:09.232Z","dependency_job_id":"19c6cf44-17e1-44b9-b459-fedfa0caadbc","html_url":"https://github.com/thisisdevelopment/mightymap","commit_stats":null,"previous_names":["thisisdevelopment/mightymap"],"tags_count":20,"template":false,"template_full_name":null,"purl":"pkg:github/thisisdevelopment/mightymap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thisisdevelopment%2Fmightymap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thisisdevelopment%2Fmightymap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thisisdevelopment%2Fmightymap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thisisdevelopment%2Fmightymap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thisisdevelopment","download_url":"https://codeload.github.com/thisisdevelopment/mightymap/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thisisdevelopment%2Fmightymap/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28429027,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T16:38:47.836Z","status":"ssl_error","status_checked_at":"2026-01-14T16:34:59.695Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["badgerdb","concurrent-programming","high-performance","maps","persistence","swiss","thread-safe"],"created_at":"2026-01-14T17:55:20.566Z","updated_at":"2026-01-14T17:55:21.289Z","avatar_url":"https://github.com/thisisdevelopment.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MightyMap\n\n[![go report card](https://goreportcard.com/badge/github.com/thisisdevelopment/mightymap \"go report card\")](https://goreportcard.com/report/github.com/thisisdevelopment/mightymap)\n[![codecov](https://codecov.io/gh/thisisdevelopment/mightymap/graph/badge.svg?token=DMJCBUY273)](https://codecov.io/gh/thisisdevelopment/mightymap)\n[![CircleCI](https://dl.circleci.com/status-badge/img/gh/thisisdevelopment/mightymap/tree/master.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/thisisdevelopment/mightymap/tree/master)\n[![GoDoc](https://godoc.org/github.com/thisisdevelopment/mightymap?status.svg)](https://godoc.org/github.com/thisisdevelopment/mightymap)\n\n\nMightyMap is a powerful, thread-safe, and concurrent generic map implementation for Go with configurable storage backends. 🔒 It provides type-safe operations through Go generics and flexibility in choosing different storage engines depending on your performance and feature requirements. 🚀 The generic implementation allows you to create strongly-typed maps with any comparable key type and any value type. 💪\n\n## Features\n\n- **Thread-safe**: Designed for concurrent use without the need for additional synchronization.\n- **Configurable Storage Backends**: Swap out the storage engine as needed. Currently supports:\n  - Default in-memory map with mutex locking.\n  - SwissStorage for optimized performance.\n  - BadgerStorage for persistent key-value storage.\n- **Generic Support**: Works with any comparable key type and any value type.\n- **Flexible Overwrite Behavior**: Choose whether keys can be overwritten or not.\n\n## Installation\n\nTo install MightyMap, use `go get`:\n\n```bash\ngo get github.com/thisisdevelopment/mightymap\n```\n\n## Note v0.3.1 to v0.4.0\nThe `ctx` parameter is now required for all methods. This is a breaking change. Since we allow multiple storage backends, we need to be able pass the context to the storage backend.\n\n## Note on swissMap Storage\nSince golang 1.24 the default internal map implementation has been switched to a swiss map implementation. If you're using golang version \u003e= 1.24 the SwissMapStorage implementation is obsolete, and you can just use the default storage (for in memory mightyMaps)\n\n## Storage Changes in v0.5.0\n\nIn version 0.5.0, we've made significant changes to the underlying storage implementation for persistent backends (BadgerDB and Redis):\n\n### MessagePack Encoding\n- **New Encoding**: All persistent storage backends now use MessagePack encoding for improved reliability, type safety, and cross-language compatibility\n- **Performance**: MessagePack provides efficient binary serialization with minimal overhead (~2-5% of total operation time)\n- **Type Safety**: Better handling of complex data types including structs, maps, slices, and interfaces\n- **Schema Evolution**: Future-proof format that supports gradual schema changes\n\n### Backward Compatibility\n⚠️ **Breaking Change**: v0.5.0 is **not backward compatible** with previous versions when using BadgerDB or Redis storage backends. Data stored with v0.4.5 and earlier versions uses direct Go encoding, while v0.5.0 uses MessagePack format.\n\n### Redis Username Support\nv0.5.1 also adds support for Redis username authentication (Redis 6.0+ ACL), providing enhanced security for Redis deployments:\n\n```go\nstore := storage.NewMightyMapRedisStorage[string, any](\n    storage.WithRedisAddr(\"localhost:6379\"),\n    storage.WithRedisUsername(\"myuser\"),     // New in v0.5.0\n    storage.WithRedisPassword(\"mypassword\"),\n    storage.WithRedisDB(0),\n)\n```\n\n### Migration Tools\nTo help with the transition from v0.4.5 to v0.5.0, we provide comprehensive CLI migration tools:\n\n#### Available Tools\n- **`cmd/migrate-badger`**: Migrates BadgerDB data from v0.4.5 (direct storage) to v0.5.0 (MessagePack)\n- **`cmd/migrate-redis`**: Migrates Redis data from v0.4.5 to v0.5.0 with username support\n\n#### Features\n- ✅ **Dry-run mode** to preview changes before migration\n- ✅ **Configurable batch processing** with progress logging\n- ✅ **Key pattern filtering** for selective migration\n- ✅ **Error handling and reporting** with detailed statistics\n- ✅ **Timeout controls** and safe operation practices\n- ✅ **YAML configuration** with example files provided\n\n#### Quick Start\n```bash\n# BadgerDB migration\ncd cmd/migrate-badger\ngo run main.go                    # Creates config file\ngo run main.go --dry-run --verbose   # Preview migration\ngo run main.go --verbose         # Run actual migration\n\n# Redis migration  \ncd cmd/migrate-redis\ngo run main.go                    # Creates config file\ngo run main.go --dry-run --verbose   # Preview migration\ngo run main.go --verbose         # Run actual migration\n```\n\nSee `cmd/README.md` for detailed setup instructions and configuration options.\n\n### Why MessagePack?\nThe migration to MessagePack provides several benefits:\n- **Reliability**: Consistent encoding/decoding across different Go versions\n- **Performance**: Efficient binary format optimized for storage\n- **Compatibility**: Cross-language support for polyglot environments  \n- **Future-proofing**: Schema evolution capabilities for long-term data integrity\n\n## Usage\nHere is a simple example of how to use MightyMap:\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"github.com/thisisdevelopment/mightymap\"\n)\n\nfunc main() {\n    // Create a new concurrent map that allows overwriting existing keys\n    ctx := context.Background()\n    cm := mightymap.New[int, string](true)\n\n    // Store a value\n    cm.Store(ctx, 1, \"one\")\n\n    // Load a value\n    value, ok := cm.Load(ctx, 1)\n    if ok {\n        fmt.Println(\"Loaded value:\", value)\n    }\n\n    // Check if a key exists\n    exists := cm.Has(ctx, 2)\n    fmt.Println(\"Key 2 exists:\", exists)\n\n    // Store more values\n    cm.Store(ctx, 2, \"two\")\n    cm.Store(ctx, 3, \"three\")\n\n    // Get all keys\n    keys := cm.Keys(ctx)\n    fmt.Println(\"All keys:\", keys)\n\n    // Delete a key\n    cm.Delete(ctx, 1)\n\n    // Get map length\n    fmt.Println(\"Map length:\", cm.Len(ctx))\n\n    // Clear the map\n    cm.Clear()\n}\n```\n\n## Storage Backends\n\nMightyMap allows you to choose different storage backends. Here's how to use them:\n\n### Default Storage\n\nUses the standard Go map with mutex locking.\n\n```go\nstore := storage.NewMightyMapDefaultStorage[int, string]()\ncm := mightymap.New[int, string](true, store)\n-- or just --\ncm := mightymap.New[int, string](true)\n```\n\n### Swiss Storage\n\nSwiss Storage leverages the [dolthub/swiss](https://github.com/dolthub/swiss) package, which implements Swiss-table hash maps in Go for high-performance in-memory storage. Swiss-table hash maps offer faster lookups and lower memory usage compared to Go's native maps, especially beneficial in concurrent environments and with large datasets.\n\nBy utilizing Swiss Storage, MightyMap achieves optimized performance for workloads that require fast access and modification of in-memory data.\n\nTo use Swiss Storage:\n\n```go\nstore := storage.NewMightyMapSwissStorage[int, string]()\ncm := mightymap.New[int, string](true, store)\n```\n\nYou can also customize the initial capacity to optimize memory usage for your specific use case:\n\n```go\nstore := storage.NewMightyMapSwissStorage[int, string](storage.WithDefaultCapacity(100_000))\ncm := mightymap.New[int, string](true, store)\n```\n\n### Badger Storage\n\nUses BadgerDB for persistent storage.\n\nBadgerDB is an embeddable, persistent, and fast key-value database written in pure Go. It provides efficient storage for key-value data, supporting ACID transactions and achieving high performance on SSDs. Using BadgerDB as a storage backend allows MightyMap to handle large datasets that don't fit entirely in memory, with persistence across application restarts. BadgerDB also supports encryption and compression, making it a versatile choice for various use cases.\n\nFor more information about BadgerDB, visit the [BadgerDB GitHub repository](https://github.com/dgraph-io/badger).\n\n\n\n```go\nstore := storage.NewMightyMapBadgerStorage[int, string](\n    storage.WithMemoryStorage(false),\n    storage.WithTempDir(\"/path/to/db\"),\n)\ncm := mightymap.New[int, string](true, store)\n```\n\nFor the BadgerDB backend, you can customize the options to optimize performance and storage behavior.\n\n```\n    WithNumCompactors(numCompactors int).         // Set the number of compaction workers.\n    WithMetricsEnabled(metricsEnabled bool).      // Enable or disable metrics collection.\n    WithDetectConflicts(detectConflicts bool).    // Enable or disable conflict detection.\n    WithLoggingLevel(loggingLevel badger.LoggerLevel). // Set the logging level.\n    WithBlockSize(blockSize int).                 // Set the size of each block in bytes.\n    WithNumVersionsToKeep(numVersionsToKeep int). // Set the number of versions to keep per key.\n    WithIndexCacheSize(indexCacheSize int64).     // Set the size of the index cache in bytes.\n    WithMemoryStorage(memoryStorage bool).        // Use in-memory storage if true.\n    WithBlockCacheSize(blockCacheSize int64)      // Set the size of the block cache in bytes.\n    WithCompression(compression bool)             // Enable or disable data compression using ZSTD.\n    WithValueThreshold(valueThreshold int64)      // Set the threshold for value storage in bytes.\n    WithEncryptionKey(encryptionKey string)       // Set the encryption key for the Badger database.\n    WithEncryptionKeyRotationDuration(encryptionKeyRotation time.Duration) // Set the rotation duration for the encryption key in Badger.\n    WithSyncWrites(syncWrites bool)               // Enable or disable synchronous writes in Badger.\n```\n\nsensible defaults are used if you don't specify options.\n\n```\t\n\treturn \u0026badgerOpts{\n\t\tdir:                   os.TempDir() + fmt.Sprintf(\"/badger-%d\", time.Now().UnixNano()),\n\t\tcompression:           false,\n\t\tmemoryStorage:         true,\n\t\tnumCompactors:         4,\n\t\tnumVersionsToKeep:     1,\n\t\tindexCacheSize:        int64(128 \u003c\u003c 20),\n\t\tblockCacheSize:        512 \u003c\u003c 20,\n\t\tblockSize:             16 * 1024,\n\t\tloggingLevel:          int(badger.ERROR),\n\t\tmetricsEnabled:        true,\n\t\tdetectConflicts:       true,\n\t\tgcInterval:            10 * time.Second,\n\t\tgcPercentage:          0.5,\n\t\tmemTableSize:          64 \u003c\u003c 20,\n\t\tvalueThreshold:        1 \u003c\u003c 20,\n\t\tencryptionKey:         \"\",\n\t\tencryptionKeyRotation: 10 * 24 * time.Hour, // 10 days default\n\t\tsyncWrites:            false,\n\t}\n```\n\n## API Reference\n\n### Methods\n\n- `Load(key K) (value V, ok bool)`: Retrieves the value for a key.\n- `Has(key K) (ok bool)`: Checks if a key exists.\n- `Store(key K, value V)`: Stores a value for a key.\n- `Delete(keys ...K)`: Deletes one or more keys.\n- `Range(f func(key K, value V) bool)`: Iterates over all key-value pairs.\n- `Keys() []K`: Returns all keys in the map in an unspecified order.\n- `Pop(key K) (value V, ok bool)`: Retrieves and deletes a value for a key.\n- `Next() (value V, key K, ok bool)`: Retrieves the next key-value pair.\n- `Len() int`: Returns the number of items in the map.\n- `Clear()`: Removes all items from the map.\n- `Close() error`: Closes the map.\n\n### Constructor\n\n- `New[K comparable, V any](allowOverwrite bool, storages ...storage.IMightyMapStorage[K, V]) *Map[K, V]`\n\n    - `allowOverwrite`: If `true`, existing keys can be overwritten when using `Store()`. If `false`, `Store()` will only insert new keys.\n    - `storages`: Optional storage implementation.\n\n## Benchmarks\n\nBenchmarks are available in the `storage` package to compare the performance of different storage backends. You can run the benchmarks using:\n\n```bash\n\u003e\u003e go test -benchmem -bench=. ./storage\ngoos: darwin\ngoarch: arm64\npkg: github.com/thisisdevelopment/mightymap/storage\ncpu: Apple M2 Max\nBenchmarkBadgerMemStorageStore-12         260845              4056 ns/op            1258 B/op         31 allocs/op\nBenchmarkBadgerMemStorageLoad-12         1000000              1001 ns/op             632 B/op         17 allocs/op\nBenchmarkBadgerMemStorageDelete-12        665391              1684 ns/op             722 B/op         18 allocs/op\nBenchmarkBadgerStorageStore-12            231421              6041 ns/op            1518 B/op         41 allocs/op\nBenchmarkBadgerStorageLoad-12            1000000              1098 ns/op             632 B/op         18 allocs/op\nBenchmarkBadgerStorageDelete-12           547647              1970 ns/op             810 B/op         21 allocs/op\nBenchmarkSwissStorageStore-12           14271470               174.7 ns/op           102 B/op          0 allocs/op\nBenchmarkSwissStorageLoad-12            21759957                52.01 ns/op            0 B/op          0 allocs/op\nBenchmarkSwissStorageDelete-12          30795424                35.53 ns/op            8 B/op          1 allocs/op\nBenchmarkDefaultStorageStore-12         11982510               135.4 ns/op            84 B/op          0 allocs/op\nBenchmarkDefaultStorageLoad-12          24373977                45.56 ns/op            0 B/op          0 allocs/op\nBenchmarkDefaultStorageDelete-12        33667726                31.96 ns/op            8 B/op          1 allocs/op\nPASS\nok      github.com/thisisdevelopment/mightymap/storage  35.461s\n```\n\n## Limitations\n- The **default storage** backend and **Swiss backend** are not thread-safe by default. Using Delete() while iterating with Range() will cause a deadlock. To safely delete items while iterating, collect the keys to delete during Range() and call Delete() with all keys after Range() completes. For example:\n\n```go\n// Collect keys to delete during iteration\nkeysToDelete := []K{}\n\ncm.Range(func(key K, value V) bool {\n    // Replace this condition with your own logic\n    if shouldDelete(key, value) {\n        keysToDelete = append(keysToDelete, key)\n    }\n    return true // Continue iteration\n})\n\n// Delete all collected keys after iteration\ncm.Delete(keysToDelete...)\n```\n\n## StorageAPI Reference\n\n### Methods\n```\n\tLoad(ctx context.Context, key K) (value V, ok bool)\n\tStore(ctx context.Context, key K, value V)\n\tDelete(ctx context.Context, keys ...K)\n\tRange(ctx context.Context, f func(key K, value V) bool)\n\tKeys(ctx context.Context) []K\n\tNext(ctx context.Context) (key K, value V, ok bool)\n\tLen(ctx context.Context) int\n\tClear(ctx context.Context)\n```\n\n## About Us Th[is]\n\n[This](https://this.nl) is a digital agency based in Utrecht, the Netherlands, specializing in crafting high-performance, resilient, and scalable digital solutions, api's, microservices, and more. Our multidisciplinary team of designers, front and backend developers and strategists collaborates closely to deliver robust and efficient products that meet the demands of today's digital landscape. We are passionate about turning ideas into reality and providing exceptional value to our clients through innovative technology and exceptional user experiences.\n\n## Contributing\n\nContributions are welcome! We especially encourage contributions of new storage backends. Please open an issue to discuss your ideas or submit a pull request with your implementation.\n\n## License\n\nThis project is licensed under the MIT License.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthisisdevelopment%2Fmightymap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthisisdevelopment%2Fmightymap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthisisdevelopment%2Fmightymap/lists"}