{"id":17726633,"url":"https://github.com/sysulq/dataloader-go","last_synced_at":"2025-08-21T10:32:16.289Z","repository":{"id":251168452,"uuid":"835627910","full_name":"sysulq/dataloader-go","owner":"sysulq","description":"Go implementation of Facebook's DataLoader with 200+ lines of code.","archived":false,"fork":false,"pushed_at":"2024-10-21T22:46:38.000Z","size":221,"stargazers_count":51,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-22T19:10:01.585Z","etag":null,"topics":["batch","cache","dataloader","deduplicate","facebook","go","golang","lru","opentelemetry"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/sysulq/dataloader-go","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sysulq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-30T08:08:22.000Z","updated_at":"2024-10-21T22:46:34.000Z","dependencies_parsed_at":"2024-10-22T23:03:37.854Z","dependency_job_id":null,"html_url":"https://github.com/sysulq/dataloader-go","commit_stats":null,"previous_names":["sysulq/dataloader-go"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysulq%2Fdataloader-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysulq%2Fdataloader-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysulq%2Fdataloader-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysulq%2Fdataloader-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sysulq","download_url":"https://codeload.github.com/sysulq/dataloader-go/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230507051,"owners_count":18236944,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","cache","dataloader","deduplicate","facebook","go","golang","lru","opentelemetry"],"created_at":"2024-10-25T17:06:20.575Z","updated_at":"2024-12-19T22:09:04.030Z","avatar_url":"https://github.com/sysulq.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"dataloader-go\n===\n\n[![Go](https://github.com/sysulq/dataloader-go/actions/workflows/go.yml/badge.svg)](https://github.com/sysulq/dataloader-go/actions/workflows/go.yml)\n[![codecov](https://codecov.io/gh/sysulq/dataloader-go/graph/badge.svg?token=KHQZ38ES45)](https://codecov.io/gh/sysulq/dataloader-go)\n\nThis is a Go implementation of Facebook's DataLoader.\n\nA generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce the number of requests to the server.\n\nFeature\n---\n\n- 200+ lines of code, easy to understand and maintain.\n- 100% test coverage, bug free and reliable.\n- Based on generics and can be used with any type of data.\n- Use hashicorp/golang-lru to cache the loaded values.\n- Can be used to batch and cache multiple requests.\n- Deduplicate identical requests, reducing the number of requests.\n- Support OpenTelemetry, trace batched requests with Links.\n\nInstallation\n---\n\n```go\nimport \"github.com/sysulq/dataloader-go\"\n```\n\nAPI Design\n---\n\n```go\n// New creates a new DataLoader with the given loader and options.\nfunc New[K comparable, V any](loader Loader[K, V], options ...Option) Interface[K, V]\n\ntype Interface[K comparable, V any] interface {\n\t// Load loads a single key\n\tLoad(context.Context, K) Result[V]\n\t// LoadMany loads multiple keys\n\tLoadMany(context.Context, []K) []Result[V]\n\t// LoadMap loads multiple keys and returns a map of results\n\tLoadMap(context.Context, []K) map[K]Result[V]\n\t// Clear removes an item from the cache\n\tClear(K) Interface[K, V]\n\t// ClearAll clears the entire cache\n\tClearAll() Interface[K, V]\n\t// Prime primes the cache with a key and value\n\tPrime(ctx context.Context, key K, value V) Interface[K, V]\n}\n```\n\nExample\n---\n\n```go\npackage dataloader_test\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/sysulq/dataloader-go\"\n)\n\nfunc TestExample(t *testing.T) {\n\tloader := dataloader.New(\n\t\tfunc(ctx context.Context, keys []int) []dataloader.Result[string] {\n\t\t\tresults := make([]dataloader.Result[string], len(keys))\n\n\t\t\tfor i, key := range keys {\n\t\t\t\tresults[i] = dataloader.Wrap(fmt.Sprintf(\"Result for %d\", key), nil)\n\t\t\t}\n\t\t\treturn results\n\t\t},\n\t\tdataloader.WithCache(100, time.Minute),\n\t\tdataloader.WithBatchSize(50),\n\t\tdataloader.WithWait(5*time.Millisecond),\n\t)\n\n\tctx := context.Background()\n\n\t// Load\n\tdata, err := loader.Load(ctx, 1).Unwrap()\n\tif err != nil {\n\t\tt.Errorf(\"Unexpected error: %v\", err)\n\t} else {\n\t\tfmt.Printf(\"Result: %s\\n\", data)\n\t\t// Output:\n\t\t// Result: Result for 1\n\t}\n\n\t// LoadMany\n\tresults := loader.LoadMany(ctx, []int{3, 4, 5})\n\tfor _, result := range results {\n\t\tdata, err := result.Unwrap()\n\t\tif err != nil {\n\t\t\tt.Errorf(\"Unexpected error: %v\", err)\n\t\t} else {\n\t\t\tfmt.Printf(\"Result: %s\\n\", data)\n\t\t\t// Output:\n\t\t\t// Result: Result for 3\n\t\t\t// Result: Result for 4\n\t\t\t// Result: Result for 5\n\t\t}\n\t}\n\n\t// LoadMap\n\tkeys := []int{6, 7, 8}\n\tresultsMap := loader.LoadMap(ctx, keys)\n\tfor _, key := range keys {\n\t\tdata, err := resultsMap[key].Unwrap()\n\t\tif err != nil {\n\t\t\tt.Errorf(\"Unexpected error: %v\", err)\n\t\t} else {\n\t\t\tfmt.Printf(\"Result: %s\\n\", data)\n\t\t\t// Output:\n\t\t\t// Result: Result for 6\n\t\t\t// Result: Result for 7\n\t\t\t// Result: Result for 8\n\t\t}\n\t}\n\n\t// Prime\n\tloader.Prime(ctx, 8, \"Prime result\")\n\tdata, err = loader.Load(ctx, 8).Unwrap()\n\tif err != nil {\n\t\tt.Errorf(\"Unexpected error: %v\", err)\n\t} else {\n\t\tfmt.Printf(\"Result: %s\\n\", data)\n\t\t// Output:\n\t\t// Result: Prime result\n\t}\n\n\t// Clear\n\tloader.Clear(7)\n\tdata, err = loader.Load(ctx, 7).Unwrap()\n\tif err != nil {\n\t\tt.Errorf(\"Unexpected error: %v\", err)\n\t} else {\n\t\tfmt.Printf(\"Result: %s\\n\", data)\n\t\t// Output:\n\t\t// Result: Result for 7\n\t}\n\n\t// ClearAll\n\tloader.ClearAll()\n\tdata, err = loader.Load(ctx, 8).Unwrap()\n\tif err != nil {\n\t\tt.Errorf(\"Unexpected error: %v\", err)\n\t} else {\n\t\tfmt.Printf(\"Result: %s\\n\", data)\n\t\t// Output:\n\t\t// Result: Result for 8\n\t}\n}\n```\n\n\nBenchmark\n---\n\n```plain\ngoos: darwin\ngoarch: amd64\npkg: github.com/sysulq/dataloader-go\ncpu: Intel(R) Core(TM) i5-10600K CPU @ 4.10GHz\nBenchmarkDataLoader/direct.Batch-12         \t 1437706\t       827.1 ns/op\t     480 B/op\t      11 allocs/op\nBenchmarkDataLoader/dataloader.Load-12      \t  513562\t      2386 ns/op\t    1280 B/op\t      20 allocs/op\nBenchmarkDataLoader/dataloader.LoadMany-12  \t  438864\t      2500 ns/op\t    1760 B/op\t      23 allocs/op\nBenchmarkDataLoader/dataloader.LoadMap-12   \t  437780\t      2711 ns/op\t    2199 B/op\t      24 allocs/op\nPASS\ncoverage: 60.7% of statements\nok  \tgithub.com/sysulq/dataloader-go\t5.938s\n```\n\nAcknowledgements\n---\n\nInspired by facebook/dataloader and graph-gophers/dataloader.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysulq%2Fdataloader-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsysulq%2Fdataloader-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysulq%2Fdataloader-go/lists"}