{"id":18017043,"url":"https://github.com/achille-roussel/sqlrange","last_synced_at":"2025-07-03T22:04:27.426Z","repository":{"id":217200298,"uuid":"742653311","full_name":"achille-roussel/sqlrange","owner":"achille-roussel","description":"Go 1.23 range functions with database/sql","archived":false,"fork":false,"pushed_at":"2024-11-18T19:09:12.000Z","size":42,"stargazers_count":147,"open_issues_count":2,"forks_count":4,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-07-03T22:03:34.754Z","etag":null,"topics":["database","golang","rangefunc","sql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/achille-roussel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-13T00:58:45.000Z","updated_at":"2025-05-19T16:13:28.000Z","dependencies_parsed_at":"2024-01-15T01:23:27.249Z","dependency_job_id":"7498d406-d458-4f19-9949-62a5374d08ec","html_url":"https://github.com/achille-roussel/sqlrange","commit_stats":null,"previous_names":["achille-roussel/sqlrange"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/achille-roussel/sqlrange","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achille-roussel%2Fsqlrange","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achille-roussel%2Fsqlrange/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achille-roussel%2Fsqlrange/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achille-roussel%2Fsqlrange/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/achille-roussel","download_url":"https://codeload.github.com/achille-roussel/sqlrange/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achille-roussel%2Fsqlrange/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263410758,"owners_count":23462295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","golang","rangefunc","sql"],"created_at":"2024-10-30T04:19:58.970Z","updated_at":"2025-07-03T22:04:27.341Z","avatar_url":"https://github.com/achille-roussel.png","language":"Go","readme":"# sqlrange [![Go Reference](https://pkg.go.dev/badge/github.com/achille-roussel/sqlrange.svg)](https://pkg.go.dev/github.com/achille-roussel/sqlrange)\n\nLibrary using the `database/sql` package and Go 1.23 range functions to execute\nqueries against SQL databases.\n\n## Installation\n\nThis package is intended to be used as a library and installed with:\n```sh\ngo get github.com/achille-roussel/sqlrange\n```\n\n:warning: The package requires Go 1.23 or later for range function support.\n\n## Usage\n\nThe `sqlrange` package contains two kinds of functions called **Exec** and\n**Query** which wrap the standard library's `database/sql` methods with the\nsame names. The package adds stronger type safety and the ability to use\nrange functions as iterators to pass values to the queries or retrieve results.\n\nNote that `sqlrange` **IS NOT AN ORM**, it is a lightweight package which does\nnot hide any of the details and simply provides library functions to structure\napplications that stream values in and out of databases.\n\n### Query\n\nThe **Query** functions are used to read streams of values from databases,\nin the same way that `sql.(*DB).Query` does, but using range functions to\nsimplify the code constructs, and type parameters to automatically decode\nSQL results into Go struct values.\n\nThe type parameter must be a struct with fields containing \"sql\" struct tags\nto define the names of columns that the fields are mapped to:\n```go\ntype Point struct {\n    X float64 `sql:\"x\"`\n    Y float64 `sql:\"y\"`\n}\n```\n```go\nfor p, err := range sqlrange.Query[Point](db, `select x, y from points`) {\n    if err != nil {\n        ...\n    }\n    ...\n}\n```\nNote that resource management here is automated by the range function\nreturned by calling **Query**, the underlying `*sql.Rows` value is automatically\nclosed when the program exits the body of the range loop consuming the rows.\n\n### Exec\n\nThe **Exec** functions are used to execute insert, update, or delete queries\nagainst databases, accepting a stream of parameters as arguments (in the form of\na range function), and returning a stream of results.\n\nSince the function will send multiple queries to the database, it is often\npreferable to apply it to a transaction (or a statement derived from a\ntransaction via `sql.(*Tx).Stmt`) to ensure atomicity of the operation.\n\n```go\ntx, err := db.Begin()\nif err != nil {\n    ...\n}\ndefer tx.Rollback()\n\nfor r, err := range sqlrange.Exec(tx,\n    `insert into table (col1, col2, col3) values (?, ?, ?)`,\n    // This range function yields the values that will be inserted into\n    // the database by executing the query above.\n    func(yield func(RowType, error) bool) {\n        ...\n    },\n    // Inject the arguments for the SQL query being executed.\n    // The function is called for each value yielded by the range\n    // function above.\n    sqlrange.ExecArgs(func(args []any, row RowType) []any {\n        return append(args, row.Col1, row.Col2, row.Col3)\n    }),\n) {\n    // Each results of each execution are streamed and must be consumed\n    // by the program to drive the operation.\n    if err != nil {\n        ...\n    }\n    ...\n}\n\nif err := tx.Commit(); err != nil {\n    ...\n}\n```\n\n### Context\n\nMirroring methods of the `sql.DB` type, functions of the `sqlrange` package have\nvariants that take a `context.Context` as first argument to support asynchronous\ncancellation or timing out the operations.\n\nReusing the example above, we could set a 10 secondstime limit for the query\nusing **QueryContext** instead of **Query**:\n\n```go\nctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)\ndefer cancel()\n\nfor p, err := range sqlrange.QueryContext[Point](ctx, db, `select x, y from points`) {\n    if err != nil {\n        ...\n    }\n    ...\n}\n```\n\nThe context is propagated to the `sql.(*DB).QueryContext` method, which then\npasses it to the underlying SQL driver.\n\n## Performance\n\nFunctions in this package are optimized to have a minimal compute and memory\nfootprint. Applications should not observe any performance degradation from\nusing it, compared to using the `database/sql` package directly. This is an\nimportant property of the package since it means that the type safety, resource\nlifecycle management, and expressiveness do not have to be a trade off.\n\nThis is a use case where the use of range functions really shines: because all\nthe code points where range functions are created get inlined, the compiler's\nescape analysis can place most of the values on the stack, keeping the memory\nand garbage collection overhead to a minimum.\n\nMost of the escaping heap allocations in this package come from the use of\nreflection to convert SQL rows into Go values, which are optimized using two\ndifferent approaches:\n\n- **Caching:** internally, the package caches the `reflect.StructField` values\n  that it needs. This is necessary to remove some of the allocations caused by\n  the `reflect` package allocating the [`Index`][structField] on the heap.\n  See https://github.com/golang/go/issues/2320 for more details.\n\n- **Amortization:** since the intended use case is to select ranges of rows,\n  or execute batch queries, the functions can reuse the local state maintained\n  to read values. The more rows are involved in the query, the great the cost of\n  allocating those values gets amortized, to the point that it quickly becomes\n  insignificant.\n\nTo illustrate, we can look at the memory profiles for the package benchmarks.\n\n**objects allocated on the heap**\n```\nFile: sqlrange.test\nType: alloc_objects\nTime: Jan 15, 2024 at 8:32am (PST)\nShowing nodes accounting for 23444929, 97.50% of 24046152 total\nDropped 43 nodes (cum \u003c= 120230)\n      flat  flat%   sum%        cum   cum%\n  21408835 89.03% 89.03%   21408835 89.03%  github.com/achille-roussel/sqlrange_test.(*fakeStmt).QueryContext /go/src/github.com/achille-roussel/sqlrange/fakedb_test.go:1040\n   1769499  7.36% 96.39%    1769499  7.36%  strconv.formatBits /sdk/go1.22rc1/src/strconv/itoa.go:199\n    217443   0.9% 97.30%     217443   0.9%  github.com/achille-roussel/sqlrange_test.(*fakeStmt).QueryContext /go/src/github.com/achille-roussel/sqlrange/fakedb_test.go:1044\n     32768  0.14% 97.43%   21926303 91.18%  database/sql.(*DB).query /sdk/go1.22rc1/src/database/sql/sql.go:1754\n     16384 0.068% 97.50%   23925181 99.50%  github.com/achille-roussel/sqlrange_test.BenchmarkQuery100Rows /go/src/github.com/achille-roussel/sqlrange/sqlrange_test.go:145\n         0     0% 97.50%   21926303 91.18%  database/sql.(*DB).QueryContext /sdk/go1.22rc1/src/database/sql/sql.go:1731\n         0     0% 97.50%   21926303 91.18%  database/sql.(*DB).QueryContext.func1 /sdk/go1.22rc1/src/database/sql/sql.go:1732\n         0     0% 97.50%   21746433 90.44%  database/sql.(*DB).queryDC /sdk/go1.22rc1/src/database/sql/sql.go:1806\n         0     0% 97.50%   22039082 91.65%  database/sql.(*DB).retry /sdk/go1.22rc1/src/database/sql/sql.go:1566\n         0     0% 97.50%    1769499  7.36%  database/sql.(*Rows).Scan /sdk/go1.22rc1/src/database/sql/sql.go:3354\n         0     0% 97.50%    1769499  7.36%  database/sql.asString /sdk/go1.22rc1/src/database/sql/convert.go:499\n         0     0% 97.50%    1769499  7.36%  database/sql.convertAssignRows /sdk/go1.22rc1/src/database/sql/convert.go:433\n         0     0% 97.50%     169852  0.71%  database/sql.ctxDriverPrepare /sdk/go1.22rc1/src/database/sql/ctxutil.go:15\n         0     0% 97.50%   21746433 90.44%  database/sql.ctxDriverStmtQuery /sdk/go1.22rc1/src/database/sql/ctxutil.go:82\n         0     0% 97.50%   21746433 90.44%  database/sql.rowsiFromStatement /sdk/go1.22rc1/src/database/sql/sql.go:2836\n         0     0% 97.50%     202620  0.84%  database/sql.withLock /sdk/go1.22rc1/src/database/sql/sql.go:3530\n         0     0% 97.50%   21926303 91.18%  github.com/achille-roussel/sqlrange.QueryContext[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }] /go/src/github.com/achille-roussel/sqlrange/sqlrange.go:213\n         0     0% 97.50%    1769499  7.36%  github.com/achille-roussel/sqlrange.QueryContext[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }].Scan[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }].func2 /go/src/github.com/achille-roussel/sqlrange/sqlrange.go:278\n         0     0% 97.50%   21926303 91.18%  github.com/achille-roussel/sqlrange.Query[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }] /go/src/github.com/achille-roussel/sqlrange/sqlrange.go:189 (inline)\n         0     0% 97.50%     120971   0.5%  github.com/achille-roussel/sqlrange_test.BenchmarkQuery100Rows /go/src/github.com/achille-roussel/sqlrange/sqlrange_test.go:129\n         0     0% 97.50%     120971   0.5%  github.com/achille-roussel/sqlrange_test.BenchmarkQuery100Rows.Exec[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }].ExecContext[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }].func4 /go/src/github.com/achille-roussel/sqlrange/sqlrange.go:162\n         0     0% 97.50%     120971   0.5%  github.com/achille-roussel/sqlrange_test.BenchmarkQuery100Rows.func1 /go/src/github.com/achille-roussel/sqlrange/sqlrange_test.go:131\n         0     0% 97.50%    1769499  7.36%  strconv.FormatInt /sdk/go1.22rc1/src/strconv/itoa.go:29\n         0     0% 97.50%   24033864 99.95%  testing.(*B).launch /sdk/go1.22rc1/src/testing/benchmark.go:316\n         0     0% 97.50%   24046152   100%  testing.(*B).runN /sdk/go1.22rc1/src/testing/benchmark.go:193\n```\n\n**memory allocated on the heap**\n```\nFile: sqlrange.test\nType: alloc_space\nTime: Jan 15, 2024 at 8:32am (PST)\nShowing nodes accounting for 626.05MB, 97.66% of 641.05MB total\nDropped 33 nodes (cum \u003c= 3.21MB)\n      flat  flat%   sum%        cum   cum%\n  408.51MB 63.72% 63.72%   408.51MB 63.72%  github.com/achille-roussel/sqlrange_test.(*fakeStmt).QueryContext /go/src/github.com/achille-roussel/sqlrange/fakedb_test.go:1040\n  174.04MB 27.15% 90.87%   174.04MB 27.15%  github.com/achille-roussel/sqlrange_test.(*fakeStmt).QueryContext /go/src/github.com/achille-roussel/sqlrange/fakedb_test.go:1044\n      27MB  4.21% 95.09%       27MB  4.21%  strconv.formatBits /sdk/go1.22rc1/src/strconv/itoa.go:199\n    5.50MB  0.86% 95.94%     5.50MB  0.86%  github.com/achille-roussel/sqlrange_test.(*fakeStmt).QueryContext /go/src/github.com/achille-roussel/sqlrange/fakedb_test.go:1064\n    5.50MB  0.86% 96.80%     5.50MB  0.86%  database/sql.(*DB).queryDC /sdk/go1.22rc1/src/database/sql/sql.go:1815\n    4.50MB   0.7% 97.50%     4.50MB   0.7%  strings.genSplit /sdk/go1.22rc1/src/strings/strings.go:249\n    0.50MB 0.078% 97.58%   635.05MB 99.06%  github.com/achille-roussel/sqlrange_test.BenchmarkQuery100Rows /go/src/github.com/achille-roussel/sqlrange/sqlrange_test.go:145\n    0.50MB 0.078% 97.66%   602.05MB 93.92%  database/sql.(*DB).query /sdk/go1.22rc1/src/database/sql/sql.go:1754\n         0     0% 97.66%     5.50MB  0.86%  database/sql.(*DB).ExecContext /sdk/go1.22rc1/src/database/sql/sql.go:1661\n         0     0% 97.66%     5.50MB  0.86%  database/sql.(*DB).ExecContext.func1 /sdk/go1.22rc1/src/database/sql/sql.go:1662\n         0     0% 97.66%   602.05MB 93.92%  database/sql.(*DB).QueryContext /sdk/go1.22rc1/src/database/sql/sql.go:1731\n         0     0% 97.66%   602.05MB 93.92%  database/sql.(*DB).QueryContext.func1 /sdk/go1.22rc1/src/database/sql/sql.go:1732\n         0     0% 97.66%     5.50MB  0.86%  database/sql.(*DB).exec /sdk/go1.22rc1/src/database/sql/sql.go:1683\n         0     0% 97.66%        5MB  0.78%  database/sql.(*DB).queryDC /sdk/go1.22rc1/src/database/sql/sql.go:1797\n         0     0% 97.66%   589.55MB 91.97%  database/sql.(*DB).queryDC /sdk/go1.22rc1/src/database/sql/sql.go:1806\n         0     0% 97.66%        5MB  0.78%  database/sql.(*DB).queryDC.func2 /sdk/go1.22rc1/src/database/sql/sql.go:1798\n         0     0% 97.66%   607.55MB 94.77%  database/sql.(*DB).retry /sdk/go1.22rc1/src/database/sql/sql.go:1566\n         0     0% 97.66%       27MB  4.21%  database/sql.(*Rows).Scan /sdk/go1.22rc1/src/database/sql/sql.go:3354\n         0     0% 97.66%       27MB  4.21%  database/sql.asString /sdk/go1.22rc1/src/database/sql/convert.go:499\n         0     0% 97.66%       27MB  4.21%  database/sql.convertAssignRows /sdk/go1.22rc1/src/database/sql/convert.go:433\n         0     0% 97.66%        8MB  1.25%  database/sql.ctxDriverPrepare /sdk/go1.22rc1/src/database/sql/ctxutil.go:15\n         0     0% 97.66%   589.55MB 91.97%  database/sql.ctxDriverStmtQuery /sdk/go1.22rc1/src/database/sql/ctxutil.go:82\n         0     0% 97.66%   589.55MB 91.97%  database/sql.rowsiFromStatement /sdk/go1.22rc1/src/database/sql/sql.go:2836\n         0     0% 97.66%     8.50MB  1.33%  database/sql.withLock /sdk/go1.22rc1/src/database/sql/sql.go:3530\n         0     0% 97.66%   602.05MB 93.92%  github.com/achille-roussel/sqlrange.QueryContext[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }] /go/src/github.com/achille-roussel/sqlrange/sqlrange.go:213\n         0     0% 97.66%       27MB  4.21%  github.com/achille-roussel/sqlrange.QueryContext[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }].Scan[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }].func2 /go/src/github.com/achille-roussel/sqlrange/sqlrange.go:278\n         0     0% 97.66%   602.05MB 93.92%  github.com/achille-roussel/sqlrange.Query[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }] /go/src/github.com/achille-roussel/sqlrange/sqlrange.go:189 (inline)\n         0     0% 97.66%        6MB  0.94%  github.com/achille-roussel/sqlrange_test.BenchmarkQuery100Rows /go/src/github.com/achille-roussel/sqlrange/sqlrange_test.go:129\n         0     0% 97.66%        6MB  0.94%  github.com/achille-roussel/sqlrange_test.BenchmarkQuery100Rows.Exec[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }].ExecContext[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }].func4 /go/src/github.com/achille-roussel/sqlrange/sqlrange.go:162\n         0     0% 97.66%     5.50MB  0.86%  github.com/achille-roussel/sqlrange_test.BenchmarkQuery100Rows.Exec[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }].ExecContext[go.shape.struct { Age int \"sql:\\\"age\\\"\"; Name string \"sql:\\\"name\\\"\"; BirthDate time.Time \"sql:\\\"bdate\\\"\" }].func4.3 /go/src/github.com/achille-roussel/sqlrange/sqlrange.go:170\n         0     0% 97.66%        6MB  0.94%  github.com/achille-roussel/sqlrange_test.BenchmarkQuery100Rows.func1 /go/src/github.com/achille-roussel/sqlrange/sqlrange_test.go:131\n         0     0% 97.66%       27MB  4.21%  strconv.FormatInt /sdk/go1.22rc1/src/strconv/itoa.go:29\n         0     0% 97.66%     4.50MB   0.7%  strings.Split /sdk/go1.22rc1/src/strings/strings.go:307 (inline)\n         0     0% 97.66%   640.05MB 99.84%  testing.(*B).launch /sdk/go1.22rc1/src/testing/benchmark.go:316\n         0     0% 97.66%   641.05MB   100%  testing.(*B).runN /sdk/go1.22rc1/src/testing/benchmark.go:193\n```\n\nAlmost all the memory allocated on the heap is done in the SQL driver.\nThe fake driver employed for tests isn't very efficient, but it still shows\nthat the package does not contribute to the majority of memory usage.\nPrograms that use SQL drivers for production databases like MySQL or Postgres\nwill have performance characteristics dictated by the driver and won't suffer\nfrom utilizing the `sqlrange` package abstractions.\n\n[structField]: https://pkg.go.dev/reflect#StructField\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fachille-roussel%2Fsqlrange","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fachille-roussel%2Fsqlrange","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fachille-roussel%2Fsqlrange/lists"}