{"id":22732934,"url":"https://github.com/acronis/go-dbkit","last_synced_at":"2025-12-25T18:57:05.985Z","repository":{"id":257820011,"uuid":"848430197","full_name":"acronis/go-dbkit","owner":"acronis","description":"Toolkit for working with SQL databases in Go","archived":false,"fork":false,"pushed_at":"2025-02-26T11:30:01.000Z","size":143,"stargazers_count":12,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-14T01:52:21.944Z","etag":null,"topics":["go","golang","mssql","mysql","postgresql","sql","sqlite3","toolkit"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/acronis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-27T18:40:39.000Z","updated_at":"2025-03-06T10:25:04.000Z","dependencies_parsed_at":"2024-10-10T07:28:19.608Z","dependency_job_id":"6557e34a-6e57-47a9-8519-441b116fa899","html_url":"https://github.com/acronis/go-dbkit","commit_stats":null,"previous_names":["acronis/go-dbkit"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acronis%2Fgo-dbkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acronis%2Fgo-dbkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acronis%2Fgo-dbkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acronis%2Fgo-dbkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/acronis","download_url":"https://codeload.github.com/acronis/go-dbkit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248809032,"owners_count":21164895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","mssql","mysql","postgresql","sql","sqlite3","toolkit"],"created_at":"2024-12-10T20:11:56.819Z","updated_at":"2025-12-25T18:57:05.915Z","avatar_url":"https://github.com/acronis.png","language":"Go","readme":"# Toolkit for working with SQL databases in Go\n\n[![GoDoc Widget]][GoDoc]\n\n`go‑dbkit` is a Go library designed to simplify and streamline working with SQL databases.\nIt provides a solid foundation for connection management, instrumentation, error retry mechanisms, and transaction handling.\nAdditionally, `go‑dbkit` offers a suite of specialized sub‑packages that address common database challenges—such as [distributed locking](./distrlock), [schema migrations](./migrate), and query builder utilities - making it a one‑stop solution for applications that needs to interact with multiple SQL databases.\n\n## Features\n- **Transaction Management**: Execute functions within transactions that automatically commit on success or roll back on error. The transaction runner abstracts the boilerplate, ensuring cleaner and more reliable code.\n- **Retryable Queries**: Built‑in support for detecting and automatically retrying transient errors (e.g., deadlocks, lock timeouts) across various databases.\n- **Distributed Locking**: Implement SQL‑based distributed locks to coordinate exclusive access to shared resources across multiple processes.\n- **Database Migrations**: Seamlessly manage schema changes with support for both embedded SQL migrations (using Go’s embed package) and programmatic migration definitions.\n- **Query Builder Utilities**: Enhance your query‐building experience with utilities for popular libraries:\n  * [dbrutil](./dbrutil): Simplifies working with the [dbr query builder](https://github.com/gocraft/dbr), adding instrumentation (Prometheus metrics, slow query logging) and transaction support.\n  * [goquutil](./goquutil): Provides helper routines for the [goqu query builder](https://github.com/doug-martin/goqu) (currently, no dedicated README exists—refer to the source for usage).\n\n## Packages Overview\n- Root `go‑dbkit` package provides configuration management, DSN generation, and the foundational retryable query functionality used across the library.\n- [dbrutil](./dbrutil) offers utilities for the dbr query builder, including:\n  * Instrumented connection opening with Prometheus metrics.\n  *\tAutomatic slow query logging based on configurable thresholds.\n  *\tA transaction runner that simplifies commit/rollback logic.\n  Read more in [dbrutil/README.md](./dbrutil/README.md).\n- [distrlock](./distrlock) implements a lightweight, SQL‑based distributed locking mechanism that ensures exclusive execution of critical sections across concurrent services.\n  Read more in [distrlock/README.md](./distrlock/README.md).\n- [migrate](./migrate):\n  Manage your database schema changes effortlessly with support for both embedded SQL files and programmatic migrations.\n  Read more in [migrate/README.md](./migration/README.md).\n- [goquutil](./goquutil) provides helper functions for working with the goqu query builder, streamlining common operations. (This package does not have its own README yet, so please refer to the source code for more details.)\n- RDBMS‑Specific dedicated sub‑packages are provided for various relational databases:\n  * [mysql](./mysql) includes DSN generation, retryable error handling, and other MySQL‑specific utilities.\n  * [sqlite](./sqlite) contains helpers to integrate SQLite seamlessly into your projects.\n  * [postgres](./postgres) \u0026 [pgx](./pgx) offers tools and error handling improvements for PostgreSQL using both the lib/pq and pgx drivers.\n  * [mssql](./mssql) provides MSSQL‑specific error handling, including registration of retryable functions for deadlocks and related transient errors.\n  Each of these packages registers its own retryable function in the init() block, ensuring that transient errors (like deadlocks or cached plan invalidations) are automatically retried.o\n\n## Installation\n\n```\ngo get -u github.com/acronis/go-dbkit\n```\n\n## Usage\n\n### Basic Example\n\nBelow is a simple example that demonstrates how to register a retryable function for a MySQL database connection and execute a query within a transaction with a custom retry policy on transient errors.\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"database/sql\"\n\t\"log\"\n\t\"os\"\n\t\"time\"\n\n\t\"github.com/acronis/go-appkit/retry\"\n\n\t\"github.com/acronis/go-dbkit\"\n\n\t// Import the `mysql` package for registering the retryable function for MySQL transient errors (like deadlocks).\n\t_ \"github.com/acronis/go-dbkit/mysql\"\n)\n\nfunc main() {\n\t// Configure the database using the dbkit.Config struct.\n\t// In this example, we're using MySQL. Adjust Dialect and config fields for your target DB.\n\tcfg := \u0026dbkit.Config{\n\t\tDialect: dbkit.DialectMySQL,\n\t\tMySQL: dbkit.MySQLConfig{\n\t\t\tHost:     os.Getenv(\"MYSQL_HOST\"),\n\t\t\tPort:     3306,\n\t\t\tUser:     os.Getenv(\"MYSQL_USER\"),\n\t\t\tPassword: os.Getenv(\"MYSQL_PASSWORD\"),\n\t\t\tDatabase: os.Getenv(\"MYSQL_DATABASE\"),\n\t\t},\n\t\tMaxOpenConns: 16,\n\t\tMaxIdleConns: 8,\n\t}\n\n\t// Open the database connection.\n\t// The 2nd parameter is a boolean that indicates whether to ping the database.\n\tdb, err := dbkit.Open(cfg, true)\n\tif err != nil {\n\t\tlog.Fatalf(\"failed to open database: %v\", err)\n\t}\n\tdefer db.Close()\n\n\t// Execute a transaction with a custom retry policy (exponential backoff with 3 retries, starting from 10ms).\n\tretryPolicy := retry.NewConstantBackoffPolicy(10*time.Millisecond, 3)\n\tif err = dbkit.DoInTx(context.Background(), db, func(tx *sql.Tx) error {\n\t\t// Execute your transactional operations here.\n\t\t// Example: _, err := tx.Exec(\"UPDATE users SET last_login = ? WHERE id = ?\", time.Now(), 1)\n\t\treturn nil\n\t}, dbkit.WithRetryPolicy(retryPolicy)); err != nil {\n\t\tlog.Fatal(err)\n\t}\n}\n```\n\n### `dbrutil` Usage Example\n\nThe following basic example demonstrates how to use `dbrutil` to open a database connection with instrumentation,\nand execute queries with an automatic slow query logging and Prometheus metrics collection within transaction.\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"database/sql\"\n\t\"errors\"\n\t\"fmt\"\n\tstdlog \"log\"\n\t\"net/http\"\n\t\"os\"\n\t\"time\"\n\n\t\"github.com/acronis/go-appkit/log\"\n\t\"github.com/gocraft/dbr/v2\"\n\t\n\t\"github.com/acronis/go-dbkit\"\n\t\"github.com/acronis/go-dbkit/dbrutil\"\n)\n\nfunc main() {\n\tlogger, loggerClose := log.NewLogger(\u0026log.Config{Output: log.OutputStderr, Level: log.LevelInfo})\n\tdefer loggerClose()\n\n\t// Create a Prometheus metrics collector.\n\tpromMetrics := dbkit.NewPrometheusMetrics()\n\tpromMetrics.MustRegister()\n\tdefer promMetrics.Unregister()\n\n\t// Open the database connection with instrumentation.\n\t// Instrumentation includes collecting metrics about SQL queries and logging slow queries.\n\teventReceiver := dbrutil.NewCompositeReceiver([]dbr.EventReceiver{\n\t\tdbrutil.NewQueryMetricsEventReceiver(promMetrics, queryAnnotationPrefix),\n\t\tdbrutil.NewSlowQueryLogEventReceiver(logger, 100*time.Millisecond, queryAnnotationPrefix),\n\t})\n\tconn, err := openDB(eventReceiver)\n\tif err != nil {\n\t\tstdlog.Fatal(err)\n\t}\n\tdefer conn.Close()\n\n\ttxRunner := dbrutil.NewTxRunner(conn, \u0026sql.TxOptions{Isolation: sql.LevelReadCommitted}, nil)\n\n\t// Execute function in a transaction.\n\t// The transaction will be automatically committed if the function returns nil, otherwise it will be rolled back.\n\tif dbErr := txRunner.DoInTx(context.Background(), func(tx dbr.SessionRunner) error {\n\t\tvar result int\n\t\treturn tx.Select(\"SLEEP(1)\").\n\t\t\tComment(annotateQuery(\"long_operation\")). // Annotate the query for Prometheus metrics and slow query log.\n\t\t\tLoadOne(\u0026result)\n\t}); dbErr != nil {\n\t\tstdlog.Fatal(dbErr)\n\t}\n\n\t// The following log message will be printed:\n\t// {\"level\":\"warn\",\"time\":\"2025-02-14T16:29:55.429257+02:00\",\"msg\":\"slow SQL query\",\"pid\":14030,\"annotation\":\"query:long_operation\",\"duration_ms\":1007}\n\n\t// Prometheus metrics will be collected:\n\t// db_query_duration_seconds_bucket{query=\"query:long_operation\",le=\"2.5\"} 1\n\t// db_query_duration_seconds_sum{query=\"query:long_operation\"} 1.004573875\n\t// db_query_duration_seconds_count{query=\"query:long_operation\"} 1\n}\n\nconst queryAnnotationPrefix = \"query:\"\n\nfunc annotateQuery(queryName string) string {\n\treturn queryAnnotationPrefix + queryName\n}\n\nfunc openDB(eventReceiver dbr.EventReceiver) (*dbr.Connection, error) {\n\tcfg := \u0026dbkit.Config{\n\t\tDialect: dbkit.DialectMySQL,\n\t\tMySQL: dbkit.MySQLConfig{\n\t\t\tHost:     os.Getenv(\"MYSQL_HOST\"),\n\t\t\tPort:     3306,\n\t\t\tUser:     os.Getenv(\"MYSQL_USER\"),\n\t\t\tPassword: os.Getenv(\"MYSQL_PASSWORD\"),\n\t\t\tDatabase: os.Getenv(\"MYSQL_DATABASE\"),\n\t\t},\n\t}\n\n\t// Open database with instrumentation based on the provided event receiver (see github.com/gocraft/dbr doc for details).\n\t// Opening includes configuring the max open/idle connections and their lifetime and pinging the database.\n\tconn, err := dbrutil.Open(cfg, true, eventReceiver)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"open database: %w\", err)\n\t}\n\treturn conn, nil\n}\n```\n\nMore examples and detailed usage instructions can be found in the `dbrutil` package [README](./dbrutil/README.md).\n\n### `distrlock` Usage Example\n\nThe following basic example demonstrates how to use `distrlock` to ensure exclusive execution of a critical section of code.\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"database/sql\"\n\t\"log\"\n\t\"os\"\n\t\"time\"\n\n\t\"github.com/acronis/go-dbkit\"\n\t\"github.com/acronis/go-dbkit/distrlock\"\n)\n\nfunc main() {\n\t// Setup database connection\n\tdb, err := sql.Open(\"mysql\", os.Getenv(\"MYSQL_DSN\"))\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tdefer db.Close()\n\n\tctx := context.Background()\n\n\t// Create \"distributed_locks\" table for locks.\n\tcreateTableSQL, err := distrlock.CreateTableSQL(dbkit.DialectMySQL)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\t_, err = db.ExecContext(ctx, createTableSQL)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\t// Do some work exclusively.\n\tconst lockKey = \"test-lock-key-1\" // Unique key that will be used to ensure exclusive execution among multiple instances\n\terr = distrlock.DoExclusively(ctx, db, dbkit.DialectMySQL, lockKey, func(ctx context.Context) error {\n\t\ttime.Sleep(10 * time.Second) // Simulate work.\n\t\treturn nil\n\t})\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n}\n```\n\nMore examples and detailed usage instructions can be found in the `distrlock` package [README](./distrlock/README.md).\n\n## License\n\nCopyright © 2024 Acronis International GmbH.\n\nLicensed under [MIT License](./LICENSE).\n\n[GoDoc]: https://pkg.go.dev/github.com/acronis/go-dbkit\n[GoDoc Widget]: https://godoc.org/github.com/acronis/go-dbkit?status.svg\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facronis%2Fgo-dbkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Facronis%2Fgo-dbkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facronis%2Fgo-dbkit/lists"}