{"id":20411633,"url":"https://github.com/muhlemmer/pbpgx","last_synced_at":"2025-04-12T16:25:49.730Z","repository":{"id":57648645,"uuid":"426608908","full_name":"muhlemmer/pbpgx","owner":"muhlemmer","description":"Package pbpgx provides a toolkit for easier Protocol Buffers interaction with PostgreSQL databases.","archived":false,"fork":false,"pushed_at":"2022-09-22T11:47:51.000Z","size":268,"stargazers_count":11,"open_issues_count":2,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-26T10:52:17.302Z","etag":null,"topics":["generics","go","go-generics","golang","grpc","grpc-go","pgx","postgres","postgresql","protobuf","protocol-buffers","protoreflect"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/muhlemmer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-10T12:11:44.000Z","updated_at":"2023-05-02T22:11:49.000Z","dependencies_parsed_at":"2022-09-15T13:30:26.943Z","dependency_job_id":null,"html_url":"https://github.com/muhlemmer/pbpgx","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhlemmer%2Fpbpgx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhlemmer%2Fpbpgx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhlemmer%2Fpbpgx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhlemmer%2Fpbpgx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/muhlemmer","download_url":"https://codeload.github.com/muhlemmer/pbpgx/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248595432,"owners_count":21130522,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generics","go","go-generics","golang","grpc","grpc-go","pgx","postgres","postgresql","protobuf","protocol-buffers","protoreflect"],"created_at":"2024-11-15T05:53:31.658Z","updated_at":"2025-04-12T16:25:49.704Z","avatar_url":"https://github.com/muhlemmer.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Go](https://github.com/muhlemmer/pbpgx/actions/workflows/go.yml/badge.svg)](https://github.com/muhlemmer/pbpgx/actions/workflows/go.yml)\n[![codecov](https://codecov.io/gh/muhlemmer/pbpgx/branch/main/graph/badge.svg?token=NS16O82CC1)](https://codecov.io/gh/muhlemmer/pbpgx)\n[![Go Report Card](https://goreportcard.com/badge/github.com/muhlemmer/pbpgx)](https://goreportcard.com/report/github.com/muhlemmer/pbpgx)\n\n# PBPGX\n\nPackage pbpgx provides a toolkit for easier Protocol Buffers interaction with PostgreSQL databases.\n\nPbpgx supports the Protocol Buffer types generated in your project, through protoreflect.\nIt is build on the pgx PostgreSQL driver toolkit for Go, for richer SQL type support.\nThefore pbpgx is targetting developers which are building protcol buffers based APIs against a PostgreSQL database.\n\nThis package is currently a WIP and is written against Go version 1.18-beta1 with type parameter support and\nwill support Go version 1.18 upward.\n\n## Features\n\nWhen not using an ORM a typical development workflow I often experience is:\n\n1. Query preparation: Load definitions from files, execute templates or [build queries](#query-building) based on the request to the API.\n2. [Query execution](#query-execution): Send the prepared query to the database, with metods such as `conn.Exec()` or `conn.Query()`.\n3. [Row scanning](#row-scanning): In case of the read action `conn.Query()`, the returned data must be scanned with `rows.Next()` and `rows.Scan()`,\n    checking for errors on each iteration.\n    If the result columns are variable based on the request data, then the scan targets must also dynamically be prepared during query preparation.\n\nPbpgx can greatly reduce this effort for most boilerplate API tasks. This package can be used from the bottom-up.\nFor instance, if your project already has a way of preparing and executing queries, this package can still be used for [scanning](#row-scanning) Rows into protocol buffer messages.\nHowever, for most [CRUD](https://en.wikipedia.org/wiki/CRUD) actions, this package provides a way to build and execute queries, returning the results in the required type in a single call.\nIt aims to minimize data transformation by the caller, by using Protocol Buffer messages as argument and return types.\n\n### Row Scanning\n\nA common design challange with Go is scanning database Rows results into structs. Ussualy, one would scan into the indivudual fields or local variables.\nBut what if the the selected columns are a variable in the application? In such cases developers have to resort to reflection, an ORM based on\nreflection or a code generator like SQLBoiler with tons of adaptor code. \nPbpgx uses [protoreflect](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect), which uses the embedded protobuf descriptors in the generated Go code.\nThis should give a potential performance edge over standard Go reflection. \n\nBy using Go generics, the `Scan()` function is able to return slices of concrete message types, allowing you to consume the result set without further transformation. For example a product list in a [proto file](example_gen/example.proto):\n\n```\nmessage Product {\n    int64 id = 1;\n    string title = 2;\n    double price = 3;\n    google.protobuf.Timestamp created = 4;\n}\n\nmessage ProductList {\n    repeated Product products = 1;\n}\n\n```\n\nCan easily be filled with:\n\n```\nrows, err := conn.Query(ctx, \"select id, title, price, created from products;\")\nif err != nil {\n    panic(err)\n}\n\nresult := new(gen.ProductList)\n\nresult.Products, err = pbpgx.Scan[*gen.Product](rows)\nif err != nil {\n    panic(err)\n}\n\n```\n\n### Query Execution\n\nThe generic `Query()` function can be used to execute a query and return a slice of Protocol Buffer messages filled with the results, through [Row Scanning](#row-scanning):\n\n```\nresult.Products, err = pbpgx.Query[*gen.Product](ctx, conn,\n    \"select title, price, created from products where id in ($1, $2, $3);\", 2, 4, 5)\nif err != nil {\n    panic(err)\n}\n```\n\n### CRUD operations\n\nThe `github.com/muhlemmer/pbpgx/crud` sub-package provides query building, execution and scanning for common CRUD operations. (Create, Read, Update and Delete).\nThis allows to easily integrate your database models with protocol buffers and derived RPC implementations, for all basic operations.\nArgument and return types are typically of `proto.Message`, where applicable, again eliminating the need of adaptor code.\nAlso the use of generic type arguments and `protoreflect` ensures a maintainable, consice, code base.\n\n#### Usage\n\nUsage is simple. First we extend on our [protocal buffer definitions](example_gen/example.proto) from above:\n\n```\nmessage ProductColumns{\n    enum Names {\n        id = 0;\n        title = 1;\n        price = 2;\n        created = 3;\n    }\n}\n\nmessage ProductQuery {\n    int64 id = 1;\n    repeated ProductColumns.Names columns = 2;\n}\n\nmessage ProductCreateQuery {\n    Product data = 1;\n    repeated ProductColumns.Names columns = 2;\n}\n```\n\nIn our implementation code we reate a `Table` once, which takes care of query builder re-use:\n\n```\ncolumns := crud.Columns{\"title\": crud.Zero}\ntab := crud.NewTable[gen.ProductColumns_Names, *gen.Product, int32](\"public\", \"example\", columns)\n```\n\nWe can use the `id` and `columns` from an incomming query to read a single message,\nreturned as the requested type `Product`:\n\n```\nquery := \u0026gen.ProductQuery{\n    Id: 2,\n    Columns: []gen.ProductColumns_Names{\n        gen.ProductColumns_title,\n        gen.ProductColumns_price,\n        gen.ProductColumns_created,\n    },\n}\n\nrecord, err := tab.ReadOne(ctx, conn, query.Id, query.Columns)\nif err != nil {\n    panic(err)\n}\n\n```\n\nOr, create a new entry (`INSERT`), returing specific fields after the database assigns the primary ID and created timestamp:\n\n```\nquery := \u0026gen.ProductCreateQuery{\n    Data: \u0026gen.Product{\n        Title: \"Great deal!\",\n        Price: 9.99,\n    },\n    Columns: []gen.ProductColumns_Names{\n        gen.ProductColumns_id,\n        gen.ProductColumns_title,\n        gen.ProductColumns_price,\n        gen.ProductColumns_created,\n    },\n}\n\nrecord, err := tab.CreateOne(ctx, conn, crud.ParseFields(query.GetData(), true), query.GetData(), query.GetColumns()...)\nif err != nil {\n    panic(err)\n}\n```\n\nLikewise, there are the `CreateOne`, `DeleteOne` and `UpdateOne` functions.\n\n### Query building\n\nThe `crud` package is build on the `github.com/muhlemmer/pbpgx/query` sub-package. The building blocks for query building are exported, so that package consumers can use them to build custom queries.\nThe `query` package uses a `sync.Pool` for efficient memory re-use.\nThe builder's capacity is stored and on the next use, the same capacity is allocated in order to reduce the amount off allocations needed during building.\nThis results in a fairly efficient builder:\n\n```\nBenchmarkBuilder_Insert-16    \t 5989922\t       477.7 ns/op\t     192 B/op\t       1 allocs/op\nBenchmarkBuilder_Select-16    \t 2476291\t       405.9 ns/op\t      96 B/op\t       1 allocs/op\nBenchmarkBuilder_Update-16    \t 3126738\t       555.4 ns/op\t     192 B/op\t       1 allocs/op\nBenchmarkBuilder_Delete-16    \t 4884342\t       303.8 ns/op\t      80 B/op\t       1 allocs/op\n```\n\n## License\n\nCopyright (C) 2021, Tim Möhlmann\n\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU Affero General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU Affero General Public License for more details.\n\nYou should have received a copy of the GNU Affero General Public License\nalong with this program.  If not, see \u003chttps://www.gnu.org/licenses/\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuhlemmer%2Fpbpgx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmuhlemmer%2Fpbpgx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuhlemmer%2Fpbpgx/lists"}